Jul 20 17:55:52.631242 ip-10-0-137-247 systemd[1]: Starting Kubernetes Kubelet...
Jul 20 17:55:53.000236 ip-10-0-137-247 kubenswrapper[2569]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 20 17:55:53.000236 ip-10-0-137-247 kubenswrapper[2569]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 20 17:55:53.000236 ip-10-0-137-247 kubenswrapper[2569]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 20 17:55:53.000236 ip-10-0-137-247 kubenswrapper[2569]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 20 17:55:53.000236 ip-10-0-137-247 kubenswrapper[2569]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 20 17:55:53.000236 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.953908    2569 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 20 17:55:53.000236 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.961314    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 20 17:55:53.000236 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.961328    2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 20 17:55:53.000236 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.961332    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 20 17:55:53.000236 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.961335    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 20 17:55:53.000236 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.961338    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 20 17:55:53.000236 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.961341    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 20 17:55:53.000236 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.961345    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 20 17:55:53.000236 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.961348    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 20 17:55:53.002182 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.961350    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 20 17:55:53.002182 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.961353    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 20 17:55:53.002182 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.961356    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 20 17:55:53.002182 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.961359    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 20 17:55:53.002182 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.961361    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 20 17:55:53.002182 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.961364    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 20 17:55:53.002182 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.961367    2569 feature_gate.go:328] unrecognized feature gate: Example2
Jul 20 17:55:53.002182 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.961369    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 20 17:55:53.002182 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.961372    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 20 17:55:53.002182 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.961375    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 20 17:55:53.002182 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.961378    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 20 17:55:53.002182 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.961380    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 20 17:55:53.002182 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.961383    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 20 17:55:53.002182 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.961386    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 20 17:55:53.002182 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.961388    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 20 17:55:53.002182 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.961391    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 20 17:55:53.002182 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.961393    2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 20 17:55:53.002182 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.961396    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 20 17:55:53.002182 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.961399    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 20 17:55:53.002182 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.961402    2569 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 20 17:55:53.004349 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.961405    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 20 17:55:53.004349 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.961410    2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 20 17:55:53.004349 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.961414    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 20 17:55:53.004349 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.961417    2569 feature_gate.go:328] unrecognized feature gate: Example
Jul 20 17:55:53.004349 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.961422    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 20 17:55:53.004349 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.961426    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 20 17:55:53.004349 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.961428    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 20 17:55:53.004349 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.961431    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 20 17:55:53.004349 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.961435    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 20 17:55:53.004349 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.961438    2569 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 20 17:55:53.004349 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.961440    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 20 17:55:53.004349 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.961443    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 20 17:55:53.004349 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.961446    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 20 17:55:53.004349 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.961448    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 20 17:55:53.004349 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.961451    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 20 17:55:53.004349 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.961454    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 20 17:55:53.004349 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.961456    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 20 17:55:53.004349 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.961459    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 20 17:55:53.004349 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.961462    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 20 17:55:53.004349 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.961464    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 20 17:55:53.006052 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.961467    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 20 17:55:53.006052 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.961469    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 20 17:55:53.006052 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.961472    2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 20 17:55:53.006052 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.961474    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 20 17:55:53.006052 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.961477    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 20 17:55:53.006052 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.961480    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 20 17:55:53.006052 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.961484    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 20 17:55:53.006052 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.961487    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 20 17:55:53.006052 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.961490    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 20 17:55:53.006052 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.961492    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 20 17:55:53.006052 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.961495    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 20 17:55:53.006052 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.961498    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 20 17:55:53.006052 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.961501    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 20 17:55:53.006052 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.961504    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 20 17:55:53.006052 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.961507    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 20 17:55:53.006052 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.961509    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 20 17:55:53.006052 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.961512    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 20 17:55:53.006052 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.961515    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 20 17:55:53.006052 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.961517    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 20 17:55:53.161664 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.961520    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 20 17:55:53.161664 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.961523    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 20 17:55:53.161664 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.961525    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 20 17:55:53.161664 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.961528    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 20 17:55:53.161664 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.961530    2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 20 17:55:53.161664 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.961533    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 20 17:55:53.161664 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.961535    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 20 17:55:53.161664 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.961539    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 20 17:55:53.161664 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.961543    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 20 17:55:53.161664 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.961545    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 20 17:55:53.161664 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.961548    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 20 17:55:53.161664 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.961551    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 20 17:55:53.161664 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.961554    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 20 17:55:53.161664 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.961556    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 20 17:55:53.161664 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.961559    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 20 17:55:53.161664 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.961563    2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 20 17:55:53.161664 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.961566    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 20 17:55:53.161664 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.961568    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 20 17:55:53.161664 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.961571    2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 20 17:55:53.023299 ip-10-0-137-247 systemd[1]: Started Kubernetes Kubelet.
Jul 20 17:55:53.179940 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.961576    2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 20 17:55:53.179940 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.961579    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 20 17:55:53.179940 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.961581    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 20 17:55:53.179940 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.961584    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 20 17:55:53.179940 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.961586    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 20 17:55:53.179940 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.961590    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 20 17:55:53.179940 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.961593    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 20 17:55:53.179940 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.961595    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 20 17:55:53.179940 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.961598    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 20 17:55:53.179940 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.961600    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 20 17:55:53.179940 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.961603    2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 20 17:55:53.179940 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.961606    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 20 17:55:53.179940 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.961608    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 20 17:55:53.179940 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.962340    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 20 17:55:53.179940 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.962347    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 20 17:55:53.179940 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.962350    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 20 17:55:53.179940 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.962353    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 20 17:55:53.179940 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.962356    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 20 17:55:53.179940 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.962358    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 20 17:55:53.180943 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.962361    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 20 17:55:53.180943 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.962364    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 20 17:55:53.180943 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.962367    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 20 17:55:53.180943 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.962370    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 20 17:55:53.180943 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.962372    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 20 17:55:53.180943 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.962375    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 20 17:55:53.180943 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.962377    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 20 17:55:53.180943 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.962380    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 20 17:55:53.180943 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.962383    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 20 17:55:53.180943 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.962385    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 20 17:55:53.180943 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.962389    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 20 17:55:53.180943 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.962394    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 20 17:55:53.180943 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.962397    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 20 17:55:53.180943 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.962402    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 20 17:55:53.180943 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.962404    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 20 17:55:53.180943 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.962408    2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 20 17:55:53.180943 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.962411    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 20 17:55:53.180943 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.962414    2569 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 20 17:55:53.180943 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.962416    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 20 17:55:53.181777 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.962419    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 20 17:55:53.181777 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.962421    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 20 17:55:53.181777 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.962424    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 20 17:55:53.181777 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.962427    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 20 17:55:53.181777 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.962429    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 20 17:55:53.181777 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.962432    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 20 17:55:53.181777 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.962434    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 20 17:55:53.181777 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.962437    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 20 17:55:53.181777 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.962439    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 20 17:55:53.181777 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.962442    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 20 17:55:53.181777 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.962445    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 20 17:55:53.181777 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.962449    2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 20 17:55:53.181777 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.962453    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 20 17:55:53.181777 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.962455    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 20 17:55:53.181777 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.962458    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 20 17:55:53.181777 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.962461    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 20 17:55:53.181777 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.962463    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 20 17:55:53.181777 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.962466    2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 20 17:55:53.181777 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.962469    2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 20 17:55:53.182530 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.962471    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 20 17:55:53.182530 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.962473    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 20 17:55:53.182530 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.962476    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 20 17:55:53.182530 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.962479    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 20 17:55:53.182530 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.962481    2569 feature_gate.go:328] unrecognized feature gate: Example
Jul 20 17:55:53.182530 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.962486    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 20 17:55:53.182530 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.962490    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 20 17:55:53.182530 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.962494    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 20 17:55:53.182530 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.962497    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 20 17:55:53.182530 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.962501    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 20 17:55:53.182530 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.962504    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 20 17:55:53.182530 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.962507    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 20 17:55:53.182530 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.962509    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 20 17:55:53.182530 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.962512    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 20 17:55:53.182530 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.962515    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 20 17:55:53.182530 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.962517    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 20 17:55:53.182530 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.962520    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 20 17:55:53.182530 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.962523    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 20 17:55:53.182530 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.962525    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 20 17:55:53.183182 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.962528    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 20 17:55:53.183182 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.962530    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 20 17:55:53.183182 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.962533    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 20 17:55:53.183182 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.962535    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 20 17:55:53.183182 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.962538    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 20 17:55:53.183182 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.962541    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 20 17:55:53.183182 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.962544    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 20 17:55:53.183182 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.962546    2569 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 20 17:55:53.183182 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.962549    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 20 17:55:53.183182 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.962553    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 20 17:55:53.183182 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.962556    2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 20 17:55:53.183182 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.962558    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 20 17:55:53.183182 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.962561    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 20 17:55:53.183182 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.962563    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 20 17:55:53.183182 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.962566    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 20 17:55:53.183182 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.962569    2569 feature_gate.go:328] unrecognized feature gate: Example2
Jul 20 17:55:53.183182 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.962571    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 20 17:55:53.183182 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.962574    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 20 17:55:53.183182 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.962576    2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 20 17:55:53.183182 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.962579    2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 20 17:55:53.184724 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.962584    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 20 17:55:53.184724 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.962588    2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 20 17:55:53.184724 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.962590    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 20 17:55:53.184724 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.962593    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 20 17:55:53.184724 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.962596    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 20 17:55:53.184724 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.962598    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 20 17:55:53.184724 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.962601    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 20 17:55:53.184724 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.962603    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 20 17:55:53.184724 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.962606    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 20 17:55:53.184724 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.962609    2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 20 17:55:53.184724 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.962611    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 20 17:55:53.184724 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.962614    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 20 17:55:53.184724 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.962616    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 20 17:55:53.184724 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.962619    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 20 17:55:53.184724 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.962621    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 20 17:55:53.184724 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.962624    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 20 17:55:53.184724 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.963040    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 20 17:55:53.184724 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.963047    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 20 17:55:53.184724 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.963051    2569 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 20 17:55:53.184724 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.963053    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 20 17:55:53.186104 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.963057    2569 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 20 17:55:53.186104 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.963060    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 20 17:55:53.186104 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.963063    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 20 17:55:53.186104 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.963065    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 20 17:55:53.186104 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.963068    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 20 17:55:53.186104 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.963071    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 20 17:55:53.186104 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.963073    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 20 17:55:53.186104 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.963076    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 20 17:55:53.186104 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.963079    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 20 17:55:53.186104 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.963082    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 20 17:55:53.186104 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.963085    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 20 17:55:53.186104 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.963087    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 20 17:55:53.186104 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.963094    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 20 17:55:53.186104 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.963097    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 20 17:55:53.186104 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.963100    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 20 17:55:53.186104 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.963102    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 20 17:55:53.186104 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.963106    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 20 17:55:53.186104 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.963109    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 20 17:55:53.186104 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.963112    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 20 17:55:53.186104 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.963114    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 20 17:55:53.188336 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.963116    2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 20 17:55:53.188336 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.963119    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 20 17:55:53.188336 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.963122    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 20 17:55:53.188336 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.963125    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 20 17:55:53.188336 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.963127    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 20 17:55:53.188336 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.963130    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 20 17:55:53.188336 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.963132    2569 feature_gate.go:328] unrecognized feature gate: Example2
Jul 20 17:55:53.188336 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.963135    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 20 17:55:53.188336 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.963137    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 20 17:55:53.188336 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.963142    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 20 17:55:53.188336 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.963145    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 20 17:55:53.188336 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.963148    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 20 17:55:53.188336 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.963152    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 20 17:55:53.188336 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.963155    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 20 17:55:53.188336 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.963157    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 20 17:55:53.188336 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.963160    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 20 17:55:53.188336 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.963162    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 20 17:55:53.188336 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.963166    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 20 17:55:53.188336 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.963170    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 20 17:55:53.190228 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.963173    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 20 17:55:53.190228 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.963175    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 20 17:55:53.190228 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.963177    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 20 17:55:53.190228 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.963180    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 20 17:55:53.190228 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.963183    2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 20 17:55:53.190228 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.963188    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 20 17:55:53.190228 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.963191    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 20 17:55:53.190228 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.963194    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 20 17:55:53.190228 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.963196    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 20 17:55:53.190228 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.963199    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 20 17:55:53.190228 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.963202    2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 20 17:55:53.190228 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.963204    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 20 17:55:53.190228 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.963207    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 20 17:55:53.190228 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.963209    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 20 17:55:53.190228 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.963212    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 20 17:55:53.190228 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.963214    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 20 17:55:53.190228 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.963217    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 20 17:55:53.190228 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.963219    2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 20 17:55:53.190228 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.963222    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 20 17:55:53.190228 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.963224    2569 feature_gate.go:328] unrecognized feature gate: Example
Jul 20 17:55:53.192055 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.963227    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 20 17:55:53.192055 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.963230    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 20 17:55:53.192055 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.963233    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 20 17:55:53.192055 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.963235    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 20 17:55:53.192055 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.963238    2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 20 17:55:53.192055 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.963241    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 20 17:55:53.192055 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.963243    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 20 17:55:53.192055 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.963246    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 20 17:55:53.192055 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.963250    2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 20 17:55:53.192055 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.963253    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 20 17:55:53.192055 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.963256    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 20 17:55:53.192055 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.963259    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 20 17:55:53.192055 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.963262    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 20 17:55:53.192055 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.963265    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 20 17:55:53.192055 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.963268    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 20 17:55:53.192055 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.963271    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 20 17:55:53.192055 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.963273    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 20 17:55:53.192055 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.963277    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 20 17:55:53.192055 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.963280    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 20 17:55:53.192643 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.963285    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 20 17:55:53.192643 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.963287    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 20 17:55:53.192643 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.963290    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 20 17:55:53.192643 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.963293    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 20 17:55:53.192643 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.963296    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 20 17:55:53.192643 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.963298    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 20 17:55:53.192643 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.963301    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 20 17:55:53.192643 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.963303    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 20 17:55:53.192643 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.963306    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 20 17:55:53.192643 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.963308    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 20 17:55:53.192643 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.963311    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 20 17:55:53.192643 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.963313    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 20 17:55:53.192643 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.963316    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 20 17:55:53.192643 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.963318    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 20 17:55:53.192643 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.963321    2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 20 17:55:53.192643 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.963324    2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 20 17:55:53.192643 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.963326    2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 20 17:55:53.192643 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963391    2569 flags.go:64] FLAG: --address="0.0.0.0"
Jul 20 17:55:53.192643 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963399    2569 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 20 17:55:53.192643 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963405    2569 flags.go:64] FLAG: --anonymous-auth="true"
Jul 20 17:55:53.192643 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963410    2569 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 20 17:55:53.194256 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963414    2569 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 20 17:55:53.194256 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963417    2569 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 20 17:55:53.194256 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963421    2569 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 20 17:55:53.194256 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963425    2569 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 20 17:55:53.194256 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963428    2569 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 20 17:55:53.194256 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963431    2569 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 20 17:55:53.194256 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963437    2569 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 20 17:55:53.194256 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963440    2569 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 20 17:55:53.194256 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963444    2569 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 20 17:55:53.194256 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963449    2569 flags.go:64] FLAG: --cgroup-root=""
Jul 20 17:55:53.194256 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963452    2569 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 20 17:55:53.194256 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963455    2569 flags.go:64] FLAG: --client-ca-file=""
Jul 20 17:55:53.194256 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963459    2569 flags.go:64] FLAG: --cloud-provider="external"
Jul 20 17:55:53.194256 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963462    2569 flags.go:64] FLAG: --cluster-dns="[]"
Jul 20 17:55:53.194256 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963466    2569 flags.go:64] FLAG: --cluster-domain=""
Jul 20 17:55:53.194256 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963468    2569 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 20 17:55:53.194256 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963472    2569 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 20 17:55:53.194256 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963475    2569 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 20 17:55:53.194256 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963478    2569 flags.go:64] FLAG: --container-log-max-files="5"
Jul 20 17:55:53.194256 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963482    2569 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 20 17:55:53.194256 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963484    2569 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 20 17:55:53.194256 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963488    2569 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 20 17:55:53.194256 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963491    2569 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 20 17:55:53.195749 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963494    2569 flags.go:64] FLAG: --contention-profiling="false"
Jul 20 17:55:53.195749 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963497    2569 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 20 17:55:53.195749 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963500    2569 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 20 17:55:53.195749 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963503    2569 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 20 17:55:53.195749 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963506    2569 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 20 17:55:53.195749 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963510    2569 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 20 17:55:53.195749 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963513    2569 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 20 17:55:53.195749 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963516    2569 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 20 17:55:53.195749 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963519    2569 flags.go:64] FLAG: --enable-load-reader="false"
Jul 20 17:55:53.195749 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963522    2569 flags.go:64] FLAG: --enable-server="true"
Jul 20 17:55:53.195749 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963525    2569 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 20 17:55:53.195749 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963529    2569 flags.go:64] FLAG: --event-burst="100"
Jul 20 17:55:53.195749 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963532    2569 flags.go:64] FLAG: --event-qps="50"
Jul 20 17:55:53.195749 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963535    2569 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 20 17:55:53.195749 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963539    2569 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 20 17:55:53.195749 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963541    2569 flags.go:64] FLAG: --eviction-hard=""
Jul 20 17:55:53.195749 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963545    2569 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 20 17:55:53.195749 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963548    2569 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 20 17:55:53.195749 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963552    2569 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 20 17:55:53.195749 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963556    2569 flags.go:64] FLAG: --eviction-soft=""
Jul 20 17:55:53.195749 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963559    2569 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 20 17:55:53.195749 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963562    2569 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 20 17:55:53.195749 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963566    2569 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 20 17:55:53.195749 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963569    2569 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 20 17:55:53.195749 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963572    2569 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 20 17:55:53.196925 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963575    2569 flags.go:64] FLAG: --fail-swap-on="true"
Jul 20 17:55:53.196925 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963578    2569 flags.go:64] FLAG: --feature-gates=""
Jul 20 17:55:53.196925 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963582    2569 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 20 17:55:53.196925 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963585    2569 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 20 17:55:53.196925 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963588    2569 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 20 17:55:53.196925 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963591    2569 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 20 17:55:53.196925 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963594    2569 flags.go:64] FLAG: --healthz-port="10248"
Jul 20 17:55:53.196925 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963597    2569 flags.go:64] FLAG: --help="false"
Jul 20 17:55:53.196925 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963600    2569 flags.go:64] FLAG: --hostname-override="ip-10-0-137-247.ec2.internal"
Jul 20 17:55:53.196925 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963603    2569 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 20 17:55:53.196925 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963606    2569 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 20 17:55:53.196925 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963609    2569 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 20 17:55:53.196925 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963612    2569 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 20 17:55:53.196925 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963616    2569 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 20 17:55:53.196925 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963619    2569 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 20 17:55:53.196925 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963622    2569 flags.go:64] FLAG: --image-service-endpoint=""
Jul 20 17:55:53.196925 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963626    2569 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 20 17:55:53.196925 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963629    2569 flags.go:64] FLAG: --kube-api-burst="100"
Jul 20 17:55:53.196925 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963632    2569 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 20 17:55:53.196925 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963635    2569 flags.go:64] FLAG: --kube-api-qps="50"
Jul 20 17:55:53.196925 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963638    2569 flags.go:64] FLAG: --kube-reserved=""
Jul 20 17:55:53.196925 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963641    2569 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 20 17:55:53.196925 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963644    2569 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 20 17:55:53.196925 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963647    2569 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 20 17:55:53.199721 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963650    2569 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 20 17:55:53.199721 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963654    2569 flags.go:64] FLAG: --lock-file=""
Jul 20 17:55:53.199721 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963657    2569 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 20 17:55:53.199721 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963660    2569 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 20 17:55:53.199721 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963663    2569 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 20 17:55:53.199721 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963668    2569 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 20 17:55:53.199721 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963671    2569 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 20 17:55:53.199721 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963674    2569 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 20 17:55:53.199721 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963677    2569 flags.go:64] FLAG: --logging-format="text"
Jul 20 17:55:53.199721 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963680    2569 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 20 17:55:53.199721 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963683    2569 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 20 17:55:53.199721 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963686    2569 flags.go:64] FLAG: --manifest-url=""
Jul 20 17:55:53.199721 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963689    2569 flags.go:64] FLAG: --manifest-url-header=""
Jul 20 17:55:53.199721 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963693    2569 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 20 17:55:53.199721 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963696    2569 flags.go:64] FLAG: --max-open-files="1000000"
Jul 20 17:55:53.199721 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963700    2569 flags.go:64] FLAG: --max-pods="110"
Jul 20 17:55:53.199721 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963703    2569 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 20 17:55:53.199721 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963706    2569 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 20 17:55:53.199721 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963708    2569 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 20 17:55:53.199721 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963711    2569 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 20 17:55:53.199721 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963714    2569 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 20 17:55:53.199721 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963717    2569 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 20 17:55:53.199721 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963720    2569 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 20 17:55:53.199721 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963727    2569 flags.go:64] FLAG: --node-status-max-images="50"
Jul 20 17:55:53.199721 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963732    2569 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 20 17:55:53.200523 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963735    2569 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 20 17:55:53.200523 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963738    2569 flags.go:64] FLAG: --pod-cidr=""
Jul 20 17:55:53.200523 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963741    2569 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0"
Jul 20 17:55:53.200523 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963747    2569 flags.go:64] FLAG: --pod-manifest-path=""
Jul 20 17:55:53.200523 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963750    2569 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 20 17:55:53.200523 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963753    2569 flags.go:64] FLAG: --pods-per-core="0"
Jul 20 17:55:53.200523 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963756    2569 flags.go:64] FLAG: --port="10250"
Jul 20 17:55:53.200523 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963759    2569 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 20 17:55:53.200523 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963762    2569 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0748c4744ba74973a"
Jul 20 17:55:53.200523 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963766    2569 flags.go:64] FLAG: --qos-reserved=""
Jul 20 17:55:53.200523 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963769    2569 flags.go:64] FLAG: --read-only-port="10255"
Jul 20 17:55:53.200523 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963772    2569 flags.go:64] FLAG: --register-node="true"
Jul 20 17:55:53.200523 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963775    2569 flags.go:64] FLAG: --register-with-taints=""
Jul 20 17:55:53.200523 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963779    2569 flags.go:64] FLAG: --registry-burst="10"
Jul 20 17:55:53.200523 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963782    2569 flags.go:64] FLAG: --registry-qps="5"
Jul 20 17:55:53.200523 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963785    2569 flags.go:64] FLAG: --reserved-cpus=""
Jul 20 17:55:53.200523 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963788    2569 flags.go:64] FLAG: --reserved-memory=""
Jul 20 17:55:53.200523 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963799    2569 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 20 17:55:53.200523 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963802    2569 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 20 17:55:53.200523 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963805    2569 flags.go:64] FLAG: --rotate-certificates="false"
Jul 20 17:55:53.200523 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963808    2569 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 20 17:55:53.200523 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963811    2569 flags.go:64] FLAG: --runonce="false"
Jul 20 17:55:53.200523 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963814    2569 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 20 17:55:53.200523 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963817    2569 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 20 17:55:53.200523 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963820    2569 flags.go:64] FLAG: --seccomp-default="false"
Jul 20 17:55:53.414193 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963823    2569 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 20 17:55:53.414193 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963825    2569 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 20 17:55:53.414193 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963828    2569 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 20 17:55:53.414193 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963831    2569 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 20 17:55:53.414193 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963834    2569 flags.go:64] FLAG: --storage-driver-password="root"
Jul 20 17:55:53.414193 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963837    2569 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 20 17:55:53.414193 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963842    2569 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 20 17:55:53.414193 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963845    2569 flags.go:64] FLAG: --storage-driver-user="root"
Jul 20 17:55:53.414193 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963848    2569 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 20 17:55:53.414193 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963851    2569 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 20 17:55:53.414193 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963853    2569 flags.go:64] FLAG: --system-cgroups=""
Jul 20 17:55:53.414193 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963856    2569 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 20 17:55:53.414193 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963862    2569 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 20 17:55:53.414193 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963864    2569 flags.go:64] FLAG: --tls-cert-file=""
Jul 20 17:55:53.414193 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963867    2569 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 20 17:55:53.414193 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963871    2569 flags.go:64] FLAG: --tls-min-version=""
Jul 20 17:55:53.414193 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963875    2569 flags.go:64] FLAG: --tls-private-key-file=""
Jul 20 17:55:53.414193 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963877    2569 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 20 17:55:53.414193 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963881    2569 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 20 17:55:53.414193 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963883    2569 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 20 17:55:53.414193 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963886    2569 flags.go:64] FLAG: --v="2"
Jul 20 17:55:53.414193 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963890    2569 flags.go:64] FLAG: --version="false"
Jul 20 17:55:53.414193 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963913    2569 flags.go:64] FLAG: --vmodule=""
Jul 20 17:55:53.414193 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963917    2569 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 20 17:55:53.414193 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.963921    2569 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 20 17:55:53.415199 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.964039    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 20 17:55:53.415199 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.964043    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 20 17:55:53.415199 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.964046    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 20 17:55:53.415199 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.964049    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 20 17:55:53.415199 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.964051    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 20 17:55:53.415199 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.964054    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 20 17:55:53.415199 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.964057    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 20 17:55:53.415199 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.964059    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 20 17:55:53.415199 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.964062    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 20 17:55:53.415199 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.964066    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 20 17:55:53.415199 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.964070    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 20 17:55:53.415199 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.964073    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 20 17:55:53.415199 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.964075    2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 20 17:55:53.415199 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.964079    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 20 17:55:53.415199 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.964082    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 20 17:55:53.415199 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.964085    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 20 17:55:53.415199 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.964088    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 20 17:55:53.415199 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.964090    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 20 17:55:53.415199 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.964093    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 20 17:55:53.415199 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.964096    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 20 17:55:53.415790 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.964098    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 20 17:55:53.415790 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.964101    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 20 17:55:53.415790 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.964105    2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 20 17:55:53.415790 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.964110    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 20 17:55:53.415790 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.964113    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 20 17:55:53.415790 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.964116    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 20 17:55:53.415790 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.964118    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 20 17:55:53.415790 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.964121    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 20 17:55:53.415790 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.964123    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 20 17:55:53.415790 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.964126    2569 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 20 17:55:53.415790 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.964129    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 20 17:55:53.415790 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.964132    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 20 17:55:53.415790 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.964134    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 20 17:55:53.415790 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.964137    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 20 17:55:53.415790 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.964139    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 20 17:55:53.415790 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.964142    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 20 17:55:53.415790 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.964144    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 20 17:55:53.415790 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.964147    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 20 17:55:53.415790 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.964150    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 20 17:55:53.416339 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.964152    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 20 17:55:53.416339 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.964156    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 20 17:55:53.416339 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.964159    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 20 17:55:53.416339 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.964162    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 20 17:55:53.416339 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.964165    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 20 17:55:53.416339 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.964168    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 20 17:55:53.416339 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.964172    2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 20 17:55:53.416339 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.964174    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 20 17:55:53.416339 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.964177    2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 20 17:55:53.416339 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.964179    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 20 17:55:53.416339 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.964182    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 20 17:55:53.416339 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.964185    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 20 17:55:53.416339 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.964187    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 20 17:55:53.416339 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.964190    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 20 17:55:53.416339 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.964193    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 20 17:55:53.416339 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.964195    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 20 17:55:53.416339 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.964199    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 20 17:55:53.416339 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.964201    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 20 17:55:53.416339 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.964204    2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 20 17:55:53.416852 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.964207    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 20 17:55:53.416852 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.964209    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 20 17:55:53.416852 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.964212    2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 20 17:55:53.416852 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.964214    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 20 17:55:53.416852 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.964217    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 20 17:55:53.416852 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.964220    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 20 17:55:53.416852 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.964223    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 20 17:55:53.416852 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.964226    2569 feature_gate.go:328] unrecognized feature gate: Example
Jul 20 17:55:53.416852 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.964228    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 20 17:55:53.416852 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.964231    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 20 17:55:53.416852 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.964233    2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 20 17:55:53.416852 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.964236    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 20 17:55:53.416852 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.964238    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 20 17:55:53.416852 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.964241    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 20 17:55:53.416852 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.964244    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 20 17:55:53.416852 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.964246    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 20 17:55:53.416852 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.964249    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 20 17:55:53.416852 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.964251    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 20 17:55:53.416852 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.964254    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 20 17:55:53.416852 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.964258    2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 20 17:55:53.417423 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.964261    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 20 17:55:53.417423 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.964263    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 20 17:55:53.417423 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.964266    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 20 17:55:53.417423 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.964269    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 20 17:55:53.417423 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.964271    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 20 17:55:53.417423 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.964274    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 20 17:55:53.417423 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.964277    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 20 17:55:53.417423 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.964279    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 20 17:55:53.417423 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.964282    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 20 17:55:53.417423 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.964286    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 20 17:55:53.417423 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.964289    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 20 17:55:53.417423 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.964291    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 20 17:55:53.417423 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.964294    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 20 17:55:53.417423 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.964297    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 20 17:55:53.417423 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.964300    2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 20 17:55:53.417423 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.964302    2569 feature_gate.go:328] unrecognized feature gate: Example2
Jul 20 17:55:53.417423 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.964305    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 20 17:55:53.417423 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.964307    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 20 17:55:53.417423 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.964310    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 20 17:55:53.417423 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.964313    2569 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 20 17:55:53.417971 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.964315    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 20 17:55:53.417971 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.964919    2569 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 20 17:55:53.417971 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.971395    2569 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 20 17:55:53.417971 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.971410    2569 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 20 17:55:53.417971 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.972044    2569 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 20 17:55:53.417971 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.972055    2569 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 20 17:55:53.417971 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972120    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 20 17:55:53.417971 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972126    2569 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 20 17:55:53.417971 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972129    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 20 17:55:53.417971 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972132    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 20 17:55:53.417971 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972134    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 20 17:55:53.417971 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972137    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 20 17:55:53.417971 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972142    2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 20 17:55:53.417971 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972146    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 20 17:55:53.417971 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972149    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 20 17:55:53.418416 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972152    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 20 17:55:53.418416 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972154    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 20 17:55:53.418416 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972157    2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 20 17:55:53.418416 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972160    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 20 17:55:53.418416 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972162    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 20 17:55:53.418416 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972165    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 20 17:55:53.418416 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972168    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 20 17:55:53.418416 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972170    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 20 17:55:53.418416 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972173    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 20 17:55:53.418416 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972176    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 20 17:55:53.418416 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972178    2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 20 17:55:53.418416 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972181    2569 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 20 17:55:53.418416 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972184    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 20 17:55:53.418416 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972187    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 20 17:55:53.418416 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972190    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 20 17:55:53.418416 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972192    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 20 17:55:53.418416 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972195    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 20 17:55:53.418416 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972198    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 20 17:55:53.418416 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972200    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 20 17:55:53.418416 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972203    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 20 17:55:53.418957 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972207    2569 feature_gate.go:328] unrecognized feature gate: Example
Jul 20 17:55:53.418957 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972210    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 20 17:55:53.418957 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972212    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 20 17:55:53.418957 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972215    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 20 17:55:53.418957 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972218    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 20 17:55:53.418957 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972221    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 20 17:55:53.418957 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972224    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 20 17:55:53.418957 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972226    2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 20 17:55:53.418957 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972229    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 20 17:55:53.418957 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972231    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 20 17:55:53.418957 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972234    2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 20 17:55:53.418957 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972237    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 20 17:55:53.418957 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972239    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 20 17:55:53.418957 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972242    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 20 17:55:53.418957 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972244    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 20 17:55:53.418957 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972247    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 20 17:55:53.418957 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972249    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 20 17:55:53.418957 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972252    2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 20 17:55:53.418957 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972254    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 20 17:55:53.419506 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972257    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 20 17:55:53.419506 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972261    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 20 17:55:53.419506 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972265    2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 20 17:55:53.419506 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972268    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 20 17:55:53.419506 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972270    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 20 17:55:53.419506 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972273    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 20 17:55:53.419506 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972275    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 20 17:55:53.419506 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972278    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 20 17:55:53.419506 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972281    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 20 17:55:53.419506 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972283    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 20 17:55:53.419506 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972286    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 20 17:55:53.419506 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972289    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 20 17:55:53.419506 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972292    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 20 17:55:53.419506 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972295    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 20 17:55:53.419506 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972301    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 20 17:55:53.419506 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972305    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 20 17:55:53.419506 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972309    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 20 17:55:53.419506 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972312    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 20 17:55:53.419506 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972315    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 20 17:55:53.420019 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972318    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 20 17:55:53.420019 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972321    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 20 17:55:53.420019 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972324    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 20 17:55:53.420019 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972330    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 20 17:55:53.420019 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972333    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 20 17:55:53.420019 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972335    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 20 17:55:53.420019 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972338    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 20 17:55:53.420019 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972340    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 20 17:55:53.420019 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972343    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 20 17:55:53.420019 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972345    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 20 17:55:53.420019 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972348    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 20 17:55:53.420019 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972351    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 20 17:55:53.420019 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972354    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 20 17:55:53.420019 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972356    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 20 17:55:53.420019 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972359    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 20 17:55:53.420019 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972361    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 20 17:55:53.420019 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972364    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 20 17:55:53.420019 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972367    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 20 17:55:53.420019 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972369    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 20 17:55:53.420019 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972372    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 20 17:55:53.420019 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972374    2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 20 17:55:53.420597 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972377    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 20 17:55:53.420597 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972379    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 20 17:55:53.420597 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972382    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 20 17:55:53.420597 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972384    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 20 17:55:53.420597 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972387    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 20 17:55:53.420597 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972389    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 20 17:55:53.420597 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972393    2569 feature_gate.go:328] unrecognized feature gate: Example2
Jul 20 17:55:53.420597 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972396    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 20 17:55:53.420597 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972399    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 20 17:55:53.420597 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972401    2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 20 17:55:53.420597 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972404    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 20 17:55:53.420597 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.972409    2569 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 20 17:55:53.420597 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972869    2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 20 17:55:53.420597 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972875    2569 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 20 17:55:53.420597 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972878    2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 20 17:55:53.421033 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972882    2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 20 17:55:53.421033 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972885    2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 20 17:55:53.421033 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972888    2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 20 17:55:53.421033 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972891    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 20 17:55:53.421033 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972909    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 20 17:55:53.421033 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972913    2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 20 17:55:53.421033 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972916    2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 20 17:55:53.421033 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972919    2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 20 17:55:53.421033 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972921    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 20 17:55:53.421033 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972924    2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 20 17:55:53.421033 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972927    2569 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 20 17:55:53.421033 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972930    2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 20 17:55:53.421033 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972932    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 20 17:55:53.421033 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972935    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 20 17:55:53.421033 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972938    2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 20 17:55:53.421033 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972940    2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 20 17:55:53.421033 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972943    2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 20 17:55:53.421033 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972946    2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 20 17:55:53.421033 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972950    2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 20 17:55:53.421601 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972954    2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 20 17:55:53.421601 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972956    2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 20 17:55:53.421601 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972959    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 20 17:55:53.421601 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972962    2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 20 17:55:53.421601 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972966    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 20 17:55:53.421601 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972969    2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 20 17:55:53.421601 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972972    2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 20 17:55:53.421601 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972975    2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 20 17:55:53.421601 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972977    2569 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 20 17:55:53.421601 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972980    2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 20 17:55:53.421601 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972983    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 20 17:55:53.421601 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972985    2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 20 17:55:53.421601 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972988    2569 feature_gate.go:328] unrecognized feature gate: Example
Jul 20 17:55:53.421601 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972992    2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 20 17:55:53.421601 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972995    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 20 17:55:53.421601 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.972998    2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 20 17:55:53.421601 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.973001    2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 20 17:55:53.421601 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.973003    2569 feature_gate.go:328] unrecognized feature gate: Example2
Jul 20 17:55:53.421601 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.973006    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 20 17:55:53.421601 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.973010    2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 20 17:55:53.422155 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.973013    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 20 17:55:53.422155 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.973016    2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 20 17:55:53.422155 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.973019    2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 20 17:55:53.422155 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.973021    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 20 17:55:53.422155 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.973024    2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 20 17:55:53.422155 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.973027    2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 20 17:55:53.422155 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.973030    2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 20 17:55:53.422155 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.973032    2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 20 17:55:53.422155 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.973035    2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 20 17:55:53.422155 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.973037    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 20 17:55:53.422155 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.973040    2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 20 17:55:53.422155 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.973042    2569 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 20 17:55:53.422155 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.973045    2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 20 17:55:53.422155 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.973047    2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 20 17:55:53.422155 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.973050    2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 20 17:55:53.422155 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.973053    2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 20 17:55:53.422155 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.973056    2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 20 17:55:53.422155 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.973060    2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 20 17:55:53.422155 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.973063    2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 20 17:55:53.422155 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.973066    2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 20 17:55:53.422750 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.973069    2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 20 17:55:53.422750 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.973071    2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 20 17:55:53.422750 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.973074    2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 20 17:55:53.422750 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.973076    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 20 17:55:53.422750 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.973079    2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 20 17:55:53.422750 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.973082    2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 20 17:55:53.422750 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.973085    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 20 17:55:53.422750 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.973087    2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 20 17:55:53.422750 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.973090    2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 20 17:55:53.422750 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.973092    2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 20 17:55:53.422750 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.973095    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 20 17:55:53.422750 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.973097    2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 20 17:55:53.422750 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.973100    2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 20 17:55:53.422750 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.973103    2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 20 17:55:53.422750 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.973105    2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 20 17:55:53.422750 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.973108    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 20 17:55:53.422750 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.973110    2569 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 20 17:55:53.422750 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.973113    2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 20 17:55:53.422750 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.973115    2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 20 17:55:53.423274 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.973118    2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 20 17:55:53.423274 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.973120    2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 20 17:55:53.423274 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.973123    2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 20 17:55:53.423274 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.973125    2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 20 17:55:53.423274 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.973128    2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 20 17:55:53.423274 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.973130    2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 20 17:55:53.423274 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.973133    2569 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 20 17:55:53.423274 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.973136    2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 20 17:55:53.423274 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.973138    2569 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 20 17:55:53.423274 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.973141    2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 20 17:55:53.423274 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.973144    2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 20 17:55:53.423274 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.973146    2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 20 17:55:53.423274 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.973149    2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 20 17:55:53.423274 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.973152    2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 20 17:55:53.423274 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.973155    2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 20 17:55:53.423274 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.973157    2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 20 17:55:53.423274 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.973160    2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 20 17:55:53.423274 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:52.973162    2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 20 17:55:53.423773 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.973169    2569 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 20 17:55:53.423773 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.973272    2569 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 20 17:55:53.423773 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.975188    2569 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 20 17:55:53.423773 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.976350    2569 server.go:1019] "Starting client certificate rotation"
Jul 20 17:55:53.423773 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.976449    2569 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 20 17:55:53.423773 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.976489    2569 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 20 17:55:53.423773 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.979914    2569 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 20 17:55:53.423773 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.982138    2569 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 20 17:55:53.423773 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.991793    2569 log.go:25] "Validated CRI v1 runtime API"
Jul 20 17:55:53.423773 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.993719    2569 log.go:25] "Validated CRI v1 image API"
Jul 20 17:55:53.423773 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.995975    2569 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 20 17:55:53.423773 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.997769    2569 fs.go:135] Filesystem UUIDs: map[74bf6367-68ed-4e0f-a2f6-b6772a3acc4b:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 fd103b1a-6ea7-4240-8648-e7bddb9d8538:/dev/nvme0n1p3]
Jul 20 17:55:53.424161 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:52.997783    2569 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 20 17:55:53.424265 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:53.003008    2569 manager.go:216] Machine: {Timestamp:2026-07-20 17:55:53.001918704 +0000 UTC m=+0.276277074 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100876 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2828172883f3f33601e0fae22f5632 SystemUUID:ec282817-2883-f3f3-3601-e0fae22f5632 BootID:7b7a420a-1d64-4e79-8ece-61adacd8ce7d Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:4c:0c:cb:da:e9 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:4c:0c:cb:da:e9 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:12:0b:a4:e0:d2:75 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 20 17:55:53.424265 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:53.003104    2569 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 20 17:55:53.424265 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:53.003181    2569 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 20 17:55:53.424265 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:53.005578    2569 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 20 17:55:53.424265 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:53.005601    2569 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-247.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 20 17:55:53.424265 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:53.006219    2569 topology_manager.go:138] "Creating topology manager with none policy"
Jul 20 17:55:53.424265 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:53.006230    2569 container_manager_linux.go:309] "Creating device plugin manager"
Jul 20 17:55:53.424265 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:53.006237    2569 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 20 17:55:53.424978 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:53.006745    2569 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 20 17:55:53.424978 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:53.008017    2569 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 20 17:55:53.424978 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:53.008786    2569 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 20 17:55:53.424978 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:53.011865    2569 state_mem.go:36] "Initialized new in-memory state store"
Jul 20 17:55:53.424978 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:53.011996    2569 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 20 17:55:53.424978 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:53.012025    2569 kubelet.go:486] "Attempting to sync node with API server"
Jul 20 17:55:53.424978 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:53.012034    2569 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 20 17:55:53.424978 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:53.012048    2569 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 20 17:55:53.424978 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:53.012058    2569 kubelet.go:398] "Adding apiserver pod source"
Jul 20 17:55:53.424978 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:53.012067    2569 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 20 17:55:53.424978 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:53.012377    2569 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 20 17:55:53.424978 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:53.012392    2569 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 20 17:55:53.424978 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:53.013980    2569 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 20 17:55:53.424978 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:53.015973    2569 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 20 17:55:53.424978 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:53.016000    2569 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 20 17:55:53.424978 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:53.017001    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 20 17:55:53.424978 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:53.017022    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 20 17:55:53.424978 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:53.017051    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 20 17:55:53.424978 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:53.017070    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 20 17:55:53.424978 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:53.017087    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 20 17:55:53.424978 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:53.017100    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 20 17:55:53.424978 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:53.017117    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 20 17:55:53.424978 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:53.017127    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 20 17:55:53.424978 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:53.017137    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 20 17:55:53.424978 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:53.017143    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 20 17:55:53.424978 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:53.017157    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 20 17:55:53.424978 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:53.017164    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 20 17:55:53.424978 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:53.018026    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 20 17:55:53.424978 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:53.018299    2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 20 17:55:53.424978 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:53.022155    2569 server.go:1268] "Started kubelet"
Jul 20 17:55:53.424978 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:53.022292    2569 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 20 17:55:53.424978 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:53.022303    2569 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 20 17:55:53.424978 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:53.022496    2569 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 20 17:55:53.424978 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:53.024042    2569 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 20 17:55:53.424978 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:53.025556    2569 server.go:310] "Adding debug handlers to kubelet server"
Jul 20 17:55:53.424978 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:53.027770    2569 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-137-247.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 20 17:55:53.424978 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:55:53.028035    2569 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-137-247.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 20 17:55:53.424978 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:55:53.028205    2569 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 20 17:55:53.424978 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:55:53.028373    2569 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 20 17:55:54.663603 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:53.032679    2569 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 20 17:55:54.663603 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:53.032686    2569 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 20 17:55:54.663603 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:55:53.032748    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-247.ec2.internal\" not found"
Jul 20 17:55:54.663603 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:53.033179    2569 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 20 17:55:54.663603 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:53.033195    2569 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 20 17:55:54.663603 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:53.033212    2569 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 20 17:55:54.663603 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:53.033320    2569 reconstruct.go:97] "Volume reconstruction finished"
Jul 20 17:55:54.663603 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:53.033329    2569 reconciler.go:29] "Reconciler: start to sync state"
Jul 20 17:55:54.663603 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:53.033655    2569 factory.go:55] Registering systemd factory
Jul 20 17:55:54.663603 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:53.033668    2569 factory.go:223] Registration of the systemd container factory successfully
Jul 20 17:55:54.663603 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:53.033853    2569 factory.go:153] Registering CRI-O factory
Jul 20 17:55:54.663603 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:53.033864    2569 factory.go:223] Registration of the crio container factory successfully
Jul 20 17:55:54.663603 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:53.033936    2569 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 20 17:55:54.663603 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:53.033959    2569 factory.go:103] Registering Raw factory
Jul 20 17:55:54.663603 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:53.033973    2569 manager.go:1196] Started watching for new ooms in manager
Jul 20 17:55:54.663603 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:53.034398    2569 manager.go:349] Starting recovery of all containers
Jul 20 17:55:54.663603 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:55:53.036747    2569 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-137-247.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 20 17:55:54.663603 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:55:53.036793    2569 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-247.ec2.internal.18c411195fa59a98  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-247.ec2.internal,UID:ip-10-0-137-247.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-137-247.ec2.internal,},FirstTimestamp:2026-07-20 17:55:53.022102168 +0000 UTC m=+0.296460537,LastTimestamp:2026-07-20 17:55:53.022102168 +0000 UTC m=+0.296460537,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-247.ec2.internal,}"
Jul 20 17:55:54.663603 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:55:53.037832    2569 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 20 17:55:54.663603 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:53.044488    2569 manager.go:354] Recovery completed
Jul 20 17:55:54.663603 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:53.050004    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 20 17:55:54.663603 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:53.051879    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-247.ec2.internal" event="NodeHasSufficientMemory"
Jul 20 17:55:54.663603 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:53.051925    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-247.ec2.internal" event="NodeHasNoDiskPressure"
Jul 20 17:55:54.663603 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:53.051936    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-247.ec2.internal" event="NodeHasSufficientPID"
Jul 20 17:55:54.663603 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:53.052474    2569 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 20 17:55:54.663603 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:53.052481    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 20 17:55:54.663603 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:53.052486    2569 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 20 17:55:54.663603 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:53.052504    2569 state_mem.go:36] "Initialized new in-memory state store"
Jul 20 17:55:54.663603 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:53.053155    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-247.ec2.internal" event="NodeHasSufficientMemory"
Jul 20 17:55:54.663603 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:53.053178    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-247.ec2.internal" event="NodeHasNoDiskPressure"
Jul 20 17:55:54.663603 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:53.053192    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-247.ec2.internal" event="NodeHasSufficientPID"
Jul 20 17:55:54.663603 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:53.054685    2569 policy_none.go:49] "None policy: Start"
Jul 20 17:55:54.663603 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:53.054697    2569 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 20 17:55:54.663603 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:53.054749    2569 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 20 17:55:54.667903 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:53.056774    2569 policy_none.go:47] "Start"
Jul 20 17:55:54.667903 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:55:53.066783    2569 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-247.ec2.internal.18c41119616c7667  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-247.ec2.internal,UID:ip-10-0-137-247.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-137-247.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-137-247.ec2.internal,},FirstTimestamp:2026-07-20 17:55:53.051911783 +0000 UTC m=+0.326270154,LastTimestamp:2026-07-20 17:55:53.051911783 +0000 UTC m=+0.326270154,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-247.ec2.internal,}"
Jul 20 17:55:54.667903 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:55:53.078182    2569 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-247.ec2.internal.18c41119616cbfd6  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-247.ec2.internal,UID:ip-10-0-137-247.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-137-247.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-137-247.ec2.internal,},FirstTimestamp:2026-07-20 17:55:53.051930582 +0000 UTC m=+0.326288951,LastTimestamp:2026-07-20 17:55:53.051930582 +0000 UTC m=+0.326288951,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-247.ec2.internal,}"
Jul 20 17:55:54.667903 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:55:53.088682    2569 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-247.ec2.internal.18c41119616ce553  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-247.ec2.internal,UID:ip-10-0-137-247.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-137-247.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-137-247.ec2.internal,},FirstTimestamp:2026-07-20 17:55:53.051940179 +0000 UTC m=+0.326298549,LastTimestamp:2026-07-20 17:55:53.051940179 +0000 UTC m=+0.326298549,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-247.ec2.internal,}"
Jul 20 17:55:54.667903 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:53.092418    2569 manager.go:337] "Starting Device Plugin manager"
Jul 20 17:55:54.667903 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:55:53.092444    2569 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 20 17:55:54.667903 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:53.092453    2569 server.go:87] "Starting device plugin registration server"
Jul 20 17:55:54.667903 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:53.092685    2569 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 20 17:55:54.667903 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:53.092694    2569 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 20 17:55:54.667903 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:53.092756    2569 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 20 17:55:54.667903 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:53.092828    2569 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 20 17:55:54.667903 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:53.092837    2569 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 20 17:55:54.667903 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:55:53.093350    2569 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 20 17:55:54.667903 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:55:53.093409    2569 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-247.ec2.internal\" not found"
Jul 20 17:55:54.668419 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:55:53.098288    2569 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-137-247.ec2.internal.18c41119616c7667\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-247.ec2.internal.18c41119616c7667  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-247.ec2.internal,UID:ip-10-0-137-247.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-137-247.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-137-247.ec2.internal,},FirstTimestamp:2026-07-20 17:55:53.051911783 +0000 UTC m=+0.326270154,LastTimestamp:2026-07-20 17:55:53.053166435 +0000 UTC m=+0.327524805,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-247.ec2.internal,}"
Jul 20 17:55:54.668419 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:55:53.108950    2569 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-137-247.ec2.internal.18c41119616cbfd6\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-247.ec2.internal.18c41119616cbfd6  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-247.ec2.internal,UID:ip-10-0-137-247.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-137-247.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-137-247.ec2.internal,},FirstTimestamp:2026-07-20 17:55:53.051930582 +0000 UTC m=+0.326288951,LastTimestamp:2026-07-20 17:55:53.053183378 +0000 UTC m=+0.327541748,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-247.ec2.internal,}"
Jul 20 17:55:54.668419 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:53.115325    2569 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-chgd6"
Jul 20 17:55:54.668419 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:55:53.118608    2569 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-137-247.ec2.internal.18c41119616ce553\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-247.ec2.internal.18c41119616ce553  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-247.ec2.internal,UID:ip-10-0-137-247.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-137-247.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-137-247.ec2.internal,},FirstTimestamp:2026-07-20 17:55:53.051940179 +0000 UTC m=+0.326298549,LastTimestamp:2026-07-20 17:55:53.05319772 +0000 UTC m=+0.327556091,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-247.ec2.internal,}"
Jul 20 17:55:54.668419 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:53.127316    2569 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-chgd6"
Jul 20 17:55:54.668419 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:55:53.128848    2569 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-247.ec2.internal.18c4111963f5f3ba  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-247.ec2.internal,UID:ip-10-0-137-247.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeAllocatableEnforced,Message:Updated Node Allocatable limit across pods,Source:EventSource{Component:kubelet,Host:ip-10-0-137-247.ec2.internal,},FirstTimestamp:2026-07-20 17:55:53.09447673 +0000 UTC m=+0.368835091,LastTimestamp:2026-07-20 17:55:53.09447673 +0000 UTC m=+0.368835091,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-247.ec2.internal,}"
Jul 20 17:55:54.668419 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:53.147248    2569 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 20 17:55:54.668419 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:53.148427    2569 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 20 17:55:54.668419 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:53.148466    2569 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 20 17:55:54.668419 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:53.148487    2569 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 20 17:55:54.668419 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:55:53.148524    2569 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 20 17:55:54.668813 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:53.159508    2569 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 20 17:55:54.668813 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:53.193655    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 20 17:55:54.668813 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:53.201002    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-247.ec2.internal" event="NodeHasSufficientMemory"
Jul 20 17:55:54.668813 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:53.201035    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-247.ec2.internal" event="NodeHasNoDiskPressure"
Jul 20 17:55:54.668813 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:53.201048    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-247.ec2.internal" event="NodeHasSufficientPID"
Jul 20 17:55:54.668813 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:53.201076    2569 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-137-247.ec2.internal"
Jul 20 17:55:54.668813 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:53.209092    2569 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-137-247.ec2.internal"
Jul 20 17:55:54.668813 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:55:53.209109    2569 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-137-247.ec2.internal\": node \"ip-10-0-137-247.ec2.internal\" not found"
Jul 20 17:55:54.668813 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:55:53.231268    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-247.ec2.internal\" not found"
Jul 20 17:55:54.668813 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:53.249467    2569 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-247.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-247.ec2.internal"]
Jul 20 17:55:54.668813 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:53.249517    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 20 17:55:54.668813 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:53.250858    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-247.ec2.internal" event="NodeHasSufficientMemory"
Jul 20 17:55:54.668813 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:53.250880    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-247.ec2.internal" event="NodeHasNoDiskPressure"
Jul 20 17:55:54.668813 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:53.250889    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-247.ec2.internal" event="NodeHasSufficientPID"
Jul 20 17:55:54.668813 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:53.253160    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 20 17:55:54.668813 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:53.253295    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-247.ec2.internal"
Jul 20 17:55:54.668813 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:53.253319    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 20 17:55:54.668813 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:53.253912    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-247.ec2.internal" event="NodeHasSufficientMemory"
Jul 20 17:55:54.668813 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:53.253935    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-247.ec2.internal" event="NodeHasNoDiskPressure"
Jul 20 17:55:54.668813 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:53.253945    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-247.ec2.internal" event="NodeHasSufficientPID"
Jul 20 17:55:54.668813 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:53.253915    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-247.ec2.internal" event="NodeHasSufficientMemory"
Jul 20 17:55:54.668813 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:53.254004    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-247.ec2.internal" event="NodeHasNoDiskPressure"
Jul 20 17:55:54.668813 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:53.254013    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-247.ec2.internal" event="NodeHasSufficientPID"
Jul 20 17:55:54.668813 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:53.256404    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-247.ec2.internal"
Jul 20 17:55:54.668813 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:53.256424    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 20 17:55:54.668813 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:53.261160    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-247.ec2.internal" event="NodeHasSufficientMemory"
Jul 20 17:55:54.668813 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:53.261187    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-247.ec2.internal" event="NodeHasNoDiskPressure"
Jul 20 17:55:54.668813 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:53.261201    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-247.ec2.internal" event="NodeHasSufficientPID"
Jul 20 17:55:54.668813 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:55:53.282425    2569 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-247.ec2.internal\" not found" node="ip-10-0-137-247.ec2.internal"
Jul 20 17:55:54.668813 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:55:53.288102    2569 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-247.ec2.internal\" not found" node="ip-10-0-137-247.ec2.internal"
Jul 20 17:55:54.668813 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:55:53.334314    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-247.ec2.internal\" not found"
Jul 20 17:55:54.668813 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:55:53.434473    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-247.ec2.internal\" not found"
Jul 20 17:55:54.669672 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:53.434529    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7c055eb6eb0be9ef4618781875e36f04-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-247.ec2.internal\" (UID: \"7c055eb6eb0be9ef4618781875e36f04\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-247.ec2.internal"
Jul 20 17:55:54.669672 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:53.434552    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7c055eb6eb0be9ef4618781875e36f04-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-247.ec2.internal\" (UID: \"7c055eb6eb0be9ef4618781875e36f04\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-247.ec2.internal"
Jul 20 17:55:54.669672 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:53.434572    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/757e30ff139839aada6b39c2403a950d-config\") pod \"kube-apiserver-proxy-ip-10-0-137-247.ec2.internal\" (UID: \"757e30ff139839aada6b39c2403a950d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-247.ec2.internal"
Jul 20 17:55:54.669672 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:55:53.535082    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-247.ec2.internal\" not found"
Jul 20 17:55:54.669672 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:53.535128    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/757e30ff139839aada6b39c2403a950d-config\") pod \"kube-apiserver-proxy-ip-10-0-137-247.ec2.internal\" (UID: \"757e30ff139839aada6b39c2403a950d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-247.ec2.internal"
Jul 20 17:55:54.669672 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:53.535149    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7c055eb6eb0be9ef4618781875e36f04-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-247.ec2.internal\" (UID: \"7c055eb6eb0be9ef4618781875e36f04\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-247.ec2.internal"
Jul 20 17:55:54.669672 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:53.535169    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7c055eb6eb0be9ef4618781875e36f04-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-247.ec2.internal\" (UID: \"7c055eb6eb0be9ef4618781875e36f04\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-247.ec2.internal"
Jul 20 17:55:54.669672 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:53.535218    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7c055eb6eb0be9ef4618781875e36f04-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-247.ec2.internal\" (UID: \"7c055eb6eb0be9ef4618781875e36f04\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-247.ec2.internal"
Jul 20 17:55:54.669672 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:53.535222    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/757e30ff139839aada6b39c2403a950d-config\") pod \"kube-apiserver-proxy-ip-10-0-137-247.ec2.internal\" (UID: \"757e30ff139839aada6b39c2403a950d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-247.ec2.internal"
Jul 20 17:55:54.669672 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:53.535248    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7c055eb6eb0be9ef4618781875e36f04-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-247.ec2.internal\" (UID: \"7c055eb6eb0be9ef4618781875e36f04\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-247.ec2.internal"
Jul 20 17:55:54.669672 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:54.613085    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-247.ec2.internal"
Jul 20 17:55:54.669672 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:54.619478    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-247.ec2.internal"
Jul 20 17:55:54.669672 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:55:54.662959    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-247.ec2.internal\" not found"
Jul 20 17:55:54.880094 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:55:54.763344    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-247.ec2.internal\" not found"
Jul 20 17:55:54.880094 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:55:54.864090    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-247.ec2.internal\" not found"
Jul 20 17:55:54.880094 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:54.864241    2569 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 20 17:55:55.094316 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:55:54.964525    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-247.ec2.internal\" not found"
Jul 20 17:55:55.094316 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:55.003747    2569 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 20 17:55:55.094316 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:55.003910    2569 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 20 17:55:55.094316 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:55.003917    2569 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 20 17:55:55.094316 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:55.028416    2569 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 20 17:55:55.094316 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:55:55.065172    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-247.ec2.internal\" not found"
Jul 20 17:55:55.303952 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:55.159710    2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-19 17:50:54 +0000 UTC" deadline="2028-01-02 22:15:36.441183543 +0000 UTC"
Jul 20 17:55:55.303952 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:55.159764    2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12748h19m41.281422386s"
Jul 20 17:55:55.303952 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:55:55.165989    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-247.ec2.internal\" not found"
Jul 20 17:55:55.303952 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:55:55.266196    2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-247.ec2.internal\" not found"
Jul 20 17:55:55.303952 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:55.270391    2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 20 17:55:55.303952 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:55.271434    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-247.ec2.internal" event="NodeHasSufficientMemory"
Jul 20 17:55:55.303952 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:55.271461    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-247.ec2.internal" event="NodeHasNoDiskPressure"
Jul 20 17:55:55.303952 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:55.271472    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-247.ec2.internal" event="NodeHasSufficientPID"
Jul 20 17:55:55.503639 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:55.317485    2569 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 20 17:55:55.503639 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:55.361431    2569 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-247.ec2.internal"
Jul 20 17:55:55.503639 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:55.381031    2569 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 20 17:55:55.503639 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:55.381862    2569 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-247.ec2.internal"
Jul 20 17:55:55.503639 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:55.391909    2569 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 20 17:55:55.708018 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:55.632597    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c055eb6eb0be9ef4618781875e36f04.slice/crio-2d80bc0956311db459e8714ebdd7dc05f2531a7a0d27726783df5f073ccb7647 WatchSource:0}: Error finding container 2d80bc0956311db459e8714ebdd7dc05f2531a7a0d27726783df5f073ccb7647: Status 404 returned error can't find the container with id 2d80bc0956311db459e8714ebdd7dc05f2531a7a0d27726783df5f073ccb7647
Jul 20 17:55:55.708018 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:55.638915    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 20 17:55:55.708018 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:55.644565    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod757e30ff139839aada6b39c2403a950d.slice/crio-a7fecbe7b43832507a90c09515b3a9b645acdbbc2a0eb061c8712ce91f8de16d WatchSource:0}: Error finding container a7fecbe7b43832507a90c09515b3a9b645acdbbc2a0eb061c8712ce91f8de16d: Status 404 returned error can't find the container with id a7fecbe7b43832507a90c09515b3a9b645acdbbc2a0eb061c8712ce91f8de16d
Jul 20 17:55:56.088217 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:55.986669    2569 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 20 17:55:56.088217 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.040682    2569 apiserver.go:52] "Watching apiserver"
Jul 20 17:55:56.088217 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.047853    2569 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 20 17:55:56.088217 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.048181    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-ql9lx","openshift-cluster-node-tuning-operator/tuned-q6gcb","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-247.ec2.internal","openshift-multus/multus-additional-cni-plugins-mzsnt","openshift-multus/multus-rqtm8","openshift-network-diagnostics/network-check-target-8v5ff","openshift-ovn-kubernetes/ovnkube-node-wjs8g","kube-system/global-pull-secret-syncer-tgg22","kube-system/konnectivity-agent-l45qp","kube-system/kube-apiserver-proxy-ip-10-0-137-247.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p74sv","openshift-dns/node-resolver-smcqb","openshift-image-registry/node-ca-q42fv","openshift-multus/network-metrics-daemon-nkmr4"]
Jul 20 17:55:56.088217 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.051056    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-wjs8g"
Jul 20 17:55:56.088217 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.053300    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-6jj8h\""
Jul 20 17:55:56.088217 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.053639    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-q42fv"
Jul 20 17:55:56.088217 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.053975    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 20 17:55:56.088217 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.054020    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 20 17:55:56.088217 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.054025    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 20 17:55:56.088217 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.054070    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 20 17:55:56.088217 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.054116    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 20 17:55:56.088217 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.054122    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 20 17:55:56.088217 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.055785    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nkmr4"
Jul 20 17:55:56.088217 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:55:56.055858    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nkmr4" podUID="26b0d75c-ecf7-4bb4-ad4b-d9077da808bd"
Jul 20 17:55:56.088217 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.058046    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-mzsnt"
Jul 20 17:55:56.088217 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.060205    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-rqtm8"
Jul 20 17:55:56.088217 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.061753    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-jl7wq\""
Jul 20 17:55:56.088217 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.061884    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 20 17:55:56.088217 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.061923    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 20 17:55:56.088217 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.061977    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 20 17:55:56.088217 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.062093    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 20 17:55:56.088217 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.062108    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 20 17:55:56.088217 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.062164    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-7hjbp\""
Jul 20 17:55:56.088217 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.062431    2569 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 20 17:55:56.088217 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.062747    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8v5ff"
Jul 20 17:55:56.088217 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.062851    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 20 17:55:56.089741 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:55:56.062825    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8v5ff" podUID="de1b439b-702a-43a3-8118-f7e20e716ac5"
Jul 20 17:55:56.089741 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.063280    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-nf987\""
Jul 20 17:55:56.089741 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.063344    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 20 17:55:56.089741 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.063830    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 20 17:55:56.089741 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.064048    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 20 17:55:56.089741 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.065368    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-l45qp"
Jul 20 17:55:56.089741 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.067737    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-ql9lx"
Jul 20 17:55:56.089741 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.068390    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 20 17:55:56.089741 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.068707    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 20 17:55:56.089741 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.068904    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-dkn9l\""
Jul 20 17:55:56.089741 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.070012    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 20 17:55:56.089741 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.070064    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-q6gcb"
Jul 20 17:55:56.089741 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.070394    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d5f812b2-2ecb-4be4-99aa-7808ab7154e4-etc-kubernetes\") pod \"multus-rqtm8\" (UID: \"d5f812b2-2ecb-4be4-99aa-7808ab7154e4\") " pod="openshift-multus/multus-rqtm8"
Jul 20 17:55:56.089741 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.070414    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/07587a78-2d3a-40ad-9aae-a4875200263a-node-log\") pod \"ovnkube-node-wjs8g\" (UID: \"07587a78-2d3a-40ad-9aae-a4875200263a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjs8g"
Jul 20 17:55:56.089741 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.070428    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/07587a78-2d3a-40ad-9aae-a4875200263a-host-cni-netd\") pod \"ovnkube-node-wjs8g\" (UID: \"07587a78-2d3a-40ad-9aae-a4875200263a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjs8g"
Jul 20 17:55:56.089741 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.070443    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1dfc0381-8b5e-4400-a1c8-da99129540c7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mzsnt\" (UID: \"1dfc0381-8b5e-4400-a1c8-da99129540c7\") " pod="openshift-multus/multus-additional-cni-plugins-mzsnt"
Jul 20 17:55:56.089741 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.070459    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sm9p5\" (UniqueName: \"kubernetes.io/projected/1dfc0381-8b5e-4400-a1c8-da99129540c7-kube-api-access-sm9p5\") pod \"multus-additional-cni-plugins-mzsnt\" (UID: \"1dfc0381-8b5e-4400-a1c8-da99129540c7\") " pod="openshift-multus/multus-additional-cni-plugins-mzsnt"
Jul 20 17:55:56.089741 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.070496    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d5f812b2-2ecb-4be4-99aa-7808ab7154e4-os-release\") pod \"multus-rqtm8\" (UID: \"d5f812b2-2ecb-4be4-99aa-7808ab7154e4\") " pod="openshift-multus/multus-rqtm8"
Jul 20 17:55:56.089741 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.070529    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m594d\" (UniqueName: \"kubernetes.io/projected/d5f812b2-2ecb-4be4-99aa-7808ab7154e4-kube-api-access-m594d\") pod \"multus-rqtm8\" (UID: \"d5f812b2-2ecb-4be4-99aa-7808ab7154e4\") " pod="openshift-multus/multus-rqtm8"
Jul 20 17:55:56.089741 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.070540    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 20 17:55:56.091076 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.070555    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/07587a78-2d3a-40ad-9aae-a4875200263a-host-kubelet\") pod \"ovnkube-node-wjs8g\" (UID: \"07587a78-2d3a-40ad-9aae-a4875200263a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjs8g"
Jul 20 17:55:56.091076 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.070581    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/07587a78-2d3a-40ad-9aae-a4875200263a-host-slash\") pod \"ovnkube-node-wjs8g\" (UID: \"07587a78-2d3a-40ad-9aae-a4875200263a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjs8g"
Jul 20 17:55:56.091076 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.070540    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 20 17:55:56.091076 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.070620    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/07587a78-2d3a-40ad-9aae-a4875200263a-host-run-netns\") pod \"ovnkube-node-wjs8g\" (UID: \"07587a78-2d3a-40ad-9aae-a4875200263a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjs8g"
Jul 20 17:55:56.091076 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.070645    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-mn7cb\""
Jul 20 17:55:56.091076 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.070653    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/07587a78-2d3a-40ad-9aae-a4875200263a-etc-openvswitch\") pod \"ovnkube-node-wjs8g\" (UID: \"07587a78-2d3a-40ad-9aae-a4875200263a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjs8g"
Jul 20 17:55:56.091076 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.070677    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/07587a78-2d3a-40ad-9aae-a4875200263a-run-openvswitch\") pod \"ovnkube-node-wjs8g\" (UID: \"07587a78-2d3a-40ad-9aae-a4875200263a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjs8g"
Jul 20 17:55:56.091076 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.070700    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/07587a78-2d3a-40ad-9aae-a4875200263a-run-ovn\") pod \"ovnkube-node-wjs8g\" (UID: \"07587a78-2d3a-40ad-9aae-a4875200263a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjs8g"
Jul 20 17:55:56.091076 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.070763    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d5f812b2-2ecb-4be4-99aa-7808ab7154e4-cni-binary-copy\") pod \"multus-rqtm8\" (UID: \"d5f812b2-2ecb-4be4-99aa-7808ab7154e4\") " pod="openshift-multus/multus-rqtm8"
Jul 20 17:55:56.091076 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.070793    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d5f812b2-2ecb-4be4-99aa-7808ab7154e4-host-run-k8s-cni-cncf-io\") pod \"multus-rqtm8\" (UID: \"d5f812b2-2ecb-4be4-99aa-7808ab7154e4\") " pod="openshift-multus/multus-rqtm8"
Jul 20 17:55:56.091076 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.070835    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/07587a78-2d3a-40ad-9aae-a4875200263a-ovn-node-metrics-cert\") pod \"ovnkube-node-wjs8g\" (UID: \"07587a78-2d3a-40ad-9aae-a4875200263a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjs8g"
Jul 20 17:55:56.091076 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.070858    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1dfc0381-8b5e-4400-a1c8-da99129540c7-cni-binary-copy\") pod \"multus-additional-cni-plugins-mzsnt\" (UID: \"1dfc0381-8b5e-4400-a1c8-da99129540c7\") " pod="openshift-multus/multus-additional-cni-plugins-mzsnt"
Jul 20 17:55:56.091076 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.070907    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d5f812b2-2ecb-4be4-99aa-7808ab7154e4-multus-cni-dir\") pod \"multus-rqtm8\" (UID: \"d5f812b2-2ecb-4be4-99aa-7808ab7154e4\") " pod="openshift-multus/multus-rqtm8"
Jul 20 17:55:56.091076 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.070934    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d5f812b2-2ecb-4be4-99aa-7808ab7154e4-host-run-netns\") pod \"multus-rqtm8\" (UID: \"d5f812b2-2ecb-4be4-99aa-7808ab7154e4\") " pod="openshift-multus/multus-rqtm8"
Jul 20 17:55:56.091076 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.070957    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d5f812b2-2ecb-4be4-99aa-7808ab7154e4-host-var-lib-cni-multus\") pod \"multus-rqtm8\" (UID: \"d5f812b2-2ecb-4be4-99aa-7808ab7154e4\") " pod="openshift-multus/multus-rqtm8"
Jul 20 17:55:56.092279 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.070987    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d5f812b2-2ecb-4be4-99aa-7808ab7154e4-multus-conf-dir\") pod \"multus-rqtm8\" (UID: \"d5f812b2-2ecb-4be4-99aa-7808ab7154e4\") " pod="openshift-multus/multus-rqtm8"
Jul 20 17:55:56.092279 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.071006    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/07587a78-2d3a-40ad-9aae-a4875200263a-host-cni-bin\") pod \"ovnkube-node-wjs8g\" (UID: \"07587a78-2d3a-40ad-9aae-a4875200263a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjs8g"
Jul 20 17:55:56.092279 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.071026    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/07587a78-2d3a-40ad-9aae-a4875200263a-env-overrides\") pod \"ovnkube-node-wjs8g\" (UID: \"07587a78-2d3a-40ad-9aae-a4875200263a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjs8g"
Jul 20 17:55:56.092279 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.071042    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9l5j\" (UniqueName: \"kubernetes.io/projected/07587a78-2d3a-40ad-9aae-a4875200263a-kube-api-access-v9l5j\") pod \"ovnkube-node-wjs8g\" (UID: \"07587a78-2d3a-40ad-9aae-a4875200263a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjs8g"
Jul 20 17:55:56.092279 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.071056    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7a564ed7-d4b4-4f1a-a7da-d1e4d0652601-host\") pod \"node-ca-q42fv\" (UID: \"7a564ed7-d4b4-4f1a-a7da-d1e4d0652601\") " pod="openshift-image-registry/node-ca-q42fv"
Jul 20 17:55:56.092279 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.071072    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d5f812b2-2ecb-4be4-99aa-7808ab7154e4-multus-socket-dir-parent\") pod \"multus-rqtm8\" (UID: \"d5f812b2-2ecb-4be4-99aa-7808ab7154e4\") " pod="openshift-multus/multus-rqtm8"
Jul 20 17:55:56.092279 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.071093    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/07587a78-2d3a-40ad-9aae-a4875200263a-log-socket\") pod \"ovnkube-node-wjs8g\" (UID: \"07587a78-2d3a-40ad-9aae-a4875200263a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjs8g"
Jul 20 17:55:56.092279 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.071118    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1dfc0381-8b5e-4400-a1c8-da99129540c7-os-release\") pod \"multus-additional-cni-plugins-mzsnt\" (UID: \"1dfc0381-8b5e-4400-a1c8-da99129540c7\") " pod="openshift-multus/multus-additional-cni-plugins-mzsnt"
Jul 20 17:55:56.092279 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.071133    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d5f812b2-2ecb-4be4-99aa-7808ab7154e4-multus-daemon-config\") pod \"multus-rqtm8\" (UID: \"d5f812b2-2ecb-4be4-99aa-7808ab7154e4\") " pod="openshift-multus/multus-rqtm8"
Jul 20 17:55:56.092279 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.071150    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/07587a78-2d3a-40ad-9aae-a4875200263a-systemd-units\") pod \"ovnkube-node-wjs8g\" (UID: \"07587a78-2d3a-40ad-9aae-a4875200263a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjs8g"
Jul 20 17:55:56.092279 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.071168    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/07587a78-2d3a-40ad-9aae-a4875200263a-var-lib-openvswitch\") pod \"ovnkube-node-wjs8g\" (UID: \"07587a78-2d3a-40ad-9aae-a4875200263a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjs8g"
Jul 20 17:55:56.092279 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.071182    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/07587a78-2d3a-40ad-9aae-a4875200263a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wjs8g\" (UID: \"07587a78-2d3a-40ad-9aae-a4875200263a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjs8g"
Jul 20 17:55:56.092279 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.071196    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7a564ed7-d4b4-4f1a-a7da-d1e4d0652601-serviceca\") pod \"node-ca-q42fv\" (UID: \"7a564ed7-d4b4-4f1a-a7da-d1e4d0652601\") " pod="openshift-image-registry/node-ca-q42fv"
Jul 20 17:55:56.092279 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.071228    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26b0d75c-ecf7-4bb4-ad4b-d9077da808bd-metrics-certs\") pod \"network-metrics-daemon-nkmr4\" (UID: \"26b0d75c-ecf7-4bb4-ad4b-d9077da808bd\") " pod="openshift-multus/network-metrics-daemon-nkmr4"
Jul 20 17:55:56.092878 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.071253    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9csjm\" (UniqueName: \"kubernetes.io/projected/26b0d75c-ecf7-4bb4-ad4b-d9077da808bd-kube-api-access-9csjm\") pod \"network-metrics-daemon-nkmr4\" (UID: \"26b0d75c-ecf7-4bb4-ad4b-d9077da808bd\") " pod="openshift-multus/network-metrics-daemon-nkmr4"
Jul 20 17:55:56.092878 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.071277    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d5f812b2-2ecb-4be4-99aa-7808ab7154e4-host-var-lib-cni-bin\") pod \"multus-rqtm8\" (UID: \"d5f812b2-2ecb-4be4-99aa-7808ab7154e4\") " pod="openshift-multus/multus-rqtm8"
Jul 20 17:55:56.092878 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.071293    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d5f812b2-2ecb-4be4-99aa-7808ab7154e4-hostroot\") pod \"multus-rqtm8\" (UID: \"d5f812b2-2ecb-4be4-99aa-7808ab7154e4\") " pod="openshift-multus/multus-rqtm8"
Jul 20 17:55:56.092878 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.071310    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mgpl\" (UniqueName: \"kubernetes.io/projected/de1b439b-702a-43a3-8118-f7e20e716ac5-kube-api-access-4mgpl\") pod \"network-check-target-8v5ff\" (UID: \"de1b439b-702a-43a3-8118-f7e20e716ac5\") " pod="openshift-network-diagnostics/network-check-target-8v5ff"
Jul 20 17:55:56.092878 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.071342    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/07587a78-2d3a-40ad-9aae-a4875200263a-run-systemd\") pod \"ovnkube-node-wjs8g\" (UID: \"07587a78-2d3a-40ad-9aae-a4875200263a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjs8g"
Jul 20 17:55:56.092878 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.071363    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/07587a78-2d3a-40ad-9aae-a4875200263a-host-run-ovn-kubernetes\") pod \"ovnkube-node-wjs8g\" (UID: \"07587a78-2d3a-40ad-9aae-a4875200263a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjs8g"
Jul 20 17:55:56.092878 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.071376    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/07587a78-2d3a-40ad-9aae-a4875200263a-ovnkube-config\") pod \"ovnkube-node-wjs8g\" (UID: \"07587a78-2d3a-40ad-9aae-a4875200263a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjs8g"
Jul 20 17:55:56.092878 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.071389    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbl5c\" (UniqueName: \"kubernetes.io/projected/7a564ed7-d4b4-4f1a-a7da-d1e4d0652601-kube-api-access-fbl5c\") pod \"node-ca-q42fv\" (UID: \"7a564ed7-d4b4-4f1a-a7da-d1e4d0652601\") " pod="openshift-image-registry/node-ca-q42fv"
Jul 20 17:55:56.092878 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.071403    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d5f812b2-2ecb-4be4-99aa-7808ab7154e4-system-cni-dir\") pod \"multus-rqtm8\" (UID: \"d5f812b2-2ecb-4be4-99aa-7808ab7154e4\") " pod="openshift-multus/multus-rqtm8"
Jul 20 17:55:56.092878 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.071424    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d5f812b2-2ecb-4be4-99aa-7808ab7154e4-host-var-lib-kubelet\") pod \"multus-rqtm8\" (UID: \"d5f812b2-2ecb-4be4-99aa-7808ab7154e4\") " pod="openshift-multus/multus-rqtm8"
Jul 20 17:55:56.092878 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.071446    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/07587a78-2d3a-40ad-9aae-a4875200263a-ovnkube-script-lib\") pod \"ovnkube-node-wjs8g\" (UID: \"07587a78-2d3a-40ad-9aae-a4875200263a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjs8g"
Jul 20 17:55:56.092878 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.071460    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1dfc0381-8b5e-4400-a1c8-da99129540c7-system-cni-dir\") pod \"multus-additional-cni-plugins-mzsnt\" (UID: \"1dfc0381-8b5e-4400-a1c8-da99129540c7\") " pod="openshift-multus/multus-additional-cni-plugins-mzsnt"
Jul 20 17:55:56.092878 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.071485    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1dfc0381-8b5e-4400-a1c8-da99129540c7-cnibin\") pod \"multus-additional-cni-plugins-mzsnt\" (UID: \"1dfc0381-8b5e-4400-a1c8-da99129540c7\") " pod="openshift-multus/multus-additional-cni-plugins-mzsnt"
Jul 20 17:55:56.093500 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.071515    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1dfc0381-8b5e-4400-a1c8-da99129540c7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mzsnt\" (UID: \"1dfc0381-8b5e-4400-a1c8-da99129540c7\") " pod="openshift-multus/multus-additional-cni-plugins-mzsnt"
Jul 20 17:55:56.093500 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.071541    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1dfc0381-8b5e-4400-a1c8-da99129540c7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mzsnt\" (UID: \"1dfc0381-8b5e-4400-a1c8-da99129540c7\") " pod="openshift-multus/multus-additional-cni-plugins-mzsnt"
Jul 20 17:55:56.093500 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.071555    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d5f812b2-2ecb-4be4-99aa-7808ab7154e4-cnibin\") pod \"multus-rqtm8\" (UID: \"d5f812b2-2ecb-4be4-99aa-7808ab7154e4\") " pod="openshift-multus/multus-rqtm8"
Jul 20 17:55:56.093500 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.071574    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d5f812b2-2ecb-4be4-99aa-7808ab7154e4-host-run-multus-certs\") pod \"multus-rqtm8\" (UID: \"d5f812b2-2ecb-4be4-99aa-7808ab7154e4\") " pod="openshift-multus/multus-rqtm8"
Jul 20 17:55:56.093500 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.071930    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 20 17:55:56.093500 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.072331    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tgg22"
Jul 20 17:55:56.093500 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:55:56.072399    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tgg22" podUID="eca0024f-2963-4ed1-817e-f1a6febd79aa"
Jul 20 17:55:56.093500 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.073545    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 20 17:55:56.093500 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.073590    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-6m6xv\""
Jul 20 17:55:56.093500 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.074693    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p74sv"
Jul 20 17:55:56.093500 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.076830    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-p7mp5\""
Jul 20 17:55:56.093500 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.076852    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 20 17:55:56.093500 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.077066    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-smcqb"
Jul 20 17:55:56.093500 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.077165    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 20 17:55:56.093500 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.077196    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 20 17:55:56.093500 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.078446    2569 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 20 17:55:56.093500 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.079557    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-lmvvp\""
Jul 20 17:55:56.093500 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.079631    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 20 17:55:56.093500 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.079717    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 20 17:55:56.263612 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.110950    2569 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-j8cd7"
Jul 20 17:55:56.263612 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.120874    2569 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-j8cd7"
Jul 20 17:55:56.263612 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.161835    2569 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 20 17:55:56.263612 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.172313    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d5f812b2-2ecb-4be4-99aa-7808ab7154e4-multus-cni-dir\") pod \"multus-rqtm8\" (UID: \"d5f812b2-2ecb-4be4-99aa-7808ab7154e4\") " pod="openshift-multus/multus-rqtm8"
Jul 20 17:55:56.263612 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.172338    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d5f812b2-2ecb-4be4-99aa-7808ab7154e4-host-var-lib-cni-multus\") pod \"multus-rqtm8\" (UID: \"d5f812b2-2ecb-4be4-99aa-7808ab7154e4\") " pod="openshift-multus/multus-rqtm8"
Jul 20 17:55:56.263612 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.172359    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/07587a78-2d3a-40ad-9aae-a4875200263a-env-overrides\") pod \"ovnkube-node-wjs8g\" (UID: \"07587a78-2d3a-40ad-9aae-a4875200263a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjs8g"
Jul 20 17:55:56.263612 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.172419    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d5f812b2-2ecb-4be4-99aa-7808ab7154e4-host-var-lib-cni-multus\") pod \"multus-rqtm8\" (UID: \"d5f812b2-2ecb-4be4-99aa-7808ab7154e4\") " pod="openshift-multus/multus-rqtm8"
Jul 20 17:55:56.263612 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.172436    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9l5j\" (UniqueName: \"kubernetes.io/projected/07587a78-2d3a-40ad-9aae-a4875200263a-kube-api-access-v9l5j\") pod \"ovnkube-node-wjs8g\" (UID: \"07587a78-2d3a-40ad-9aae-a4875200263a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjs8g"
Jul 20 17:55:56.263612 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.172479    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bb3035ba-fece-492c-aa33-43135c1925da-host\") pod \"tuned-q6gcb\" (UID: \"bb3035ba-fece-492c-aa33-43135c1925da\") " pod="openshift-cluster-node-tuning-operator/tuned-q6gcb"
Jul 20 17:55:56.263612 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.172499    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhthz\" (UniqueName: \"kubernetes.io/projected/91b53241-763d-4868-81c3-786a3d6d4080-kube-api-access-rhthz\") pod \"aws-ebs-csi-driver-node-p74sv\" (UID: \"91b53241-763d-4868-81c3-786a3d6d4080\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p74sv"
Jul 20 17:55:56.263612 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.172520    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea740a69-374a-483b-a045-a2c795b99d4c-tmp-dir\") pod \"node-resolver-smcqb\" (UID: \"ea740a69-374a-483b-a045-a2c795b99d4c\") " pod="openshift-dns/node-resolver-smcqb"
Jul 20 17:55:56.263612 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.172542    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d5f812b2-2ecb-4be4-99aa-7808ab7154e4-multus-socket-dir-parent\") pod \"multus-rqtm8\" (UID: \"d5f812b2-2ecb-4be4-99aa-7808ab7154e4\") " pod="openshift-multus/multus-rqtm8"
Jul 20 17:55:56.263612 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.172561    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1dfc0381-8b5e-4400-a1c8-da99129540c7-os-release\") pod \"multus-additional-cni-plugins-mzsnt\" (UID: \"1dfc0381-8b5e-4400-a1c8-da99129540c7\") " pod="openshift-multus/multus-additional-cni-plugins-mzsnt"
Jul 20 17:55:56.263612 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.172592    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d5f812b2-2ecb-4be4-99aa-7808ab7154e4-multus-cni-dir\") pod \"multus-rqtm8\" (UID: \"d5f812b2-2ecb-4be4-99aa-7808ab7154e4\") " pod="openshift-multus/multus-rqtm8"
Jul 20 17:55:56.263612 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.172602    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/91b53241-763d-4868-81c3-786a3d6d4080-socket-dir\") pod \"aws-ebs-csi-driver-node-p74sv\" (UID: \"91b53241-763d-4868-81c3-786a3d6d4080\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p74sv"
Jul 20 17:55:56.263612 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.172633    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/07587a78-2d3a-40ad-9aae-a4875200263a-systemd-units\") pod \"ovnkube-node-wjs8g\" (UID: \"07587a78-2d3a-40ad-9aae-a4875200263a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjs8g"
Jul 20 17:55:56.264771 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.172659    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d5f812b2-2ecb-4be4-99aa-7808ab7154e4-multus-socket-dir-parent\") pod \"multus-rqtm8\" (UID: \"d5f812b2-2ecb-4be4-99aa-7808ab7154e4\") " pod="openshift-multus/multus-rqtm8"
Jul 20 17:55:56.264771 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.172686    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1dfc0381-8b5e-4400-a1c8-da99129540c7-os-release\") pod \"multus-additional-cni-plugins-mzsnt\" (UID: \"1dfc0381-8b5e-4400-a1c8-da99129540c7\") " pod="openshift-multus/multus-additional-cni-plugins-mzsnt"
Jul 20 17:55:56.264771 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.172717    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/07587a78-2d3a-40ad-9aae-a4875200263a-var-lib-openvswitch\") pod \"ovnkube-node-wjs8g\" (UID: \"07587a78-2d3a-40ad-9aae-a4875200263a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjs8g"
Jul 20 17:55:56.264771 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.172751    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/07587a78-2d3a-40ad-9aae-a4875200263a-systemd-units\") pod \"ovnkube-node-wjs8g\" (UID: \"07587a78-2d3a-40ad-9aae-a4875200263a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjs8g"
Jul 20 17:55:56.264771 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.172780    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/07587a78-2d3a-40ad-9aae-a4875200263a-var-lib-openvswitch\") pod \"ovnkube-node-wjs8g\" (UID: \"07587a78-2d3a-40ad-9aae-a4875200263a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjs8g"
Jul 20 17:55:56.264771 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.172808    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/07587a78-2d3a-40ad-9aae-a4875200263a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wjs8g\" (UID: \"07587a78-2d3a-40ad-9aae-a4875200263a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjs8g"
Jul 20 17:55:56.264771 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.172836    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1dfc0381-8b5e-4400-a1c8-da99129540c7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mzsnt\" (UID: \"1dfc0381-8b5e-4400-a1c8-da99129540c7\") " pod="openshift-multus/multus-additional-cni-plugins-mzsnt"
Jul 20 17:55:56.264771 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.172836    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/07587a78-2d3a-40ad-9aae-a4875200263a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wjs8g\" (UID: \"07587a78-2d3a-40ad-9aae-a4875200263a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjs8g"
Jul 20 17:55:56.264771 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.172849    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/07587a78-2d3a-40ad-9aae-a4875200263a-env-overrides\") pod \"ovnkube-node-wjs8g\" (UID: \"07587a78-2d3a-40ad-9aae-a4875200263a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjs8g"
Jul 20 17:55:56.264771 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.172854    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0a7d39a3-ee65-40f5-940c-149a0e87e580-iptables-alerter-script\") pod \"iptables-alerter-ql9lx\" (UID: \"0a7d39a3-ee65-40f5-940c-149a0e87e580\") " pod="openshift-network-operator/iptables-alerter-ql9lx"
Jul 20 17:55:56.264771 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.172906    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/bb3035ba-fece-492c-aa33-43135c1925da-etc-tuned\") pod \"tuned-q6gcb\" (UID: \"bb3035ba-fece-492c-aa33-43135c1925da\") " pod="openshift-cluster-node-tuning-operator/tuned-q6gcb"
Jul 20 17:55:56.264771 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.172928    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d5f812b2-2ecb-4be4-99aa-7808ab7154e4-host-var-lib-cni-bin\") pod \"multus-rqtm8\" (UID: \"d5f812b2-2ecb-4be4-99aa-7808ab7154e4\") " pod="openshift-multus/multus-rqtm8"
Jul 20 17:55:56.264771 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.172948    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/07587a78-2d3a-40ad-9aae-a4875200263a-run-systemd\") pod \"ovnkube-node-wjs8g\" (UID: \"07587a78-2d3a-40ad-9aae-a4875200263a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjs8g"
Jul 20 17:55:56.264771 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.172963    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/07587a78-2d3a-40ad-9aae-a4875200263a-ovnkube-config\") pod \"ovnkube-node-wjs8g\" (UID: \"07587a78-2d3a-40ad-9aae-a4875200263a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjs8g"
Jul 20 17:55:56.265558 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.172978    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/76c83a34-e7fe-46a1-a01e-944840b62724-konnectivity-ca\") pod \"konnectivity-agent-l45qp\" (UID: \"76c83a34-e7fe-46a1-a01e-944840b62724\") " pod="kube-system/konnectivity-agent-l45qp"
Jul 20 17:55:56.265558 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.172984    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1dfc0381-8b5e-4400-a1c8-da99129540c7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mzsnt\" (UID: \"1dfc0381-8b5e-4400-a1c8-da99129540c7\") " pod="openshift-multus/multus-additional-cni-plugins-mzsnt"
Jul 20 17:55:56.265558 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.172998    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bb3035ba-fece-492c-aa33-43135c1925da-etc-systemd\") pod \"tuned-q6gcb\" (UID: \"bb3035ba-fece-492c-aa33-43135c1925da\") " pod="openshift-cluster-node-tuning-operator/tuned-q6gcb"
Jul 20 17:55:56.265558 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.173014    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/07587a78-2d3a-40ad-9aae-a4875200263a-run-systemd\") pod \"ovnkube-node-wjs8g\" (UID: \"07587a78-2d3a-40ad-9aae-a4875200263a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjs8g"
Jul 20 17:55:56.265558 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.173020    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d5f812b2-2ecb-4be4-99aa-7808ab7154e4-system-cni-dir\") pod \"multus-rqtm8\" (UID: \"d5f812b2-2ecb-4be4-99aa-7808ab7154e4\") " pod="openshift-multus/multus-rqtm8"
Jul 20 17:55:56.265558 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.173041    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d5f812b2-2ecb-4be4-99aa-7808ab7154e4-host-var-lib-cni-bin\") pod \"multus-rqtm8\" (UID: \"d5f812b2-2ecb-4be4-99aa-7808ab7154e4\") " pod="openshift-multus/multus-rqtm8"
Jul 20 17:55:56.265558 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.173044    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d5f812b2-2ecb-4be4-99aa-7808ab7154e4-host-var-lib-kubelet\") pod \"multus-rqtm8\" (UID: \"d5f812b2-2ecb-4be4-99aa-7808ab7154e4\") " pod="openshift-multus/multus-rqtm8"
Jul 20 17:55:56.265558 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.173090    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d5f812b2-2ecb-4be4-99aa-7808ab7154e4-host-var-lib-kubelet\") pod \"multus-rqtm8\" (UID: \"d5f812b2-2ecb-4be4-99aa-7808ab7154e4\") " pod="openshift-multus/multus-rqtm8"
Jul 20 17:55:56.265558 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.173094    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d5f812b2-2ecb-4be4-99aa-7808ab7154e4-system-cni-dir\") pod \"multus-rqtm8\" (UID: \"d5f812b2-2ecb-4be4-99aa-7808ab7154e4\") " pod="openshift-multus/multus-rqtm8"
Jul 20 17:55:56.265558 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.173105    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1dfc0381-8b5e-4400-a1c8-da99129540c7-cnibin\") pod \"multus-additional-cni-plugins-mzsnt\" (UID: \"1dfc0381-8b5e-4400-a1c8-da99129540c7\") " pod="openshift-multus/multus-additional-cni-plugins-mzsnt"
Jul 20 17:55:56.265558 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.173126    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1dfc0381-8b5e-4400-a1c8-da99129540c7-cnibin\") pod \"multus-additional-cni-plugins-mzsnt\" (UID: \"1dfc0381-8b5e-4400-a1c8-da99129540c7\") " pod="openshift-multus/multus-additional-cni-plugins-mzsnt"
Jul 20 17:55:56.265558 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.173131    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d5f812b2-2ecb-4be4-99aa-7808ab7154e4-host-run-multus-certs\") pod \"multus-rqtm8\" (UID: \"d5f812b2-2ecb-4be4-99aa-7808ab7154e4\") " pod="openshift-multus/multus-rqtm8"
Jul 20 17:55:56.265558 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.173148    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/07587a78-2d3a-40ad-9aae-a4875200263a-node-log\") pod \"ovnkube-node-wjs8g\" (UID: \"07587a78-2d3a-40ad-9aae-a4875200263a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjs8g"
Jul 20 17:55:56.265558 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.173169    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/07587a78-2d3a-40ad-9aae-a4875200263a-node-log\") pod \"ovnkube-node-wjs8g\" (UID: \"07587a78-2d3a-40ad-9aae-a4875200263a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjs8g"
Jul 20 17:55:56.265558 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.173150    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d5f812b2-2ecb-4be4-99aa-7808ab7154e4-host-run-multus-certs\") pod \"multus-rqtm8\" (UID: \"d5f812b2-2ecb-4be4-99aa-7808ab7154e4\") " pod="openshift-multus/multus-rqtm8"
Jul 20 17:55:56.266395 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.173169    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/07587a78-2d3a-40ad-9aae-a4875200263a-host-cni-netd\") pod \"ovnkube-node-wjs8g\" (UID: \"07587a78-2d3a-40ad-9aae-a4875200263a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjs8g"
Jul 20 17:55:56.266395 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.173193    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/07587a78-2d3a-40ad-9aae-a4875200263a-host-cni-netd\") pod \"ovnkube-node-wjs8g\" (UID: \"07587a78-2d3a-40ad-9aae-a4875200263a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjs8g"
Jul 20 17:55:56.266395 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.173195    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/76c83a34-e7fe-46a1-a01e-944840b62724-agent-certs\") pod \"konnectivity-agent-l45qp\" (UID: \"76c83a34-e7fe-46a1-a01e-944840b62724\") " pod="kube-system/konnectivity-agent-l45qp"
Jul 20 17:55:56.266395 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.173217    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nd2w\" (UniqueName: \"kubernetes.io/projected/bb3035ba-fece-492c-aa33-43135c1925da-kube-api-access-4nd2w\") pod \"tuned-q6gcb\" (UID: \"bb3035ba-fece-492c-aa33-43135c1925da\") " pod="openshift-cluster-node-tuning-operator/tuned-q6gcb"
Jul 20 17:55:56.266395 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.173239    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/91b53241-763d-4868-81c3-786a3d6d4080-kubelet-dir\") pod \"aws-ebs-csi-driver-node-p74sv\" (UID: \"91b53241-763d-4868-81c3-786a3d6d4080\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p74sv"
Jul 20 17:55:56.266395 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.173291    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m594d\" (UniqueName: \"kubernetes.io/projected/d5f812b2-2ecb-4be4-99aa-7808ab7154e4-kube-api-access-m594d\") pod \"multus-rqtm8\" (UID: \"d5f812b2-2ecb-4be4-99aa-7808ab7154e4\") " pod="openshift-multus/multus-rqtm8"
Jul 20 17:55:56.266395 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.173315    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/07587a78-2d3a-40ad-9aae-a4875200263a-host-kubelet\") pod \"ovnkube-node-wjs8g\" (UID: \"07587a78-2d3a-40ad-9aae-a4875200263a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjs8g"
Jul 20 17:55:56.266395 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.173351    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/07587a78-2d3a-40ad-9aae-a4875200263a-host-kubelet\") pod \"ovnkube-node-wjs8g\" (UID: \"07587a78-2d3a-40ad-9aae-a4875200263a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjs8g"
Jul 20 17:55:56.266395 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.173403    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/07587a78-2d3a-40ad-9aae-a4875200263a-host-run-netns\") pod \"ovnkube-node-wjs8g\" (UID: \"07587a78-2d3a-40ad-9aae-a4875200263a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjs8g"
Jul 20 17:55:56.266395 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.173421    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/07587a78-2d3a-40ad-9aae-a4875200263a-run-openvswitch\") pod \"ovnkube-node-wjs8g\" (UID: \"07587a78-2d3a-40ad-9aae-a4875200263a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjs8g"
Jul 20 17:55:56.266395 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.173435    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bb3035ba-fece-492c-aa33-43135c1925da-etc-sysconfig\") pod \"tuned-q6gcb\" (UID: \"bb3035ba-fece-492c-aa33-43135c1925da\") " pod="openshift-cluster-node-tuning-operator/tuned-q6gcb"
Jul 20 17:55:56.266395 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.173453    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bb3035ba-fece-492c-aa33-43135c1925da-etc-kubernetes\") pod \"tuned-q6gcb\" (UID: \"bb3035ba-fece-492c-aa33-43135c1925da\") " pod="openshift-cluster-node-tuning-operator/tuned-q6gcb"
Jul 20 17:55:56.266395 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.173470    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/07587a78-2d3a-40ad-9aae-a4875200263a-host-run-netns\") pod \"ovnkube-node-wjs8g\" (UID: \"07587a78-2d3a-40ad-9aae-a4875200263a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjs8g"
Jul 20 17:55:56.266395 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.173433    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/07587a78-2d3a-40ad-9aae-a4875200263a-ovnkube-config\") pod \"ovnkube-node-wjs8g\" (UID: \"07587a78-2d3a-40ad-9aae-a4875200263a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjs8g"
Jul 20 17:55:56.267087 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.173484    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/eca0024f-2963-4ed1-817e-f1a6febd79aa-kubelet-config\") pod \"global-pull-secret-syncer-tgg22\" (UID: \"eca0024f-2963-4ed1-817e-f1a6febd79aa\") " pod="kube-system/global-pull-secret-syncer-tgg22"
Jul 20 17:55:56.267087 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.173499    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/07587a78-2d3a-40ad-9aae-a4875200263a-run-openvswitch\") pod \"ovnkube-node-wjs8g\" (UID: \"07587a78-2d3a-40ad-9aae-a4875200263a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjs8g"
Jul 20 17:55:56.267087 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.173511    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eca0024f-2963-4ed1-817e-f1a6febd79aa-original-pull-secret\") pod \"global-pull-secret-syncer-tgg22\" (UID: \"eca0024f-2963-4ed1-817e-f1a6febd79aa\") " pod="kube-system/global-pull-secret-syncer-tgg22"
Jul 20 17:55:56.267087 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.173547    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d5f812b2-2ecb-4be4-99aa-7808ab7154e4-host-run-k8s-cni-cncf-io\") pod \"multus-rqtm8\" (UID: \"d5f812b2-2ecb-4be4-99aa-7808ab7154e4\") " pod="openshift-multus/multus-rqtm8"
Jul 20 17:55:56.267087 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.173563    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pnrb\" (UniqueName: \"kubernetes.io/projected/0a7d39a3-ee65-40f5-940c-149a0e87e580-kube-api-access-9pnrb\") pod \"iptables-alerter-ql9lx\" (UID: \"0a7d39a3-ee65-40f5-940c-149a0e87e580\") " pod="openshift-network-operator/iptables-alerter-ql9lx"
Jul 20 17:55:56.267087 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.173582    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bb3035ba-fece-492c-aa33-43135c1925da-etc-sysctl-d\") pod \"tuned-q6gcb\" (UID: \"bb3035ba-fece-492c-aa33-43135c1925da\") " pod="openshift-cluster-node-tuning-operator/tuned-q6gcb"
Jul 20 17:55:56.267087 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.173590    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d5f812b2-2ecb-4be4-99aa-7808ab7154e4-host-run-k8s-cni-cncf-io\") pod \"multus-rqtm8\" (UID: \"d5f812b2-2ecb-4be4-99aa-7808ab7154e4\") " pod="openshift-multus/multus-rqtm8"
Jul 20 17:55:56.267087 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.173599    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bb3035ba-fece-492c-aa33-43135c1925da-lib-modules\") pod \"tuned-q6gcb\" (UID: \"bb3035ba-fece-492c-aa33-43135c1925da\") " pod="openshift-cluster-node-tuning-operator/tuned-q6gcb"
Jul 20 17:55:56.267087 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.173624    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bb3035ba-fece-492c-aa33-43135c1925da-var-lib-kubelet\") pod \"tuned-q6gcb\" (UID: \"bb3035ba-fece-492c-aa33-43135c1925da\") " pod="openshift-cluster-node-tuning-operator/tuned-q6gcb"
Jul 20 17:55:56.267087 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.173647    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/91b53241-763d-4868-81c3-786a3d6d4080-device-dir\") pod \"aws-ebs-csi-driver-node-p74sv\" (UID: \"91b53241-763d-4868-81c3-786a3d6d4080\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p74sv"
Jul 20 17:55:56.267087 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.173671    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d5f812b2-2ecb-4be4-99aa-7808ab7154e4-host-run-netns\") pod \"multus-rqtm8\" (UID: \"d5f812b2-2ecb-4be4-99aa-7808ab7154e4\") " pod="openshift-multus/multus-rqtm8"
Jul 20 17:55:56.267087 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.173692    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d5f812b2-2ecb-4be4-99aa-7808ab7154e4-multus-conf-dir\") pod \"multus-rqtm8\" (UID: \"d5f812b2-2ecb-4be4-99aa-7808ab7154e4\") " pod="openshift-multus/multus-rqtm8"
Jul 20 17:55:56.267087 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.173711    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/07587a78-2d3a-40ad-9aae-a4875200263a-host-cni-bin\") pod \"ovnkube-node-wjs8g\" (UID: \"07587a78-2d3a-40ad-9aae-a4875200263a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjs8g"
Jul 20 17:55:56.267087 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.173747    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7a564ed7-d4b4-4f1a-a7da-d1e4d0652601-host\") pod \"node-ca-q42fv\" (UID: \"7a564ed7-d4b4-4f1a-a7da-d1e4d0652601\") " pod="openshift-image-registry/node-ca-q42fv"
Jul 20 17:55:56.267813 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.173753    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d5f812b2-2ecb-4be4-99aa-7808ab7154e4-host-run-netns\") pod \"multus-rqtm8\" (UID: \"d5f812b2-2ecb-4be4-99aa-7808ab7154e4\") " pod="openshift-multus/multus-rqtm8"
Jul 20 17:55:56.267813 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.173768    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0a7d39a3-ee65-40f5-940c-149a0e87e580-host-slash\") pod \"iptables-alerter-ql9lx\" (UID: \"0a7d39a3-ee65-40f5-940c-149a0e87e580\") " pod="openshift-network-operator/iptables-alerter-ql9lx"
Jul 20 17:55:56.267813 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.173769    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d5f812b2-2ecb-4be4-99aa-7808ab7154e4-multus-conf-dir\") pod \"multus-rqtm8\" (UID: \"d5f812b2-2ecb-4be4-99aa-7808ab7154e4\") " pod="openshift-multus/multus-rqtm8"
Jul 20 17:55:56.267813 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.173784    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7a564ed7-d4b4-4f1a-a7da-d1e4d0652601-host\") pod \"node-ca-q42fv\" (UID: \"7a564ed7-d4b4-4f1a-a7da-d1e4d0652601\") " pod="openshift-image-registry/node-ca-q42fv"
Jul 20 17:55:56.267813 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.173784    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/07587a78-2d3a-40ad-9aae-a4875200263a-host-cni-bin\") pod \"ovnkube-node-wjs8g\" (UID: \"07587a78-2d3a-40ad-9aae-a4875200263a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjs8g"
Jul 20 17:55:56.267813 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.173789    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/91b53241-763d-4868-81c3-786a3d6d4080-sys-fs\") pod \"aws-ebs-csi-driver-node-p74sv\" (UID: \"91b53241-763d-4868-81c3-786a3d6d4080\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p74sv"
Jul 20 17:55:56.267813 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.173813    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/07587a78-2d3a-40ad-9aae-a4875200263a-log-socket\") pod \"ovnkube-node-wjs8g\" (UID: \"07587a78-2d3a-40ad-9aae-a4875200263a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjs8g"
Jul 20 17:55:56.267813 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.173834    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d5f812b2-2ecb-4be4-99aa-7808ab7154e4-multus-daemon-config\") pod \"multus-rqtm8\" (UID: \"d5f812b2-2ecb-4be4-99aa-7808ab7154e4\") " pod="openshift-multus/multus-rqtm8"
Jul 20 17:55:56.267813 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.173853    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7a564ed7-d4b4-4f1a-a7da-d1e4d0652601-serviceca\") pod \"node-ca-q42fv\" (UID: \"7a564ed7-d4b4-4f1a-a7da-d1e4d0652601\") " pod="openshift-image-registry/node-ca-q42fv"
Jul 20 17:55:56.267813 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.173865    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/07587a78-2d3a-40ad-9aae-a4875200263a-log-socket\") pod \"ovnkube-node-wjs8g\" (UID: \"07587a78-2d3a-40ad-9aae-a4875200263a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjs8g"
Jul 20 17:55:56.267813 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.173873    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26b0d75c-ecf7-4bb4-ad4b-d9077da808bd-metrics-certs\") pod \"network-metrics-daemon-nkmr4\" (UID: \"26b0d75c-ecf7-4bb4-ad4b-d9077da808bd\") " pod="openshift-multus/network-metrics-daemon-nkmr4"
Jul 20 17:55:56.267813 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.173899    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9csjm\" (UniqueName: \"kubernetes.io/projected/26b0d75c-ecf7-4bb4-ad4b-d9077da808bd-kube-api-access-9csjm\") pod \"network-metrics-daemon-nkmr4\" (UID: \"26b0d75c-ecf7-4bb4-ad4b-d9077da808bd\") " pod="openshift-multus/network-metrics-daemon-nkmr4"
Jul 20 17:55:56.267813 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.173920    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sm9p5\" (UniqueName: \"kubernetes.io/projected/1dfc0381-8b5e-4400-a1c8-da99129540c7-kube-api-access-sm9p5\") pod \"multus-additional-cni-plugins-mzsnt\" (UID: \"1dfc0381-8b5e-4400-a1c8-da99129540c7\") " pod="openshift-multus/multus-additional-cni-plugins-mzsnt"
Jul 20 17:55:56.267813 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.173942    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ea740a69-374a-483b-a045-a2c795b99d4c-hosts-file\") pod \"node-resolver-smcqb\" (UID: \"ea740a69-374a-483b-a045-a2c795b99d4c\") " pod="openshift-dns/node-resolver-smcqb"
Jul 20 17:55:56.267813 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:55:56.173949    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 20 17:55:56.268343 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.173966    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d5f812b2-2ecb-4be4-99aa-7808ab7154e4-hostroot\") pod \"multus-rqtm8\" (UID: \"d5f812b2-2ecb-4be4-99aa-7808ab7154e4\") " pod="openshift-multus/multus-rqtm8"
Jul 20 17:55:56.268343 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.173990    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mgpl\" (UniqueName: \"kubernetes.io/projected/de1b439b-702a-43a3-8118-f7e20e716ac5-kube-api-access-4mgpl\") pod \"network-check-target-8v5ff\" (UID: \"de1b439b-702a-43a3-8118-f7e20e716ac5\") " pod="openshift-network-diagnostics/network-check-target-8v5ff"
Jul 20 17:55:56.268343 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:55:56.174031    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26b0d75c-ecf7-4bb4-ad4b-d9077da808bd-metrics-certs podName:26b0d75c-ecf7-4bb4-ad4b-d9077da808bd nodeName:}" failed. No retries permitted until 2026-07-20 17:55:56.67399474 +0000 UTC m=+2.920528128 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/26b0d75c-ecf7-4bb4-ad4b-d9077da808bd-metrics-certs") pod "network-metrics-daemon-nkmr4" (UID: "26b0d75c-ecf7-4bb4-ad4b-d9077da808bd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 20 17:55:56.268343 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.174061    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/07587a78-2d3a-40ad-9aae-a4875200263a-host-run-ovn-kubernetes\") pod \"ovnkube-node-wjs8g\" (UID: \"07587a78-2d3a-40ad-9aae-a4875200263a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjs8g"
Jul 20 17:55:56.268343 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.174070    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d5f812b2-2ecb-4be4-99aa-7808ab7154e4-hostroot\") pod \"multus-rqtm8\" (UID: \"d5f812b2-2ecb-4be4-99aa-7808ab7154e4\") " pod="openshift-multus/multus-rqtm8"
Jul 20 17:55:56.268343 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.174088    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbl5c\" (UniqueName: \"kubernetes.io/projected/7a564ed7-d4b4-4f1a-a7da-d1e4d0652601-kube-api-access-fbl5c\") pod \"node-ca-q42fv\" (UID: \"7a564ed7-d4b4-4f1a-a7da-d1e4d0652601\") " pod="openshift-image-registry/node-ca-q42fv"
Jul 20 17:55:56.268343 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.174112    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1dfc0381-8b5e-4400-a1c8-da99129540c7-system-cni-dir\") pod \"multus-additional-cni-plugins-mzsnt\" (UID: \"1dfc0381-8b5e-4400-a1c8-da99129540c7\") " pod="openshift-multus/multus-additional-cni-plugins-mzsnt"
Jul 20 17:55:56.268343 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.174133    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/07587a78-2d3a-40ad-9aae-a4875200263a-host-run-ovn-kubernetes\") pod \"ovnkube-node-wjs8g\" (UID: \"07587a78-2d3a-40ad-9aae-a4875200263a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjs8g"
Jul 20 17:55:56.268343 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.174136    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1dfc0381-8b5e-4400-a1c8-da99129540c7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mzsnt\" (UID: \"1dfc0381-8b5e-4400-a1c8-da99129540c7\") " pod="openshift-multus/multus-additional-cni-plugins-mzsnt"
Jul 20 17:55:56.268343 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.174164    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bb3035ba-fece-492c-aa33-43135c1925da-etc-modprobe-d\") pod \"tuned-q6gcb\" (UID: \"bb3035ba-fece-492c-aa33-43135c1925da\") " pod="openshift-cluster-node-tuning-operator/tuned-q6gcb"
Jul 20 17:55:56.268343 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.174165    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1dfc0381-8b5e-4400-a1c8-da99129540c7-system-cni-dir\") pod \"multus-additional-cni-plugins-mzsnt\" (UID: \"1dfc0381-8b5e-4400-a1c8-da99129540c7\") " pod="openshift-multus/multus-additional-cni-plugins-mzsnt"
Jul 20 17:55:56.268343 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.174187    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bb3035ba-fece-492c-aa33-43135c1925da-sys\") pod \"tuned-q6gcb\" (UID: \"bb3035ba-fece-492c-aa33-43135c1925da\") " pod="openshift-cluster-node-tuning-operator/tuned-q6gcb"
Jul 20 17:55:56.268343 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.174217    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/07587a78-2d3a-40ad-9aae-a4875200263a-ovnkube-script-lib\") pod \"ovnkube-node-wjs8g\" (UID: \"07587a78-2d3a-40ad-9aae-a4875200263a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjs8g"
Jul 20 17:55:56.268996 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.174249    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7a564ed7-d4b4-4f1a-a7da-d1e4d0652601-serviceca\") pod \"node-ca-q42fv\" (UID: \"7a564ed7-d4b4-4f1a-a7da-d1e4d0652601\") " pod="openshift-image-registry/node-ca-q42fv"
Jul 20 17:55:56.268996 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.174259    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1dfc0381-8b5e-4400-a1c8-da99129540c7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mzsnt\" (UID: \"1dfc0381-8b5e-4400-a1c8-da99129540c7\") " pod="openshift-multus/multus-additional-cni-plugins-mzsnt"
Jul 20 17:55:56.268996 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.174365    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bb3035ba-fece-492c-aa33-43135c1925da-tmp\") pod \"tuned-q6gcb\" (UID: \"bb3035ba-fece-492c-aa33-43135c1925da\") " pod="openshift-cluster-node-tuning-operator/tuned-q6gcb"
Jul 20 17:55:56.268996 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.174398    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/eca0024f-2963-4ed1-817e-f1a6febd79aa-dbus\") pod \"global-pull-secret-syncer-tgg22\" (UID: \"eca0024f-2963-4ed1-817e-f1a6febd79aa\") " pod="kube-system/global-pull-secret-syncer-tgg22"
Jul 20 17:55:56.268996 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.174422    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d5f812b2-2ecb-4be4-99aa-7808ab7154e4-cnibin\") pod \"multus-rqtm8\" (UID: \"d5f812b2-2ecb-4be4-99aa-7808ab7154e4\") " pod="openshift-multus/multus-rqtm8"
Jul 20 17:55:56.268996 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.174477    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d5f812b2-2ecb-4be4-99aa-7808ab7154e4-etc-kubernetes\") pod \"multus-rqtm8\" (UID: \"d5f812b2-2ecb-4be4-99aa-7808ab7154e4\") " pod="openshift-multus/multus-rqtm8"
Jul 20 17:55:56.268996 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.174503    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d5f812b2-2ecb-4be4-99aa-7808ab7154e4-cnibin\") pod \"multus-rqtm8\" (UID: \"d5f812b2-2ecb-4be4-99aa-7808ab7154e4\") " pod="openshift-multus/multus-rqtm8"
Jul 20 17:55:56.268996 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.174542    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bb3035ba-fece-492c-aa33-43135c1925da-run\") pod \"tuned-q6gcb\" (UID: \"bb3035ba-fece-492c-aa33-43135c1925da\") " pod="openshift-cluster-node-tuning-operator/tuned-q6gcb"
Jul 20 17:55:56.268996 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.174563    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d5f812b2-2ecb-4be4-99aa-7808ab7154e4-etc-kubernetes\") pod \"multus-rqtm8\" (UID: \"d5f812b2-2ecb-4be4-99aa-7808ab7154e4\") " pod="openshift-multus/multus-rqtm8"
Jul 20 17:55:56.268996 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.174580    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcklf\" (UniqueName: \"kubernetes.io/projected/ea740a69-374a-483b-a045-a2c795b99d4c-kube-api-access-hcklf\") pod \"node-resolver-smcqb\" (UID: \"ea740a69-374a-483b-a045-a2c795b99d4c\") " pod="openshift-dns/node-resolver-smcqb"
Jul 20 17:55:56.268996 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.174594    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d5f812b2-2ecb-4be4-99aa-7808ab7154e4-multus-daemon-config\") pod \"multus-rqtm8\" (UID: \"d5f812b2-2ecb-4be4-99aa-7808ab7154e4\") " pod="openshift-multus/multus-rqtm8"
Jul 20 17:55:56.268996 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.174631    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d5f812b2-2ecb-4be4-99aa-7808ab7154e4-os-release\") pod \"multus-rqtm8\" (UID: \"d5f812b2-2ecb-4be4-99aa-7808ab7154e4\") " pod="openshift-multus/multus-rqtm8"
Jul 20 17:55:56.268996 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.174637    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1dfc0381-8b5e-4400-a1c8-da99129540c7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mzsnt\" (UID: \"1dfc0381-8b5e-4400-a1c8-da99129540c7\") " pod="openshift-multus/multus-additional-cni-plugins-mzsnt"
Jul 20 17:55:56.268996 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.174654    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/07587a78-2d3a-40ad-9aae-a4875200263a-host-slash\") pod \"ovnkube-node-wjs8g\" (UID: \"07587a78-2d3a-40ad-9aae-a4875200263a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjs8g"
Jul 20 17:55:56.268996 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.174685    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/07587a78-2d3a-40ad-9aae-a4875200263a-etc-openvswitch\") pod \"ovnkube-node-wjs8g\" (UID: \"07587a78-2d3a-40ad-9aae-a4875200263a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjs8g"
Jul 20 17:55:56.270079 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.174706    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/07587a78-2d3a-40ad-9aae-a4875200263a-run-ovn\") pod \"ovnkube-node-wjs8g\" (UID: \"07587a78-2d3a-40ad-9aae-a4875200263a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjs8g"
Jul 20 17:55:56.270079 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.174745    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/07587a78-2d3a-40ad-9aae-a4875200263a-ovnkube-script-lib\") pod \"ovnkube-node-wjs8g\" (UID: \"07587a78-2d3a-40ad-9aae-a4875200263a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjs8g"
Jul 20 17:55:56.270079 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.174757    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d5f812b2-2ecb-4be4-99aa-7808ab7154e4-os-release\") pod \"multus-rqtm8\" (UID: \"d5f812b2-2ecb-4be4-99aa-7808ab7154e4\") " pod="openshift-multus/multus-rqtm8"
Jul 20 17:55:56.270079 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.174745    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/07587a78-2d3a-40ad-9aae-a4875200263a-host-slash\") pod \"ovnkube-node-wjs8g\" (UID: \"07587a78-2d3a-40ad-9aae-a4875200263a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjs8g"
Jul 20 17:55:56.270079 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.174758    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bb3035ba-fece-492c-aa33-43135c1925da-etc-sysctl-conf\") pod \"tuned-q6gcb\" (UID: \"bb3035ba-fece-492c-aa33-43135c1925da\") " pod="openshift-cluster-node-tuning-operator/tuned-q6gcb"
Jul 20 17:55:56.270079 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.174798    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1dfc0381-8b5e-4400-a1c8-da99129540c7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mzsnt\" (UID: \"1dfc0381-8b5e-4400-a1c8-da99129540c7\") " pod="openshift-multus/multus-additional-cni-plugins-mzsnt"
Jul 20 17:55:56.270079 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.174816    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/07587a78-2d3a-40ad-9aae-a4875200263a-run-ovn\") pod \"ovnkube-node-wjs8g\" (UID: \"07587a78-2d3a-40ad-9aae-a4875200263a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjs8g"
Jul 20 17:55:56.270079 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.174817    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/91b53241-763d-4868-81c3-786a3d6d4080-registration-dir\") pod \"aws-ebs-csi-driver-node-p74sv\" (UID: \"91b53241-763d-4868-81c3-786a3d6d4080\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p74sv"
Jul 20 17:55:56.270079 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.174844    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d5f812b2-2ecb-4be4-99aa-7808ab7154e4-cni-binary-copy\") pod \"multus-rqtm8\" (UID: \"d5f812b2-2ecb-4be4-99aa-7808ab7154e4\") " pod="openshift-multus/multus-rqtm8"
Jul 20 17:55:56.270079 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.174861    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/07587a78-2d3a-40ad-9aae-a4875200263a-ovn-node-metrics-cert\") pod \"ovnkube-node-wjs8g\" (UID: \"07587a78-2d3a-40ad-9aae-a4875200263a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjs8g"
Jul 20 17:55:56.270079 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.174878    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1dfc0381-8b5e-4400-a1c8-da99129540c7-cni-binary-copy\") pod \"multus-additional-cni-plugins-mzsnt\" (UID: \"1dfc0381-8b5e-4400-a1c8-da99129540c7\") " pod="openshift-multus/multus-additional-cni-plugins-mzsnt"
Jul 20 17:55:56.270079 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.174901    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/91b53241-763d-4868-81c3-786a3d6d4080-etc-selinux\") pod \"aws-ebs-csi-driver-node-p74sv\" (UID: \"91b53241-763d-4868-81c3-786a3d6d4080\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p74sv"
Jul 20 17:55:56.270079 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.174821    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/07587a78-2d3a-40ad-9aae-a4875200263a-etc-openvswitch\") pod \"ovnkube-node-wjs8g\" (UID: \"07587a78-2d3a-40ad-9aae-a4875200263a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjs8g"
Jul 20 17:55:56.270079 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.175260    2569 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 20 17:55:56.270079 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.175280    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d5f812b2-2ecb-4be4-99aa-7808ab7154e4-cni-binary-copy\") pod \"multus-rqtm8\" (UID: \"d5f812b2-2ecb-4be4-99aa-7808ab7154e4\") " pod="openshift-multus/multus-rqtm8"
Jul 20 17:55:56.270659 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.175619    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1dfc0381-8b5e-4400-a1c8-da99129540c7-cni-binary-copy\") pod \"multus-additional-cni-plugins-mzsnt\" (UID: \"1dfc0381-8b5e-4400-a1c8-da99129540c7\") " pod="openshift-multus/multus-additional-cni-plugins-mzsnt"
Jul 20 17:55:56.270659 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.178870    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/07587a78-2d3a-40ad-9aae-a4875200263a-ovn-node-metrics-cert\") pod \"ovnkube-node-wjs8g\" (UID: \"07587a78-2d3a-40ad-9aae-a4875200263a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjs8g"
Jul 20 17:55:56.270659 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.181360    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-247.ec2.internal" event={"ID":"757e30ff139839aada6b39c2403a950d","Type":"ContainerStarted","Data":"a7fecbe7b43832507a90c09515b3a9b645acdbbc2a0eb061c8712ce91f8de16d"}
Jul 20 17:55:56.270659 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:55:56.182082    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 20 17:55:56.270659 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:55:56.182107    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 20 17:55:56.270659 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:55:56.182123    2569 projected.go:196] Error preparing data for projected volume kube-api-access-4mgpl for pod openshift-network-diagnostics/network-check-target-8v5ff: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 20 17:55:56.270659 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:55:56.182171    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/de1b439b-702a-43a3-8118-f7e20e716ac5-kube-api-access-4mgpl podName:de1b439b-702a-43a3-8118-f7e20e716ac5 nodeName:}" failed. No retries permitted until 2026-07-20 17:55:56.682158025 +0000 UTC m=+2.928691435 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-4mgpl" (UniqueName: "kubernetes.io/projected/de1b439b-702a-43a3-8118-f7e20e716ac5-kube-api-access-4mgpl") pod "network-check-target-8v5ff" (UID: "de1b439b-702a-43a3-8118-f7e20e716ac5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 20 17:55:56.270659 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.182710    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-247.ec2.internal" event={"ID":"7c055eb6eb0be9ef4618781875e36f04","Type":"ContainerStarted","Data":"2d80bc0956311db459e8714ebdd7dc05f2531a7a0d27726783df5f073ccb7647"}
Jul 20 17:55:56.270659 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.185541    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m594d\" (UniqueName: \"kubernetes.io/projected/d5f812b2-2ecb-4be4-99aa-7808ab7154e4-kube-api-access-m594d\") pod \"multus-rqtm8\" (UID: \"d5f812b2-2ecb-4be4-99aa-7808ab7154e4\") " pod="openshift-multus/multus-rqtm8"
Jul 20 17:55:56.270659 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.185585    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9l5j\" (UniqueName: \"kubernetes.io/projected/07587a78-2d3a-40ad-9aae-a4875200263a-kube-api-access-v9l5j\") pod \"ovnkube-node-wjs8g\" (UID: \"07587a78-2d3a-40ad-9aae-a4875200263a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wjs8g"
Jul 20 17:55:56.270659 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.185698    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9csjm\" (UniqueName: \"kubernetes.io/projected/26b0d75c-ecf7-4bb4-ad4b-d9077da808bd-kube-api-access-9csjm\") pod \"network-metrics-daemon-nkmr4\" (UID: \"26b0d75c-ecf7-4bb4-ad4b-d9077da808bd\") " pod="openshift-multus/network-metrics-daemon-nkmr4"
Jul 20 17:55:56.270659 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.185865    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sm9p5\" (UniqueName: \"kubernetes.io/projected/1dfc0381-8b5e-4400-a1c8-da99129540c7-kube-api-access-sm9p5\") pod \"multus-additional-cni-plugins-mzsnt\" (UID: \"1dfc0381-8b5e-4400-a1c8-da99129540c7\") " pod="openshift-multus/multus-additional-cni-plugins-mzsnt"
Jul 20 17:55:56.270659 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.186091    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbl5c\" (UniqueName: \"kubernetes.io/projected/7a564ed7-d4b4-4f1a-a7da-d1e4d0652601-kube-api-access-fbl5c\") pod \"node-ca-q42fv\" (UID: \"7a564ed7-d4b4-4f1a-a7da-d1e4d0652601\") " pod="openshift-image-registry/node-ca-q42fv"
Jul 20 17:55:56.408870 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.275503    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ea740a69-374a-483b-a045-a2c795b99d4c-hosts-file\") pod \"node-resolver-smcqb\" (UID: \"ea740a69-374a-483b-a045-a2c795b99d4c\") " pod="openshift-dns/node-resolver-smcqb"
Jul 20 17:55:56.408870 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.275542    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bb3035ba-fece-492c-aa33-43135c1925da-etc-modprobe-d\") pod \"tuned-q6gcb\" (UID: \"bb3035ba-fece-492c-aa33-43135c1925da\") " pod="openshift-cluster-node-tuning-operator/tuned-q6gcb"
Jul 20 17:55:56.408870 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.275562    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bb3035ba-fece-492c-aa33-43135c1925da-sys\") pod \"tuned-q6gcb\" (UID: \"bb3035ba-fece-492c-aa33-43135c1925da\") " pod="openshift-cluster-node-tuning-operator/tuned-q6gcb"
Jul 20 17:55:56.408870 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.275578    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bb3035ba-fece-492c-aa33-43135c1925da-tmp\") pod \"tuned-q6gcb\" (UID: \"bb3035ba-fece-492c-aa33-43135c1925da\") " pod="openshift-cluster-node-tuning-operator/tuned-q6gcb"
Jul 20 17:55:56.408870 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.275598    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/eca0024f-2963-4ed1-817e-f1a6febd79aa-dbus\") pod \"global-pull-secret-syncer-tgg22\" (UID: \"eca0024f-2963-4ed1-817e-f1a6febd79aa\") " pod="kube-system/global-pull-secret-syncer-tgg22"
Jul 20 17:55:56.408870 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.275618    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bb3035ba-fece-492c-aa33-43135c1925da-run\") pod \"tuned-q6gcb\" (UID: \"bb3035ba-fece-492c-aa33-43135c1925da\") " pod="openshift-cluster-node-tuning-operator/tuned-q6gcb"
Jul 20 17:55:56.408870 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.275625    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ea740a69-374a-483b-a045-a2c795b99d4c-hosts-file\") pod \"node-resolver-smcqb\" (UID: \"ea740a69-374a-483b-a045-a2c795b99d4c\") " pod="openshift-dns/node-resolver-smcqb"
Jul 20 17:55:56.408870 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.275637    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bb3035ba-fece-492c-aa33-43135c1925da-sys\") pod \"tuned-q6gcb\" (UID: \"bb3035ba-fece-492c-aa33-43135c1925da\") " pod="openshift-cluster-node-tuning-operator/tuned-q6gcb"
Jul 20 17:55:56.408870 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.275643    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcklf\" (UniqueName: \"kubernetes.io/projected/ea740a69-374a-483b-a045-a2c795b99d4c-kube-api-access-hcklf\") pod \"node-resolver-smcqb\" (UID: \"ea740a69-374a-483b-a045-a2c795b99d4c\") " pod="openshift-dns/node-resolver-smcqb"
Jul 20 17:55:56.408870 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.275676    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/eca0024f-2963-4ed1-817e-f1a6febd79aa-dbus\") pod \"global-pull-secret-syncer-tgg22\" (UID: \"eca0024f-2963-4ed1-817e-f1a6febd79aa\") " pod="kube-system/global-pull-secret-syncer-tgg22"
Jul 20 17:55:56.408870 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.275676    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bb3035ba-fece-492c-aa33-43135c1925da-etc-modprobe-d\") pod \"tuned-q6gcb\" (UID: \"bb3035ba-fece-492c-aa33-43135c1925da\") " pod="openshift-cluster-node-tuning-operator/tuned-q6gcb"
Jul 20 17:55:56.408870 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.275692    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bb3035ba-fece-492c-aa33-43135c1925da-run\") pod \"tuned-q6gcb\" (UID: \"bb3035ba-fece-492c-aa33-43135c1925da\") " pod="openshift-cluster-node-tuning-operator/tuned-q6gcb"
Jul 20 17:55:56.408870 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.275684    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bb3035ba-fece-492c-aa33-43135c1925da-etc-sysctl-conf\") pod \"tuned-q6gcb\" (UID: \"bb3035ba-fece-492c-aa33-43135c1925da\") " pod="openshift-cluster-node-tuning-operator/tuned-q6gcb"
Jul 20 17:55:56.408870 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.275767    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/91b53241-763d-4868-81c3-786a3d6d4080-registration-dir\") pod \"aws-ebs-csi-driver-node-p74sv\" (UID: \"91b53241-763d-4868-81c3-786a3d6d4080\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p74sv"
Jul 20 17:55:56.408870 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.275793    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/91b53241-763d-4868-81c3-786a3d6d4080-etc-selinux\") pod \"aws-ebs-csi-driver-node-p74sv\" (UID: \"91b53241-763d-4868-81c3-786a3d6d4080\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p74sv"
Jul 20 17:55:56.412402 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.275843    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bb3035ba-fece-492c-aa33-43135c1925da-host\") pod \"tuned-q6gcb\" (UID: \"bb3035ba-fece-492c-aa33-43135c1925da\") " pod="openshift-cluster-node-tuning-operator/tuned-q6gcb"
Jul 20 17:55:56.412402 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.275860    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhthz\" (UniqueName: \"kubernetes.io/projected/91b53241-763d-4868-81c3-786a3d6d4080-kube-api-access-rhthz\") pod \"aws-ebs-csi-driver-node-p74sv\" (UID: \"91b53241-763d-4868-81c3-786a3d6d4080\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p74sv"
Jul 20 17:55:56.412402 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.275867    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/91b53241-763d-4868-81c3-786a3d6d4080-registration-dir\") pod \"aws-ebs-csi-driver-node-p74sv\" (UID: \"91b53241-763d-4868-81c3-786a3d6d4080\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p74sv"
Jul 20 17:55:56.412402 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.275875    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea740a69-374a-483b-a045-a2c795b99d4c-tmp-dir\") pod \"node-resolver-smcqb\" (UID: \"ea740a69-374a-483b-a045-a2c795b99d4c\") " pod="openshift-dns/node-resolver-smcqb"
Jul 20 17:55:56.412402 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.275906    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/91b53241-763d-4868-81c3-786a3d6d4080-socket-dir\") pod \"aws-ebs-csi-driver-node-p74sv\" (UID: \"91b53241-763d-4868-81c3-786a3d6d4080\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p74sv"
Jul 20 17:55:56.412402 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.275917    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bb3035ba-fece-492c-aa33-43135c1925da-etc-sysctl-conf\") pod \"tuned-q6gcb\" (UID: \"bb3035ba-fece-492c-aa33-43135c1925da\") " pod="openshift-cluster-node-tuning-operator/tuned-q6gcb"
Jul 20 17:55:56.412402 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.275935    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0a7d39a3-ee65-40f5-940c-149a0e87e580-iptables-alerter-script\") pod \"iptables-alerter-ql9lx\" (UID: \"0a7d39a3-ee65-40f5-940c-149a0e87e580\") " pod="openshift-network-operator/iptables-alerter-ql9lx"
Jul 20 17:55:56.412402 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.275930    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bb3035ba-fece-492c-aa33-43135c1925da-host\") pod \"tuned-q6gcb\" (UID: \"bb3035ba-fece-492c-aa33-43135c1925da\") " pod="openshift-cluster-node-tuning-operator/tuned-q6gcb"
Jul 20 17:55:56.412402 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.275961    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/bb3035ba-fece-492c-aa33-43135c1925da-etc-tuned\") pod \"tuned-q6gcb\" (UID: \"bb3035ba-fece-492c-aa33-43135c1925da\") " pod="openshift-cluster-node-tuning-operator/tuned-q6gcb"
Jul 20 17:55:56.412402 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.276373    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/91b53241-763d-4868-81c3-786a3d6d4080-socket-dir\") pod \"aws-ebs-csi-driver-node-p74sv\" (UID: \"91b53241-763d-4868-81c3-786a3d6d4080\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p74sv"
Jul 20 17:55:56.412402 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.276023    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/91b53241-763d-4868-81c3-786a3d6d4080-etc-selinux\") pod \"aws-ebs-csi-driver-node-p74sv\" (UID: \"91b53241-763d-4868-81c3-786a3d6d4080\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p74sv"
Jul 20 17:55:56.412402 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.276648    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea740a69-374a-483b-a045-a2c795b99d4c-tmp-dir\") pod \"node-resolver-smcqb\" (UID: \"ea740a69-374a-483b-a045-a2c795b99d4c\") " pod="openshift-dns/node-resolver-smcqb"
Jul 20 17:55:56.412402 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.275986    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/76c83a34-e7fe-46a1-a01e-944840b62724-konnectivity-ca\") pod \"konnectivity-agent-l45qp\" (UID: \"76c83a34-e7fe-46a1-a01e-944840b62724\") " pod="kube-system/konnectivity-agent-l45qp"
Jul 20 17:55:56.412402 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.276795    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bb3035ba-fece-492c-aa33-43135c1925da-etc-systemd\") pod \"tuned-q6gcb\" (UID: \"bb3035ba-fece-492c-aa33-43135c1925da\") " pod="openshift-cluster-node-tuning-operator/tuned-q6gcb"
Jul 20 17:55:56.414073 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.276883    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bb3035ba-fece-492c-aa33-43135c1925da-etc-systemd\") pod \"tuned-q6gcb\" (UID: \"bb3035ba-fece-492c-aa33-43135c1925da\") " pod="openshift-cluster-node-tuning-operator/tuned-q6gcb"
Jul 20 17:55:56.414073 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.277120    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0a7d39a3-ee65-40f5-940c-149a0e87e580-iptables-alerter-script\") pod \"iptables-alerter-ql9lx\" (UID: \"0a7d39a3-ee65-40f5-940c-149a0e87e580\") " pod="openshift-network-operator/iptables-alerter-ql9lx"
Jul 20 17:55:56.414073 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.277259    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/76c83a34-e7fe-46a1-a01e-944840b62724-agent-certs\") pod \"konnectivity-agent-l45qp\" (UID: \"76c83a34-e7fe-46a1-a01e-944840b62724\") " pod="kube-system/konnectivity-agent-l45qp"
Jul 20 17:55:56.414073 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.277300    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nd2w\" (UniqueName: \"kubernetes.io/projected/bb3035ba-fece-492c-aa33-43135c1925da-kube-api-access-4nd2w\") pod \"tuned-q6gcb\" (UID: \"bb3035ba-fece-492c-aa33-43135c1925da\") " pod="openshift-cluster-node-tuning-operator/tuned-q6gcb"
Jul 20 17:55:56.414073 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.277334    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/91b53241-763d-4868-81c3-786a3d6d4080-kubelet-dir\") pod \"aws-ebs-csi-driver-node-p74sv\" (UID: \"91b53241-763d-4868-81c3-786a3d6d4080\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p74sv"
Jul 20 17:55:56.414073 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.277388    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bb3035ba-fece-492c-aa33-43135c1925da-etc-sysconfig\") pod \"tuned-q6gcb\" (UID: \"bb3035ba-fece-492c-aa33-43135c1925da\") " pod="openshift-cluster-node-tuning-operator/tuned-q6gcb"
Jul 20 17:55:56.414073 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.277377    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/76c83a34-e7fe-46a1-a01e-944840b62724-konnectivity-ca\") pod \"konnectivity-agent-l45qp\" (UID: \"76c83a34-e7fe-46a1-a01e-944840b62724\") " pod="kube-system/konnectivity-agent-l45qp"
Jul 20 17:55:56.414073 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.277423    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bb3035ba-fece-492c-aa33-43135c1925da-etc-kubernetes\") pod \"tuned-q6gcb\" (UID: \"bb3035ba-fece-492c-aa33-43135c1925da\") " pod="openshift-cluster-node-tuning-operator/tuned-q6gcb"
Jul 20 17:55:56.414073 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.277461    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/eca0024f-2963-4ed1-817e-f1a6febd79aa-kubelet-config\") pod \"global-pull-secret-syncer-tgg22\" (UID: \"eca0024f-2963-4ed1-817e-f1a6febd79aa\") " pod="kube-system/global-pull-secret-syncer-tgg22"
Jul 20 17:55:56.414073 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.277469    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/91b53241-763d-4868-81c3-786a3d6d4080-kubelet-dir\") pod \"aws-ebs-csi-driver-node-p74sv\" (UID: \"91b53241-763d-4868-81c3-786a3d6d4080\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p74sv"
Jul 20 17:55:56.414073 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.277497    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eca0024f-2963-4ed1-817e-f1a6febd79aa-original-pull-secret\") pod \"global-pull-secret-syncer-tgg22\" (UID: \"eca0024f-2963-4ed1-817e-f1a6febd79aa\") " pod="kube-system/global-pull-secret-syncer-tgg22"
Jul 20 17:55:56.414073 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.277528    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pnrb\" (UniqueName: \"kubernetes.io/projected/0a7d39a3-ee65-40f5-940c-149a0e87e580-kube-api-access-9pnrb\") pod \"iptables-alerter-ql9lx\" (UID: \"0a7d39a3-ee65-40f5-940c-149a0e87e580\") " pod="openshift-network-operator/iptables-alerter-ql9lx"
Jul 20 17:55:56.414073 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.277560    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bb3035ba-fece-492c-aa33-43135c1925da-etc-sysctl-d\") pod \"tuned-q6gcb\" (UID: \"bb3035ba-fece-492c-aa33-43135c1925da\") " pod="openshift-cluster-node-tuning-operator/tuned-q6gcb"
Jul 20 17:55:56.414073 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.277594    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bb3035ba-fece-492c-aa33-43135c1925da-lib-modules\") pod \"tuned-q6gcb\" (UID: \"bb3035ba-fece-492c-aa33-43135c1925da\") " pod="openshift-cluster-node-tuning-operator/tuned-q6gcb"
Jul 20 17:55:56.416184 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.277624    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bb3035ba-fece-492c-aa33-43135c1925da-var-lib-kubelet\") pod \"tuned-q6gcb\" (UID: \"bb3035ba-fece-492c-aa33-43135c1925da\") " pod="openshift-cluster-node-tuning-operator/tuned-q6gcb"
Jul 20 17:55:56.416184 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.277654    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/91b53241-763d-4868-81c3-786a3d6d4080-device-dir\") pod \"aws-ebs-csi-driver-node-p74sv\" (UID: \"91b53241-763d-4868-81c3-786a3d6d4080\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p74sv"
Jul 20 17:55:56.416184 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.277701    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0a7d39a3-ee65-40f5-940c-149a0e87e580-host-slash\") pod \"iptables-alerter-ql9lx\" (UID: \"0a7d39a3-ee65-40f5-940c-149a0e87e580\") " pod="openshift-network-operator/iptables-alerter-ql9lx"
Jul 20 17:55:56.416184 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.277749    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/91b53241-763d-4868-81c3-786a3d6d4080-sys-fs\") pod \"aws-ebs-csi-driver-node-p74sv\" (UID: \"91b53241-763d-4868-81c3-786a3d6d4080\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p74sv"
Jul 20 17:55:56.416184 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.277870    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/91b53241-763d-4868-81c3-786a3d6d4080-sys-fs\") pod \"aws-ebs-csi-driver-node-p74sv\" (UID: \"91b53241-763d-4868-81c3-786a3d6d4080\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p74sv"
Jul 20 17:55:56.416184 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.277975    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bb3035ba-fece-492c-aa33-43135c1925da-etc-sysconfig\") pod \"tuned-q6gcb\" (UID: \"bb3035ba-fece-492c-aa33-43135c1925da\") " pod="openshift-cluster-node-tuning-operator/tuned-q6gcb"
Jul 20 17:55:56.416184 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.278005    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bb3035ba-fece-492c-aa33-43135c1925da-etc-sysctl-d\") pod \"tuned-q6gcb\" (UID: \"bb3035ba-fece-492c-aa33-43135c1925da\") " pod="openshift-cluster-node-tuning-operator/tuned-q6gcb"
Jul 20 17:55:56.416184 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.278036    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bb3035ba-fece-492c-aa33-43135c1925da-etc-kubernetes\") pod \"tuned-q6gcb\" (UID: \"bb3035ba-fece-492c-aa33-43135c1925da\") " pod="openshift-cluster-node-tuning-operator/tuned-q6gcb"
Jul 20 17:55:56.416184 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.278095    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bb3035ba-fece-492c-aa33-43135c1925da-lib-modules\") pod \"tuned-q6gcb\" (UID: \"bb3035ba-fece-492c-aa33-43135c1925da\") " pod="openshift-cluster-node-tuning-operator/tuned-q6gcb"
Jul 20 17:55:56.416184 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.278108    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/eca0024f-2963-4ed1-817e-f1a6febd79aa-kubelet-config\") pod \"global-pull-secret-syncer-tgg22\" (UID: \"eca0024f-2963-4ed1-817e-f1a6febd79aa\") " pod="kube-system/global-pull-secret-syncer-tgg22"
Jul 20 17:55:56.416184 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.278159    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bb3035ba-fece-492c-aa33-43135c1925da-var-lib-kubelet\") pod \"tuned-q6gcb\" (UID: \"bb3035ba-fece-492c-aa33-43135c1925da\") " pod="openshift-cluster-node-tuning-operator/tuned-q6gcb"
Jul 20 17:55:56.416184 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.278213    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0a7d39a3-ee65-40f5-940c-149a0e87e580-host-slash\") pod \"iptables-alerter-ql9lx\" (UID: \"0a7d39a3-ee65-40f5-940c-149a0e87e580\") " pod="openshift-network-operator/iptables-alerter-ql9lx"
Jul 20 17:55:56.416184 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:55:56.278253    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 20 17:55:56.416184 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:55:56.278321    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eca0024f-2963-4ed1-817e-f1a6febd79aa-original-pull-secret podName:eca0024f-2963-4ed1-817e-f1a6febd79aa nodeName:}" failed. No retries permitted until 2026-07-20 17:55:56.778300744 +0000 UTC m=+3.024834146 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/eca0024f-2963-4ed1-817e-f1a6febd79aa-original-pull-secret") pod "global-pull-secret-syncer-tgg22" (UID: "eca0024f-2963-4ed1-817e-f1a6febd79aa") : object "kube-system"/"original-pull-secret" not registered
Jul 20 17:55:56.417209 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.278319    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bb3035ba-fece-492c-aa33-43135c1925da-tmp\") pod \"tuned-q6gcb\" (UID: \"bb3035ba-fece-492c-aa33-43135c1925da\") " pod="openshift-cluster-node-tuning-operator/tuned-q6gcb"
Jul 20 17:55:56.417209 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.278357    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/91b53241-763d-4868-81c3-786a3d6d4080-device-dir\") pod \"aws-ebs-csi-driver-node-p74sv\" (UID: \"91b53241-763d-4868-81c3-786a3d6d4080\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p74sv"
Jul 20 17:55:56.417209 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.279368    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/bb3035ba-fece-492c-aa33-43135c1925da-etc-tuned\") pod \"tuned-q6gcb\" (UID: \"bb3035ba-fece-492c-aa33-43135c1925da\") " pod="openshift-cluster-node-tuning-operator/tuned-q6gcb"
Jul 20 17:55:56.417209 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.282092    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/76c83a34-e7fe-46a1-a01e-944840b62724-agent-certs\") pod \"konnectivity-agent-l45qp\" (UID: \"76c83a34-e7fe-46a1-a01e-944840b62724\") " pod="kube-system/konnectivity-agent-l45qp"
Jul 20 17:55:56.417209 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.296980    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcklf\" (UniqueName: \"kubernetes.io/projected/ea740a69-374a-483b-a045-a2c795b99d4c-kube-api-access-hcklf\") pod \"node-resolver-smcqb\" (UID: \"ea740a69-374a-483b-a045-a2c795b99d4c\") " pod="openshift-dns/node-resolver-smcqb"
Jul 20 17:55:56.417209 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.297105    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nd2w\" (UniqueName: \"kubernetes.io/projected/bb3035ba-fece-492c-aa33-43135c1925da-kube-api-access-4nd2w\") pod \"tuned-q6gcb\" (UID: \"bb3035ba-fece-492c-aa33-43135c1925da\") " pod="openshift-cluster-node-tuning-operator/tuned-q6gcb"
Jul 20 17:55:56.417209 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.300511    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pnrb\" (UniqueName: \"kubernetes.io/projected/0a7d39a3-ee65-40f5-940c-149a0e87e580-kube-api-access-9pnrb\") pod \"iptables-alerter-ql9lx\" (UID: \"0a7d39a3-ee65-40f5-940c-149a0e87e580\") " pod="openshift-network-operator/iptables-alerter-ql9lx"
Jul 20 17:55:56.417209 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.302056    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhthz\" (UniqueName: \"kubernetes.io/projected/91b53241-763d-4868-81c3-786a3d6d4080-kube-api-access-rhthz\") pod \"aws-ebs-csi-driver-node-p74sv\" (UID: \"91b53241-763d-4868-81c3-786a3d6d4080\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p74sv"
Jul 20 17:55:56.417209 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.365917    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-wjs8g"
Jul 20 17:55:56.417209 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.373737    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-q42fv"
Jul 20 17:55:56.417209 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.381040    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-mzsnt"
Jul 20 17:55:56.417209 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:56.382873    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07587a78_2d3a_40ad_9aae_a4875200263a.slice/crio-3f5c472cc4241104f563bf03785c3a17ad1bd565c6a44d003fb3e930466a4c8f WatchSource:0}: Error finding container 3f5c472cc4241104f563bf03785c3a17ad1bd565c6a44d003fb3e930466a4c8f: Status 404 returned error can't find the container with id 3f5c472cc4241104f563bf03785c3a17ad1bd565c6a44d003fb3e930466a4c8f
Jul 20 17:55:56.417209 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.385180    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-rqtm8"
Jul 20 17:55:56.417209 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:56.385246    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a564ed7_d4b4_4f1a_a7da_d1e4d0652601.slice/crio-8b4fe240fbd0fd1126da9a682748ee6f1908abbd57d9b339abf8d0d9f9eaad9f WatchSource:0}: Error finding container 8b4fe240fbd0fd1126da9a682748ee6f1908abbd57d9b339abf8d0d9f9eaad9f: Status 404 returned error can't find the container with id 8b4fe240fbd0fd1126da9a682748ee6f1908abbd57d9b339abf8d0d9f9eaad9f
Jul 20 17:55:56.417209 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.395052    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-l45qp"
Jul 20 17:55:56.417209 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.400175    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-ql9lx"
Jul 20 17:55:56.417209 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.405513    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-q6gcb"
Jul 20 17:55:56.418202 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:56.406841    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5f812b2_2ecb_4be4_99aa_7808ab7154e4.slice/crio-48c671e2255254bb57fe9a73206095e034c6c9a309eccfa6dc41e14c13f9d056 WatchSource:0}: Error finding container 48c671e2255254bb57fe9a73206095e034c6c9a309eccfa6dc41e14c13f9d056: Status 404 returned error can't find the container with id 48c671e2255254bb57fe9a73206095e034c6c9a309eccfa6dc41e14c13f9d056
Jul 20 17:55:56.418202 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:56.408256    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76c83a34_e7fe_46a1_a01e_944840b62724.slice/crio-f21b7fd18a6d0a1803e3ce2e3474e07f6ff398e4e477e4a8a18ea7c952b9195f WatchSource:0}: Error finding container f21b7fd18a6d0a1803e3ce2e3474e07f6ff398e4e477e4a8a18ea7c952b9195f: Status 404 returned error can't find the container with id f21b7fd18a6d0a1803e3ce2e3474e07f6ff398e4e477e4a8a18ea7c952b9195f
Jul 20 17:55:56.418202 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.410469    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p74sv"
Jul 20 17:55:56.418202 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:56.410735    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a7d39a3_ee65_40f5_940c_149a0e87e580.slice/crio-c9e4fe4034ab3bbb0f3793e2d504b901f7c4b58ee47e4aa15f29ef98eb60e4ef WatchSource:0}: Error finding container c9e4fe4034ab3bbb0f3793e2d504b901f7c4b58ee47e4aa15f29ef98eb60e4ef: Status 404 returned error can't find the container with id c9e4fe4034ab3bbb0f3793e2d504b901f7c4b58ee47e4aa15f29ef98eb60e4ef
Jul 20 17:55:56.418202 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.415422    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-smcqb"
Jul 20 17:55:56.418202 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:56.415912    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb3035ba_fece_492c_aa33_43135c1925da.slice/crio-3a268628deff0b10778e4d1961427d40288cc559e2fef89f91b717518c9cf1f2 WatchSource:0}: Error finding container 3a268628deff0b10778e4d1961427d40288cc559e2fef89f91b717518c9cf1f2: Status 404 returned error can't find the container with id 3a268628deff0b10778e4d1961427d40288cc559e2fef89f91b717518c9cf1f2
Jul 20 17:55:56.498111 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:56.425471    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91b53241_763d_4868_81c3_786a3d6d4080.slice/crio-e39ae81681fe14e88ccda87d1b5db0449ac5f31a924d335bfaf4be01caba6800 WatchSource:0}: Error finding container e39ae81681fe14e88ccda87d1b5db0449ac5f31a924d335bfaf4be01caba6800: Status 404 returned error can't find the container with id e39ae81681fe14e88ccda87d1b5db0449ac5f31a924d335bfaf4be01caba6800
Jul 20 17:55:56.498111 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:55:56.431470    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea740a69_374a_483b_a045_a2c795b99d4c.slice/crio-d085d7c6fb7c0aec226512982da725dc4655bde1d99d5f944a4ce8d2100d10ea WatchSource:0}: Error finding container d085d7c6fb7c0aec226512982da725dc4655bde1d99d5f944a4ce8d2100d10ea: Status 404 returned error can't find the container with id d085d7c6fb7c0aec226512982da725dc4655bde1d99d5f944a4ce8d2100d10ea
Jul 20 17:55:56.749253 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.680671    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26b0d75c-ecf7-4bb4-ad4b-d9077da808bd-metrics-certs\") pod \"network-metrics-daemon-nkmr4\" (UID: \"26b0d75c-ecf7-4bb4-ad4b-d9077da808bd\") " pod="openshift-multus/network-metrics-daemon-nkmr4"
Jul 20 17:55:56.749253 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:55:56.680842    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 20 17:55:56.749253 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:55:56.680916    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26b0d75c-ecf7-4bb4-ad4b-d9077da808bd-metrics-certs podName:26b0d75c-ecf7-4bb4-ad4b-d9077da808bd nodeName:}" failed. No retries permitted until 2026-07-20 17:55:57.680896321 +0000 UTC m=+3.927429711 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/26b0d75c-ecf7-4bb4-ad4b-d9077da808bd-metrics-certs") pod "network-metrics-daemon-nkmr4" (UID: "26b0d75c-ecf7-4bb4-ad4b-d9077da808bd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 20 17:55:56.749253 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.691285    2569 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 20 17:55:56.822742 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.781635    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eca0024f-2963-4ed1-817e-f1a6febd79aa-original-pull-secret\") pod \"global-pull-secret-syncer-tgg22\" (UID: \"eca0024f-2963-4ed1-817e-f1a6febd79aa\") " pod="kube-system/global-pull-secret-syncer-tgg22"
Jul 20 17:55:56.822742 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:56.781700    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mgpl\" (UniqueName: \"kubernetes.io/projected/de1b439b-702a-43a3-8118-f7e20e716ac5-kube-api-access-4mgpl\") pod \"network-check-target-8v5ff\" (UID: \"de1b439b-702a-43a3-8118-f7e20e716ac5\") " pod="openshift-network-diagnostics/network-check-target-8v5ff"
Jul 20 17:55:56.822742 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:55:56.781791    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 20 17:55:56.822742 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:55:56.781864    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 20 17:55:56.822742 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:55:56.781885    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 20 17:55:56.822742 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:55:56.781898    2569 projected.go:196] Error preparing data for projected volume kube-api-access-4mgpl for pod openshift-network-diagnostics/network-check-target-8v5ff: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 20 17:55:56.822742 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:55:56.781858    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eca0024f-2963-4ed1-817e-f1a6febd79aa-original-pull-secret podName:eca0024f-2963-4ed1-817e-f1a6febd79aa nodeName:}" failed. No retries permitted until 2026-07-20 17:55:57.781840949 +0000 UTC m=+4.028374338 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/eca0024f-2963-4ed1-817e-f1a6febd79aa-original-pull-secret") pod "global-pull-secret-syncer-tgg22" (UID: "eca0024f-2963-4ed1-817e-f1a6febd79aa") : object "kube-system"/"original-pull-secret" not registered
Jul 20 17:55:56.822742 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:55:56.781969    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/de1b439b-702a-43a3-8118-f7e20e716ac5-kube-api-access-4mgpl podName:de1b439b-702a-43a3-8118-f7e20e716ac5 nodeName:}" failed. No retries permitted until 2026-07-20 17:55:57.781956571 +0000 UTC m=+4.028489958 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-4mgpl" (UniqueName: "kubernetes.io/projected/de1b439b-702a-43a3-8118-f7e20e716ac5-kube-api-access-4mgpl") pod "network-check-target-8v5ff" (UID: "de1b439b-702a-43a3-8118-f7e20e716ac5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 20 17:55:57.123637 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:57.123597    2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-19 17:50:56 +0000 UTC" deadline="2027-12-29 11:46:09.078074731 +0000 UTC"
Jul 20 17:55:57.123637 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:57.123628    2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12641h50m11.954450049s"
Jul 20 17:55:57.177060 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:57.177027    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tgg22"
Jul 20 17:55:57.177209 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:55:57.177140    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tgg22" podUID="eca0024f-2963-4ed1-817e-f1a6febd79aa"
Jul 20 17:55:57.186152 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:57.186121    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-l45qp" event={"ID":"76c83a34-e7fe-46a1-a01e-944840b62724","Type":"ContainerStarted","Data":"f21b7fd18a6d0a1803e3ce2e3474e07f6ff398e4e477e4a8a18ea7c952b9195f"}
Jul 20 17:55:57.187421 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:57.187381    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mzsnt" event={"ID":"1dfc0381-8b5e-4400-a1c8-da99129540c7","Type":"ContainerStarted","Data":"a0cc9a69d33c6433bb395b0566891fb4b2d9bb4af987df424261ab640bd9489a"}
Jul 20 17:55:57.188563 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:57.188539    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wjs8g" event={"ID":"07587a78-2d3a-40ad-9aae-a4875200263a","Type":"ContainerStarted","Data":"3f5c472cc4241104f563bf03785c3a17ad1bd565c6a44d003fb3e930466a4c8f"}
Jul 20 17:55:57.189622 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:57.189595    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-smcqb" event={"ID":"ea740a69-374a-483b-a045-a2c795b99d4c","Type":"ContainerStarted","Data":"d085d7c6fb7c0aec226512982da725dc4655bde1d99d5f944a4ce8d2100d10ea"}
Jul 20 17:55:57.190749 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:57.190713    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p74sv" event={"ID":"91b53241-763d-4868-81c3-786a3d6d4080","Type":"ContainerStarted","Data":"e39ae81681fe14e88ccda87d1b5db0449ac5f31a924d335bfaf4be01caba6800"}
Jul 20 17:55:57.191714 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:57.191693    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-rqtm8" event={"ID":"d5f812b2-2ecb-4be4-99aa-7808ab7154e4","Type":"ContainerStarted","Data":"48c671e2255254bb57fe9a73206095e034c6c9a309eccfa6dc41e14c13f9d056"}
Jul 20 17:55:57.192759 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:57.192737    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-q42fv" event={"ID":"7a564ed7-d4b4-4f1a-a7da-d1e4d0652601","Type":"ContainerStarted","Data":"8b4fe240fbd0fd1126da9a682748ee6f1908abbd57d9b339abf8d0d9f9eaad9f"}
Jul 20 17:55:57.193749 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:57.193713    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-q6gcb" event={"ID":"bb3035ba-fece-492c-aa33-43135c1925da","Type":"ContainerStarted","Data":"3a268628deff0b10778e4d1961427d40288cc559e2fef89f91b717518c9cf1f2"}
Jul 20 17:55:57.194752 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:57.194715    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-ql9lx" event={"ID":"0a7d39a3-ee65-40f5-940c-149a0e87e580","Type":"ContainerStarted","Data":"c9e4fe4034ab3bbb0f3793e2d504b901f7c4b58ee47e4aa15f29ef98eb60e4ef"}
Jul 20 17:55:57.689081 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:57.688798    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26b0d75c-ecf7-4bb4-ad4b-d9077da808bd-metrics-certs\") pod \"network-metrics-daemon-nkmr4\" (UID: \"26b0d75c-ecf7-4bb4-ad4b-d9077da808bd\") " pod="openshift-multus/network-metrics-daemon-nkmr4"
Jul 20 17:55:57.689081 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:55:57.689011    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 20 17:55:57.689081 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:55:57.689075    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26b0d75c-ecf7-4bb4-ad4b-d9077da808bd-metrics-certs podName:26b0d75c-ecf7-4bb4-ad4b-d9077da808bd nodeName:}" failed. No retries permitted until 2026-07-20 17:55:59.689057117 +0000 UTC m=+5.935590519 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/26b0d75c-ecf7-4bb4-ad4b-d9077da808bd-metrics-certs") pod "network-metrics-daemon-nkmr4" (UID: "26b0d75c-ecf7-4bb4-ad4b-d9077da808bd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 20 17:55:57.789219 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:57.789176    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eca0024f-2963-4ed1-817e-f1a6febd79aa-original-pull-secret\") pod \"global-pull-secret-syncer-tgg22\" (UID: \"eca0024f-2963-4ed1-817e-f1a6febd79aa\") " pod="kube-system/global-pull-secret-syncer-tgg22"
Jul 20 17:55:57.789392 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:57.789252    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mgpl\" (UniqueName: \"kubernetes.io/projected/de1b439b-702a-43a3-8118-f7e20e716ac5-kube-api-access-4mgpl\") pod \"network-check-target-8v5ff\" (UID: \"de1b439b-702a-43a3-8118-f7e20e716ac5\") " pod="openshift-network-diagnostics/network-check-target-8v5ff"
Jul 20 17:55:57.789457 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:55:57.789423    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 20 17:55:57.789457 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:55:57.789448    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 20 17:55:57.789559 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:55:57.789461    2569 projected.go:196] Error preparing data for projected volume kube-api-access-4mgpl for pod openshift-network-diagnostics/network-check-target-8v5ff: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 20 17:55:57.789559 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:55:57.789519    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/de1b439b-702a-43a3-8118-f7e20e716ac5-kube-api-access-4mgpl podName:de1b439b-702a-43a3-8118-f7e20e716ac5 nodeName:}" failed. No retries permitted until 2026-07-20 17:55:59.789501684 +0000 UTC m=+6.036035079 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-4mgpl" (UniqueName: "kubernetes.io/projected/de1b439b-702a-43a3-8118-f7e20e716ac5-kube-api-access-4mgpl") pod "network-check-target-8v5ff" (UID: "de1b439b-702a-43a3-8118-f7e20e716ac5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 20 17:55:57.789942 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:55:57.789921    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 20 17:55:57.790051 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:55:57.789977    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eca0024f-2963-4ed1-817e-f1a6febd79aa-original-pull-secret podName:eca0024f-2963-4ed1-817e-f1a6febd79aa nodeName:}" failed. No retries permitted until 2026-07-20 17:55:59.789962496 +0000 UTC m=+6.036495887 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/eca0024f-2963-4ed1-817e-f1a6febd79aa-original-pull-secret") pod "global-pull-secret-syncer-tgg22" (UID: "eca0024f-2963-4ed1-817e-f1a6febd79aa") : object "kube-system"/"original-pull-secret" not registered
Jul 20 17:55:58.124030 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:58.123982    2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-19 17:50:56 +0000 UTC" deadline="2028-03-04 01:40:04.839709384 +0000 UTC"
Jul 20 17:55:58.124030 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:58.124026    2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14215h44m6.715688716s"
Jul 20 17:55:58.176656 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:58.176624    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8v5ff"
Jul 20 17:55:58.176839 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:55:58.176765    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8v5ff" podUID="de1b439b-702a-43a3-8118-f7e20e716ac5"
Jul 20 17:55:58.177186 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:58.177164    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nkmr4"
Jul 20 17:55:58.177335 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:55:58.177278    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nkmr4" podUID="26b0d75c-ecf7-4bb4-ad4b-d9077da808bd"
Jul 20 17:55:59.176764 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:59.176717    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tgg22"
Jul 20 17:55:59.177213 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:55:59.176865    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tgg22" podUID="eca0024f-2963-4ed1-817e-f1a6febd79aa"
Jul 20 17:55:59.705821 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:59.705242    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26b0d75c-ecf7-4bb4-ad4b-d9077da808bd-metrics-certs\") pod \"network-metrics-daemon-nkmr4\" (UID: \"26b0d75c-ecf7-4bb4-ad4b-d9077da808bd\") " pod="openshift-multus/network-metrics-daemon-nkmr4"
Jul 20 17:55:59.705821 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:55:59.705414    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 20 17:55:59.705821 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:55:59.705478    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26b0d75c-ecf7-4bb4-ad4b-d9077da808bd-metrics-certs podName:26b0d75c-ecf7-4bb4-ad4b-d9077da808bd nodeName:}" failed. No retries permitted until 2026-07-20 17:56:03.705460346 +0000 UTC m=+9.951993747 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/26b0d75c-ecf7-4bb4-ad4b-d9077da808bd-metrics-certs") pod "network-metrics-daemon-nkmr4" (UID: "26b0d75c-ecf7-4bb4-ad4b-d9077da808bd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 20 17:55:59.806173 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:59.805532    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eca0024f-2963-4ed1-817e-f1a6febd79aa-original-pull-secret\") pod \"global-pull-secret-syncer-tgg22\" (UID: \"eca0024f-2963-4ed1-817e-f1a6febd79aa\") " pod="kube-system/global-pull-secret-syncer-tgg22"
Jul 20 17:55:59.806173 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:55:59.805597    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mgpl\" (UniqueName: \"kubernetes.io/projected/de1b439b-702a-43a3-8118-f7e20e716ac5-kube-api-access-4mgpl\") pod \"network-check-target-8v5ff\" (UID: \"de1b439b-702a-43a3-8118-f7e20e716ac5\") " pod="openshift-network-diagnostics/network-check-target-8v5ff"
Jul 20 17:55:59.806173 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:55:59.805734    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 20 17:55:59.806173 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:55:59.805750    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 20 17:55:59.806173 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:55:59.805759    2569 projected.go:196] Error preparing data for projected volume kube-api-access-4mgpl for pod openshift-network-diagnostics/network-check-target-8v5ff: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 20 17:55:59.806173 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:55:59.805801    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/de1b439b-702a-43a3-8118-f7e20e716ac5-kube-api-access-4mgpl podName:de1b439b-702a-43a3-8118-f7e20e716ac5 nodeName:}" failed. No retries permitted until 2026-07-20 17:56:03.805789151 +0000 UTC m=+10.052322538 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-4mgpl" (UniqueName: "kubernetes.io/projected/de1b439b-702a-43a3-8118-f7e20e716ac5-kube-api-access-4mgpl") pod "network-check-target-8v5ff" (UID: "de1b439b-702a-43a3-8118-f7e20e716ac5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 20 17:55:59.806173 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:55:59.806101    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 20 17:55:59.806173 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:55:59.806142    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eca0024f-2963-4ed1-817e-f1a6febd79aa-original-pull-secret podName:eca0024f-2963-4ed1-817e-f1a6febd79aa nodeName:}" failed. No retries permitted until 2026-07-20 17:56:03.806132033 +0000 UTC m=+10.052665421 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/eca0024f-2963-4ed1-817e-f1a6febd79aa-original-pull-secret") pod "global-pull-secret-syncer-tgg22" (UID: "eca0024f-2963-4ed1-817e-f1a6febd79aa") : object "kube-system"/"original-pull-secret" not registered
Jul 20 17:56:00.176928 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:00.176893    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nkmr4"
Jul 20 17:56:00.177350 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:56:00.177102    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nkmr4" podUID="26b0d75c-ecf7-4bb4-ad4b-d9077da808bd"
Jul 20 17:56:00.177350 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:00.177121    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8v5ff"
Jul 20 17:56:00.177350 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:56:00.177199    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8v5ff" podUID="de1b439b-702a-43a3-8118-f7e20e716ac5"
Jul 20 17:56:01.176912 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:01.176882    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tgg22"
Jul 20 17:56:01.177074 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:56:01.177001    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tgg22" podUID="eca0024f-2963-4ed1-817e-f1a6febd79aa"
Jul 20 17:56:02.177437 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:02.177405    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nkmr4"
Jul 20 17:56:02.177933 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:02.177409    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8v5ff"
Jul 20 17:56:02.177933 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:56:02.177543    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nkmr4" podUID="26b0d75c-ecf7-4bb4-ad4b-d9077da808bd"
Jul 20 17:56:02.177933 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:56:02.177602    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8v5ff" podUID="de1b439b-702a-43a3-8118-f7e20e716ac5"
Jul 20 17:56:03.176661 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:03.176626    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tgg22"
Jul 20 17:56:03.176847 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:56:03.176763    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tgg22" podUID="eca0024f-2963-4ed1-817e-f1a6febd79aa"
Jul 20 17:56:03.742645 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:03.742601    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26b0d75c-ecf7-4bb4-ad4b-d9077da808bd-metrics-certs\") pod \"network-metrics-daemon-nkmr4\" (UID: \"26b0d75c-ecf7-4bb4-ad4b-d9077da808bd\") " pod="openshift-multus/network-metrics-daemon-nkmr4"
Jul 20 17:56:03.743244 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:56:03.742814    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 20 17:56:03.743244 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:56:03.742892    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26b0d75c-ecf7-4bb4-ad4b-d9077da808bd-metrics-certs podName:26b0d75c-ecf7-4bb4-ad4b-d9077da808bd nodeName:}" failed. No retries permitted until 2026-07-20 17:56:11.742869586 +0000 UTC m=+17.989402999 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/26b0d75c-ecf7-4bb4-ad4b-d9077da808bd-metrics-certs") pod "network-metrics-daemon-nkmr4" (UID: "26b0d75c-ecf7-4bb4-ad4b-d9077da808bd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 20 17:56:03.843715 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:03.843665    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eca0024f-2963-4ed1-817e-f1a6febd79aa-original-pull-secret\") pod \"global-pull-secret-syncer-tgg22\" (UID: \"eca0024f-2963-4ed1-817e-f1a6febd79aa\") " pod="kube-system/global-pull-secret-syncer-tgg22"
Jul 20 17:56:03.843913 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:03.843757    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mgpl\" (UniqueName: \"kubernetes.io/projected/de1b439b-702a-43a3-8118-f7e20e716ac5-kube-api-access-4mgpl\") pod \"network-check-target-8v5ff\" (UID: \"de1b439b-702a-43a3-8118-f7e20e716ac5\") " pod="openshift-network-diagnostics/network-check-target-8v5ff"
Jul 20 17:56:03.843913 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:56:03.843857    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 20 17:56:03.843913 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:56:03.843897    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 20 17:56:03.844069 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:56:03.843918    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 20 17:56:03.844069 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:56:03.843929    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eca0024f-2963-4ed1-817e-f1a6febd79aa-original-pull-secret podName:eca0024f-2963-4ed1-817e-f1a6febd79aa nodeName:}" failed. No retries permitted until 2026-07-20 17:56:11.843910433 +0000 UTC m=+18.090443827 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/eca0024f-2963-4ed1-817e-f1a6febd79aa-original-pull-secret") pod "global-pull-secret-syncer-tgg22" (UID: "eca0024f-2963-4ed1-817e-f1a6febd79aa") : object "kube-system"/"original-pull-secret" not registered
Jul 20 17:56:03.844069 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:56:03.843931    2569 projected.go:196] Error preparing data for projected volume kube-api-access-4mgpl for pod openshift-network-diagnostics/network-check-target-8v5ff: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 20 17:56:03.844069 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:56:03.843973    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/de1b439b-702a-43a3-8118-f7e20e716ac5-kube-api-access-4mgpl podName:de1b439b-702a-43a3-8118-f7e20e716ac5 nodeName:}" failed. No retries permitted until 2026-07-20 17:56:11.843963899 +0000 UTC m=+18.090497293 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-4mgpl" (UniqueName: "kubernetes.io/projected/de1b439b-702a-43a3-8118-f7e20e716ac5-kube-api-access-4mgpl") pod "network-check-target-8v5ff" (UID: "de1b439b-702a-43a3-8118-f7e20e716ac5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 20 17:56:04.178382 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:04.178298    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nkmr4"
Jul 20 17:56:04.178568 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:04.178438    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8v5ff"
Jul 20 17:56:04.178568 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:56:04.178550    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8v5ff" podUID="de1b439b-702a-43a3-8118-f7e20e716ac5"
Jul 20 17:56:04.178681 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:56:04.178434    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nkmr4" podUID="26b0d75c-ecf7-4bb4-ad4b-d9077da808bd"
Jul 20 17:56:05.177405 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:05.177368    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tgg22"
Jul 20 17:56:05.177826 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:56:05.177520    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tgg22" podUID="eca0024f-2963-4ed1-817e-f1a6febd79aa"
Jul 20 17:56:06.177536 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:06.177508    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8v5ff"
Jul 20 17:56:06.178006 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:06.177446    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nkmr4"
Jul 20 17:56:06.178006 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:56:06.177864    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nkmr4" podUID="26b0d75c-ecf7-4bb4-ad4b-d9077da808bd"
Jul 20 17:56:06.178006 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:56:06.177636    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8v5ff" podUID="de1b439b-702a-43a3-8118-f7e20e716ac5"
Jul 20 17:56:07.177098 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:07.177076    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tgg22"
Jul 20 17:56:07.177186 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:56:07.177170    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tgg22" podUID="eca0024f-2963-4ed1-817e-f1a6febd79aa"
Jul 20 17:56:08.177035 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:08.176963    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nkmr4"
Jul 20 17:56:08.177501 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:56:08.177109    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nkmr4" podUID="26b0d75c-ecf7-4bb4-ad4b-d9077da808bd"
Jul 20 17:56:08.177501 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:08.177148    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8v5ff"
Jul 20 17:56:08.177501 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:56:08.177261    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8v5ff" podUID="de1b439b-702a-43a3-8118-f7e20e716ac5"
Jul 20 17:56:08.226805 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:08.226758    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-q6gcb" event={"ID":"bb3035ba-fece-492c-aa33-43135c1925da","Type":"ContainerStarted","Data":"fa8a49f61810731720c300869d65f7461f3dd91d8d64d6d6abb17dd6b8bb4d6b"}
Jul 20 17:56:08.228877 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:08.228850    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-247.ec2.internal" event={"ID":"757e30ff139839aada6b39c2403a950d","Type":"ContainerStarted","Data":"6bc8342242e502461e978c1ac2e31e5b19b924379e707dd293075b7506abae39"}
Jul 20 17:56:08.246142 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:08.245789    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-q6gcb" podStartSLOduration=3.530695652 podStartE2EDuration="14.245770032s" totalImagesPullingTime="10.71507438s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 17:55:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 17:56:08.245140649 +0000 UTC m=+14.491674059" watchObservedRunningTime="2026-07-20 17:56:08.245770032 +0000 UTC m=+14.492303442"
Jul 20 17:56:08.259653 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:08.259606    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-247.ec2.internal" podStartSLOduration=13.2595896 podStartE2EDuration="13.2595896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 17:55:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 17:56:08.259046281 +0000 UTC m=+14.505579691" watchObservedRunningTime="2026-07-20 17:56:08.2595896 +0000 UTC m=+14.506123009"
Jul 20 17:56:09.176822 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:09.176564    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tgg22"
Jul 20 17:56:09.176964 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:56:09.176932    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tgg22" podUID="eca0024f-2963-4ed1-817e-f1a6febd79aa"
Jul 20 17:56:09.232761 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:09.232692    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-247.ec2.internal" event={"ID":"7c055eb6eb0be9ef4618781875e36f04","Type":"ContainerStarted","Data":"d9456f6b409b5701f31e9e86a9ff95c18d4705da773dd6b9156b4f7d71df4e16"}
Jul 20 17:56:09.234310 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:09.234281    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-ql9lx" event={"ID":"0a7d39a3-ee65-40f5-940c-149a0e87e580","Type":"ContainerStarted","Data":"ecbd2ddd6241efbfa2530d986885a0b2a618838c35bf2776ae419581442b0e3a"}
Jul 20 17:56:09.235757 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:09.235706    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-l45qp" event={"ID":"76c83a34-e7fe-46a1-a01e-944840b62724","Type":"ContainerStarted","Data":"b2145dfa90c6c51e48b5f581f28515e5ed04c0be0afbac09bbd0c1f7933a19ff"}
Jul 20 17:56:09.237498 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:09.237466    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mzsnt" event={"ID":"1dfc0381-8b5e-4400-a1c8-da99129540c7","Type":"ContainerStarted","Data":"bc23440eff42f7724f8c481163694cbbe9bb30dddd2d901f1fabaa0f022b7d51"}
Jul 20 17:56:09.238955 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:09.238919    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-smcqb" event={"ID":"ea740a69-374a-483b-a045-a2c795b99d4c","Type":"ContainerStarted","Data":"01cf52d00a4a200f92836a98238e711febdada7118e21c3aad4ecee8dfc28eb8"}
Jul 20 17:56:09.240329 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:09.240307    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p74sv" event={"ID":"91b53241-763d-4868-81c3-786a3d6d4080","Type":"ContainerStarted","Data":"171d1af67f26ead908b38523f4b497ff3285412957e100a8b872fc80a4120b59"}
Jul 20 17:56:09.241785 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:09.241763    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-q42fv" event={"ID":"7a564ed7-d4b4-4f1a-a7da-d1e4d0652601","Type":"ContainerStarted","Data":"f3fa782aadf4eee69b5b929d5ec1c9fb28744bd4ab3ea12fd631a5fb694c5073"}
Jul 20 17:56:09.272887 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:09.272829    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-l45qp" podStartSLOduration=4.599664035 podStartE2EDuration="15.272811111s" totalImagesPullingTime="10.673147076s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 17:55:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 17:56:09.27281163 +0000 UTC m=+15.519345043" watchObservedRunningTime="2026-07-20 17:56:09.272811111 +0000 UTC m=+15.519344522"
Jul 20 17:56:09.284932 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:09.284883    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-ql9lx" podStartSLOduration=4.559705961 podStartE2EDuration="15.284868144s" totalImagesPullingTime="10.725162183s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 17:55:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 17:56:09.284489483 +0000 UTC m=+15.531022894" watchObservedRunningTime="2026-07-20 17:56:09.284868144 +0000 UTC m=+15.531401555"
Jul 20 17:56:09.296575 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:09.296536    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-smcqb" podStartSLOduration=4.623331187 podStartE2EDuration="15.296521832s" totalImagesPullingTime="10.673190645s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 17:55:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 17:56:09.296131323 +0000 UTC m=+15.542664746" watchObservedRunningTime="2026-07-20 17:56:09.296521832 +0000 UTC m=+15.543055242"
Jul 20 17:56:09.307623 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:09.307579    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-q42fv" podStartSLOduration=4.571614376 podStartE2EDuration="15.307565438s" totalImagesPullingTime="10.735951062s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 17:55:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 17:56:09.30743486 +0000 UTC m=+15.553968273" watchObservedRunningTime="2026-07-20 17:56:09.307565438 +0000 UTC m=+15.554098848"
Jul 20 17:56:10.176684 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:10.176648    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nkmr4"
Jul 20 17:56:10.176898 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:56:10.176811    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nkmr4" podUID="26b0d75c-ecf7-4bb4-ad4b-d9077da808bd"
Jul 20 17:56:10.176898 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:10.176874    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8v5ff"
Jul 20 17:56:10.177017 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:56:10.176959    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8v5ff" podUID="de1b439b-702a-43a3-8118-f7e20e716ac5"
Jul 20 17:56:11.176672 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:11.176635    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tgg22"
Jul 20 17:56:11.177192 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:56:11.176768    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tgg22" podUID="eca0024f-2963-4ed1-817e-f1a6febd79aa"
Jul 20 17:56:11.809641 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:11.809603    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26b0d75c-ecf7-4bb4-ad4b-d9077da808bd-metrics-certs\") pod \"network-metrics-daemon-nkmr4\" (UID: \"26b0d75c-ecf7-4bb4-ad4b-d9077da808bd\") " pod="openshift-multus/network-metrics-daemon-nkmr4"
Jul 20 17:56:11.809888 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:56:11.809777    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 20 17:56:11.809888 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:56:11.809844    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26b0d75c-ecf7-4bb4-ad4b-d9077da808bd-metrics-certs podName:26b0d75c-ecf7-4bb4-ad4b-d9077da808bd nodeName:}" failed. No retries permitted until 2026-07-20 17:56:27.809827664 +0000 UTC m=+34.056361055 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/26b0d75c-ecf7-4bb4-ad4b-d9077da808bd-metrics-certs") pod "network-metrics-daemon-nkmr4" (UID: "26b0d75c-ecf7-4bb4-ad4b-d9077da808bd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 20 17:56:11.910510 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:11.910474    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eca0024f-2963-4ed1-817e-f1a6febd79aa-original-pull-secret\") pod \"global-pull-secret-syncer-tgg22\" (UID: \"eca0024f-2963-4ed1-817e-f1a6febd79aa\") " pod="kube-system/global-pull-secret-syncer-tgg22"
Jul 20 17:56:11.910751 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:11.910540    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mgpl\" (UniqueName: \"kubernetes.io/projected/de1b439b-702a-43a3-8118-f7e20e716ac5-kube-api-access-4mgpl\") pod \"network-check-target-8v5ff\" (UID: \"de1b439b-702a-43a3-8118-f7e20e716ac5\") " pod="openshift-network-diagnostics/network-check-target-8v5ff"
Jul 20 17:56:11.910751 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:56:11.910639    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 20 17:56:11.910751 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:56:11.910653    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 20 17:56:11.910751 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:56:11.910669    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 20 17:56:11.910751 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:56:11.910681    2569 projected.go:196] Error preparing data for projected volume kube-api-access-4mgpl for pod openshift-network-diagnostics/network-check-target-8v5ff: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 20 17:56:11.910751 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:56:11.910703    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eca0024f-2963-4ed1-817e-f1a6febd79aa-original-pull-secret podName:eca0024f-2963-4ed1-817e-f1a6febd79aa nodeName:}" failed. No retries permitted until 2026-07-20 17:56:27.910685381 +0000 UTC m=+34.157218792 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/eca0024f-2963-4ed1-817e-f1a6febd79aa-original-pull-secret") pod "global-pull-secret-syncer-tgg22" (UID: "eca0024f-2963-4ed1-817e-f1a6febd79aa") : object "kube-system"/"original-pull-secret" not registered
Jul 20 17:56:11.910751 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:56:11.910736    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/de1b439b-702a-43a3-8118-f7e20e716ac5-kube-api-access-4mgpl podName:de1b439b-702a-43a3-8118-f7e20e716ac5 nodeName:}" failed. No retries permitted until 2026-07-20 17:56:27.910711735 +0000 UTC m=+34.157245125 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-4mgpl" (UniqueName: "kubernetes.io/projected/de1b439b-702a-43a3-8118-f7e20e716ac5-kube-api-access-4mgpl") pod "network-check-target-8v5ff" (UID: "de1b439b-702a-43a3-8118-f7e20e716ac5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 20 17:56:12.177220 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:12.177138    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nkmr4"
Jul 20 17:56:12.177220 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:12.177162    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8v5ff"
Jul 20 17:56:12.177929 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:56:12.177267    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8v5ff" podUID="de1b439b-702a-43a3-8118-f7e20e716ac5"
Jul 20 17:56:12.177929 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:56:12.177373    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nkmr4" podUID="26b0d75c-ecf7-4bb4-ad4b-d9077da808bd"
Jul 20 17:56:12.249769 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:12.249734    2569 generic.go:356] "Generic (PLEG): container finished" podID="7c055eb6eb0be9ef4618781875e36f04" containerID="d9456f6b409b5701f31e9e86a9ff95c18d4705da773dd6b9156b4f7d71df4e16" exitCode=0
Jul 20 17:56:12.249957 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:12.249804    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-247.ec2.internal" event={"ID":"7c055eb6eb0be9ef4618781875e36f04","Type":"ContainerDied","Data":"d9456f6b409b5701f31e9e86a9ff95c18d4705da773dd6b9156b4f7d71df4e16"}
Jul 20 17:56:12.251486 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:12.251464    2569 generic.go:356] "Generic (PLEG): container finished" podID="1dfc0381-8b5e-4400-a1c8-da99129540c7" containerID="bc23440eff42f7724f8c481163694cbbe9bb30dddd2d901f1fabaa0f022b7d51" exitCode=0
Jul 20 17:56:12.251591 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:12.251507    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mzsnt" event={"ID":"1dfc0381-8b5e-4400-a1c8-da99129540c7","Type":"ContainerDied","Data":"bc23440eff42f7724f8c481163694cbbe9bb30dddd2d901f1fabaa0f022b7d51"}
Jul 20 17:56:13.177323 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:13.177289    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tgg22"
Jul 20 17:56:13.177774 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:56:13.177417    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tgg22" podUID="eca0024f-2963-4ed1-817e-f1a6febd79aa"
Jul 20 17:56:13.964373 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:13.964332    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-l45qp"
Jul 20 17:56:13.965098 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:13.965073    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-l45qp"
Jul 20 17:56:14.177562 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:14.177527    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nkmr4"
Jul 20 17:56:14.178249 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:14.177592    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8v5ff"
Jul 20 17:56:14.178434 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:56:14.178394    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nkmr4" podUID="26b0d75c-ecf7-4bb4-ad4b-d9077da808bd"
Jul 20 17:56:14.178554 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:56:14.178520    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8v5ff" podUID="de1b439b-702a-43a3-8118-f7e20e716ac5"
Jul 20 17:56:14.256053 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:14.256017    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-l45qp"
Jul 20 17:56:14.256551 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:14.256533    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-l45qp"
Jul 20 17:56:15.177277 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:15.177245    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tgg22"
Jul 20 17:56:15.177485 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:56:15.177395    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tgg22" podUID="eca0024f-2963-4ed1-817e-f1a6febd79aa"
Jul 20 17:56:16.176987 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:16.176952    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nkmr4"
Jul 20 17:56:16.177415 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:16.176958    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8v5ff"
Jul 20 17:56:16.177415 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:56:16.177103    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nkmr4" podUID="26b0d75c-ecf7-4bb4-ad4b-d9077da808bd"
Jul 20 17:56:16.177415 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:56:16.177165    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8v5ff" podUID="de1b439b-702a-43a3-8118-f7e20e716ac5"
Jul 20 17:56:16.674497 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:16.674279    2569 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 20 17:56:17.130406 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:17.130019    2569 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-20T17:56:16.674496518Z","UUID":"f421dbb0-81f0-47df-84b3-797fb87c5833","Handler":null,"Name":"","Endpoint":""}
Jul 20 17:56:17.131707 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:17.131664    2569 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 20 17:56:17.131707 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:17.131709    2569 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 20 17:56:17.176867 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:17.176830    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tgg22"
Jul 20 17:56:17.177011 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:56:17.176931    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tgg22" podUID="eca0024f-2963-4ed1-817e-f1a6febd79aa"
Jul 20 17:56:17.265560 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:17.265530    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wjs8g_07587a78-2d3a-40ad-9aae-a4875200263a/ovn-acl-logging/0.log"
Jul 20 17:56:17.265960 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:17.265936    2569 generic.go:356] "Generic (PLEG): container finished" podID="07587a78-2d3a-40ad-9aae-a4875200263a" containerID="20d167c54e3afef402dad67e5ab9e4fbc614198c1743dbc551fcd01fb55afa46" exitCode=1
Jul 20 17:56:17.266073 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:17.266014    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wjs8g" event={"ID":"07587a78-2d3a-40ad-9aae-a4875200263a","Type":"ContainerStarted","Data":"b23213facea587c639a6e60c43b89b318d43f894fa560dab58e04b240e8ddcc2"}
Jul 20 17:56:17.266133 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:17.266104    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wjs8g" event={"ID":"07587a78-2d3a-40ad-9aae-a4875200263a","Type":"ContainerStarted","Data":"bb89bac548ce0ff1f3ccf5389235c08a6b6d6c620c507cf5024cf71449b05810"}
Jul 20 17:56:17.266181 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:17.266153    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wjs8g" event={"ID":"07587a78-2d3a-40ad-9aae-a4875200263a","Type":"ContainerStarted","Data":"60917e05d6184a36211c135a300c59ff779419b600883fbf107cbd25c8ce5370"}
Jul 20 17:56:17.266238 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:17.266198    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wjs8g" event={"ID":"07587a78-2d3a-40ad-9aae-a4875200263a","Type":"ContainerStarted","Data":"d1f00cb3071ba0ecf2a72cacb07f701d91bd755150e26b879a0961769a783779"}
Jul 20 17:56:17.266290 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:17.266243    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wjs8g" event={"ID":"07587a78-2d3a-40ad-9aae-a4875200263a","Type":"ContainerDied","Data":"20d167c54e3afef402dad67e5ab9e4fbc614198c1743dbc551fcd01fb55afa46"}
Jul 20 17:56:17.266341 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:17.266291    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wjs8g" event={"ID":"07587a78-2d3a-40ad-9aae-a4875200263a","Type":"ContainerStarted","Data":"4d907941ccdfd134c02e61713711728fc1bb3db73fe8ca91bdd5f32d085ef4c9"}
Jul 20 17:56:17.268108 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:17.268073    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p74sv" event={"ID":"91b53241-763d-4868-81c3-786a3d6d4080","Type":"ContainerStarted","Data":"96013202c9e812a22987ca838cc81d1fe9d203f434e40720d3561bc84ff276b4"}
Jul 20 17:56:17.269807 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:17.269785    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-rqtm8" event={"ID":"d5f812b2-2ecb-4be4-99aa-7808ab7154e4","Type":"ContainerStarted","Data":"3f515fd085f15be60e0b1c7f797745f84fe29c6e36a7feaa6c3fb49dd7c00c55"}
Jul 20 17:56:17.271778 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:17.271757    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-247.ec2.internal" event={"ID":"7c055eb6eb0be9ef4618781875e36f04","Type":"ContainerStarted","Data":"fdae85c4535104de14efc8531fbec5018a703dd3e086373e8cfdc50d463af776"}
Jul 20 17:56:17.295302 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:17.295256    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-rqtm8" podStartSLOduration=3.456372329 podStartE2EDuration="23.295240635s" totalImagesPullingTime="19.838868306s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 17:55:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 17:56:17.295138165 +0000 UTC m=+23.541671587" watchObservedRunningTime="2026-07-20 17:56:17.295240635 +0000 UTC m=+23.541774045"
Jul 20 17:56:18.180830 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:18.180803    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8v5ff"
Jul 20 17:56:18.181501 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:56:18.180909    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8v5ff" podUID="de1b439b-702a-43a3-8118-f7e20e716ac5"
Jul 20 17:56:18.181501 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:18.180967    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nkmr4"
Jul 20 17:56:18.181501 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:56:18.181133    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nkmr4" podUID="26b0d75c-ecf7-4bb4-ad4b-d9077da808bd"
Jul 20 17:56:19.176628 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:19.176608    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tgg22"
Jul 20 17:56:19.176779 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:56:19.176714    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tgg22" podUID="eca0024f-2963-4ed1-817e-f1a6febd79aa"
Jul 20 17:56:19.279776 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:19.279752    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wjs8g_07587a78-2d3a-40ad-9aae-a4875200263a/ovn-acl-logging/0.log"
Jul 20 17:56:19.280260 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:19.280142    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wjs8g" event={"ID":"07587a78-2d3a-40ad-9aae-a4875200263a","Type":"ContainerStarted","Data":"84ec4f1bd08ca25e9939e61a9472e3d8fabe152daf5d0c0356c78836281a8b25"}
Jul 20 17:56:19.282186 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:19.282160    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p74sv" event={"ID":"91b53241-763d-4868-81c3-786a3d6d4080","Type":"ContainerStarted","Data":"90c80b5d1efa705bb1e995f5facb37225189b9dbfb62dd83630f3dc0530d3f67"}
Jul 20 17:56:19.303114 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:19.303065    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-247.ec2.internal" podStartSLOduration=24.303050318 podStartE2EDuration="24.303050318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 17:55:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 17:56:17.311768594 +0000 UTC m=+23.558302003" watchObservedRunningTime="2026-07-20 17:56:19.303050318 +0000 UTC m=+25.549583728"
Jul 20 17:56:20.177014 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:20.176588    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nkmr4"
Jul 20 17:56:20.177014 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:56:20.176757    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nkmr4" podUID="26b0d75c-ecf7-4bb4-ad4b-d9077da808bd"
Jul 20 17:56:20.177014 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:20.176770    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8v5ff"
Jul 20 17:56:20.177014 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:56:20.176866    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8v5ff" podUID="de1b439b-702a-43a3-8118-f7e20e716ac5"
Jul 20 17:56:21.176888 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:21.176841    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tgg22"
Jul 20 17:56:21.177257 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:56:21.176976    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tgg22" podUID="eca0024f-2963-4ed1-817e-f1a6febd79aa"
Jul 20 17:56:22.177324 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:22.177290    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8v5ff"
Jul 20 17:56:22.177324 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:22.177319    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nkmr4"
Jul 20 17:56:22.177939 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:56:22.177459    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8v5ff" podUID="de1b439b-702a-43a3-8118-f7e20e716ac5"
Jul 20 17:56:22.177939 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:56:22.177533    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nkmr4" podUID="26b0d75c-ecf7-4bb4-ad4b-d9077da808bd"
Jul 20 17:56:22.289902 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:22.289873    2569 generic.go:356] "Generic (PLEG): container finished" podID="1dfc0381-8b5e-4400-a1c8-da99129540c7" containerID="ede956ed06609c8ea58509213dd13451890b7eacd756d25e63d87ecb4b4abbf8" exitCode=0
Jul 20 17:56:22.290104 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:22.289948    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mzsnt" event={"ID":"1dfc0381-8b5e-4400-a1c8-da99129540c7","Type":"ContainerDied","Data":"ede956ed06609c8ea58509213dd13451890b7eacd756d25e63d87ecb4b4abbf8"}
Jul 20 17:56:22.293578 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:22.293562    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wjs8g_07587a78-2d3a-40ad-9aae-a4875200263a/ovn-acl-logging/0.log"
Jul 20 17:56:22.293917 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:22.293901    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wjs8g" event={"ID":"07587a78-2d3a-40ad-9aae-a4875200263a","Type":"ContainerStarted","Data":"509ea1797c9adfb32e3577bfbfe2a6f79c3bce46367c34bacba75ab54a6267d3"}
Jul 20 17:56:22.294121 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:22.294107    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wjs8g"
Jul 20 17:56:22.294164 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:22.294156    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wjs8g"
Jul 20 17:56:22.294321 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:22.294305    2569 scope.go:117] "RemoveContainer" containerID="20d167c54e3afef402dad67e5ab9e4fbc614198c1743dbc551fcd01fb55afa46"
Jul 20 17:56:22.308900 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:22.308885    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wjs8g"
Jul 20 17:56:22.311783 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:22.311748    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p74sv" podStartSLOduration=7.314326045 podStartE2EDuration="28.311734216s" totalImagesPullingTime="20.997408171s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 17:55:54 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 17:56:19.303024135 +0000 UTC m=+25.549557544" watchObservedRunningTime="2026-07-20 17:56:22.311734216 +0000 UTC m=+28.558267621"
Jul 20 17:56:23.177262 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:23.177225    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tgg22"
Jul 20 17:56:23.177432 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:56:23.177360    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tgg22" podUID="eca0024f-2963-4ed1-817e-f1a6febd79aa"
Jul 20 17:56:23.299492 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:23.299464    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wjs8g_07587a78-2d3a-40ad-9aae-a4875200263a/ovn-acl-logging/0.log"
Jul 20 17:56:23.299829 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:23.299805    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wjs8g" event={"ID":"07587a78-2d3a-40ad-9aae-a4875200263a","Type":"ContainerStarted","Data":"4824660d02c0e9697b38388c53e202c4ecbd833f250ccf1dfc1b210b615c95bd"}
Jul 20 17:56:23.300024 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:23.300002    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wjs8g"
Jul 20 17:56:23.315088 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:23.315067    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wjs8g"
Jul 20 17:56:23.326302 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:23.326261    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-wjs8g" podStartSLOduration=9.356654886 podStartE2EDuration="29.326247104s" totalImagesPullingTime="19.969592218s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 17:55:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 17:56:23.325905173 +0000 UTC m=+29.572438584" watchObservedRunningTime="2026-07-20 17:56:23.326247104 +0000 UTC m=+29.572780513"
Jul 20 17:56:23.328141 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:23.328116    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-nkmr4"]
Jul 20 17:56:23.328251 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:23.328211    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nkmr4"
Jul 20 17:56:23.328317 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:56:23.328295    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nkmr4" podUID="26b0d75c-ecf7-4bb4-ad4b-d9077da808bd"
Jul 20 17:56:23.331746 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:23.331711    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-tgg22"]
Jul 20 17:56:23.331928 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:23.331913    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tgg22"
Jul 20 17:56:23.332114 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:56:23.332086    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tgg22" podUID="eca0024f-2963-4ed1-817e-f1a6febd79aa"
Jul 20 17:56:23.332254 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:23.332231    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8v5ff"]
Jul 20 17:56:23.332354 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:23.332341    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8v5ff"
Jul 20 17:56:23.332459 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:56:23.332435    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8v5ff" podUID="de1b439b-702a-43a3-8118-f7e20e716ac5"
Jul 20 17:56:24.303736 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:24.303699    2569 generic.go:356] "Generic (PLEG): container finished" podID="1dfc0381-8b5e-4400-a1c8-da99129540c7" containerID="6127774f6f0716d0a49c7b0dcaf5eef161acbe351d44236dbf1ad817b5d423dc" exitCode=0
Jul 20 17:56:24.304164 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:24.303781    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mzsnt" event={"ID":"1dfc0381-8b5e-4400-a1c8-da99129540c7","Type":"ContainerDied","Data":"6127774f6f0716d0a49c7b0dcaf5eef161acbe351d44236dbf1ad817b5d423dc"}
Jul 20 17:56:25.177279 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:25.177248    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tgg22"
Jul 20 17:56:25.177435 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:25.177263    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8v5ff"
Jul 20 17:56:25.177435 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:56:25.177347    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tgg22" podUID="eca0024f-2963-4ed1-817e-f1a6febd79aa"
Jul 20 17:56:25.177435 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:25.177262    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nkmr4"
Jul 20 17:56:25.177550 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:56:25.177419    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8v5ff" podUID="de1b439b-702a-43a3-8118-f7e20e716ac5"
Jul 20 17:56:25.177550 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:56:25.177514    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nkmr4" podUID="26b0d75c-ecf7-4bb4-ad4b-d9077da808bd"
Jul 20 17:56:26.310912 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:26.310715    2569 generic.go:356] "Generic (PLEG): container finished" podID="1dfc0381-8b5e-4400-a1c8-da99129540c7" containerID="5ac28224dbe92d4068457a66af135ea523810440e56dbc6929a34ad2ef69cba0" exitCode=0
Jul 20 17:56:26.311255 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:26.310749    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mzsnt" event={"ID":"1dfc0381-8b5e-4400-a1c8-da99129540c7","Type":"ContainerDied","Data":"5ac28224dbe92d4068457a66af135ea523810440e56dbc6929a34ad2ef69cba0"}
Jul 20 17:56:27.177081 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:27.177055    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nkmr4"
Jul 20 17:56:27.177081 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:27.177066    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tgg22"
Jul 20 17:56:27.177298 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:56:27.177149    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nkmr4" podUID="26b0d75c-ecf7-4bb4-ad4b-d9077da808bd"
Jul 20 17:56:27.177298 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:27.177187    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8v5ff"
Jul 20 17:56:27.177298 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:56:27.177248    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8v5ff" podUID="de1b439b-702a-43a3-8118-f7e20e716ac5"
Jul 20 17:56:27.177403 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:56:27.177323    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tgg22" podUID="eca0024f-2963-4ed1-817e-f1a6febd79aa"
Jul 20 17:56:27.830571 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:27.830539    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26b0d75c-ecf7-4bb4-ad4b-d9077da808bd-metrics-certs\") pod \"network-metrics-daemon-nkmr4\" (UID: \"26b0d75c-ecf7-4bb4-ad4b-d9077da808bd\") " pod="openshift-multus/network-metrics-daemon-nkmr4"
Jul 20 17:56:27.830974 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:56:27.830675    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 20 17:56:27.830974 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:56:27.830751    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26b0d75c-ecf7-4bb4-ad4b-d9077da808bd-metrics-certs podName:26b0d75c-ecf7-4bb4-ad4b-d9077da808bd nodeName:}" failed. No retries permitted until 2026-07-20 17:56:59.830733649 +0000 UTC m=+66.077267052 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/26b0d75c-ecf7-4bb4-ad4b-d9077da808bd-metrics-certs") pod "network-metrics-daemon-nkmr4" (UID: "26b0d75c-ecf7-4bb4-ad4b-d9077da808bd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 20 17:56:27.931081 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:27.931042    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mgpl\" (UniqueName: \"kubernetes.io/projected/de1b439b-702a-43a3-8118-f7e20e716ac5-kube-api-access-4mgpl\") pod \"network-check-target-8v5ff\" (UID: \"de1b439b-702a-43a3-8118-f7e20e716ac5\") " pod="openshift-network-diagnostics/network-check-target-8v5ff"
Jul 20 17:56:27.931245 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:27.931099    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eca0024f-2963-4ed1-817e-f1a6febd79aa-original-pull-secret\") pod \"global-pull-secret-syncer-tgg22\" (UID: \"eca0024f-2963-4ed1-817e-f1a6febd79aa\") " pod="kube-system/global-pull-secret-syncer-tgg22"
Jul 20 17:56:27.931245 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:56:27.931195    2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 20 17:56:27.931245 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:56:27.931222    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 20 17:56:27.931245 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:56:27.931243    2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 20 17:56:27.931410 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:56:27.931253    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eca0024f-2963-4ed1-817e-f1a6febd79aa-original-pull-secret podName:eca0024f-2963-4ed1-817e-f1a6febd79aa nodeName:}" failed. No retries permitted until 2026-07-20 17:56:59.931234838 +0000 UTC m=+66.177768227 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/eca0024f-2963-4ed1-817e-f1a6febd79aa-original-pull-secret") pod "global-pull-secret-syncer-tgg22" (UID: "eca0024f-2963-4ed1-817e-f1a6febd79aa") : object "kube-system"/"original-pull-secret" not registered
Jul 20 17:56:27.931410 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:56:27.931261    2569 projected.go:196] Error preparing data for projected volume kube-api-access-4mgpl for pod openshift-network-diagnostics/network-check-target-8v5ff: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 20 17:56:27.931410 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:56:27.931310    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/de1b439b-702a-43a3-8118-f7e20e716ac5-kube-api-access-4mgpl podName:de1b439b-702a-43a3-8118-f7e20e716ac5 nodeName:}" failed. No retries permitted until 2026-07-20 17:56:59.931292841 +0000 UTC m=+66.177826234 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-4mgpl" (UniqueName: "kubernetes.io/projected/de1b439b-702a-43a3-8118-f7e20e716ac5-kube-api-access-4mgpl") pod "network-check-target-8v5ff" (UID: "de1b439b-702a-43a3-8118-f7e20e716ac5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 20 17:56:29.094300 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:29.094273    2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-247.ec2.internal" event="NodeReady"
Jul 20 17:56:29.094668 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:29.094401    2569 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 20 17:56:29.134127 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:29.134078    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-794579c464-cq4k2"]
Jul 20 17:56:29.137645 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:29.137622    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-794579c464-cq4k2"
Jul 20 17:56:29.140500 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:29.140357    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 20 17:56:29.140769 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:29.140483    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-dl9wc\""
Jul 20 17:56:29.140999 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:29.140982    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 20 17:56:29.141271 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:29.141254    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 20 17:56:29.146889 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:29.146870    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 20 17:56:29.150834 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:29.150641    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-67b26"]
Jul 20 17:56:29.154604 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:29.154573    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-794579c464-cq4k2"]
Jul 20 17:56:29.154701 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:29.154665    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-67b26"
Jul 20 17:56:29.156864 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:29.156846    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 20 17:56:29.157016 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:29.156996    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 20 17:56:29.157068 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:29.157055    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 20 17:56:29.157123 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:29.157085    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-9mph2\""
Jul 20 17:56:29.164468 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:29.164440    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-67b26"]
Jul 20 17:56:29.177042 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:29.177017    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tgg22"
Jul 20 17:56:29.177162 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:29.177028    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nkmr4"
Jul 20 17:56:29.177235 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:29.177031    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8v5ff"
Jul 20 17:56:29.179428 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:29.179271    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 20 17:56:29.179428 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:29.179303    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-nnhlg\""
Jul 20 17:56:29.179428 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:29.179329    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 20 17:56:29.179428 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:29.179353    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 20 17:56:29.179428 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:29.179359    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-rwl6b\""
Jul 20 17:56:29.179678 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:29.179493    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 20 17:56:29.241591 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:29.241542    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8357c262-2ca2-4ff5-8b48-70017efdb56b-registry-tls\") pod \"image-registry-794579c464-cq4k2\" (UID: \"8357c262-2ca2-4ff5-8b48-70017efdb56b\") " pod="openshift-image-registry/image-registry-794579c464-cq4k2"
Jul 20 17:56:29.241773 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:29.241606    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8357c262-2ca2-4ff5-8b48-70017efdb56b-bound-sa-token\") pod \"image-registry-794579c464-cq4k2\" (UID: \"8357c262-2ca2-4ff5-8b48-70017efdb56b\") " pod="openshift-image-registry/image-registry-794579c464-cq4k2"
Jul 20 17:56:29.241773 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:29.241629    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8357c262-2ca2-4ff5-8b48-70017efdb56b-installation-pull-secrets\") pod \"image-registry-794579c464-cq4k2\" (UID: \"8357c262-2ca2-4ff5-8b48-70017efdb56b\") " pod="openshift-image-registry/image-registry-794579c464-cq4k2"
Jul 20 17:56:29.241773 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:29.241664    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8357c262-2ca2-4ff5-8b48-70017efdb56b-image-registry-private-configuration\") pod \"image-registry-794579c464-cq4k2\" (UID: \"8357c262-2ca2-4ff5-8b48-70017efdb56b\") " pod="openshift-image-registry/image-registry-794579c464-cq4k2"
Jul 20 17:56:29.241773 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:29.241682    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8357c262-2ca2-4ff5-8b48-70017efdb56b-ca-trust-extracted\") pod \"image-registry-794579c464-cq4k2\" (UID: \"8357c262-2ca2-4ff5-8b48-70017efdb56b\") " pod="openshift-image-registry/image-registry-794579c464-cq4k2"
Jul 20 17:56:29.241940 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:29.241810    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8357c262-2ca2-4ff5-8b48-70017efdb56b-registry-certificates\") pod \"image-registry-794579c464-cq4k2\" (UID: \"8357c262-2ca2-4ff5-8b48-70017efdb56b\") " pod="openshift-image-registry/image-registry-794579c464-cq4k2"
Jul 20 17:56:29.241940 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:29.241848    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8357c262-2ca2-4ff5-8b48-70017efdb56b-trusted-ca\") pod \"image-registry-794579c464-cq4k2\" (UID: \"8357c262-2ca2-4ff5-8b48-70017efdb56b\") " pod="openshift-image-registry/image-registry-794579c464-cq4k2"
Jul 20 17:56:29.241940 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:29.241874    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxrdz\" (UniqueName: \"kubernetes.io/projected/69347ff0-9f35-4334-be7f-b02bca61a64c-kube-api-access-vxrdz\") pod \"ingress-canary-67b26\" (UID: \"69347ff0-9f35-4334-be7f-b02bca61a64c\") " pod="openshift-ingress-canary/ingress-canary-67b26"
Jul 20 17:56:29.242048 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:29.241980    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjzfw\" (UniqueName: \"kubernetes.io/projected/8357c262-2ca2-4ff5-8b48-70017efdb56b-kube-api-access-sjzfw\") pod \"image-registry-794579c464-cq4k2\" (UID: \"8357c262-2ca2-4ff5-8b48-70017efdb56b\") " pod="openshift-image-registry/image-registry-794579c464-cq4k2"
Jul 20 17:56:29.242048 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:29.242020    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/69347ff0-9f35-4334-be7f-b02bca61a64c-cert\") pod \"ingress-canary-67b26\" (UID: \"69347ff0-9f35-4334-be7f-b02bca61a64c\") " pod="openshift-ingress-canary/ingress-canary-67b26"
Jul 20 17:56:29.251117 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:29.251090    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-ktc25"]
Jul 20 17:56:29.255103 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:29.255080    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ktc25"
Jul 20 17:56:29.257845 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:29.257818    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 20 17:56:29.257995 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:29.257979    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-m6nkv\""
Jul 20 17:56:29.258957 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:29.258938    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 20 17:56:29.265453 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:29.265435    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-ktc25"]
Jul 20 17:56:29.342653 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:29.342624    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jrpq\" (UniqueName: \"kubernetes.io/projected/bfe61b3e-4aac-4ddd-bde9-10fb1c7e6e07-kube-api-access-9jrpq\") pod \"dns-default-ktc25\" (UID: \"bfe61b3e-4aac-4ddd-bde9-10fb1c7e6e07\") " pod="openshift-dns/dns-default-ktc25"
Jul 20 17:56:29.342829 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:29.342672    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8357c262-2ca2-4ff5-8b48-70017efdb56b-registry-certificates\") pod \"image-registry-794579c464-cq4k2\" (UID: \"8357c262-2ca2-4ff5-8b48-70017efdb56b\") " pod="openshift-image-registry/image-registry-794579c464-cq4k2"
Jul 20 17:56:29.342829 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:29.342694    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8357c262-2ca2-4ff5-8b48-70017efdb56b-trusted-ca\") pod \"image-registry-794579c464-cq4k2\" (UID: \"8357c262-2ca2-4ff5-8b48-70017efdb56b\") " pod="openshift-image-registry/image-registry-794579c464-cq4k2"
Jul 20 17:56:29.342829 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:29.342711    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxrdz\" (UniqueName: \"kubernetes.io/projected/69347ff0-9f35-4334-be7f-b02bca61a64c-kube-api-access-vxrdz\") pod \"ingress-canary-67b26\" (UID: \"69347ff0-9f35-4334-be7f-b02bca61a64c\") " pod="openshift-ingress-canary/ingress-canary-67b26"
Jul 20 17:56:29.342829 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:29.342742    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bfe61b3e-4aac-4ddd-bde9-10fb1c7e6e07-config-volume\") pod \"dns-default-ktc25\" (UID: \"bfe61b3e-4aac-4ddd-bde9-10fb1c7e6e07\") " pod="openshift-dns/dns-default-ktc25"
Jul 20 17:56:29.343045 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:29.342917    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjzfw\" (UniqueName: \"kubernetes.io/projected/8357c262-2ca2-4ff5-8b48-70017efdb56b-kube-api-access-sjzfw\") pod \"image-registry-794579c464-cq4k2\" (UID: \"8357c262-2ca2-4ff5-8b48-70017efdb56b\") " pod="openshift-image-registry/image-registry-794579c464-cq4k2"
Jul 20 17:56:29.343045 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:29.342948    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/69347ff0-9f35-4334-be7f-b02bca61a64c-cert\") pod \"ingress-canary-67b26\" (UID: \"69347ff0-9f35-4334-be7f-b02bca61a64c\") " pod="openshift-ingress-canary/ingress-canary-67b26"
Jul 20 17:56:29.343045 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:29.342978    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8357c262-2ca2-4ff5-8b48-70017efdb56b-registry-tls\") pod \"image-registry-794579c464-cq4k2\" (UID: \"8357c262-2ca2-4ff5-8b48-70017efdb56b\") " pod="openshift-image-registry/image-registry-794579c464-cq4k2"
Jul 20 17:56:29.343045 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:29.343009    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8357c262-2ca2-4ff5-8b48-70017efdb56b-bound-sa-token\") pod \"image-registry-794579c464-cq4k2\" (UID: \"8357c262-2ca2-4ff5-8b48-70017efdb56b\") " pod="openshift-image-registry/image-registry-794579c464-cq4k2"
Jul 20 17:56:29.343230 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:56:29.343092    2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 20 17:56:29.343230 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:56:29.343110    2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-794579c464-cq4k2: secret "image-registry-tls" not found
Jul 20 17:56:29.343230 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:56:29.343162    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8357c262-2ca2-4ff5-8b48-70017efdb56b-registry-tls podName:8357c262-2ca2-4ff5-8b48-70017efdb56b nodeName:}" failed. No retries permitted until 2026-07-20 17:56:29.843145169 +0000 UTC m=+36.089678573 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8357c262-2ca2-4ff5-8b48-70017efdb56b-registry-tls") pod "image-registry-794579c464-cq4k2" (UID: "8357c262-2ca2-4ff5-8b48-70017efdb56b") : secret "image-registry-tls" not found
Jul 20 17:56:29.343230 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:29.343183    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8357c262-2ca2-4ff5-8b48-70017efdb56b-installation-pull-secrets\") pod \"image-registry-794579c464-cq4k2\" (UID: \"8357c262-2ca2-4ff5-8b48-70017efdb56b\") " pod="openshift-image-registry/image-registry-794579c464-cq4k2"
Jul 20 17:56:29.343230 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:29.343217    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bfe61b3e-4aac-4ddd-bde9-10fb1c7e6e07-metrics-tls\") pod \"dns-default-ktc25\" (UID: \"bfe61b3e-4aac-4ddd-bde9-10fb1c7e6e07\") " pod="openshift-dns/dns-default-ktc25"
Jul 20 17:56:29.343452 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:29.343255    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bfe61b3e-4aac-4ddd-bde9-10fb1c7e6e07-tmp-dir\") pod \"dns-default-ktc25\" (UID: \"bfe61b3e-4aac-4ddd-bde9-10fb1c7e6e07\") " pod="openshift-dns/dns-default-ktc25"
Jul 20 17:56:29.343452 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:29.343296    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8357c262-2ca2-4ff5-8b48-70017efdb56b-image-registry-private-configuration\") pod \"image-registry-794579c464-cq4k2\" (UID: \"8357c262-2ca2-4ff5-8b48-70017efdb56b\") " pod="openshift-image-registry/image-registry-794579c464-cq4k2"
Jul 20 17:56:29.343452 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:29.343310    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8357c262-2ca2-4ff5-8b48-70017efdb56b-registry-certificates\") pod \"image-registry-794579c464-cq4k2\" (UID: \"8357c262-2ca2-4ff5-8b48-70017efdb56b\") " pod="openshift-image-registry/image-registry-794579c464-cq4k2"
Jul 20 17:56:29.343452 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:29.343321    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8357c262-2ca2-4ff5-8b48-70017efdb56b-ca-trust-extracted\") pod \"image-registry-794579c464-cq4k2\" (UID: \"8357c262-2ca2-4ff5-8b48-70017efdb56b\") " pod="openshift-image-registry/image-registry-794579c464-cq4k2"
Jul 20 17:56:29.343690 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:29.343669    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8357c262-2ca2-4ff5-8b48-70017efdb56b-ca-trust-extracted\") pod \"image-registry-794579c464-cq4k2\" (UID: \"8357c262-2ca2-4ff5-8b48-70017efdb56b\") " pod="openshift-image-registry/image-registry-794579c464-cq4k2"
Jul 20 17:56:29.343768 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:29.343705    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8357c262-2ca2-4ff5-8b48-70017efdb56b-trusted-ca\") pod \"image-registry-794579c464-cq4k2\" (UID: \"8357c262-2ca2-4ff5-8b48-70017efdb56b\") " pod="openshift-image-registry/image-registry-794579c464-cq4k2"
Jul 20 17:56:29.346956 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:29.346898    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8357c262-2ca2-4ff5-8b48-70017efdb56b-installation-pull-secrets\") pod \"image-registry-794579c464-cq4k2\" (UID: \"8357c262-2ca2-4ff5-8b48-70017efdb56b\") " pod="openshift-image-registry/image-registry-794579c464-cq4k2"
Jul 20 17:56:29.346956 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:29.346908    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8357c262-2ca2-4ff5-8b48-70017efdb56b-image-registry-private-configuration\") pod \"image-registry-794579c464-cq4k2\" (UID: \"8357c262-2ca2-4ff5-8b48-70017efdb56b\") " pod="openshift-image-registry/image-registry-794579c464-cq4k2"
Jul 20 17:56:29.346956 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:29.346906    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/69347ff0-9f35-4334-be7f-b02bca61a64c-cert\") pod \"ingress-canary-67b26\" (UID: \"69347ff0-9f35-4334-be7f-b02bca61a64c\") " pod="openshift-ingress-canary/ingress-canary-67b26"
Jul 20 17:56:29.352191 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:29.352157    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjzfw\" (UniqueName: \"kubernetes.io/projected/8357c262-2ca2-4ff5-8b48-70017efdb56b-kube-api-access-sjzfw\") pod \"image-registry-794579c464-cq4k2\" (UID: \"8357c262-2ca2-4ff5-8b48-70017efdb56b\") " pod="openshift-image-registry/image-registry-794579c464-cq4k2"
Jul 20 17:56:29.352318 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:29.352294    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxrdz\" (UniqueName: \"kubernetes.io/projected/69347ff0-9f35-4334-be7f-b02bca61a64c-kube-api-access-vxrdz\") pod \"ingress-canary-67b26\" (UID: \"69347ff0-9f35-4334-be7f-b02bca61a64c\") " pod="openshift-ingress-canary/ingress-canary-67b26"
Jul 20 17:56:29.353453 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:29.353437    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8357c262-2ca2-4ff5-8b48-70017efdb56b-bound-sa-token\") pod \"image-registry-794579c464-cq4k2\" (UID: \"8357c262-2ca2-4ff5-8b48-70017efdb56b\") " pod="openshift-image-registry/image-registry-794579c464-cq4k2"
Jul 20 17:56:29.444755 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:29.444709    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bfe61b3e-4aac-4ddd-bde9-10fb1c7e6e07-metrics-tls\") pod \"dns-default-ktc25\" (UID: \"bfe61b3e-4aac-4ddd-bde9-10fb1c7e6e07\") " pod="openshift-dns/dns-default-ktc25"
Jul 20 17:56:29.444755 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:29.444758    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bfe61b3e-4aac-4ddd-bde9-10fb1c7e6e07-tmp-dir\") pod \"dns-default-ktc25\" (UID: \"bfe61b3e-4aac-4ddd-bde9-10fb1c7e6e07\") " pod="openshift-dns/dns-default-ktc25"
Jul 20 17:56:29.444963 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:29.444789    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jrpq\" (UniqueName: \"kubernetes.io/projected/bfe61b3e-4aac-4ddd-bde9-10fb1c7e6e07-kube-api-access-9jrpq\") pod \"dns-default-ktc25\" (UID: \"bfe61b3e-4aac-4ddd-bde9-10fb1c7e6e07\") " pod="openshift-dns/dns-default-ktc25"
Jul 20 17:56:29.444963 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:29.444842    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bfe61b3e-4aac-4ddd-bde9-10fb1c7e6e07-config-volume\") pod \"dns-default-ktc25\" (UID: \"bfe61b3e-4aac-4ddd-bde9-10fb1c7e6e07\") " pod="openshift-dns/dns-default-ktc25"
Jul 20 17:56:29.444963 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:56:29.444875    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 20 17:56:29.444963 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:56:29.444953    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bfe61b3e-4aac-4ddd-bde9-10fb1c7e6e07-metrics-tls podName:bfe61b3e-4aac-4ddd-bde9-10fb1c7e6e07 nodeName:}" failed. No retries permitted until 2026-07-20 17:56:29.944936045 +0000 UTC m=+36.191469447 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bfe61b3e-4aac-4ddd-bde9-10fb1c7e6e07-metrics-tls") pod "dns-default-ktc25" (UID: "bfe61b3e-4aac-4ddd-bde9-10fb1c7e6e07") : secret "dns-default-metrics-tls" not found
Jul 20 17:56:29.445155 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:29.445136    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bfe61b3e-4aac-4ddd-bde9-10fb1c7e6e07-tmp-dir\") pod \"dns-default-ktc25\" (UID: \"bfe61b3e-4aac-4ddd-bde9-10fb1c7e6e07\") " pod="openshift-dns/dns-default-ktc25"
Jul 20 17:56:29.445411 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:29.445393    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bfe61b3e-4aac-4ddd-bde9-10fb1c7e6e07-config-volume\") pod \"dns-default-ktc25\" (UID: \"bfe61b3e-4aac-4ddd-bde9-10fb1c7e6e07\") " pod="openshift-dns/dns-default-ktc25"
Jul 20 17:56:29.452816 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:29.452788    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jrpq\" (UniqueName: \"kubernetes.io/projected/bfe61b3e-4aac-4ddd-bde9-10fb1c7e6e07-kube-api-access-9jrpq\") pod \"dns-default-ktc25\" (UID: \"bfe61b3e-4aac-4ddd-bde9-10fb1c7e6e07\") " pod="openshift-dns/dns-default-ktc25"
Jul 20 17:56:29.464715 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:29.464698    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-67b26"
Jul 20 17:56:29.598350 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:29.598270    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-67b26"]
Jul 20 17:56:29.602614 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:56:29.602586    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69347ff0_9f35_4334_be7f_b02bca61a64c.slice/crio-88a2217dd3dc640011c830dc9aa0f3dd5f71906ba3e8fd0da5a945e6a7770bd3 WatchSource:0}: Error finding container 88a2217dd3dc640011c830dc9aa0f3dd5f71906ba3e8fd0da5a945e6a7770bd3: Status 404 returned error can't find the container with id 88a2217dd3dc640011c830dc9aa0f3dd5f71906ba3e8fd0da5a945e6a7770bd3
Jul 20 17:56:29.849014 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:29.848938    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8357c262-2ca2-4ff5-8b48-70017efdb56b-registry-tls\") pod \"image-registry-794579c464-cq4k2\" (UID: \"8357c262-2ca2-4ff5-8b48-70017efdb56b\") " pod="openshift-image-registry/image-registry-794579c464-cq4k2"
Jul 20 17:56:29.849151 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:56:29.849090    2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 20 17:56:29.849151 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:56:29.849109    2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-794579c464-cq4k2: secret "image-registry-tls" not found
Jul 20 17:56:29.849223 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:56:29.849161    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8357c262-2ca2-4ff5-8b48-70017efdb56b-registry-tls podName:8357c262-2ca2-4ff5-8b48-70017efdb56b nodeName:}" failed. No retries permitted until 2026-07-20 17:56:30.849145869 +0000 UTC m=+37.095679256 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8357c262-2ca2-4ff5-8b48-70017efdb56b-registry-tls") pod "image-registry-794579c464-cq4k2" (UID: "8357c262-2ca2-4ff5-8b48-70017efdb56b") : secret "image-registry-tls" not found
Jul 20 17:56:29.949864 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:29.949831    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bfe61b3e-4aac-4ddd-bde9-10fb1c7e6e07-metrics-tls\") pod \"dns-default-ktc25\" (UID: \"bfe61b3e-4aac-4ddd-bde9-10fb1c7e6e07\") " pod="openshift-dns/dns-default-ktc25"
Jul 20 17:56:29.950031 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:56:29.950012    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 20 17:56:29.950100 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:56:29.950088    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bfe61b3e-4aac-4ddd-bde9-10fb1c7e6e07-metrics-tls podName:bfe61b3e-4aac-4ddd-bde9-10fb1c7e6e07 nodeName:}" failed. No retries permitted until 2026-07-20 17:56:30.950065996 +0000 UTC m=+37.196599387 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bfe61b3e-4aac-4ddd-bde9-10fb1c7e6e07-metrics-tls") pod "dns-default-ktc25" (UID: "bfe61b3e-4aac-4ddd-bde9-10fb1c7e6e07") : secret "dns-default-metrics-tls" not found
Jul 20 17:56:30.320797 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:30.320758    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-67b26" event={"ID":"69347ff0-9f35-4334-be7f-b02bca61a64c","Type":"ContainerStarted","Data":"88a2217dd3dc640011c830dc9aa0f3dd5f71906ba3e8fd0da5a945e6a7770bd3"}
Jul 20 17:56:30.857158 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:30.857118    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8357c262-2ca2-4ff5-8b48-70017efdb56b-registry-tls\") pod \"image-registry-794579c464-cq4k2\" (UID: \"8357c262-2ca2-4ff5-8b48-70017efdb56b\") " pod="openshift-image-registry/image-registry-794579c464-cq4k2"
Jul 20 17:56:30.857323 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:56:30.857303    2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 20 17:56:30.857381 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:56:30.857326    2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-794579c464-cq4k2: secret "image-registry-tls" not found
Jul 20 17:56:30.857441 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:56:30.857404    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8357c262-2ca2-4ff5-8b48-70017efdb56b-registry-tls podName:8357c262-2ca2-4ff5-8b48-70017efdb56b nodeName:}" failed. No retries permitted until 2026-07-20 17:56:32.857380956 +0000 UTC m=+39.103914347 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8357c262-2ca2-4ff5-8b48-70017efdb56b-registry-tls") pod "image-registry-794579c464-cq4k2" (UID: "8357c262-2ca2-4ff5-8b48-70017efdb56b") : secret "image-registry-tls" not found
Jul 20 17:56:30.958142 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:30.958054    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bfe61b3e-4aac-4ddd-bde9-10fb1c7e6e07-metrics-tls\") pod \"dns-default-ktc25\" (UID: \"bfe61b3e-4aac-4ddd-bde9-10fb1c7e6e07\") " pod="openshift-dns/dns-default-ktc25"
Jul 20 17:56:30.958320 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:56:30.958245    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 20 17:56:30.958380 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:56:30.958325    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bfe61b3e-4aac-4ddd-bde9-10fb1c7e6e07-metrics-tls podName:bfe61b3e-4aac-4ddd-bde9-10fb1c7e6e07 nodeName:}" failed. No retries permitted until 2026-07-20 17:56:32.958305744 +0000 UTC m=+39.204839151 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bfe61b3e-4aac-4ddd-bde9-10fb1c7e6e07-metrics-tls") pod "dns-default-ktc25" (UID: "bfe61b3e-4aac-4ddd-bde9-10fb1c7e6e07") : secret "dns-default-metrics-tls" not found
Jul 20 17:56:32.875384 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:32.875340    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8357c262-2ca2-4ff5-8b48-70017efdb56b-registry-tls\") pod \"image-registry-794579c464-cq4k2\" (UID: \"8357c262-2ca2-4ff5-8b48-70017efdb56b\") " pod="openshift-image-registry/image-registry-794579c464-cq4k2"
Jul 20 17:56:32.875885 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:56:32.875514    2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 20 17:56:32.875885 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:56:32.875537    2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-794579c464-cq4k2: secret "image-registry-tls" not found
Jul 20 17:56:32.875885 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:56:32.875608    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8357c262-2ca2-4ff5-8b48-70017efdb56b-registry-tls podName:8357c262-2ca2-4ff5-8b48-70017efdb56b nodeName:}" failed. No retries permitted until 2026-07-20 17:56:36.875587391 +0000 UTC m=+43.122120791 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8357c262-2ca2-4ff5-8b48-70017efdb56b-registry-tls") pod "image-registry-794579c464-cq4k2" (UID: "8357c262-2ca2-4ff5-8b48-70017efdb56b") : secret "image-registry-tls" not found
Jul 20 17:56:32.975677 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:32.975645    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bfe61b3e-4aac-4ddd-bde9-10fb1c7e6e07-metrics-tls\") pod \"dns-default-ktc25\" (UID: \"bfe61b3e-4aac-4ddd-bde9-10fb1c7e6e07\") " pod="openshift-dns/dns-default-ktc25"
Jul 20 17:56:32.975833 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:56:32.975805    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 20 17:56:32.975881 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:56:32.975868    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bfe61b3e-4aac-4ddd-bde9-10fb1c7e6e07-metrics-tls podName:bfe61b3e-4aac-4ddd-bde9-10fb1c7e6e07 nodeName:}" failed. No retries permitted until 2026-07-20 17:56:36.97585126 +0000 UTC m=+43.222384649 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bfe61b3e-4aac-4ddd-bde9-10fb1c7e6e07-metrics-tls") pod "dns-default-ktc25" (UID: "bfe61b3e-4aac-4ddd-bde9-10fb1c7e6e07") : secret "dns-default-metrics-tls" not found
Jul 20 17:56:34.333813 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:34.333773    2569 generic.go:356] "Generic (PLEG): container finished" podID="1dfc0381-8b5e-4400-a1c8-da99129540c7" containerID="5529fbeb2d783a327e7ba02db36e98d61fc7d8879c5f0357556de2febde44bc1" exitCode=0
Jul 20 17:56:34.334295 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:34.333833    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mzsnt" event={"ID":"1dfc0381-8b5e-4400-a1c8-da99129540c7","Type":"ContainerDied","Data":"5529fbeb2d783a327e7ba02db36e98d61fc7d8879c5f0357556de2febde44bc1"}
Jul 20 17:56:34.335289 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:34.335263    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-67b26" event={"ID":"69347ff0-9f35-4334-be7f-b02bca61a64c","Type":"ContainerStarted","Data":"083b3463461468c4e13d629409ecdd177a0222d990a553c2a33f7b7b2ef2f377"}
Jul 20 17:56:34.389296 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:34.389239    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-67b26" podStartSLOduration=1.6182851710000001 podStartE2EDuration="5.389217244s" totalImagesPullingTime="3.770932073s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 17:56:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 17:56:34.388634997 +0000 UTC m=+40.635168408" watchObservedRunningTime="2026-07-20 17:56:34.389217244 +0000 UTC m=+40.635750653"
Jul 20 17:56:35.340327 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:35.340289    2569 generic.go:356] "Generic (PLEG): container finished" podID="1dfc0381-8b5e-4400-a1c8-da99129540c7" containerID="0dd81556e03f0be0f9e96e02dfe9b36f99572746ba313e4af25e2d68fdfbe739" exitCode=0
Jul 20 17:56:35.340779 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:35.340356    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mzsnt" event={"ID":"1dfc0381-8b5e-4400-a1c8-da99129540c7","Type":"ContainerDied","Data":"0dd81556e03f0be0f9e96e02dfe9b36f99572746ba313e4af25e2d68fdfbe739"}
Jul 20 17:56:36.345903 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:36.345708    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mzsnt" event={"ID":"1dfc0381-8b5e-4400-a1c8-da99129540c7","Type":"ContainerStarted","Data":"33703a64b8ca39a363336c73c56d30af0b27554887df570ca4c63929757bc602"}
Jul 20 17:56:36.370429 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:36.368298    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-mzsnt" podStartSLOduration=12.513479859 podStartE2EDuration="42.368278201s" totalImagesPullingTime="29.854798342s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 17:55:54 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 17:56:36.367376969 +0000 UTC m=+42.613910381" watchObservedRunningTime="2026-07-20 17:56:36.368278201 +0000 UTC m=+42.614811615"
Jul 20 17:56:36.903033 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:36.903003    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8357c262-2ca2-4ff5-8b48-70017efdb56b-registry-tls\") pod \"image-registry-794579c464-cq4k2\" (UID: \"8357c262-2ca2-4ff5-8b48-70017efdb56b\") " pod="openshift-image-registry/image-registry-794579c464-cq4k2"
Jul 20 17:56:36.903163 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:56:36.903135    2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 20 17:56:36.903163 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:56:36.903150    2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-794579c464-cq4k2: secret "image-registry-tls" not found
Jul 20 17:56:36.903244 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:56:36.903222    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8357c262-2ca2-4ff5-8b48-70017efdb56b-registry-tls podName:8357c262-2ca2-4ff5-8b48-70017efdb56b nodeName:}" failed. No retries permitted until 2026-07-20 17:56:44.903207404 +0000 UTC m=+51.149740795 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8357c262-2ca2-4ff5-8b48-70017efdb56b-registry-tls") pod "image-registry-794579c464-cq4k2" (UID: "8357c262-2ca2-4ff5-8b48-70017efdb56b") : secret "image-registry-tls" not found
Jul 20 17:56:37.004201 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:37.004162    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bfe61b3e-4aac-4ddd-bde9-10fb1c7e6e07-metrics-tls\") pod \"dns-default-ktc25\" (UID: \"bfe61b3e-4aac-4ddd-bde9-10fb1c7e6e07\") " pod="openshift-dns/dns-default-ktc25"
Jul 20 17:56:37.004311 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:56:37.004293    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 20 17:56:37.004358 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:56:37.004348    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bfe61b3e-4aac-4ddd-bde9-10fb1c7e6e07-metrics-tls podName:bfe61b3e-4aac-4ddd-bde9-10fb1c7e6e07 nodeName:}" failed. No retries permitted until 2026-07-20 17:56:45.004330446 +0000 UTC m=+51.250863853 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bfe61b3e-4aac-4ddd-bde9-10fb1c7e6e07-metrics-tls") pod "dns-default-ktc25" (UID: "bfe61b3e-4aac-4ddd-bde9-10fb1c7e6e07") : secret "dns-default-metrics-tls" not found
Jul 20 17:56:44.958486 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:44.958447    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8357c262-2ca2-4ff5-8b48-70017efdb56b-registry-tls\") pod \"image-registry-794579c464-cq4k2\" (UID: \"8357c262-2ca2-4ff5-8b48-70017efdb56b\") " pod="openshift-image-registry/image-registry-794579c464-cq4k2"
Jul 20 17:56:44.959269 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:56:44.958595    2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 20 17:56:44.959269 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:56:44.958614    2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-794579c464-cq4k2: secret "image-registry-tls" not found
Jul 20 17:56:44.959269 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:56:44.958671    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8357c262-2ca2-4ff5-8b48-70017efdb56b-registry-tls podName:8357c262-2ca2-4ff5-8b48-70017efdb56b nodeName:}" failed. No retries permitted until 2026-07-20 17:57:00.958655898 +0000 UTC m=+67.205189286 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8357c262-2ca2-4ff5-8b48-70017efdb56b-registry-tls") pod "image-registry-794579c464-cq4k2" (UID: "8357c262-2ca2-4ff5-8b48-70017efdb56b") : secret "image-registry-tls" not found
Jul 20 17:56:45.059844 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:45.059809    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bfe61b3e-4aac-4ddd-bde9-10fb1c7e6e07-metrics-tls\") pod \"dns-default-ktc25\" (UID: \"bfe61b3e-4aac-4ddd-bde9-10fb1c7e6e07\") " pod="openshift-dns/dns-default-ktc25"
Jul 20 17:56:45.059974 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:56:45.059930    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 20 17:56:45.060021 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:56:45.059988    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bfe61b3e-4aac-4ddd-bde9-10fb1c7e6e07-metrics-tls podName:bfe61b3e-4aac-4ddd-bde9-10fb1c7e6e07 nodeName:}" failed. No retries permitted until 2026-07-20 17:57:01.059975075 +0000 UTC m=+67.306508464 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bfe61b3e-4aac-4ddd-bde9-10fb1c7e6e07-metrics-tls") pod "dns-default-ktc25" (UID: "bfe61b3e-4aac-4ddd-bde9-10fb1c7e6e07") : secret "dns-default-metrics-tls" not found
Jul 20 17:56:55.317119 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:55.317092    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wjs8g"
Jul 20 17:56:59.861003 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:59.860965    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26b0d75c-ecf7-4bb4-ad4b-d9077da808bd-metrics-certs\") pod \"network-metrics-daemon-nkmr4\" (UID: \"26b0d75c-ecf7-4bb4-ad4b-d9077da808bd\") " pod="openshift-multus/network-metrics-daemon-nkmr4"
Jul 20 17:56:59.863087 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:59.863067    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 20 17:56:59.871852 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:56:59.871833    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 20 17:56:59.871934 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:56:59.871886    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26b0d75c-ecf7-4bb4-ad4b-d9077da808bd-metrics-certs podName:26b0d75c-ecf7-4bb4-ad4b-d9077da808bd nodeName:}" failed. No retries permitted until 2026-07-20 17:58:03.871871478 +0000 UTC m=+130.118404867 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/26b0d75c-ecf7-4bb4-ad4b-d9077da808bd-metrics-certs") pod "network-metrics-daemon-nkmr4" (UID: "26b0d75c-ecf7-4bb4-ad4b-d9077da808bd") : secret "metrics-daemon-secret" not found
Jul 20 17:56:59.961903 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:59.961873    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mgpl\" (UniqueName: \"kubernetes.io/projected/de1b439b-702a-43a3-8118-f7e20e716ac5-kube-api-access-4mgpl\") pod \"network-check-target-8v5ff\" (UID: \"de1b439b-702a-43a3-8118-f7e20e716ac5\") " pod="openshift-network-diagnostics/network-check-target-8v5ff"
Jul 20 17:56:59.962006 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:59.961940    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eca0024f-2963-4ed1-817e-f1a6febd79aa-original-pull-secret\") pod \"global-pull-secret-syncer-tgg22\" (UID: \"eca0024f-2963-4ed1-817e-f1a6febd79aa\") " pod="kube-system/global-pull-secret-syncer-tgg22"
Jul 20 17:56:59.964464 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:59.964443    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 20 17:56:59.964561 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:59.964465    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 20 17:56:59.974309 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:59.974286    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 20 17:56:59.976037 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:59.976020    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eca0024f-2963-4ed1-817e-f1a6febd79aa-original-pull-secret\") pod \"global-pull-secret-syncer-tgg22\" (UID: \"eca0024f-2963-4ed1-817e-f1a6febd79aa\") " pod="kube-system/global-pull-secret-syncer-tgg22"
Jul 20 17:56:59.985130 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:56:59.985102    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mgpl\" (UniqueName: \"kubernetes.io/projected/de1b439b-702a-43a3-8118-f7e20e716ac5-kube-api-access-4mgpl\") pod \"network-check-target-8v5ff\" (UID: \"de1b439b-702a-43a3-8118-f7e20e716ac5\") " pod="openshift-network-diagnostics/network-check-target-8v5ff"
Jul 20 17:57:00.088125 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:57:00.088087    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tgg22"
Jul 20 17:57:00.100901 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:57:00.100879    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-rwl6b\""
Jul 20 17:57:00.108844 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:57:00.108813    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8v5ff"
Jul 20 17:57:00.238562 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:57:00.238522    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-tgg22"]
Jul 20 17:57:00.242402 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:57:00.242371    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeca0024f_2963_4ed1_817e_f1a6febd79aa.slice/crio-70ecf96ae96b3de12240578c90201f51e7855a2824c8b7b796888aae09bf6f85 WatchSource:0}: Error finding container 70ecf96ae96b3de12240578c90201f51e7855a2824c8b7b796888aae09bf6f85: Status 404 returned error can't find the container with id 70ecf96ae96b3de12240578c90201f51e7855a2824c8b7b796888aae09bf6f85
Jul 20 17:57:00.256873 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:57:00.256852    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8v5ff"]
Jul 20 17:57:00.259256 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:57:00.259235    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde1b439b_702a_43a3_8118_f7e20e716ac5.slice/crio-cbe74669c65ddbeb7a523a887be02151fdb173ae9e7bab78d5c389557a6ea289 WatchSource:0}: Error finding container cbe74669c65ddbeb7a523a887be02151fdb173ae9e7bab78d5c389557a6ea289: Status 404 returned error can't find the container with id cbe74669c65ddbeb7a523a887be02151fdb173ae9e7bab78d5c389557a6ea289
Jul 20 17:57:00.404112 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:57:00.404041    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8v5ff" event={"ID":"de1b439b-702a-43a3-8118-f7e20e716ac5","Type":"ContainerStarted","Data":"cbe74669c65ddbeb7a523a887be02151fdb173ae9e7bab78d5c389557a6ea289"}
Jul 20 17:57:00.405140 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:57:00.405117    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-tgg22" event={"ID":"eca0024f-2963-4ed1-817e-f1a6febd79aa","Type":"ContainerStarted","Data":"70ecf96ae96b3de12240578c90201f51e7855a2824c8b7b796888aae09bf6f85"}
Jul 20 17:57:00.970976 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:57:00.970922    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8357c262-2ca2-4ff5-8b48-70017efdb56b-registry-tls\") pod \"image-registry-794579c464-cq4k2\" (UID: \"8357c262-2ca2-4ff5-8b48-70017efdb56b\") " pod="openshift-image-registry/image-registry-794579c464-cq4k2"
Jul 20 17:57:00.971505 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:57:00.971085    2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 20 17:57:00.971505 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:57:00.971108    2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-794579c464-cq4k2: secret "image-registry-tls" not found
Jul 20 17:57:00.971505 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:57:00.971184    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8357c262-2ca2-4ff5-8b48-70017efdb56b-registry-tls podName:8357c262-2ca2-4ff5-8b48-70017efdb56b nodeName:}" failed. No retries permitted until 2026-07-20 17:57:32.971163126 +0000 UTC m=+99.217696534 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8357c262-2ca2-4ff5-8b48-70017efdb56b-registry-tls") pod "image-registry-794579c464-cq4k2" (UID: "8357c262-2ca2-4ff5-8b48-70017efdb56b") : secret "image-registry-tls" not found
Jul 20 17:57:01.072160 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:57:01.072124    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bfe61b3e-4aac-4ddd-bde9-10fb1c7e6e07-metrics-tls\") pod \"dns-default-ktc25\" (UID: \"bfe61b3e-4aac-4ddd-bde9-10fb1c7e6e07\") " pod="openshift-dns/dns-default-ktc25"
Jul 20 17:57:01.072335 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:57:01.072309    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 20 17:57:01.072399 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:57:01.072388    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bfe61b3e-4aac-4ddd-bde9-10fb1c7e6e07-metrics-tls podName:bfe61b3e-4aac-4ddd-bde9-10fb1c7e6e07 nodeName:}" failed. No retries permitted until 2026-07-20 17:57:33.072364824 +0000 UTC m=+99.318898227 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bfe61b3e-4aac-4ddd-bde9-10fb1c7e6e07-metrics-tls") pod "dns-default-ktc25" (UID: "bfe61b3e-4aac-4ddd-bde9-10fb1c7e6e07") : secret "dns-default-metrics-tls" not found
Jul 20 17:57:05.419915 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:57:05.419872    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-tgg22" event={"ID":"eca0024f-2963-4ed1-817e-f1a6febd79aa","Type":"ContainerStarted","Data":"e5f46a99599da33ee6a3c4e421a246c7448853bd5f2eafbf23decb3a3cacacbd"}
Jul 20 17:57:05.421383 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:57:05.421349    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8v5ff" event={"ID":"de1b439b-702a-43a3-8118-f7e20e716ac5","Type":"ContainerStarted","Data":"041d2170f4911662923c761b1441680ea9833254d442a722c30eb0e7fd0f0e7b"}
Jul 20 17:57:05.421489 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:57:05.421473    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-8v5ff"
Jul 20 17:57:05.436868 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:57:05.436814    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-tgg22" podStartSLOduration=66.352185959 podStartE2EDuration="1m11.436795894s" totalImagesPullingTime="5.084609935s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 17:55:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 17:57:05.435929503 +0000 UTC m=+71.682462916" watchObservedRunningTime="2026-07-20 17:57:05.436795894 +0000 UTC m=+71.683329306"
Jul 20 17:57:05.455926 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:57:05.455882    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-8v5ff" podStartSLOduration=66.393100381 podStartE2EDuration="1m11.455866894s" totalImagesPullingTime="5.062766513s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 17:55:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 17:57:05.455041847 +0000 UTC m=+71.701575260" watchObservedRunningTime="2026-07-20 17:57:05.455866894 +0000 UTC m=+71.702400304"
Jul 20 17:57:32.989524 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:57:32.989493    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8357c262-2ca2-4ff5-8b48-70017efdb56b-registry-tls\") pod \"image-registry-794579c464-cq4k2\" (UID: \"8357c262-2ca2-4ff5-8b48-70017efdb56b\") " pod="openshift-image-registry/image-registry-794579c464-cq4k2"
Jul 20 17:57:32.989905 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:57:32.989610    2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 20 17:57:32.989905 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:57:32.989624    2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-794579c464-cq4k2: secret "image-registry-tls" not found
Jul 20 17:57:32.989905 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:57:32.989684    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8357c262-2ca2-4ff5-8b48-70017efdb56b-registry-tls podName:8357c262-2ca2-4ff5-8b48-70017efdb56b nodeName:}" failed. No retries permitted until 2026-07-20 17:58:36.989668763 +0000 UTC m=+163.236202150 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8357c262-2ca2-4ff5-8b48-70017efdb56b-registry-tls") pod "image-registry-794579c464-cq4k2" (UID: "8357c262-2ca2-4ff5-8b48-70017efdb56b") : secret "image-registry-tls" not found
Jul 20 17:57:33.090268 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:57:33.090239    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bfe61b3e-4aac-4ddd-bde9-10fb1c7e6e07-metrics-tls\") pod \"dns-default-ktc25\" (UID: \"bfe61b3e-4aac-4ddd-bde9-10fb1c7e6e07\") " pod="openshift-dns/dns-default-ktc25"
Jul 20 17:57:33.090391 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:57:33.090358    2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 20 17:57:33.090428 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:57:33.090415    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bfe61b3e-4aac-4ddd-bde9-10fb1c7e6e07-metrics-tls podName:bfe61b3e-4aac-4ddd-bde9-10fb1c7e6e07 nodeName:}" failed. No retries permitted until 2026-07-20 17:58:37.090401269 +0000 UTC m=+163.336934657 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bfe61b3e-4aac-4ddd-bde9-10fb1c7e6e07-metrics-tls") pod "dns-default-ktc25" (UID: "bfe61b3e-4aac-4ddd-bde9-10fb1c7e6e07") : secret "dns-default-metrics-tls" not found
Jul 20 17:57:36.425917 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:57:36.425886    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-8v5ff"
Jul 20 17:57:45.085856 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:57:45.085820    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-kh4zt"]
Jul 20 17:57:45.088656 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:57:45.088637    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-kh4zt"
Jul 20 17:57:45.092881 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:57:45.092859    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 20 17:57:45.093010 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:57:45.092943    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 20 17:57:45.093092 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:57:45.093077    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-krpk2\""
Jul 20 17:57:45.093374 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:57:45.093357    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 20 17:57:45.093778 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:57:45.093762    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 20 17:57:45.109098 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:57:45.109079    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-kh4zt"]
Jul 20 17:57:45.173181 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:57:45.173150    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/24ac2f79-9d88-4cd0-88ef-8fb14fd65674-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kh4zt\" (UID: \"24ac2f79-9d88-4cd0-88ef-8fb14fd65674\") " pod="openshift-insights/insights-runtime-extractor-kh4zt"
Jul 20 17:57:45.173181 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:57:45.173180    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gg8zb\" (UniqueName: \"kubernetes.io/projected/24ac2f79-9d88-4cd0-88ef-8fb14fd65674-kube-api-access-gg8zb\") pod \"insights-runtime-extractor-kh4zt\" (UID: \"24ac2f79-9d88-4cd0-88ef-8fb14fd65674\") " pod="openshift-insights/insights-runtime-extractor-kh4zt"
Jul 20 17:57:45.173362 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:57:45.173214    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/24ac2f79-9d88-4cd0-88ef-8fb14fd65674-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kh4zt\" (UID: \"24ac2f79-9d88-4cd0-88ef-8fb14fd65674\") " pod="openshift-insights/insights-runtime-extractor-kh4zt"
Jul 20 17:57:45.173362 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:57:45.173323    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/24ac2f79-9d88-4cd0-88ef-8fb14fd65674-crio-socket\") pod \"insights-runtime-extractor-kh4zt\" (UID: \"24ac2f79-9d88-4cd0-88ef-8fb14fd65674\") " pod="openshift-insights/insights-runtime-extractor-kh4zt"
Jul 20 17:57:45.173362 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:57:45.173355    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/24ac2f79-9d88-4cd0-88ef-8fb14fd65674-data-volume\") pod \"insights-runtime-extractor-kh4zt\" (UID: \"24ac2f79-9d88-4cd0-88ef-8fb14fd65674\") " pod="openshift-insights/insights-runtime-extractor-kh4zt"
Jul 20 17:57:45.273998 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:57:45.273951    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/24ac2f79-9d88-4cd0-88ef-8fb14fd65674-crio-socket\") pod \"insights-runtime-extractor-kh4zt\" (UID: \"24ac2f79-9d88-4cd0-88ef-8fb14fd65674\") " pod="openshift-insights/insights-runtime-extractor-kh4zt"
Jul 20 17:57:45.274157 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:57:45.274009    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/24ac2f79-9d88-4cd0-88ef-8fb14fd65674-data-volume\") pod \"insights-runtime-extractor-kh4zt\" (UID: \"24ac2f79-9d88-4cd0-88ef-8fb14fd65674\") " pod="openshift-insights/insights-runtime-extractor-kh4zt"
Jul 20 17:57:45.274157 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:57:45.274066    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/24ac2f79-9d88-4cd0-88ef-8fb14fd65674-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kh4zt\" (UID: \"24ac2f79-9d88-4cd0-88ef-8fb14fd65674\") " pod="openshift-insights/insights-runtime-extractor-kh4zt"
Jul 20 17:57:45.274157 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:57:45.274091    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gg8zb\" (UniqueName: \"kubernetes.io/projected/24ac2f79-9d88-4cd0-88ef-8fb14fd65674-kube-api-access-gg8zb\") pod \"insights-runtime-extractor-kh4zt\" (UID: \"24ac2f79-9d88-4cd0-88ef-8fb14fd65674\") " pod="openshift-insights/insights-runtime-extractor-kh4zt"
Jul 20 17:57:45.274157 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:57:45.274124    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/24ac2f79-9d88-4cd0-88ef-8fb14fd65674-crio-socket\") pod \"insights-runtime-extractor-kh4zt\" (UID: \"24ac2f79-9d88-4cd0-88ef-8fb14fd65674\") " pod="openshift-insights/insights-runtime-extractor-kh4zt"
Jul 20 17:57:45.274368 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:57:45.274178    2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 20 17:57:45.274368 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:57:45.274261    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/24ac2f79-9d88-4cd0-88ef-8fb14fd65674-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kh4zt\" (UID: \"24ac2f79-9d88-4cd0-88ef-8fb14fd65674\") " pod="openshift-insights/insights-runtime-extractor-kh4zt"
Jul 20 17:57:45.274368 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:57:45.274301    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/24ac2f79-9d88-4cd0-88ef-8fb14fd65674-kube-rbac-proxy-cm podName:24ac2f79-9d88-4cd0-88ef-8fb14fd65674 nodeName:}" failed. No retries permitted until 2026-07-20 17:57:45.774279041 +0000 UTC m=+112.020812448 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/24ac2f79-9d88-4cd0-88ef-8fb14fd65674-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kh4zt" (UID: "24ac2f79-9d88-4cd0-88ef-8fb14fd65674") : configmap "kube-rbac-proxy" not found
Jul 20 17:57:45.274368 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:57:45.274344    2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 20 17:57:45.274507 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:57:45.274392    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24ac2f79-9d88-4cd0-88ef-8fb14fd65674-insights-runtime-extractor-tls podName:24ac2f79-9d88-4cd0-88ef-8fb14fd65674 nodeName:}" failed. No retries permitted until 2026-07-20 17:57:45.774379688 +0000 UTC m=+112.020913081 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/24ac2f79-9d88-4cd0-88ef-8fb14fd65674-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kh4zt" (UID: "24ac2f79-9d88-4cd0-88ef-8fb14fd65674") : secret "insights-runtime-extractor-tls" not found
Jul 20 17:57:45.274507 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:57:45.274416    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/24ac2f79-9d88-4cd0-88ef-8fb14fd65674-data-volume\") pod \"insights-runtime-extractor-kh4zt\" (UID: \"24ac2f79-9d88-4cd0-88ef-8fb14fd65674\") " pod="openshift-insights/insights-runtime-extractor-kh4zt"
Jul 20 17:57:45.282356 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:57:45.282328    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gg8zb\" (UniqueName: \"kubernetes.io/projected/24ac2f79-9d88-4cd0-88ef-8fb14fd65674-kube-api-access-gg8zb\") pod \"insights-runtime-extractor-kh4zt\" (UID: \"24ac2f79-9d88-4cd0-88ef-8fb14fd65674\") " pod="openshift-insights/insights-runtime-extractor-kh4zt"
Jul 20 17:57:45.777906 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:57:45.777866    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/24ac2f79-9d88-4cd0-88ef-8fb14fd65674-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kh4zt\" (UID: \"24ac2f79-9d88-4cd0-88ef-8fb14fd65674\") " pod="openshift-insights/insights-runtime-extractor-kh4zt"
Jul 20 17:57:45.777906 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:57:45.777915    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/24ac2f79-9d88-4cd0-88ef-8fb14fd65674-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kh4zt\" (UID: \"24ac2f79-9d88-4cd0-88ef-8fb14fd65674\") " pod="openshift-insights/insights-runtime-extractor-kh4zt"
Jul 20 17:57:45.778111 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:57:45.777984    2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 20 17:57:45.778111 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:57:45.778029    2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 20 17:57:45.778111 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:57:45.778048    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/24ac2f79-9d88-4cd0-88ef-8fb14fd65674-kube-rbac-proxy-cm podName:24ac2f79-9d88-4cd0-88ef-8fb14fd65674 nodeName:}" failed. No retries permitted until 2026-07-20 17:57:46.77803128 +0000 UTC m=+113.024564668 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/24ac2f79-9d88-4cd0-88ef-8fb14fd65674-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kh4zt" (UID: "24ac2f79-9d88-4cd0-88ef-8fb14fd65674") : configmap "kube-rbac-proxy" not found
Jul 20 17:57:45.778111 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:57:45.778076    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24ac2f79-9d88-4cd0-88ef-8fb14fd65674-insights-runtime-extractor-tls podName:24ac2f79-9d88-4cd0-88ef-8fb14fd65674 nodeName:}" failed. No retries permitted until 2026-07-20 17:57:46.778065391 +0000 UTC m=+113.024598778 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/24ac2f79-9d88-4cd0-88ef-8fb14fd65674-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kh4zt" (UID: "24ac2f79-9d88-4cd0-88ef-8fb14fd65674") : secret "insights-runtime-extractor-tls" not found
Jul 20 17:57:46.785425 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:57:46.785396    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/24ac2f79-9d88-4cd0-88ef-8fb14fd65674-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kh4zt\" (UID: \"24ac2f79-9d88-4cd0-88ef-8fb14fd65674\") " pod="openshift-insights/insights-runtime-extractor-kh4zt"
Jul 20 17:57:46.785843 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:57:46.785436    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/24ac2f79-9d88-4cd0-88ef-8fb14fd65674-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kh4zt\" (UID: \"24ac2f79-9d88-4cd0-88ef-8fb14fd65674\") " pod="openshift-insights/insights-runtime-extractor-kh4zt"
Jul 20 17:57:46.785843 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:57:46.785511    2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 20 17:57:46.785843 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:57:46.785566    2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 20 17:57:46.785843 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:57:46.785568    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/24ac2f79-9d88-4cd0-88ef-8fb14fd65674-kube-rbac-proxy-cm podName:24ac2f79-9d88-4cd0-88ef-8fb14fd65674 nodeName:}" failed. No retries permitted until 2026-07-20 17:57:48.785553905 +0000 UTC m=+115.032087294 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/24ac2f79-9d88-4cd0-88ef-8fb14fd65674-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kh4zt" (UID: "24ac2f79-9d88-4cd0-88ef-8fb14fd65674") : configmap "kube-rbac-proxy" not found
Jul 20 17:57:46.785843 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:57:46.785610    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24ac2f79-9d88-4cd0-88ef-8fb14fd65674-insights-runtime-extractor-tls podName:24ac2f79-9d88-4cd0-88ef-8fb14fd65674 nodeName:}" failed. No retries permitted until 2026-07-20 17:57:48.785599225 +0000 UTC m=+115.032132613 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/24ac2f79-9d88-4cd0-88ef-8fb14fd65674-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kh4zt" (UID: "24ac2f79-9d88-4cd0-88ef-8fb14fd65674") : secret "insights-runtime-extractor-tls" not found
Jul 20 17:57:47.082180 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:57:47.082108    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-smcqb_ea740a69-374a-483b-a045-a2c795b99d4c/dns-node-resolver/0.log"
Jul 20 17:57:47.880132 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:57:47.880107    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-q42fv_7a564ed7-d4b4-4f1a-a7da-d1e4d0652601/node-ca/0.log"
Jul 20 17:57:48.479841 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:57:48.479811    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-67b26_69347ff0-9f35-4334-be7f-b02bca61a64c/serve-healthcheck-canary/0.log"
Jul 20 17:57:48.801664 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:57:48.801622    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/24ac2f79-9d88-4cd0-88ef-8fb14fd65674-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kh4zt\" (UID: \"24ac2f79-9d88-4cd0-88ef-8fb14fd65674\") " pod="openshift-insights/insights-runtime-extractor-kh4zt"
Jul 20 17:57:48.801842 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:57:48.801674    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/24ac2f79-9d88-4cd0-88ef-8fb14fd65674-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kh4zt\" (UID: \"24ac2f79-9d88-4cd0-88ef-8fb14fd65674\") " pod="openshift-insights/insights-runtime-extractor-kh4zt"
Jul 20 17:57:48.801842 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:57:48.801753    2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 20 17:57:48.801842 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:57:48.801775    2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 20 17:57:48.801842 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:57:48.801824    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24ac2f79-9d88-4cd0-88ef-8fb14fd65674-insights-runtime-extractor-tls podName:24ac2f79-9d88-4cd0-88ef-8fb14fd65674 nodeName:}" failed. No retries permitted until 2026-07-20 17:57:52.80181066 +0000 UTC m=+119.048344048 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/24ac2f79-9d88-4cd0-88ef-8fb14fd65674-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kh4zt" (UID: "24ac2f79-9d88-4cd0-88ef-8fb14fd65674") : secret "insights-runtime-extractor-tls" not found
Jul 20 17:57:48.801842 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:57:48.801838    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/24ac2f79-9d88-4cd0-88ef-8fb14fd65674-kube-rbac-proxy-cm podName:24ac2f79-9d88-4cd0-88ef-8fb14fd65674 nodeName:}" failed. No retries permitted until 2026-07-20 17:57:52.801830949 +0000 UTC m=+119.048364336 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/24ac2f79-9d88-4cd0-88ef-8fb14fd65674-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kh4zt" (UID: "24ac2f79-9d88-4cd0-88ef-8fb14fd65674") : configmap "kube-rbac-proxy" not found
Jul 20 17:57:52.216800 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:57:52.216766    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-fcjsx"]
Jul 20 17:57:52.219801 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:57:52.219784    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-fcjsx"
Jul 20 17:57:52.221916 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:57:52.221891    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-ch7gl\""
Jul 20 17:57:52.222605 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:57:52.222587    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Jul 20 17:57:52.222689 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:57:52.222594    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Jul 20 17:57:52.228330 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:57:52.228309    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-fcjsx"]
Jul 20 17:57:52.329541 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:57:52.329509    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5bpr\" (UniqueName: \"kubernetes.io/projected/8a0b6c5f-6bca-49b3-b6dd-8ba1677ed470-kube-api-access-p5bpr\") pod \"migrator-7cc9c7ffd7-fcjsx\" (UID: \"8a0b6c5f-6bca-49b3-b6dd-8ba1677ed470\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-fcjsx"
Jul 20 17:57:52.430096 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:57:52.430064    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5bpr\" (UniqueName: \"kubernetes.io/projected/8a0b6c5f-6bca-49b3-b6dd-8ba1677ed470-kube-api-access-p5bpr\") pod \"migrator-7cc9c7ffd7-fcjsx\" (UID: \"8a0b6c5f-6bca-49b3-b6dd-8ba1677ed470\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-fcjsx"
Jul 20 17:57:52.437840 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:57:52.437810    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5bpr\" (UniqueName: \"kubernetes.io/projected/8a0b6c5f-6bca-49b3-b6dd-8ba1677ed470-kube-api-access-p5bpr\") pod \"migrator-7cc9c7ffd7-fcjsx\" (UID: \"8a0b6c5f-6bca-49b3-b6dd-8ba1677ed470\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-fcjsx"
Jul 20 17:57:52.528716 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:57:52.528686    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-fcjsx"
Jul 20 17:57:52.641630 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:57:52.641600    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-fcjsx"]
Jul 20 17:57:52.645110 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:57:52.645085    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a0b6c5f_6bca_49b3_b6dd_8ba1677ed470.slice/crio-36016f700981c304e2f496f127cad7a85fd6c7d345913a94fb8b77e31e141c76 WatchSource:0}: Error finding container 36016f700981c304e2f496f127cad7a85fd6c7d345913a94fb8b77e31e141c76: Status 404 returned error can't find the container with id 36016f700981c304e2f496f127cad7a85fd6c7d345913a94fb8b77e31e141c76
Jul 20 17:57:52.832327 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:57:52.832251    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/24ac2f79-9d88-4cd0-88ef-8fb14fd65674-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kh4zt\" (UID: \"24ac2f79-9d88-4cd0-88ef-8fb14fd65674\") " pod="openshift-insights/insights-runtime-extractor-kh4zt"
Jul 20 17:57:52.832327 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:57:52.832295    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/24ac2f79-9d88-4cd0-88ef-8fb14fd65674-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kh4zt\" (UID: \"24ac2f79-9d88-4cd0-88ef-8fb14fd65674\") " pod="openshift-insights/insights-runtime-extractor-kh4zt"
Jul 20 17:57:52.832494 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:57:52.832369    2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 20 17:57:52.832494 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:57:52.832434    2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 20 17:57:52.832494 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:57:52.832437    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/24ac2f79-9d88-4cd0-88ef-8fb14fd65674-kube-rbac-proxy-cm podName:24ac2f79-9d88-4cd0-88ef-8fb14fd65674 nodeName:}" failed. No retries permitted until 2026-07-20 17:58:00.832420435 +0000 UTC m=+127.078953822 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/24ac2f79-9d88-4cd0-88ef-8fb14fd65674-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kh4zt" (UID: "24ac2f79-9d88-4cd0-88ef-8fb14fd65674") : configmap "kube-rbac-proxy" not found
Jul 20 17:57:52.832494 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:57:52.832489    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24ac2f79-9d88-4cd0-88ef-8fb14fd65674-insights-runtime-extractor-tls podName:24ac2f79-9d88-4cd0-88ef-8fb14fd65674 nodeName:}" failed. No retries permitted until 2026-07-20 17:58:00.832474452 +0000 UTC m=+127.079007844 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/24ac2f79-9d88-4cd0-88ef-8fb14fd65674-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kh4zt" (UID: "24ac2f79-9d88-4cd0-88ef-8fb14fd65674") : secret "insights-runtime-extractor-tls" not found
Jul 20 17:57:53.534624 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:57:53.534589    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-fcjsx" event={"ID":"8a0b6c5f-6bca-49b3-b6dd-8ba1677ed470","Type":"ContainerStarted","Data":"36016f700981c304e2f496f127cad7a85fd6c7d345913a94fb8b77e31e141c76"}
Jul 20 17:57:54.539666 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:57:54.539625    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-fcjsx" event={"ID":"8a0b6c5f-6bca-49b3-b6dd-8ba1677ed470","Type":"ContainerStarted","Data":"283738a8501d4c75c619ffa31f890bc602a1280160c541186e922a434837e910"}
Jul 20 17:57:54.540058 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:57:54.539686    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-fcjsx" event={"ID":"8a0b6c5f-6bca-49b3-b6dd-8ba1677ed470","Type":"ContainerStarted","Data":"a67a98d0cc984c07111ad1853c1157b0b5fe385d9ab15b51798b3bd6ffaa3c55"}
Jul 20 17:57:54.554973 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:57:54.554933    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-fcjsx" podStartSLOduration=0.956712007 podStartE2EDuration="2.554919487s" totalImagesPullingTime="1.59820748s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 17:57:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 17:57:54.554529672 +0000 UTC m=+120.801063082" watchObservedRunningTime="2026-07-20 17:57:54.554919487 +0000 UTC m=+120.801452897"
Jul 20 17:58:00.890486 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:00.890447    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/24ac2f79-9d88-4cd0-88ef-8fb14fd65674-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kh4zt\" (UID: \"24ac2f79-9d88-4cd0-88ef-8fb14fd65674\") " pod="openshift-insights/insights-runtime-extractor-kh4zt"
Jul 20 17:58:00.890486 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:00.890492    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/24ac2f79-9d88-4cd0-88ef-8fb14fd65674-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kh4zt\" (UID: \"24ac2f79-9d88-4cd0-88ef-8fb14fd65674\") " pod="openshift-insights/insights-runtime-extractor-kh4zt"
Jul 20 17:58:00.890934 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:58:00.890592    2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 20 17:58:00.890934 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:58:00.890653    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24ac2f79-9d88-4cd0-88ef-8fb14fd65674-insights-runtime-extractor-tls podName:24ac2f79-9d88-4cd0-88ef-8fb14fd65674 nodeName:}" failed. No retries permitted until 2026-07-20 17:58:16.890635729 +0000 UTC m=+143.137169121 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/24ac2f79-9d88-4cd0-88ef-8fb14fd65674-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kh4zt" (UID: "24ac2f79-9d88-4cd0-88ef-8fb14fd65674") : secret "insights-runtime-extractor-tls" not found
Jul 20 17:58:00.891021 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:00.890983    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/24ac2f79-9d88-4cd0-88ef-8fb14fd65674-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kh4zt\" (UID: \"24ac2f79-9d88-4cd0-88ef-8fb14fd65674\") " pod="openshift-insights/insights-runtime-extractor-kh4zt"
Jul 20 17:58:03.915337 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:03.915300    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26b0d75c-ecf7-4bb4-ad4b-d9077da808bd-metrics-certs\") pod \"network-metrics-daemon-nkmr4\" (UID: \"26b0d75c-ecf7-4bb4-ad4b-d9077da808bd\") " pod="openshift-multus/network-metrics-daemon-nkmr4"
Jul 20 17:58:03.915829 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:58:03.915444    2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 20 17:58:03.915829 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:58:03.915508    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26b0d75c-ecf7-4bb4-ad4b-d9077da808bd-metrics-certs podName:26b0d75c-ecf7-4bb4-ad4b-d9077da808bd nodeName:}" failed. No retries permitted until 2026-07-20 18:00:05.915493499 +0000 UTC m=+252.162026888 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/26b0d75c-ecf7-4bb4-ad4b-d9077da808bd-metrics-certs") pod "network-metrics-daemon-nkmr4" (UID: "26b0d75c-ecf7-4bb4-ad4b-d9077da808bd") : secret "metrics-daemon-secret" not found
Jul 20 17:58:16.910689 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:16.910651    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/24ac2f79-9d88-4cd0-88ef-8fb14fd65674-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kh4zt\" (UID: \"24ac2f79-9d88-4cd0-88ef-8fb14fd65674\") " pod="openshift-insights/insights-runtime-extractor-kh4zt"
Jul 20 17:58:16.912926 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:16.912901    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/24ac2f79-9d88-4cd0-88ef-8fb14fd65674-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kh4zt\" (UID: \"24ac2f79-9d88-4cd0-88ef-8fb14fd65674\") " pod="openshift-insights/insights-runtime-extractor-kh4zt"
Jul 20 17:58:17.199685 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:17.199608    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-krpk2\""
Jul 20 17:58:17.207705 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:17.207684    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-kh4zt"
Jul 20 17:58:17.340119 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:17.340094    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-kh4zt"]
Jul 20 17:58:17.342309 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:58:17.342283    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24ac2f79_9d88_4cd0_88ef_8fb14fd65674.slice/crio-3acb8fd482c2c6c9d2db04d2093a2dbdf8d1b24b19cfa3c113fab62d5ea883ca WatchSource:0}: Error finding container 3acb8fd482c2c6c9d2db04d2093a2dbdf8d1b24b19cfa3c113fab62d5ea883ca: Status 404 returned error can't find the container with id 3acb8fd482c2c6c9d2db04d2093a2dbdf8d1b24b19cfa3c113fab62d5ea883ca
Jul 20 17:58:17.605332 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:17.605294    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kh4zt" event={"ID":"24ac2f79-9d88-4cd0-88ef-8fb14fd65674","Type":"ContainerStarted","Data":"3acb8fd482c2c6c9d2db04d2093a2dbdf8d1b24b19cfa3c113fab62d5ea883ca"}
Jul 20 17:58:21.416251 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:21.416216    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-677468d475-jp6gg"]
Jul 20 17:58:21.448869 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:21.448840    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-677468d475-jp6gg"
Jul 20 17:58:21.452327 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:21.452296    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 20 17:58:21.453606 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:21.453278    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-677468d475-jp6gg"]
Jul 20 17:58:21.457850 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:21.457606    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 20 17:58:21.457850 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:21.457634    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 20 17:58:21.457850 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:21.457679    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 20 17:58:21.457850 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:21.457691    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 20 17:58:21.458148 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:21.457983    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 20 17:58:21.458148 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:21.457986    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 20 17:58:21.549083 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:21.549053    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/5669b001-62a9-4e0d-ae68-d0b09a4b98fa-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-677468d475-jp6gg\" (UID: \"5669b001-62a9-4e0d-ae68-d0b09a4b98fa\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-677468d475-jp6gg"
Jul 20 17:58:21.549083 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:21.549091    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwclv\" (UniqueName: \"kubernetes.io/projected/5669b001-62a9-4e0d-ae68-d0b09a4b98fa-kube-api-access-pwclv\") pod \"cluster-proxy-proxy-agent-677468d475-jp6gg\" (UID: \"5669b001-62a9-4e0d-ae68-d0b09a4b98fa\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-677468d475-jp6gg"
Jul 20 17:58:21.549282 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:21.549114    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/5669b001-62a9-4e0d-ae68-d0b09a4b98fa-hub\") pod \"cluster-proxy-proxy-agent-677468d475-jp6gg\" (UID: \"5669b001-62a9-4e0d-ae68-d0b09a4b98fa\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-677468d475-jp6gg"
Jul 20 17:58:21.549282 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:21.549169    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5669b001-62a9-4e0d-ae68-d0b09a4b98fa-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-677468d475-jp6gg\" (UID: \"5669b001-62a9-4e0d-ae68-d0b09a4b98fa\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-677468d475-jp6gg"
Jul 20 17:58:21.549282 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:21.549186    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/5669b001-62a9-4e0d-ae68-d0b09a4b98fa-additional-service-ca\") pod \"cluster-proxy-proxy-agent-677468d475-jp6gg\" (UID: \"5669b001-62a9-4e0d-ae68-d0b09a4b98fa\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-677468d475-jp6gg"
Jul 20 17:58:21.549282 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:21.549239    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/5669b001-62a9-4e0d-ae68-d0b09a4b98fa-ca\") pod \"cluster-proxy-proxy-agent-677468d475-jp6gg\" (UID: \"5669b001-62a9-4e0d-ae68-d0b09a4b98fa\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-677468d475-jp6gg"
Jul 20 17:58:21.650502 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:21.650467    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/5669b001-62a9-4e0d-ae68-d0b09a4b98fa-ca\") pod \"cluster-proxy-proxy-agent-677468d475-jp6gg\" (UID: \"5669b001-62a9-4e0d-ae68-d0b09a4b98fa\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-677468d475-jp6gg"
Jul 20 17:58:21.650664 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:21.650621    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/5669b001-62a9-4e0d-ae68-d0b09a4b98fa-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-677468d475-jp6gg\" (UID: \"5669b001-62a9-4e0d-ae68-d0b09a4b98fa\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-677468d475-jp6gg"
Jul 20 17:58:21.650755 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:21.650661    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwclv\" (UniqueName: \"kubernetes.io/projected/5669b001-62a9-4e0d-ae68-d0b09a4b98fa-kube-api-access-pwclv\") pod \"cluster-proxy-proxy-agent-677468d475-jp6gg\" (UID: \"5669b001-62a9-4e0d-ae68-d0b09a4b98fa\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-677468d475-jp6gg"
Jul 20 17:58:21.650755 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:21.650687    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/5669b001-62a9-4e0d-ae68-d0b09a4b98fa-hub\") pod \"cluster-proxy-proxy-agent-677468d475-jp6gg\" (UID: \"5669b001-62a9-4e0d-ae68-d0b09a4b98fa\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-677468d475-jp6gg"
Jul 20 17:58:21.650865 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:21.650753    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5669b001-62a9-4e0d-ae68-d0b09a4b98fa-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-677468d475-jp6gg\" (UID: \"5669b001-62a9-4e0d-ae68-d0b09a4b98fa\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-677468d475-jp6gg"
Jul 20 17:58:21.650865 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:21.650818    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/5669b001-62a9-4e0d-ae68-d0b09a4b98fa-additional-service-ca\") pod \"cluster-proxy-proxy-agent-677468d475-jp6gg\" (UID: \"5669b001-62a9-4e0d-ae68-d0b09a4b98fa\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-677468d475-jp6gg"
Jul 20 17:58:21.651528 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:21.651479    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/5669b001-62a9-4e0d-ae68-d0b09a4b98fa-additional-service-ca\") pod \"cluster-proxy-proxy-agent-677468d475-jp6gg\" (UID: \"5669b001-62a9-4e0d-ae68-d0b09a4b98fa\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-677468d475-jp6gg"
Jul 20 17:58:21.652856 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:21.652836    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/5669b001-62a9-4e0d-ae68-d0b09a4b98fa-ca\") pod \"cluster-proxy-proxy-agent-677468d475-jp6gg\" (UID: \"5669b001-62a9-4e0d-ae68-d0b09a4b98fa\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-677468d475-jp6gg"
Jul 20 17:58:21.653574 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:21.653552    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/5669b001-62a9-4e0d-ae68-d0b09a4b98fa-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-677468d475-jp6gg\" (UID: \"5669b001-62a9-4e0d-ae68-d0b09a4b98fa\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-677468d475-jp6gg"
Jul 20 17:58:21.653629 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:21.653557    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/5669b001-62a9-4e0d-ae68-d0b09a4b98fa-hub\") pod \"cluster-proxy-proxy-agent-677468d475-jp6gg\" (UID: \"5669b001-62a9-4e0d-ae68-d0b09a4b98fa\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-677468d475-jp6gg"
Jul 20 17:58:21.654432 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:21.654413    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5669b001-62a9-4e0d-ae68-d0b09a4b98fa-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-677468d475-jp6gg\" (UID: \"5669b001-62a9-4e0d-ae68-d0b09a4b98fa\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-677468d475-jp6gg"
Jul 20 17:58:21.664090 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:21.664069    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwclv\" (UniqueName: \"kubernetes.io/projected/5669b001-62a9-4e0d-ae68-d0b09a4b98fa-kube-api-access-pwclv\") pod \"cluster-proxy-proxy-agent-677468d475-jp6gg\" (UID: \"5669b001-62a9-4e0d-ae68-d0b09a4b98fa\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-677468d475-jp6gg"
Jul 20 17:58:21.770101 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:21.770068    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-677468d475-jp6gg"
Jul 20 17:58:22.465421 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:22.465392    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-677468d475-jp6gg"]
Jul 20 17:58:22.471276 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:58:22.471249    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5669b001_62a9_4e0d_ae68_d0b09a4b98fa.slice/crio-648c3d734ad58e6c5c28254e47c8d5e12221390e4d27e39acddd6b3f108ca4cf WatchSource:0}: Error finding container 648c3d734ad58e6c5c28254e47c8d5e12221390e4d27e39acddd6b3f108ca4cf: Status 404 returned error can't find the container with id 648c3d734ad58e6c5c28254e47c8d5e12221390e4d27e39acddd6b3f108ca4cf
Jul 20 17:58:22.623519 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:22.623431    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-677468d475-jp6gg" event={"ID":"5669b001-62a9-4e0d-ae68-d0b09a4b98fa","Type":"ContainerStarted","Data":"648c3d734ad58e6c5c28254e47c8d5e12221390e4d27e39acddd6b3f108ca4cf"}
Jul 20 17:58:22.624799 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:22.624777    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kh4zt" event={"ID":"24ac2f79-9d88-4cd0-88ef-8fb14fd65674","Type":"ContainerStarted","Data":"575c8cda7548374bf40c794f2913eb266fa063e0d0f2938b4e72e4e3dd3cdfde"}
Jul 20 17:58:28.217798 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:28.217713    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-mw5n7"]
Jul 20 17:58:28.220707 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:28.220692    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-mw5n7"
Jul 20 17:58:28.227438 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:28.227394    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 20 17:58:28.227438 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:28.227411    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 20 17:58:28.227584 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:28.227450    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 20 17:58:28.227642 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:28.227606    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-hvccw\""
Jul 20 17:58:28.227862 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:28.227846    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 20 17:58:28.230542 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:28.230526    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 20 17:58:28.243496 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:28.243474    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-mw5n7"]
Jul 20 17:58:28.308296 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:28.308265    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/46e0d218-80cf-4517-9265-3acbb2d30631-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-mw5n7\" (UID: \"46e0d218-80cf-4517-9265-3acbb2d30631\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-mw5n7"
Jul 20 17:58:28.308445 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:28.308303    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/46e0d218-80cf-4517-9265-3acbb2d30631-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-mw5n7\" (UID: \"46e0d218-80cf-4517-9265-3acbb2d30631\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-mw5n7"
Jul 20 17:58:28.308445 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:28.308345    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/46e0d218-80cf-4517-9265-3acbb2d30631-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-mw5n7\" (UID: \"46e0d218-80cf-4517-9265-3acbb2d30631\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-mw5n7"
Jul 20 17:58:28.308445 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:28.308384    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qvb6\" (UniqueName: \"kubernetes.io/projected/46e0d218-80cf-4517-9265-3acbb2d30631-kube-api-access-5qvb6\") pod \"prometheus-operator-5945b5cfd-mw5n7\" (UID: \"46e0d218-80cf-4517-9265-3acbb2d30631\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-mw5n7"
Jul 20 17:58:28.409291 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:28.409256    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/46e0d218-80cf-4517-9265-3acbb2d30631-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-mw5n7\" (UID: \"46e0d218-80cf-4517-9265-3acbb2d30631\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-mw5n7"
Jul 20 17:58:28.409465 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:28.409320    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/46e0d218-80cf-4517-9265-3acbb2d30631-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-mw5n7\" (UID: \"46e0d218-80cf-4517-9265-3acbb2d30631\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-mw5n7"
Jul 20 17:58:28.409465 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:28.409342    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qvb6\" (UniqueName: \"kubernetes.io/projected/46e0d218-80cf-4517-9265-3acbb2d30631-kube-api-access-5qvb6\") pod \"prometheus-operator-5945b5cfd-mw5n7\" (UID: \"46e0d218-80cf-4517-9265-3acbb2d30631\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-mw5n7"
Jul 20 17:58:28.409465 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:28.409399    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/46e0d218-80cf-4517-9265-3acbb2d30631-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-mw5n7\" (UID: \"46e0d218-80cf-4517-9265-3acbb2d30631\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-mw5n7"
Jul 20 17:58:28.409631 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:58:28.409502    2569 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Jul 20 17:58:28.409631 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:58:28.409581    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46e0d218-80cf-4517-9265-3acbb2d30631-prometheus-operator-tls podName:46e0d218-80cf-4517-9265-3acbb2d30631 nodeName:}" failed. No retries permitted until 2026-07-20 17:58:28.909561003 +0000 UTC m=+155.156094403 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/46e0d218-80cf-4517-9265-3acbb2d30631-prometheus-operator-tls") pod "prometheus-operator-5945b5cfd-mw5n7" (UID: "46e0d218-80cf-4517-9265-3acbb2d30631") : secret "prometheus-operator-tls" not found
Jul 20 17:58:28.410134 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:28.410111    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/46e0d218-80cf-4517-9265-3acbb2d30631-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-mw5n7\" (UID: \"46e0d218-80cf-4517-9265-3acbb2d30631\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-mw5n7"
Jul 20 17:58:28.412025 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:28.412001    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/46e0d218-80cf-4517-9265-3acbb2d30631-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-mw5n7\" (UID: \"46e0d218-80cf-4517-9265-3acbb2d30631\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-mw5n7"
Jul 20 17:58:28.419207 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:28.419184    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qvb6\" (UniqueName: \"kubernetes.io/projected/46e0d218-80cf-4517-9265-3acbb2d30631-kube-api-access-5qvb6\") pod \"prometheus-operator-5945b5cfd-mw5n7\" (UID: \"46e0d218-80cf-4517-9265-3acbb2d30631\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-mw5n7"
Jul 20 17:58:28.646247 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:28.646204    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kh4zt" event={"ID":"24ac2f79-9d88-4cd0-88ef-8fb14fd65674","Type":"ContainerStarted","Data":"2aa77e198eab17a999d6cedf7297603f22d0a23e6a75da83e48a0909de5e3317"}
Jul 20 17:58:28.648027 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:28.648002    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-677468d475-jp6gg" event={"ID":"5669b001-62a9-4e0d-ae68-d0b09a4b98fa","Type":"ContainerStarted","Data":"f3bfeea7c38e2ca676826011163d059878d86707ecac39bd76920416167b456d"}
Jul 20 17:58:28.648139 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:28.648035    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-677468d475-jp6gg" event={"ID":"5669b001-62a9-4e0d-ae68-d0b09a4b98fa","Type":"ContainerStarted","Data":"14bda78e0df993e307a7fab3bf1e258303b656449a8051561dd110ce7838a906"}
Jul 20 17:58:28.648139 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:28.648048    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-677468d475-jp6gg" event={"ID":"5669b001-62a9-4e0d-ae68-d0b09a4b98fa","Type":"ContainerStarted","Data":"2bc3cc996041b2d26da986fa3ecfe2eadc28f9dc18acf15ad5a61e08722f6069"}
Jul 20 17:58:28.676657 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:28.676611    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-677468d475-jp6gg" podStartSLOduration=2.215800454 podStartE2EDuration="7.676595153s" totalImagesPullingTime="5.460794699s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 17:58:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 17:58:28.674389391 +0000 UTC m=+154.920922813" watchObservedRunningTime="2026-07-20 17:58:28.676595153 +0000 UTC m=+154.923128600"
Jul 20 17:58:28.914381 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:28.914298    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/46e0d218-80cf-4517-9265-3acbb2d30631-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-mw5n7\" (UID: \"46e0d218-80cf-4517-9265-3acbb2d30631\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-mw5n7"
Jul 20 17:58:28.914535 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:58:28.914468    2569 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Jul 20 17:58:28.914604 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:58:28.914546    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46e0d218-80cf-4517-9265-3acbb2d30631-prometheus-operator-tls podName:46e0d218-80cf-4517-9265-3acbb2d30631 nodeName:}" failed. No retries permitted until 2026-07-20 17:58:29.914526184 +0000 UTC m=+156.161059585 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/46e0d218-80cf-4517-9265-3acbb2d30631-prometheus-operator-tls") pod "prometheus-operator-5945b5cfd-mw5n7" (UID: "46e0d218-80cf-4517-9265-3acbb2d30631") : secret "prometheus-operator-tls" not found
Jul 20 17:58:29.921966 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:29.921930    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/46e0d218-80cf-4517-9265-3acbb2d30631-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-mw5n7\" (UID: \"46e0d218-80cf-4517-9265-3acbb2d30631\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-mw5n7"
Jul 20 17:58:29.924633 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:29.924606    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/46e0d218-80cf-4517-9265-3acbb2d30631-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-mw5n7\" (UID: \"46e0d218-80cf-4517-9265-3acbb2d30631\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-mw5n7"
Jul 20 17:58:30.029313 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:30.029283    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-mw5n7"
Jul 20 17:58:31.547953 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:31.547925    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-mw5n7"]
Jul 20 17:58:31.550460 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:58:31.550433    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46e0d218_80cf_4517_9265_3acbb2d30631.slice/crio-319dacfa8facabe03165368438933f5b5ed34a98103d1459e22718f9903f05b9 WatchSource:0}: Error finding container 319dacfa8facabe03165368438933f5b5ed34a98103d1459e22718f9903f05b9: Status 404 returned error can't find the container with id 319dacfa8facabe03165368438933f5b5ed34a98103d1459e22718f9903f05b9
Jul 20 17:58:31.659323 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:31.659288    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-mw5n7" event={"ID":"46e0d218-80cf-4517-9265-3acbb2d30631","Type":"ContainerStarted","Data":"319dacfa8facabe03165368438933f5b5ed34a98103d1459e22718f9903f05b9"}
Jul 20 17:58:31.661104 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:31.661082    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kh4zt" event={"ID":"24ac2f79-9d88-4cd0-88ef-8fb14fd65674","Type":"ContainerStarted","Data":"904d801492b9218eeba363a9b2ddeae021951cd6daafbd32f9e1bd62d757358f"}
Jul 20 17:58:32.151047 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:58:32.151007    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-794579c464-cq4k2" podUID="8357c262-2ca2-4ff5-8b48-70017efdb56b"
Jul 20 17:58:32.194232 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:58:32.194202    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-nkmr4" podUID="26b0d75c-ecf7-4bb4-ad4b-d9077da808bd"
Jul 20 17:58:32.265905 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:58:32.265870    2569 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-ktc25" podUID="bfe61b3e-4aac-4ddd-bde9-10fb1c7e6e07"
Jul 20 17:58:32.664114 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:32.664078    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ktc25"
Jul 20 17:58:32.664114 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:32.664086    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-794579c464-cq4k2"
Jul 20 17:58:33.669097 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:33.669059    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-mw5n7" event={"ID":"46e0d218-80cf-4517-9265-3acbb2d30631","Type":"ContainerStarted","Data":"f06d798962ab02d75e78d6837c842a358eefee62d010c2f4eabbb66350007a44"}
Jul 20 17:58:34.673926 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:34.673883    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-mw5n7" event={"ID":"46e0d218-80cf-4517-9265-3acbb2d30631","Type":"ContainerStarted","Data":"e5bbf815cfae0ae274d16c03762ea931e7e78857435a4dbd6356e89373d51d73"}
Jul 20 17:58:34.690950 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:34.690898    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-kh4zt" podStartSLOduration=35.664998491 podStartE2EDuration="49.690884122s" totalImagesPullingTime="14.025885631s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 17:57:45 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 17:58:31.704564781 +0000 UTC m=+157.951098201" watchObservedRunningTime="2026-07-20 17:58:34.690884122 +0000 UTC m=+160.937417532"
Jul 20 17:58:34.691118 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:34.691040    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-5945b5cfd-mw5n7" podStartSLOduration=4.67440375 podStartE2EDuration="6.691034452s" totalImagesPullingTime="2.016630702s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 17:58:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 17:58:34.690155968 +0000 UTC m=+160.936689413" watchObservedRunningTime="2026-07-20 17:58:34.691034452 +0000 UTC m=+160.937567862"
Jul 20 17:58:36.669867 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:36.669834    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-dj4gt"]
Jul 20 17:58:36.673011 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:36.672994    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-dj4gt"
Jul 20 17:58:36.676069 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:36.676049    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 20 17:58:36.676291 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:36.676278    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-bqxsm\""
Jul 20 17:58:36.676592 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:36.676576    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 20 17:58:36.676708 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:36.676687    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 20 17:58:36.776138 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:36.776110    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4c02e13a-6eb6-4c56-87d0-ea9258d3912f-root\") pod \"node-exporter-dj4gt\" (UID: \"4c02e13a-6eb6-4c56-87d0-ea9258d3912f\") " pod="openshift-monitoring/node-exporter-dj4gt"
Jul 20 17:58:36.776266 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:36.776144    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4c02e13a-6eb6-4c56-87d0-ea9258d3912f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dj4gt\" (UID: \"4c02e13a-6eb6-4c56-87d0-ea9258d3912f\") " pod="openshift-monitoring/node-exporter-dj4gt"
Jul 20 17:58:36.776266 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:36.776181    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4c02e13a-6eb6-4c56-87d0-ea9258d3912f-metrics-client-ca\") pod \"node-exporter-dj4gt\" (UID: \"4c02e13a-6eb6-4c56-87d0-ea9258d3912f\") " pod="openshift-monitoring/node-exporter-dj4gt"
Jul 20 17:58:36.776266 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:36.776226    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4c02e13a-6eb6-4c56-87d0-ea9258d3912f-node-exporter-tls\") pod \"node-exporter-dj4gt\" (UID: \"4c02e13a-6eb6-4c56-87d0-ea9258d3912f\") " pod="openshift-monitoring/node-exporter-dj4gt"
Jul 20 17:58:36.776360 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:36.776278    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4c02e13a-6eb6-4c56-87d0-ea9258d3912f-node-exporter-textfile\") pod \"node-exporter-dj4gt\" (UID: \"4c02e13a-6eb6-4c56-87d0-ea9258d3912f\") " pod="openshift-monitoring/node-exporter-dj4gt"
Jul 20 17:58:36.776360 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:36.776300    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4c02e13a-6eb6-4c56-87d0-ea9258d3912f-node-exporter-wtmp\") pod \"node-exporter-dj4gt\" (UID: \"4c02e13a-6eb6-4c56-87d0-ea9258d3912f\") " pod="openshift-monitoring/node-exporter-dj4gt"
Jul 20 17:58:36.776360 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:36.776315    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54nbg\" (UniqueName: \"kubernetes.io/projected/4c02e13a-6eb6-4c56-87d0-ea9258d3912f-kube-api-access-54nbg\") pod \"node-exporter-dj4gt\" (UID: \"4c02e13a-6eb6-4c56-87d0-ea9258d3912f\") " pod="openshift-monitoring/node-exporter-dj4gt"
Jul 20 17:58:36.776360 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:36.776345    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4c02e13a-6eb6-4c56-87d0-ea9258d3912f-node-exporter-accelerators-collector-config\") pod \"node-exporter-dj4gt\" (UID: \"4c02e13a-6eb6-4c56-87d0-ea9258d3912f\") " pod="openshift-monitoring/node-exporter-dj4gt"
Jul 20 17:58:36.776478 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:36.776377    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4c02e13a-6eb6-4c56-87d0-ea9258d3912f-sys\") pod \"node-exporter-dj4gt\" (UID: \"4c02e13a-6eb6-4c56-87d0-ea9258d3912f\") " pod="openshift-monitoring/node-exporter-dj4gt"
Jul 20 17:58:36.877577 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:36.877542    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4c02e13a-6eb6-4c56-87d0-ea9258d3912f-node-exporter-textfile\") pod \"node-exporter-dj4gt\" (UID: \"4c02e13a-6eb6-4c56-87d0-ea9258d3912f\") " pod="openshift-monitoring/node-exporter-dj4gt"
Jul 20 17:58:36.877577 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:36.877579    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4c02e13a-6eb6-4c56-87d0-ea9258d3912f-node-exporter-wtmp\") pod \"node-exporter-dj4gt\" (UID: \"4c02e13a-6eb6-4c56-87d0-ea9258d3912f\") " pod="openshift-monitoring/node-exporter-dj4gt"
Jul 20 17:58:36.877790 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:36.877597    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54nbg\" (UniqueName: \"kubernetes.io/projected/4c02e13a-6eb6-4c56-87d0-ea9258d3912f-kube-api-access-54nbg\") pod \"node-exporter-dj4gt\" (UID: \"4c02e13a-6eb6-4c56-87d0-ea9258d3912f\") " pod="openshift-monitoring/node-exporter-dj4gt"
Jul 20 17:58:36.877790 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:36.877628    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4c02e13a-6eb6-4c56-87d0-ea9258d3912f-node-exporter-accelerators-collector-config\") pod \"node-exporter-dj4gt\" (UID: \"4c02e13a-6eb6-4c56-87d0-ea9258d3912f\") " pod="openshift-monitoring/node-exporter-dj4gt"
Jul 20 17:58:36.877790 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:36.877654    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4c02e13a-6eb6-4c56-87d0-ea9258d3912f-sys\") pod \"node-exporter-dj4gt\" (UID: \"4c02e13a-6eb6-4c56-87d0-ea9258d3912f\") " pod="openshift-monitoring/node-exporter-dj4gt"
Jul 20 17:58:36.877790 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:36.877743    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4c02e13a-6eb6-4c56-87d0-ea9258d3912f-root\") pod \"node-exporter-dj4gt\" (UID: \"4c02e13a-6eb6-4c56-87d0-ea9258d3912f\") " pod="openshift-monitoring/node-exporter-dj4gt"
Jul 20 17:58:36.877790 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:36.877775    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4c02e13a-6eb6-4c56-87d0-ea9258d3912f-node-exporter-wtmp\") pod \"node-exporter-dj4gt\" (UID: \"4c02e13a-6eb6-4c56-87d0-ea9258d3912f\") " pod="openshift-monitoring/node-exporter-dj4gt"
Jul 20 17:58:36.877980 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:36.877787    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4c02e13a-6eb6-4c56-87d0-ea9258d3912f-sys\") pod \"node-exporter-dj4gt\" (UID: \"4c02e13a-6eb6-4c56-87d0-ea9258d3912f\") " pod="openshift-monitoring/node-exporter-dj4gt"
Jul 20 17:58:36.877980 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:36.877776    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4c02e13a-6eb6-4c56-87d0-ea9258d3912f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dj4gt\" (UID: \"4c02e13a-6eb6-4c56-87d0-ea9258d3912f\") " pod="openshift-monitoring/node-exporter-dj4gt"
Jul 20 17:58:36.877980 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:36.877841    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4c02e13a-6eb6-4c56-87d0-ea9258d3912f-root\") pod \"node-exporter-dj4gt\" (UID: \"4c02e13a-6eb6-4c56-87d0-ea9258d3912f\") " pod="openshift-monitoring/node-exporter-dj4gt"
Jul 20 17:58:36.877980 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:36.877863    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4c02e13a-6eb6-4c56-87d0-ea9258d3912f-metrics-client-ca\") pod \"node-exporter-dj4gt\" (UID: \"4c02e13a-6eb6-4c56-87d0-ea9258d3912f\") " pod="openshift-monitoring/node-exporter-dj4gt"
Jul 20 17:58:36.877980 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:36.877893    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4c02e13a-6eb6-4c56-87d0-ea9258d3912f-node-exporter-tls\") pod \"node-exporter-dj4gt\" (UID: \"4c02e13a-6eb6-4c56-87d0-ea9258d3912f\") " pod="openshift-monitoring/node-exporter-dj4gt"
Jul 20 17:58:36.878139 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:36.877993    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4c02e13a-6eb6-4c56-87d0-ea9258d3912f-node-exporter-textfile\") pod \"node-exporter-dj4gt\" (UID: \"4c02e13a-6eb6-4c56-87d0-ea9258d3912f\") " pod="openshift-monitoring/node-exporter-dj4gt"
Jul 20 17:58:36.878139 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:58:36.878001    2569 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 20 17:58:36.878139 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:58:36.878069    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c02e13a-6eb6-4c56-87d0-ea9258d3912f-node-exporter-tls podName:4c02e13a-6eb6-4c56-87d0-ea9258d3912f nodeName:}" failed. No retries permitted until 2026-07-20 17:58:37.378047395 +0000 UTC m=+163.624580783 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/4c02e13a-6eb6-4c56-87d0-ea9258d3912f-node-exporter-tls") pod "node-exporter-dj4gt" (UID: "4c02e13a-6eb6-4c56-87d0-ea9258d3912f") : secret "node-exporter-tls" not found
Jul 20 17:58:36.878288 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:36.878266    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4c02e13a-6eb6-4c56-87d0-ea9258d3912f-node-exporter-accelerators-collector-config\") pod \"node-exporter-dj4gt\" (UID: \"4c02e13a-6eb6-4c56-87d0-ea9258d3912f\") " pod="openshift-monitoring/node-exporter-dj4gt"
Jul 20 17:58:36.878325 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:36.878281    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4c02e13a-6eb6-4c56-87d0-ea9258d3912f-metrics-client-ca\") pod \"node-exporter-dj4gt\" (UID: \"4c02e13a-6eb6-4c56-87d0-ea9258d3912f\") " pod="openshift-monitoring/node-exporter-dj4gt"
Jul 20 17:58:36.879975 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:36.879958    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4c02e13a-6eb6-4c56-87d0-ea9258d3912f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dj4gt\" (UID: \"4c02e13a-6eb6-4c56-87d0-ea9258d3912f\") " pod="openshift-monitoring/node-exporter-dj4gt"
Jul 20 17:58:36.887085 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:36.887059    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-54nbg\" (UniqueName: \"kubernetes.io/projected/4c02e13a-6eb6-4c56-87d0-ea9258d3912f-kube-api-access-54nbg\") pod \"node-exporter-dj4gt\" (UID: \"4c02e13a-6eb6-4c56-87d0-ea9258d3912f\") " pod="openshift-monitoring/node-exporter-dj4gt"
Jul 20 17:58:37.080073 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:37.080039    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8357c262-2ca2-4ff5-8b48-70017efdb56b-registry-tls\") pod \"image-registry-794579c464-cq4k2\" (UID: \"8357c262-2ca2-4ff5-8b48-70017efdb56b\") " pod="openshift-image-registry/image-registry-794579c464-cq4k2"
Jul 20 17:58:37.082288 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:37.082266    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8357c262-2ca2-4ff5-8b48-70017efdb56b-registry-tls\") pod \"image-registry-794579c464-cq4k2\" (UID: \"8357c262-2ca2-4ff5-8b48-70017efdb56b\") " pod="openshift-image-registry/image-registry-794579c464-cq4k2"
Jul 20 17:58:37.167168 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:37.167142    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-dl9wc\""
Jul 20 17:58:37.175059 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:37.175037    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-794579c464-cq4k2"
Jul 20 17:58:37.181198 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:37.181178    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bfe61b3e-4aac-4ddd-bde9-10fb1c7e6e07-metrics-tls\") pod \"dns-default-ktc25\" (UID: \"bfe61b3e-4aac-4ddd-bde9-10fb1c7e6e07\") " pod="openshift-dns/dns-default-ktc25"
Jul 20 17:58:37.183368 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:37.183349    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bfe61b3e-4aac-4ddd-bde9-10fb1c7e6e07-metrics-tls\") pod \"dns-default-ktc25\" (UID: \"bfe61b3e-4aac-4ddd-bde9-10fb1c7e6e07\") " pod="openshift-dns/dns-default-ktc25"
Jul 20 17:58:37.303215 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:37.303190    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-794579c464-cq4k2"]
Jul 20 17:58:37.304965 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:58:37.304943    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8357c262_2ca2_4ff5_8b48_70017efdb56b.slice/crio-6ff6e7fdd26e4783f14bd10dfdc1606308bdeb7888c5d56b6b010a9effcfb7e0 WatchSource:0}: Error finding container 6ff6e7fdd26e4783f14bd10dfdc1606308bdeb7888c5d56b6b010a9effcfb7e0: Status 404 returned error can't find the container with id 6ff6e7fdd26e4783f14bd10dfdc1606308bdeb7888c5d56b6b010a9effcfb7e0
Jul 20 17:58:37.382774 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:37.382745    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4c02e13a-6eb6-4c56-87d0-ea9258d3912f-node-exporter-tls\") pod \"node-exporter-dj4gt\" (UID: \"4c02e13a-6eb6-4c56-87d0-ea9258d3912f\") " pod="openshift-monitoring/node-exporter-dj4gt"
Jul 20 17:58:37.384784 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:37.384763    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4c02e13a-6eb6-4c56-87d0-ea9258d3912f-node-exporter-tls\") pod \"node-exporter-dj4gt\" (UID: \"4c02e13a-6eb6-4c56-87d0-ea9258d3912f\") " pod="openshift-monitoring/node-exporter-dj4gt"
Jul 20 17:58:37.468098 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:37.468071    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-m6nkv\""
Jul 20 17:58:37.476271 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:37.476237    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ktc25"
Jul 20 17:58:37.582103 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:37.582073    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-dj4gt"
Jul 20 17:58:37.593304 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:58:37.593276    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c02e13a_6eb6_4c56_87d0_ea9258d3912f.slice/crio-ff0e228a8be930313c7736c58e5098aa41c638c7866d4780c47b9d9c50d89f54 WatchSource:0}: Error finding container ff0e228a8be930313c7736c58e5098aa41c638c7866d4780c47b9d9c50d89f54: Status 404 returned error can't find the container with id ff0e228a8be930313c7736c58e5098aa41c638c7866d4780c47b9d9c50d89f54
Jul 20 17:58:37.606574 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:37.606550    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-ktc25"]
Jul 20 17:58:37.609644 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:58:37.609621    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfe61b3e_4aac_4ddd_bde9_10fb1c7e6e07.slice/crio-bb9148c05a7a6dfd2de69e7416837967f47343434656fbb82e20ceae27b15c28 WatchSource:0}: Error finding container bb9148c05a7a6dfd2de69e7416837967f47343434656fbb82e20ceae27b15c28: Status 404 returned error can't find the container with id bb9148c05a7a6dfd2de69e7416837967f47343434656fbb82e20ceae27b15c28
Jul 20 17:58:37.686038 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:37.685993    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dj4gt" event={"ID":"4c02e13a-6eb6-4c56-87d0-ea9258d3912f","Type":"ContainerStarted","Data":"ff0e228a8be930313c7736c58e5098aa41c638c7866d4780c47b9d9c50d89f54"}
Jul 20 17:58:37.687250 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:37.687226    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ktc25" event={"ID":"bfe61b3e-4aac-4ddd-bde9-10fb1c7e6e07","Type":"ContainerStarted","Data":"bb9148c05a7a6dfd2de69e7416837967f47343434656fbb82e20ceae27b15c28"}
Jul 20 17:58:37.688646 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:37.688625    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-794579c464-cq4k2" event={"ID":"8357c262-2ca2-4ff5-8b48-70017efdb56b","Type":"ContainerStarted","Data":"02aa7637e909cc166a9ddd11f25865e28593089836ae2980c06aeccad181dc52"}
Jul 20 17:58:37.688784 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:37.688658    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-794579c464-cq4k2" event={"ID":"8357c262-2ca2-4ff5-8b48-70017efdb56b","Type":"ContainerStarted","Data":"6ff6e7fdd26e4783f14bd10dfdc1606308bdeb7888c5d56b6b010a9effcfb7e0"}
Jul 20 17:58:37.688784 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:37.688762    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-794579c464-cq4k2"
Jul 20 17:58:37.713684 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:37.713630    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-794579c464-cq4k2" podStartSLOduration=163.713609856 podStartE2EDuration="2m43.713609856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 17:55:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 17:58:37.712747642 +0000 UTC m=+163.959281055" watchObservedRunningTime="2026-07-20 17:58:37.713609856 +0000 UTC m=+163.960143264"
Jul 20 17:58:39.700698 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:39.700656    2569 generic.go:356] "Generic (PLEG): container finished" podID="4c02e13a-6eb6-4c56-87d0-ea9258d3912f" containerID="567ee837126ce080fd52f719d9414456a13051f04d538403eb2fc1f83f774d37" exitCode=0
Jul 20 17:58:39.701122 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:39.700710    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dj4gt" event={"ID":"4c02e13a-6eb6-4c56-87d0-ea9258d3912f","Type":"ContainerDied","Data":"567ee837126ce080fd52f719d9414456a13051f04d538403eb2fc1f83f774d37"}
Jul 20 17:58:40.705471 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:40.705435    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dj4gt" event={"ID":"4c02e13a-6eb6-4c56-87d0-ea9258d3912f","Type":"ContainerStarted","Data":"9828c00fc02c216b3a1b0b072c4cac3625cded67959a3f1235351cb21790a045"}
Jul 20 17:58:40.705941 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:40.705485    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dj4gt" event={"ID":"4c02e13a-6eb6-4c56-87d0-ea9258d3912f","Type":"ContainerStarted","Data":"0af8d48cdb12eadf5ea488d520d95ec9e5ee01a0662f4a31cd3863b7a7a7a7e8"}
Jul 20 17:58:40.707207 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:40.707181    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ktc25" event={"ID":"bfe61b3e-4aac-4ddd-bde9-10fb1c7e6e07","Type":"ContainerStarted","Data":"79be435bae7754912230b22cc92e24273c9adf8fe0f0d3f24473238346117f81"}
Jul 20 17:58:40.707311 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:40.707222    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ktc25" event={"ID":"bfe61b3e-4aac-4ddd-bde9-10fb1c7e6e07","Type":"ContainerStarted","Data":"95aee4f700471df228ad3f5e0849ced8efbe8a4149189a7a7c2d74dea09a8539"}
Jul 20 17:58:40.707311 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:40.707244    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-ktc25"
Jul 20 17:58:40.731457 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:40.731412    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-dj4gt" podStartSLOduration=3.630503081 podStartE2EDuration="4.731398015s" totalImagesPullingTime="1.040133882s" totalInitContainerRuntime="60.761052ms" isStatefulPod=true podCreationTimestamp="2026-07-20 17:58:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 17:58:40.72974121 +0000 UTC m=+166.976274682" watchObservedRunningTime="2026-07-20 17:58:40.731398015 +0000 UTC m=+166.977931423"
Jul 20 17:58:40.751853 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:40.751645    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-ktc25" podStartSLOduration=129.208315619 podStartE2EDuration="2m11.751627501s" totalImagesPullingTime="2.543311882s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 17:56:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 17:58:40.751623452 +0000 UTC m=+166.998156863" watchObservedRunningTime="2026-07-20 17:58:40.751627501 +0000 UTC m=+166.998160912"
Jul 20 17:58:41.080175 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:41.080140    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-664668c958-tt77p"]
Jul 20 17:58:41.083358 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:41.083336    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-664668c958-tt77p"
Jul 20 17:58:41.087029 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:41.087001    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Jul 20 17:58:41.087029 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:41.087025    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Jul 20 17:58:41.087180 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:41.087039    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 20 17:58:41.087180 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:41.087023    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-cnu9u4b1u9a0i\""
Jul 20 17:58:41.087180 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:41.087003    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Jul 20 17:58:41.087180 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:41.087119    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-h67sl\""
Jul 20 17:58:41.097533 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:41.097510    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-664668c958-tt77p"]
Jul 20 17:58:41.215693 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:41.215658    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/89d7700b-5ef7-4b01-8e3f-e9fd631bd59c-audit-log\") pod \"metrics-server-664668c958-tt77p\" (UID: \"89d7700b-5ef7-4b01-8e3f-e9fd631bd59c\") " pod="openshift-monitoring/metrics-server-664668c958-tt77p"
Jul 20 17:58:41.215877 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:41.215701    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcmg4\" (UniqueName: \"kubernetes.io/projected/89d7700b-5ef7-4b01-8e3f-e9fd631bd59c-kube-api-access-qcmg4\") pod \"metrics-server-664668c958-tt77p\" (UID: \"89d7700b-5ef7-4b01-8e3f-e9fd631bd59c\") " pod="openshift-monitoring/metrics-server-664668c958-tt77p"
Jul 20 17:58:41.215877 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:41.215817    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/89d7700b-5ef7-4b01-8e3f-e9fd631bd59c-secret-metrics-server-tls\") pod \"metrics-server-664668c958-tt77p\" (UID: \"89d7700b-5ef7-4b01-8e3f-e9fd631bd59c\") " pod="openshift-monitoring/metrics-server-664668c958-tt77p"
Jul 20 17:58:41.215877 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:41.215851    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/89d7700b-5ef7-4b01-8e3f-e9fd631bd59c-client-ca-bundle\") pod \"metrics-server-664668c958-tt77p\" (UID: \"89d7700b-5ef7-4b01-8e3f-e9fd631bd59c\") " pod="openshift-monitoring/metrics-server-664668c958-tt77p"
Jul 20 17:58:41.215877 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:41.215872    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/89d7700b-5ef7-4b01-8e3f-e9fd631bd59c-secret-metrics-server-client-certs\") pod \"metrics-server-664668c958-tt77p\" (UID: \"89d7700b-5ef7-4b01-8e3f-e9fd631bd59c\") " pod="openshift-monitoring/metrics-server-664668c958-tt77p"
Jul 20 17:58:41.216008 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:41.215912    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/89d7700b-5ef7-4b01-8e3f-e9fd631bd59c-metrics-server-audit-profiles\") pod \"metrics-server-664668c958-tt77p\" (UID: \"89d7700b-5ef7-4b01-8e3f-e9fd631bd59c\") " pod="openshift-monitoring/metrics-server-664668c958-tt77p"
Jul 20 17:58:41.216008 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:41.215967    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/89d7700b-5ef7-4b01-8e3f-e9fd631bd59c-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-664668c958-tt77p\" (UID: \"89d7700b-5ef7-4b01-8e3f-e9fd631bd59c\") " pod="openshift-monitoring/metrics-server-664668c958-tt77p"
Jul 20 17:58:41.316989 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:41.316958    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/89d7700b-5ef7-4b01-8e3f-e9fd631bd59c-audit-log\") pod \"metrics-server-664668c958-tt77p\" (UID: \"89d7700b-5ef7-4b01-8e3f-e9fd631bd59c\") " pod="openshift-monitoring/metrics-server-664668c958-tt77p"
Jul 20 17:58:41.317136 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:41.317000    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcmg4\" (UniqueName: \"kubernetes.io/projected/89d7700b-5ef7-4b01-8e3f-e9fd631bd59c-kube-api-access-qcmg4\") pod \"metrics-server-664668c958-tt77p\" (UID: \"89d7700b-5ef7-4b01-8e3f-e9fd631bd59c\") " pod="openshift-monitoring/metrics-server-664668c958-tt77p"
Jul 20 17:58:41.317136 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:41.317062    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/89d7700b-5ef7-4b01-8e3f-e9fd631bd59c-secret-metrics-server-tls\") pod \"metrics-server-664668c958-tt77p\" (UID: \"89d7700b-5ef7-4b01-8e3f-e9fd631bd59c\") " pod="openshift-monitoring/metrics-server-664668c958-tt77p"
Jul 20 17:58:41.317136 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:41.317090    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/89d7700b-5ef7-4b01-8e3f-e9fd631bd59c-client-ca-bundle\") pod \"metrics-server-664668c958-tt77p\" (UID: \"89d7700b-5ef7-4b01-8e3f-e9fd631bd59c\") " pod="openshift-monitoring/metrics-server-664668c958-tt77p"
Jul 20 17:58:41.317136 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:41.317123    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/89d7700b-5ef7-4b01-8e3f-e9fd631bd59c-secret-metrics-server-client-certs\") pod \"metrics-server-664668c958-tt77p\" (UID: \"89d7700b-5ef7-4b01-8e3f-e9fd631bd59c\") " pod="openshift-monitoring/metrics-server-664668c958-tt77p"
Jul 20 17:58:41.317332 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:41.317151    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/89d7700b-5ef7-4b01-8e3f-e9fd631bd59c-metrics-server-audit-profiles\") pod \"metrics-server-664668c958-tt77p\" (UID: \"89d7700b-5ef7-4b01-8e3f-e9fd631bd59c\") " pod="openshift-monitoring/metrics-server-664668c958-tt77p"
Jul 20 17:58:41.317332 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:41.317184    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/89d7700b-5ef7-4b01-8e3f-e9fd631bd59c-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-664668c958-tt77p\" (UID: \"89d7700b-5ef7-4b01-8e3f-e9fd631bd59c\") " pod="openshift-monitoring/metrics-server-664668c958-tt77p"
Jul 20 17:58:41.317442 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:41.317422    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/89d7700b-5ef7-4b01-8e3f-e9fd631bd59c-audit-log\") pod \"metrics-server-664668c958-tt77p\" (UID: \"89d7700b-5ef7-4b01-8e3f-e9fd631bd59c\") " pod="openshift-monitoring/metrics-server-664668c958-tt77p"
Jul 20 17:58:41.317957 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:41.317933    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/89d7700b-5ef7-4b01-8e3f-e9fd631bd59c-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-664668c958-tt77p\" (UID: \"89d7700b-5ef7-4b01-8e3f-e9fd631bd59c\") " pod="openshift-monitoring/metrics-server-664668c958-tt77p"
Jul 20 17:58:41.318088 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:41.318070    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/89d7700b-5ef7-4b01-8e3f-e9fd631bd59c-metrics-server-audit-profiles\") pod \"metrics-server-664668c958-tt77p\" (UID: \"89d7700b-5ef7-4b01-8e3f-e9fd631bd59c\") " pod="openshift-monitoring/metrics-server-664668c958-tt77p"
Jul 20 17:58:41.319588 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:41.319565    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/89d7700b-5ef7-4b01-8e3f-e9fd631bd59c-client-ca-bundle\") pod \"metrics-server-664668c958-tt77p\" (UID: \"89d7700b-5ef7-4b01-8e3f-e9fd631bd59c\") " pod="openshift-monitoring/metrics-server-664668c958-tt77p"
Jul 20 17:58:41.319953 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:41.319938    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/89d7700b-5ef7-4b01-8e3f-e9fd631bd59c-secret-metrics-server-client-certs\") pod \"metrics-server-664668c958-tt77p\" (UID: \"89d7700b-5ef7-4b01-8e3f-e9fd631bd59c\") " pod="openshift-monitoring/metrics-server-664668c958-tt77p"
Jul 20 17:58:41.320116 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:41.320096    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/89d7700b-5ef7-4b01-8e3f-e9fd631bd59c-secret-metrics-server-tls\") pod \"metrics-server-664668c958-tt77p\" (UID: \"89d7700b-5ef7-4b01-8e3f-e9fd631bd59c\") " pod="openshift-monitoring/metrics-server-664668c958-tt77p"
Jul 20 17:58:41.331307 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:41.331234    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcmg4\" (UniqueName: \"kubernetes.io/projected/89d7700b-5ef7-4b01-8e3f-e9fd631bd59c-kube-api-access-qcmg4\") pod \"metrics-server-664668c958-tt77p\" (UID: \"89d7700b-5ef7-4b01-8e3f-e9fd631bd59c\") " pod="openshift-monitoring/metrics-server-664668c958-tt77p"
Jul 20 17:58:41.392435 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:41.392403    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-664668c958-tt77p"
Jul 20 17:58:41.533503 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:41.533474    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-664668c958-tt77p"]
Jul 20 17:58:41.536716 ip-10-0-137-247 kubenswrapper[2569]: W0720 17:58:41.536689    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89d7700b_5ef7_4b01_8e3f_e9fd631bd59c.slice/crio-6ca0a1cda1aea9001cb865c2302ab895dd86c5379c88227db007a1a0233277eb WatchSource:0}: Error finding container 6ca0a1cda1aea9001cb865c2302ab895dd86c5379c88227db007a1a0233277eb: Status 404 returned error can't find the container with id 6ca0a1cda1aea9001cb865c2302ab895dd86c5379c88227db007a1a0233277eb
Jul 20 17:58:41.711630 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:41.711540    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-664668c958-tt77p" event={"ID":"89d7700b-5ef7-4b01-8e3f-e9fd631bd59c","Type":"ContainerStarted","Data":"6ca0a1cda1aea9001cb865c2302ab895dd86c5379c88227db007a1a0233277eb"}
Jul 20 17:58:43.721802 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:43.721766    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-664668c958-tt77p" event={"ID":"89d7700b-5ef7-4b01-8e3f-e9fd631bd59c","Type":"ContainerStarted","Data":"e2789225b7c481e08d0f70e7811d7a425db9c5354c1e8047083ab0f1d9721390"}
Jul 20 17:58:43.739212 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:43.739101    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-664668c958-tt77p" podStartSLOduration=1.071781556 podStartE2EDuration="2.739084653s" totalImagesPullingTime="1.667303097s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 17:58:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 17:58:43.73858079 +0000 UTC m=+169.985114199" watchObservedRunningTime="2026-07-20 17:58:43.739084653 +0000 UTC m=+169.985618064"
Jul 20 17:58:46.176801 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:46.176768    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nkmr4"
Jul 20 17:58:50.713634 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:50.713596    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-ktc25"
Jul 20 17:58:53.998460 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:53.998421    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-794579c464-cq4k2"]
Jul 20 17:58:54.002697 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:54.002666    2569 patch_prober.go:28] interesting pod/image-registry-794579c464-cq4k2 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 20 17:58:54.002829 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:58:54.002714    2569 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-794579c464-cq4k2" podUID="8357c262-2ca2-4ff5-8b48-70017efdb56b" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 20 17:59:01.392830 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:59:01.392780    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-664668c958-tt77p"
Jul 20 17:59:01.393221 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:59:01.392851    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-664668c958-tt77p"
Jul 20 17:59:04.002824 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:59:04.002795    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-794579c464-cq4k2"
Jul 20 17:59:06.057310 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:59:06.057282    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-ktc25_bfe61b3e-4aac-4ddd-bde9-10fb1c7e6e07/dns/0.log"
Jul 20 17:59:06.063985 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:59:06.063963    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-ktc25_bfe61b3e-4aac-4ddd-bde9-10fb1c7e6e07/kube-rbac-proxy/0.log"
Jul 20 17:59:06.590238 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:59:06.590212    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-smcqb_ea740a69-374a-483b-a045-a2c795b99d4c/dns-node-resolver/0.log"
Jul 20 17:59:19.018131 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:59:19.018072    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-794579c464-cq4k2" podUID="8357c262-2ca2-4ff5-8b48-70017efdb56b" containerName="registry" containerID="cri-o://02aa7637e909cc166a9ddd11f25865e28593089836ae2980c06aeccad181dc52" gracePeriod=30
Jul 20 17:59:19.269299 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:59:19.269240    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-794579c464-cq4k2"
Jul 20 17:59:19.316587 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:59:19.316558    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8357c262-2ca2-4ff5-8b48-70017efdb56b-registry-tls\") pod \"8357c262-2ca2-4ff5-8b48-70017efdb56b\" (UID: \"8357c262-2ca2-4ff5-8b48-70017efdb56b\") "
Jul 20 17:59:19.316761 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:59:19.316613    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8357c262-2ca2-4ff5-8b48-70017efdb56b-bound-sa-token\") pod \"8357c262-2ca2-4ff5-8b48-70017efdb56b\" (UID: \"8357c262-2ca2-4ff5-8b48-70017efdb56b\") "
Jul 20 17:59:19.316761 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:59:19.316649    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sjzfw\" (UniqueName: \"kubernetes.io/projected/8357c262-2ca2-4ff5-8b48-70017efdb56b-kube-api-access-sjzfw\") pod \"8357c262-2ca2-4ff5-8b48-70017efdb56b\" (UID: \"8357c262-2ca2-4ff5-8b48-70017efdb56b\") "
Jul 20 17:59:19.316761 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:59:19.316683    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8357c262-2ca2-4ff5-8b48-70017efdb56b-image-registry-private-configuration\") pod \"8357c262-2ca2-4ff5-8b48-70017efdb56b\" (UID: \"8357c262-2ca2-4ff5-8b48-70017efdb56b\") "
Jul 20 17:59:19.316761 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:59:19.316757    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8357c262-2ca2-4ff5-8b48-70017efdb56b-ca-trust-extracted\") pod \"8357c262-2ca2-4ff5-8b48-70017efdb56b\" (UID: \"8357c262-2ca2-4ff5-8b48-70017efdb56b\") "
Jul 20 17:59:19.316983 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:59:19.316810    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8357c262-2ca2-4ff5-8b48-70017efdb56b-installation-pull-secrets\") pod \"8357c262-2ca2-4ff5-8b48-70017efdb56b\" (UID: \"8357c262-2ca2-4ff5-8b48-70017efdb56b\") "
Jul 20 17:59:19.316983 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:59:19.316836    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8357c262-2ca2-4ff5-8b48-70017efdb56b-trusted-ca\") pod \"8357c262-2ca2-4ff5-8b48-70017efdb56b\" (UID: \"8357c262-2ca2-4ff5-8b48-70017efdb56b\") "
Jul 20 17:59:19.316983 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:59:19.316875    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8357c262-2ca2-4ff5-8b48-70017efdb56b-registry-certificates\") pod \"8357c262-2ca2-4ff5-8b48-70017efdb56b\" (UID: \"8357c262-2ca2-4ff5-8b48-70017efdb56b\") "
Jul 20 17:59:19.317626 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:59:19.317457    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8357c262-2ca2-4ff5-8b48-70017efdb56b-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "8357c262-2ca2-4ff5-8b48-70017efdb56b" (UID: "8357c262-2ca2-4ff5-8b48-70017efdb56b"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 17:59:19.318613 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:59:19.318577    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8357c262-2ca2-4ff5-8b48-70017efdb56b-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "8357c262-2ca2-4ff5-8b48-70017efdb56b" (UID: "8357c262-2ca2-4ff5-8b48-70017efdb56b"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 17:59:19.319574 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:59:19.319530    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8357c262-2ca2-4ff5-8b48-70017efdb56b-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "8357c262-2ca2-4ff5-8b48-70017efdb56b" (UID: "8357c262-2ca2-4ff5-8b48-70017efdb56b"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 17:59:19.319574 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:59:19.319544    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8357c262-2ca2-4ff5-8b48-70017efdb56b-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "8357c262-2ca2-4ff5-8b48-70017efdb56b" (UID: "8357c262-2ca2-4ff5-8b48-70017efdb56b"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 17:59:19.319751 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:59:19.319642    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8357c262-2ca2-4ff5-8b48-70017efdb56b-kube-api-access-sjzfw" (OuterVolumeSpecName: "kube-api-access-sjzfw") pod "8357c262-2ca2-4ff5-8b48-70017efdb56b" (UID: "8357c262-2ca2-4ff5-8b48-70017efdb56b"). InnerVolumeSpecName "kube-api-access-sjzfw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 17:59:19.319906 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:59:19.319878    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8357c262-2ca2-4ff5-8b48-70017efdb56b-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "8357c262-2ca2-4ff5-8b48-70017efdb56b" (UID: "8357c262-2ca2-4ff5-8b48-70017efdb56b"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 17:59:19.319994 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:59:19.319941    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8357c262-2ca2-4ff5-8b48-70017efdb56b-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "8357c262-2ca2-4ff5-8b48-70017efdb56b" (UID: "8357c262-2ca2-4ff5-8b48-70017efdb56b"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 17:59:19.326025 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:59:19.325999    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8357c262-2ca2-4ff5-8b48-70017efdb56b-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "8357c262-2ca2-4ff5-8b48-70017efdb56b" (UID: "8357c262-2ca2-4ff5-8b48-70017efdb56b"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 17:59:19.417373 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:59:19.417341    2569 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8357c262-2ca2-4ff5-8b48-70017efdb56b-ca-trust-extracted\") on node \"ip-10-0-137-247.ec2.internal\" DevicePath \"\""
Jul 20 17:59:19.417373 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:59:19.417368    2569 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8357c262-2ca2-4ff5-8b48-70017efdb56b-installation-pull-secrets\") on node \"ip-10-0-137-247.ec2.internal\" DevicePath \"\""
Jul 20 17:59:19.417373 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:59:19.417379    2569 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8357c262-2ca2-4ff5-8b48-70017efdb56b-trusted-ca\") on node \"ip-10-0-137-247.ec2.internal\" DevicePath \"\""
Jul 20 17:59:19.417582 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:59:19.417387    2569 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8357c262-2ca2-4ff5-8b48-70017efdb56b-registry-certificates\") on node \"ip-10-0-137-247.ec2.internal\" DevicePath \"\""
Jul 20 17:59:19.417582 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:59:19.417396    2569 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8357c262-2ca2-4ff5-8b48-70017efdb56b-registry-tls\") on node \"ip-10-0-137-247.ec2.internal\" DevicePath \"\""
Jul 20 17:59:19.417582 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:59:19.417404    2569 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8357c262-2ca2-4ff5-8b48-70017efdb56b-bound-sa-token\") on node \"ip-10-0-137-247.ec2.internal\" DevicePath \"\""
Jul 20 17:59:19.417582 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:59:19.417412    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjzfw\" (UniqueName: \"kubernetes.io/projected/8357c262-2ca2-4ff5-8b48-70017efdb56b-kube-api-access-sjzfw\") on node \"ip-10-0-137-247.ec2.internal\" DevicePath \"\""
Jul 20 17:59:19.417582 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:59:19.417420    2569 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8357c262-2ca2-4ff5-8b48-70017efdb56b-image-registry-private-configuration\") on node \"ip-10-0-137-247.ec2.internal\" DevicePath \"\""
Jul 20 17:59:19.843769 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:59:19.843709    2569 generic.go:356] "Generic (PLEG): container finished" podID="8357c262-2ca2-4ff5-8b48-70017efdb56b" containerID="02aa7637e909cc166a9ddd11f25865e28593089836ae2980c06aeccad181dc52" exitCode=0
Jul 20 17:59:19.843954 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:59:19.843778    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-794579c464-cq4k2"
Jul 20 17:59:19.843954 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:59:19.843799    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-794579c464-cq4k2" event={"ID":"8357c262-2ca2-4ff5-8b48-70017efdb56b","Type":"ContainerDied","Data":"02aa7637e909cc166a9ddd11f25865e28593089836ae2980c06aeccad181dc52"}
Jul 20 17:59:19.843954 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:59:19.843863    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-794579c464-cq4k2" event={"ID":"8357c262-2ca2-4ff5-8b48-70017efdb56b","Type":"ContainerDied","Data":"6ff6e7fdd26e4783f14bd10dfdc1606308bdeb7888c5d56b6b010a9effcfb7e0"}
Jul 20 17:59:19.843954 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:59:19.843904    2569 scope.go:117] "RemoveContainer" containerID="02aa7637e909cc166a9ddd11f25865e28593089836ae2980c06aeccad181dc52"
Jul 20 17:59:19.853818 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:59:19.852897    2569 scope.go:117] "RemoveContainer" containerID="02aa7637e909cc166a9ddd11f25865e28593089836ae2980c06aeccad181dc52"
Jul 20 17:59:19.853818 ip-10-0-137-247 kubenswrapper[2569]: E0720 17:59:19.853770    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02aa7637e909cc166a9ddd11f25865e28593089836ae2980c06aeccad181dc52\": container with ID starting with 02aa7637e909cc166a9ddd11f25865e28593089836ae2980c06aeccad181dc52 not found: ID does not exist" containerID="02aa7637e909cc166a9ddd11f25865e28593089836ae2980c06aeccad181dc52"
Jul 20 17:59:19.853929 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:59:19.853797    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02aa7637e909cc166a9ddd11f25865e28593089836ae2980c06aeccad181dc52"} err="failed to get container status \"02aa7637e909cc166a9ddd11f25865e28593089836ae2980c06aeccad181dc52\": rpc error: code = NotFound desc = could not find container \"02aa7637e909cc166a9ddd11f25865e28593089836ae2980c06aeccad181dc52\": container with ID starting with 02aa7637e909cc166a9ddd11f25865e28593089836ae2980c06aeccad181dc52 not found: ID does not exist"
Jul 20 17:59:19.873553 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:59:19.873530    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-794579c464-cq4k2"]
Jul 20 17:59:19.877498 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:59:19.877478    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-794579c464-cq4k2"]
Jul 20 17:59:20.181550 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:59:20.181477    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8357c262-2ca2-4ff5-8b48-70017efdb56b" path="/var/lib/kubelet/pods/8357c262-2ca2-4ff5-8b48-70017efdb56b/volumes"
Jul 20 17:59:21.398253 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:59:21.398226    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-664668c958-tt77p"
Jul 20 17:59:21.401888 ip-10-0-137-247 kubenswrapper[2569]: I0720 17:59:21.401865    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-664668c958-tt77p"
Jul 20 18:00:05.973185 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:00:05.973127    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26b0d75c-ecf7-4bb4-ad4b-d9077da808bd-metrics-certs\") pod \"network-metrics-daemon-nkmr4\" (UID: \"26b0d75c-ecf7-4bb4-ad4b-d9077da808bd\") " pod="openshift-multus/network-metrics-daemon-nkmr4"
Jul 20 18:00:05.975426 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:00:05.975405    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26b0d75c-ecf7-4bb4-ad4b-d9077da808bd-metrics-certs\") pod \"network-metrics-daemon-nkmr4\" (UID: \"26b0d75c-ecf7-4bb4-ad4b-d9077da808bd\") " pod="openshift-multus/network-metrics-daemon-nkmr4"
Jul 20 18:00:05.979966 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:00:05.979942    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-nnhlg\""
Jul 20 18:00:05.987455 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:00:05.987441    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nkmr4"
Jul 20 18:00:06.108812 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:00:06.108782    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-nkmr4"]
Jul 20 18:00:06.112058 ip-10-0-137-247 kubenswrapper[2569]: W0720 18:00:06.112033    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26b0d75c_ecf7_4bb4_ad4b_d9077da808bd.slice/crio-460255d11b47dda93d6187cbf55ace9689b436fb51b79591f3802bb834c91bed WatchSource:0}: Error finding container 460255d11b47dda93d6187cbf55ace9689b436fb51b79591f3802bb834c91bed: Status 404 returned error can't find the container with id 460255d11b47dda93d6187cbf55ace9689b436fb51b79591f3802bb834c91bed
Jul 20 18:00:06.996269 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:00:06.996227    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nkmr4" event={"ID":"26b0d75c-ecf7-4bb4-ad4b-d9077da808bd","Type":"ContainerStarted","Data":"460255d11b47dda93d6187cbf55ace9689b436fb51b79591f3802bb834c91bed"}
Jul 20 18:00:08.001244 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:00:08.001209    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nkmr4" event={"ID":"26b0d75c-ecf7-4bb4-ad4b-d9077da808bd","Type":"ContainerStarted","Data":"6c497507f6e8b64053cb81652a733db1cfa64c6bf8bcee4950c8b614770552b4"}
Jul 20 18:00:08.001635 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:00:08.001263    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nkmr4" event={"ID":"26b0d75c-ecf7-4bb4-ad4b-d9077da808bd","Type":"ContainerStarted","Data":"be60789b1f172d73c2d8080ab42bd645918e022f52f1f16010f03937e3a1118c"}
Jul 20 18:00:08.017901 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:00:08.017855    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-nkmr4" podStartSLOduration=252.71856726 podStartE2EDuration="4m14.017841087s" totalImagesPullingTime="1.299273827s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 17:55:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 18:00:08.017075841 +0000 UTC m=+254.263609252" watchObservedRunningTime="2026-07-20 18:00:08.017841087 +0000 UTC m=+254.264374497"
Jul 20 18:00:54.073238 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:00:54.073209    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wjs8g_07587a78-2d3a-40ad-9aae-a4875200263a/ovn-acl-logging/0.log"
Jul 20 18:00:54.075772 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:00:54.075748    2569 kubelet.go:1648] "Image garbage collection succeeded"
Jul 20 18:00:54.077872 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:00:54.077855    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wjs8g_07587a78-2d3a-40ad-9aae-a4875200263a/ovn-acl-logging/0.log"
Jul 20 18:02:16.766833 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:02:16.766794    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-z4jb5"]
Jul 20 18:02:16.767441 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:02:16.767157    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8357c262-2ca2-4ff5-8b48-70017efdb56b" containerName="registry"
Jul 20 18:02:16.767441 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:02:16.767175    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="8357c262-2ca2-4ff5-8b48-70017efdb56b" containerName="registry"
Jul 20 18:02:16.767441 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:02:16.767231    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="8357c262-2ca2-4ff5-8b48-70017efdb56b" containerName="registry"
Jul 20 18:02:16.769971 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:02:16.769949    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-z4jb5"
Jul 20 18:02:16.772768 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:02:16.772747    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 20 18:02:16.772981 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:02:16.772963    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-jh4vw\""
Jul 20 18:02:16.773581 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:02:16.773563    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 20 18:02:16.779426 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:02:16.779401    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-z4jb5"]
Jul 20 18:02:16.879574 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:02:16.879541    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhkw9\" (UniqueName: \"kubernetes.io/projected/219cdb3c-e9f0-4f47-810c-c7ba57538804-kube-api-access-qhkw9\") pod \"cert-manager-webhook-6486ffd997-z4jb5\" (UID: \"219cdb3c-e9f0-4f47-810c-c7ba57538804\") " pod="cert-manager/cert-manager-webhook-6486ffd997-z4jb5"
Jul 20 18:02:16.879764 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:02:16.879589    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/219cdb3c-e9f0-4f47-810c-c7ba57538804-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-z4jb5\" (UID: \"219cdb3c-e9f0-4f47-810c-c7ba57538804\") " pod="cert-manager/cert-manager-webhook-6486ffd997-z4jb5"
Jul 20 18:02:16.979962 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:02:16.979931    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhkw9\" (UniqueName: \"kubernetes.io/projected/219cdb3c-e9f0-4f47-810c-c7ba57538804-kube-api-access-qhkw9\") pod \"cert-manager-webhook-6486ffd997-z4jb5\" (UID: \"219cdb3c-e9f0-4f47-810c-c7ba57538804\") " pod="cert-manager/cert-manager-webhook-6486ffd997-z4jb5"
Jul 20 18:02:16.980062 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:02:16.979982    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/219cdb3c-e9f0-4f47-810c-c7ba57538804-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-z4jb5\" (UID: \"219cdb3c-e9f0-4f47-810c-c7ba57538804\") " pod="cert-manager/cert-manager-webhook-6486ffd997-z4jb5"
Jul 20 18:02:16.988414 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:02:16.988388    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/219cdb3c-e9f0-4f47-810c-c7ba57538804-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-z4jb5\" (UID: \"219cdb3c-e9f0-4f47-810c-c7ba57538804\") " pod="cert-manager/cert-manager-webhook-6486ffd997-z4jb5"
Jul 20 18:02:16.988525 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:02:16.988443    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhkw9\" (UniqueName: \"kubernetes.io/projected/219cdb3c-e9f0-4f47-810c-c7ba57538804-kube-api-access-qhkw9\") pod \"cert-manager-webhook-6486ffd997-z4jb5\" (UID: \"219cdb3c-e9f0-4f47-810c-c7ba57538804\") " pod="cert-manager/cert-manager-webhook-6486ffd997-z4jb5"
Jul 20 18:02:17.079340 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:02:17.079267    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-z4jb5"
Jul 20 18:02:17.143416 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:02:17.143383    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-nlcl8"]
Jul 20 18:02:17.146605 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:02:17.146584    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-nlcl8"
Jul 20 18:02:17.149104 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:02:17.149051    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 20 18:02:17.149104 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:02:17.149075    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-dqc7t\""
Jul 20 18:02:17.149370 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:02:17.149344    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 20 18:02:17.156157 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:02:17.156116    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-nlcl8"]
Jul 20 18:02:17.223831 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:02:17.223800    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-z4jb5"]
Jul 20 18:02:17.228406 ip-10-0-137-247 kubenswrapper[2569]: W0720 18:02:17.228375    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod219cdb3c_e9f0_4f47_810c_c7ba57538804.slice/crio-c1abd307975fbf20f80ef25993349cf523163345236e7b77798bc5214a3a92e1 WatchSource:0}: Error finding container c1abd307975fbf20f80ef25993349cf523163345236e7b77798bc5214a3a92e1: Status 404 returned error can't find the container with id c1abd307975fbf20f80ef25993349cf523163345236e7b77798bc5214a3a92e1
Jul 20 18:02:17.230441 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:02:17.230422    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 20 18:02:17.282472 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:02:17.282446    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b32cfd04-a0db-4334-9ff6-5136f1ab4f39-tmp\") pod \"openshift-lws-operator-fd8ccff4c-nlcl8\" (UID: \"b32cfd04-a0db-4334-9ff6-5136f1ab4f39\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-nlcl8"
Jul 20 18:02:17.282607 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:02:17.282485    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kr9p5\" (UniqueName: \"kubernetes.io/projected/b32cfd04-a0db-4334-9ff6-5136f1ab4f39-kube-api-access-kr9p5\") pod \"openshift-lws-operator-fd8ccff4c-nlcl8\" (UID: \"b32cfd04-a0db-4334-9ff6-5136f1ab4f39\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-nlcl8"
Jul 20 18:02:17.383784 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:02:17.383684    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kr9p5\" (UniqueName: \"kubernetes.io/projected/b32cfd04-a0db-4334-9ff6-5136f1ab4f39-kube-api-access-kr9p5\") pod \"openshift-lws-operator-fd8ccff4c-nlcl8\" (UID: \"b32cfd04-a0db-4334-9ff6-5136f1ab4f39\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-nlcl8"
Jul 20 18:02:17.383784 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:02:17.383776    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b32cfd04-a0db-4334-9ff6-5136f1ab4f39-tmp\") pod \"openshift-lws-operator-fd8ccff4c-nlcl8\" (UID: \"b32cfd04-a0db-4334-9ff6-5136f1ab4f39\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-nlcl8"
Jul 20 18:02:17.384081 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:02:17.384065    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b32cfd04-a0db-4334-9ff6-5136f1ab4f39-tmp\") pod \"openshift-lws-operator-fd8ccff4c-nlcl8\" (UID: \"b32cfd04-a0db-4334-9ff6-5136f1ab4f39\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-nlcl8"
Jul 20 18:02:17.396841 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:02:17.396818    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kr9p5\" (UniqueName: \"kubernetes.io/projected/b32cfd04-a0db-4334-9ff6-5136f1ab4f39-kube-api-access-kr9p5\") pod \"openshift-lws-operator-fd8ccff4c-nlcl8\" (UID: \"b32cfd04-a0db-4334-9ff6-5136f1ab4f39\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-nlcl8"
Jul 20 18:02:17.414374 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:02:17.414338    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-z4jb5" event={"ID":"219cdb3c-e9f0-4f47-810c-c7ba57538804","Type":"ContainerStarted","Data":"c1abd307975fbf20f80ef25993349cf523163345236e7b77798bc5214a3a92e1"}
Jul 20 18:02:17.456625 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:02:17.456606    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-nlcl8"
Jul 20 18:02:17.577095 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:02:17.577064    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-nlcl8"]
Jul 20 18:02:17.580246 ip-10-0-137-247 kubenswrapper[2569]: W0720 18:02:17.580215    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb32cfd04_a0db_4334_9ff6_5136f1ab4f39.slice/crio-cb43000fd0221246d7429454da0f05dff3c805911390345facb86c9444b2f425 WatchSource:0}: Error finding container cb43000fd0221246d7429454da0f05dff3c805911390345facb86c9444b2f425: Status 404 returned error can't find the container with id cb43000fd0221246d7429454da0f05dff3c805911390345facb86c9444b2f425
Jul 20 18:02:18.419372 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:02:18.419342    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-nlcl8" event={"ID":"b32cfd04-a0db-4334-9ff6-5136f1ab4f39","Type":"ContainerStarted","Data":"cb43000fd0221246d7429454da0f05dff3c805911390345facb86c9444b2f425"}
Jul 20 18:02:20.429274 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:02:20.429187    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-z4jb5" event={"ID":"219cdb3c-e9f0-4f47-810c-c7ba57538804","Type":"ContainerStarted","Data":"f1facc79f404f4d884a12f6505d04a2e332824f280656b85126838c6f0af5741"}
Jul 20 18:02:20.429709 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:02:20.429304    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-z4jb5"
Jul 20 18:02:20.447998 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:02:20.447947    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-z4jb5" podStartSLOduration=1.598159527 podStartE2EDuration="4.447926772s" totalImagesPullingTime="2.849767245s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 18:02:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 18:02:20.446148861 +0000 UTC m=+386.692682280" watchObservedRunningTime="2026-07-20 18:02:20.447926772 +0000 UTC m=+386.694460183"
Jul 20 18:02:22.438009 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:02:22.437978    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-nlcl8" event={"ID":"b32cfd04-a0db-4334-9ff6-5136f1ab4f39","Type":"ContainerStarted","Data":"20cfc453731187731f7ce82f5ba6a02477825d745d31491c6e35ce9d72bba85e"}
Jul 20 18:02:22.470300 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:02:22.470238    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-nlcl8" podStartSLOduration=1.115012831 podStartE2EDuration="5.470217399s" totalImagesPullingTime="4.355204568s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 18:02:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 18:02:22.469146194 +0000 UTC m=+388.715679603" watchObservedRunningTime="2026-07-20 18:02:22.470217399 +0000 UTC m=+388.716750812"
Jul 20 18:02:26.434770 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:02:26.434716    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-z4jb5"
Jul 20 18:02:38.105125 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:02:38.105080    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-79747b4d4c-cks45"]
Jul 20 18:02:38.108433 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:02:38.108410    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-79747b4d4c-cks45"
Jul 20 18:02:38.119949 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:02:38.119923    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 20 18:02:38.120090 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:02:38.120072    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 20 18:02:38.120651 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:02:38.120636    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 20 18:02:38.122047 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:02:38.122032    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 20 18:02:38.122196 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:02:38.122181    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-4q2sn\""
Jul 20 18:02:38.154090 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:02:38.154065    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-79747b4d4c-cks45"]
Jul 20 18:02:38.225084 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:02:38.225052    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6caac6ef-ec22-491d-8b82-41665d8ca54c-tmp\") pod \"opendatahub-operator-controller-manager-79747b4d4c-cks45\" (UID: \"6caac6ef-ec22-491d-8b82-41665d8ca54c\") " pod="opendatahub/opendatahub-operator-controller-manager-79747b4d4c-cks45"
Jul 20 18:02:38.225084 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:02:38.225085    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6caac6ef-ec22-491d-8b82-41665d8ca54c-webhook-cert\") pod \"opendatahub-operator-controller-manager-79747b4d4c-cks45\" (UID: \"6caac6ef-ec22-491d-8b82-41665d8ca54c\") " pod="opendatahub/opendatahub-operator-controller-manager-79747b4d4c-cks45"
Jul 20 18:02:38.225308 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:02:38.225114    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cl2vp\" (UniqueName: \"kubernetes.io/projected/6caac6ef-ec22-491d-8b82-41665d8ca54c-kube-api-access-cl2vp\") pod \"opendatahub-operator-controller-manager-79747b4d4c-cks45\" (UID: \"6caac6ef-ec22-491d-8b82-41665d8ca54c\") " pod="opendatahub/opendatahub-operator-controller-manager-79747b4d4c-cks45"
Jul 20 18:02:38.225308 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:02:38.225207    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6caac6ef-ec22-491d-8b82-41665d8ca54c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-79747b4d4c-cks45\" (UID: \"6caac6ef-ec22-491d-8b82-41665d8ca54c\") " pod="opendatahub/opendatahub-operator-controller-manager-79747b4d4c-cks45"
Jul 20 18:02:38.225308 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:02:38.225234    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6caac6ef-ec22-491d-8b82-41665d8ca54c-manifests\") pod \"opendatahub-operator-controller-manager-79747b4d4c-cks45\" (UID: \"6caac6ef-ec22-491d-8b82-41665d8ca54c\") " pod="opendatahub/opendatahub-operator-controller-manager-79747b4d4c-cks45"
Jul 20 18:02:38.326336 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:02:38.326303    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6caac6ef-ec22-491d-8b82-41665d8ca54c-tmp\") pod \"opendatahub-operator-controller-manager-79747b4d4c-cks45\" (UID: \"6caac6ef-ec22-491d-8b82-41665d8ca54c\") " pod="opendatahub/opendatahub-operator-controller-manager-79747b4d4c-cks45"
Jul 20 18:02:38.326336 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:02:38.326334    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6caac6ef-ec22-491d-8b82-41665d8ca54c-webhook-cert\") pod \"opendatahub-operator-controller-manager-79747b4d4c-cks45\" (UID: \"6caac6ef-ec22-491d-8b82-41665d8ca54c\") " pod="opendatahub/opendatahub-operator-controller-manager-79747b4d4c-cks45"
Jul 20 18:02:38.326537 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:02:38.326354    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cl2vp\" (UniqueName: \"kubernetes.io/projected/6caac6ef-ec22-491d-8b82-41665d8ca54c-kube-api-access-cl2vp\") pod \"opendatahub-operator-controller-manager-79747b4d4c-cks45\" (UID: \"6caac6ef-ec22-491d-8b82-41665d8ca54c\") " pod="opendatahub/opendatahub-operator-controller-manager-79747b4d4c-cks45"
Jul 20 18:02:38.326537 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:02:38.326385    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6caac6ef-ec22-491d-8b82-41665d8ca54c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-79747b4d4c-cks45\" (UID: \"6caac6ef-ec22-491d-8b82-41665d8ca54c\") " pod="opendatahub/opendatahub-operator-controller-manager-79747b4d4c-cks45"
Jul 20 18:02:38.326537 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:02:38.326409    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6caac6ef-ec22-491d-8b82-41665d8ca54c-manifests\") pod \"opendatahub-operator-controller-manager-79747b4d4c-cks45\" (UID: \"6caac6ef-ec22-491d-8b82-41665d8ca54c\") " pod="opendatahub/opendatahub-operator-controller-manager-79747b4d4c-cks45"
Jul 20 18:02:38.326909 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:02:38.326869    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6caac6ef-ec22-491d-8b82-41665d8ca54c-manifests\") pod \"opendatahub-operator-controller-manager-79747b4d4c-cks45\" (UID: \"6caac6ef-ec22-491d-8b82-41665d8ca54c\") " pod="opendatahub/opendatahub-operator-controller-manager-79747b4d4c-cks45"
Jul 20 18:02:38.328628 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:02:38.328609    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6caac6ef-ec22-491d-8b82-41665d8ca54c-tmp\") pod \"opendatahub-operator-controller-manager-79747b4d4c-cks45\" (UID: \"6caac6ef-ec22-491d-8b82-41665d8ca54c\") " pod="opendatahub/opendatahub-operator-controller-manager-79747b4d4c-cks45"
Jul 20 18:02:38.328877 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:02:38.328854    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6caac6ef-ec22-491d-8b82-41665d8ca54c-webhook-cert\") pod \"opendatahub-operator-controller-manager-79747b4d4c-cks45\" (UID: \"6caac6ef-ec22-491d-8b82-41665d8ca54c\") " pod="opendatahub/opendatahub-operator-controller-manager-79747b4d4c-cks45"
Jul 20 18:02:38.328961 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:02:38.328945    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6caac6ef-ec22-491d-8b82-41665d8ca54c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-79747b4d4c-cks45\" (UID: \"6caac6ef-ec22-491d-8b82-41665d8ca54c\") " pod="opendatahub/opendatahub-operator-controller-manager-79747b4d4c-cks45"
Jul 20 18:02:38.336909 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:02:38.336882    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cl2vp\" (UniqueName: \"kubernetes.io/projected/6caac6ef-ec22-491d-8b82-41665d8ca54c-kube-api-access-cl2vp\") pod \"opendatahub-operator-controller-manager-79747b4d4c-cks45\" (UID: \"6caac6ef-ec22-491d-8b82-41665d8ca54c\") " pod="opendatahub/opendatahub-operator-controller-manager-79747b4d4c-cks45"
Jul 20 18:02:38.417768 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:02:38.417683    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-79747b4d4c-cks45"
Jul 20 18:02:38.550932 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:02:38.550883    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-79747b4d4c-cks45"]
Jul 20 18:02:38.554807 ip-10-0-137-247 kubenswrapper[2569]: W0720 18:02:38.554779    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6caac6ef_ec22_491d_8b82_41665d8ca54c.slice/crio-920d901c157ca525383c6bf99e572eecdec7e29ff9ddbdb78c377e92d0c5a860 WatchSource:0}: Error finding container 920d901c157ca525383c6bf99e572eecdec7e29ff9ddbdb78c377e92d0c5a860: Status 404 returned error can't find the container with id 920d901c157ca525383c6bf99e572eecdec7e29ff9ddbdb78c377e92d0c5a860
Jul 20 18:02:39.495344 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:02:39.495275    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-79747b4d4c-cks45" event={"ID":"6caac6ef-ec22-491d-8b82-41665d8ca54c","Type":"ContainerStarted","Data":"920d901c157ca525383c6bf99e572eecdec7e29ff9ddbdb78c377e92d0c5a860"}
Jul 20 18:02:42.508135 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:02:42.508091    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-79747b4d4c-cks45" event={"ID":"6caac6ef-ec22-491d-8b82-41665d8ca54c","Type":"ContainerStarted","Data":"bbbaab569bdb7274a43ee87ff837bcab49752b245d5f4b631578aafd688f6407"}
Jul 20 18:02:43.513261 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:02:43.513224    2569 generic.go:356] "Generic (PLEG): container finished" podID="6caac6ef-ec22-491d-8b82-41665d8ca54c" containerID="bbbaab569bdb7274a43ee87ff837bcab49752b245d5f4b631578aafd688f6407" exitCode=0
Jul 20 18:02:43.513261 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:02:43.513269    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-79747b4d4c-cks45" event={"ID":"6caac6ef-ec22-491d-8b82-41665d8ca54c","Type":"ContainerDied","Data":"bbbaab569bdb7274a43ee87ff837bcab49752b245d5f4b631578aafd688f6407"}
Jul 20 18:02:44.519154 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:02:44.519117    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-79747b4d4c-cks45" event={"ID":"6caac6ef-ec22-491d-8b82-41665d8ca54c","Type":"ContainerStarted","Data":"a72f956d2f3a96c9b3ddc92f3b97fc7b21dd2946bbef8936ff6d38efa093d80a"}
Jul 20 18:02:44.519596 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:02:44.519253    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-79747b4d4c-cks45"
Jul 20 18:02:44.540482 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:02:44.540429    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-79747b4d4c-cks45" podStartSLOduration=1.994951905 podStartE2EDuration="6.540415197s" totalImagesPullingTime="3.474630237s" totalInitContainerRuntime="1.070833055s" isStatefulPod=false podCreationTimestamp="2026-07-20 18:02:38 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 18:02:44.539746371 +0000 UTC m=+410.786279780" watchObservedRunningTime="2026-07-20 18:02:44.540415197 +0000 UTC m=+410.786948664"
Jul 20 18:02:55.524923 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:02:55.524893    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-79747b4d4c-cks45"
Jul 20 18:02:58.057096 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:02:58.057063    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-75cc6b94bb-wcbx8"]
Jul 20 18:02:58.189426 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:02:58.189396    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-75cc6b94bb-wcbx8"]
Jul 20 18:02:58.189557 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:02:58.189437    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-57d4489cf8-2q6sw"]
Jul 20 18:02:58.189557 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:02:58.189505    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-75cc6b94bb-wcbx8"
Jul 20 18:02:58.192166 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:02:58.192141    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-fpgjs\""
Jul 20 18:02:58.192513 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:02:58.192500    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 20 18:02:58.192676 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:02:58.192664    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 20 18:02:58.193126 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:02:58.193108    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 20 18:02:58.193516 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:02:58.193499    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 20 18:02:58.202563 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:02:58.202546    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-57d4489cf8-2q6sw"
Jul 20 18:02:58.204892 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:02:58.204874    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 20 18:02:58.205106 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:02:58.205086    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 20 18:02:58.205222 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:02:58.205147    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 20 18:02:58.205222 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:02:58.205148    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-tpzld\""
Jul 20 18:02:58.207992 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:02:58.207972    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-57d4489cf8-2q6sw"]
Jul 20 18:02:58.276050 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:02:58.276018    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cghm7\" (UniqueName: \"kubernetes.io/projected/a8cfc3b1-73da-4052-9e03-cb0437933008-kube-api-access-cghm7\") pod \"kube-auth-proxy-75cc6b94bb-wcbx8\" (UID: \"a8cfc3b1-73da-4052-9e03-cb0437933008\") " pod="openshift-ingress/kube-auth-proxy-75cc6b94bb-wcbx8"
Jul 20 18:02:58.276208 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:02:58.276071    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a8cfc3b1-73da-4052-9e03-cb0437933008-tmp\") pod \"kube-auth-proxy-75cc6b94bb-wcbx8\" (UID: \"a8cfc3b1-73da-4052-9e03-cb0437933008\") " pod="openshift-ingress/kube-auth-proxy-75cc6b94bb-wcbx8"
Jul 20 18:02:58.276208 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:02:58.276191    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8cfc3b1-73da-4052-9e03-cb0437933008-tls-certs\") pod \"kube-auth-proxy-75cc6b94bb-wcbx8\" (UID: \"a8cfc3b1-73da-4052-9e03-cb0437933008\") " pod="openshift-ingress/kube-auth-proxy-75cc6b94bb-wcbx8"
Jul 20 18:02:58.376984 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:02:58.376906    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b6f825b2-5581-4dd1-a340-6eab11b76c3b-metrics-cert\") pod \"lws-controller-manager-57d4489cf8-2q6sw\" (UID: \"b6f825b2-5581-4dd1-a340-6eab11b76c3b\") " pod="openshift-lws-operator/lws-controller-manager-57d4489cf8-2q6sw"
Jul 20 18:02:58.376984 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:02:58.376938    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b6f825b2-5581-4dd1-a340-6eab11b76c3b-cert\") pod \"lws-controller-manager-57d4489cf8-2q6sw\" (UID: \"b6f825b2-5581-4dd1-a340-6eab11b76c3b\") " pod="openshift-lws-operator/lws-controller-manager-57d4489cf8-2q6sw"
Jul 20 18:02:58.376984 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:02:58.376965    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cghm7\" (UniqueName: \"kubernetes.io/projected/a8cfc3b1-73da-4052-9e03-cb0437933008-kube-api-access-cghm7\") pod \"kube-auth-proxy-75cc6b94bb-wcbx8\" (UID: \"a8cfc3b1-73da-4052-9e03-cb0437933008\") " pod="openshift-ingress/kube-auth-proxy-75cc6b94bb-wcbx8"
Jul 20 18:02:58.377238 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:02:58.377168    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqv7s\" (UniqueName: \"kubernetes.io/projected/b6f825b2-5581-4dd1-a340-6eab11b76c3b-kube-api-access-xqv7s\") pod \"lws-controller-manager-57d4489cf8-2q6sw\" (UID: \"b6f825b2-5581-4dd1-a340-6eab11b76c3b\") " pod="openshift-lws-operator/lws-controller-manager-57d4489cf8-2q6sw"
Jul 20 18:02:58.377238 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:02:58.377209    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/b6f825b2-5581-4dd1-a340-6eab11b76c3b-manager-config\") pod \"lws-controller-manager-57d4489cf8-2q6sw\" (UID: \"b6f825b2-5581-4dd1-a340-6eab11b76c3b\") " pod="openshift-lws-operator/lws-controller-manager-57d4489cf8-2q6sw"
Jul 20 18:02:58.377238 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:02:58.377237    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a8cfc3b1-73da-4052-9e03-cb0437933008-tmp\") pod \"kube-auth-proxy-75cc6b94bb-wcbx8\" (UID: \"a8cfc3b1-73da-4052-9e03-cb0437933008\") " pod="openshift-ingress/kube-auth-proxy-75cc6b94bb-wcbx8"
Jul 20 18:02:58.377366 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:02:58.377283    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8cfc3b1-73da-4052-9e03-cb0437933008-tls-certs\") pod \"kube-auth-proxy-75cc6b94bb-wcbx8\" (UID: \"a8cfc3b1-73da-4052-9e03-cb0437933008\") " pod="openshift-ingress/kube-auth-proxy-75cc6b94bb-wcbx8"
Jul 20 18:02:58.379429 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:02:58.379401    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a8cfc3b1-73da-4052-9e03-cb0437933008-tmp\") pod \"kube-auth-proxy-75cc6b94bb-wcbx8\" (UID: \"a8cfc3b1-73da-4052-9e03-cb0437933008\") " pod="openshift-ingress/kube-auth-proxy-75cc6b94bb-wcbx8"
Jul 20 18:02:58.379621 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:02:58.379605    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8cfc3b1-73da-4052-9e03-cb0437933008-tls-certs\") pod \"kube-auth-proxy-75cc6b94bb-wcbx8\" (UID: \"a8cfc3b1-73da-4052-9e03-cb0437933008\") " pod="openshift-ingress/kube-auth-proxy-75cc6b94bb-wcbx8"
Jul 20 18:02:58.396451 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:02:58.396418    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cghm7\" (UniqueName: \"kubernetes.io/projected/a8cfc3b1-73da-4052-9e03-cb0437933008-kube-api-access-cghm7\") pod \"kube-auth-proxy-75cc6b94bb-wcbx8\" (UID: \"a8cfc3b1-73da-4052-9e03-cb0437933008\") " pod="openshift-ingress/kube-auth-proxy-75cc6b94bb-wcbx8"
Jul 20 18:02:58.478370 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:02:58.478339    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b6f825b2-5581-4dd1-a340-6eab11b76c3b-metrics-cert\") pod \"lws-controller-manager-57d4489cf8-2q6sw\" (UID: \"b6f825b2-5581-4dd1-a340-6eab11b76c3b\") " pod="openshift-lws-operator/lws-controller-manager-57d4489cf8-2q6sw"
Jul 20 18:02:58.478370 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:02:58.478370    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b6f825b2-5581-4dd1-a340-6eab11b76c3b-cert\") pod \"lws-controller-manager-57d4489cf8-2q6sw\" (UID: \"b6f825b2-5581-4dd1-a340-6eab11b76c3b\") " pod="openshift-lws-operator/lws-controller-manager-57d4489cf8-2q6sw"
Jul 20 18:02:58.478547 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:02:58.478393    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqv7s\" (UniqueName: \"kubernetes.io/projected/b6f825b2-5581-4dd1-a340-6eab11b76c3b-kube-api-access-xqv7s\") pod \"lws-controller-manager-57d4489cf8-2q6sw\" (UID: \"b6f825b2-5581-4dd1-a340-6eab11b76c3b\") " pod="openshift-lws-operator/lws-controller-manager-57d4489cf8-2q6sw"
Jul 20 18:02:58.478547 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:02:58.478412    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/b6f825b2-5581-4dd1-a340-6eab11b76c3b-manager-config\") pod \"lws-controller-manager-57d4489cf8-2q6sw\" (UID: \"b6f825b2-5581-4dd1-a340-6eab11b76c3b\") " pod="openshift-lws-operator/lws-controller-manager-57d4489cf8-2q6sw"
Jul 20 18:02:58.478986 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:02:58.478966    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/b6f825b2-5581-4dd1-a340-6eab11b76c3b-manager-config\") pod \"lws-controller-manager-57d4489cf8-2q6sw\" (UID: \"b6f825b2-5581-4dd1-a340-6eab11b76c3b\") " pod="openshift-lws-operator/lws-controller-manager-57d4489cf8-2q6sw"
Jul 20 18:02:58.480701 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:02:58.480679    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b6f825b2-5581-4dd1-a340-6eab11b76c3b-metrics-cert\") pod \"lws-controller-manager-57d4489cf8-2q6sw\" (UID: \"b6f825b2-5581-4dd1-a340-6eab11b76c3b\") " pod="openshift-lws-operator/lws-controller-manager-57d4489cf8-2q6sw"
Jul 20 18:02:58.480798 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:02:58.480712    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b6f825b2-5581-4dd1-a340-6eab11b76c3b-cert\") pod \"lws-controller-manager-57d4489cf8-2q6sw\" (UID: \"b6f825b2-5581-4dd1-a340-6eab11b76c3b\") " pod="openshift-lws-operator/lws-controller-manager-57d4489cf8-2q6sw"
Jul 20 18:02:58.491315 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:02:58.491295    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqv7s\" (UniqueName: \"kubernetes.io/projected/b6f825b2-5581-4dd1-a340-6eab11b76c3b-kube-api-access-xqv7s\") pod \"lws-controller-manager-57d4489cf8-2q6sw\" (UID: \"b6f825b2-5581-4dd1-a340-6eab11b76c3b\") " pod="openshift-lws-operator/lws-controller-manager-57d4489cf8-2q6sw"
Jul 20 18:02:58.499165 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:02:58.499150    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-75cc6b94bb-wcbx8"
Jul 20 18:02:58.511209 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:02:58.511183    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-57d4489cf8-2q6sw"
Jul 20 18:02:58.660967 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:02:58.660891    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-75cc6b94bb-wcbx8"]
Jul 20 18:02:58.664375 ip-10-0-137-247 kubenswrapper[2569]: W0720 18:02:58.664351    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8cfc3b1_73da_4052_9e03_cb0437933008.slice/crio-703341831a2595774c70ba50d273efde902edd89bbca8a36fa3dc5326709b53c WatchSource:0}: Error finding container 703341831a2595774c70ba50d273efde902edd89bbca8a36fa3dc5326709b53c: Status 404 returned error can't find the container with id 703341831a2595774c70ba50d273efde902edd89bbca8a36fa3dc5326709b53c
Jul 20 18:02:58.715219 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:02:58.715188    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-57d4489cf8-2q6sw"]
Jul 20 18:02:58.728361 ip-10-0-137-247 kubenswrapper[2569]: W0720 18:02:58.728339    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6f825b2_5581_4dd1_a340_6eab11b76c3b.slice/crio-818e446600298fe88ab53536a8799b2cc8ff62849c095bf5ab8804ecfa4ec650 WatchSource:0}: Error finding container 818e446600298fe88ab53536a8799b2cc8ff62849c095bf5ab8804ecfa4ec650: Status 404 returned error can't find the container with id 818e446600298fe88ab53536a8799b2cc8ff62849c095bf5ab8804ecfa4ec650
Jul 20 18:02:59.575399 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:02:59.575351    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-57d4489cf8-2q6sw" event={"ID":"b6f825b2-5581-4dd1-a340-6eab11b76c3b","Type":"ContainerStarted","Data":"818e446600298fe88ab53536a8799b2cc8ff62849c095bf5ab8804ecfa4ec650"}
Jul 20 18:02:59.576791 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:02:59.576753    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-75cc6b94bb-wcbx8" event={"ID":"a8cfc3b1-73da-4052-9e03-cb0437933008","Type":"ContainerStarted","Data":"703341831a2595774c70ba50d273efde902edd89bbca8a36fa3dc5326709b53c"}
Jul 20 18:03:02.591534 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:03:02.591495    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-57d4489cf8-2q6sw" event={"ID":"b6f825b2-5581-4dd1-a340-6eab11b76c3b","Type":"ContainerStarted","Data":"3e1cf1f72993285e223393bd486fc8cfb0291917e671c8d4b8f7bcc3ffca1e49"}
Jul 20 18:03:02.591958 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:03:02.591617    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-57d4489cf8-2q6sw"
Jul 20 18:03:02.592942 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:03:02.592907    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-75cc6b94bb-wcbx8" event={"ID":"a8cfc3b1-73da-4052-9e03-cb0437933008","Type":"ContainerStarted","Data":"34988a0c5ac514f6a375d0835ac1cd358097dda31b160b7cabcd7ca1b0a198e9"}
Jul 20 18:03:02.608659 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:03:02.608612    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-57d4489cf8-2q6sw" podStartSLOduration=1.357810921 podStartE2EDuration="4.608598283s" totalImagesPullingTime="3.250787362s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 18:02:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 18:03:02.608354244 +0000 UTC m=+428.854887645" watchObservedRunningTime="2026-07-20 18:03:02.608598283 +0000 UTC m=+428.855131694"
Jul 20 18:03:02.625225 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:03:02.625183    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-75cc6b94bb-wcbx8" podStartSLOduration=1.310479428 podStartE2EDuration="4.625169125s" totalImagesPullingTime="3.314689697s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 18:02:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 18:03:02.623692844 +0000 UTC m=+428.870226256" watchObservedRunningTime="2026-07-20 18:03:02.625169125 +0000 UTC m=+428.871702535"
Jul 20 18:03:13.600217 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:03:13.600187    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-57d4489cf8-2q6sw"
Jul 20 18:04:53.342033 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:04:53.341994    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-wpwhf"]
Jul 20 18:04:53.345037 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:04:53.345020    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-wpwhf"
Jul 20 18:04:53.349254 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:04:53.349231    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Jul 20 18:04:53.351614 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:04:53.351596    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Jul 20 18:04:53.358065 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:04:53.358049    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 20 18:04:53.358781 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:04:53.358768    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-5hqz2\""
Jul 20 18:04:53.360416 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:04:53.360403    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 20 18:04:53.378979 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:04:53.378949    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-wpwhf"]
Jul 20 18:04:53.439986 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:04:53.439953    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7wkj\" (UniqueName: \"kubernetes.io/projected/4d880c15-e2ed-4921-8494-154405091530-kube-api-access-q7wkj\") pod \"kuadrant-console-plugin-66b5774985-wpwhf\" (UID: \"4d880c15-e2ed-4921-8494-154405091530\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-wpwhf"
Jul 20 18:04:53.439986 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:04:53.439989    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/4d880c15-e2ed-4921-8494-154405091530-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-wpwhf\" (UID: \"4d880c15-e2ed-4921-8494-154405091530\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-wpwhf"
Jul 20 18:04:53.440171 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:04:53.440028    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/4d880c15-e2ed-4921-8494-154405091530-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-wpwhf\" (UID: \"4d880c15-e2ed-4921-8494-154405091530\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-wpwhf"
Jul 20 18:04:53.541166 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:04:53.541120    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7wkj\" (UniqueName: \"kubernetes.io/projected/4d880c15-e2ed-4921-8494-154405091530-kube-api-access-q7wkj\") pod \"kuadrant-console-plugin-66b5774985-wpwhf\" (UID: \"4d880c15-e2ed-4921-8494-154405091530\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-wpwhf"
Jul 20 18:04:53.541166 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:04:53.541173    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/4d880c15-e2ed-4921-8494-154405091530-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-wpwhf\" (UID: \"4d880c15-e2ed-4921-8494-154405091530\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-wpwhf"
Jul 20 18:04:53.541433 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:04:53.541209    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/4d880c15-e2ed-4921-8494-154405091530-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-wpwhf\" (UID: \"4d880c15-e2ed-4921-8494-154405091530\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-wpwhf"
Jul 20 18:04:53.541433 ip-10-0-137-247 kubenswrapper[2569]: E0720 18:04:53.541306    2569 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Jul 20 18:04:53.541433 ip-10-0-137-247 kubenswrapper[2569]: E0720 18:04:53.541371    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d880c15-e2ed-4921-8494-154405091530-plugin-serving-cert podName:4d880c15-e2ed-4921-8494-154405091530 nodeName:}" failed. No retries permitted until 2026-07-20 18:04:54.041351306 +0000 UTC m=+540.287884697 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/4d880c15-e2ed-4921-8494-154405091530-plugin-serving-cert") pod "kuadrant-console-plugin-66b5774985-wpwhf" (UID: "4d880c15-e2ed-4921-8494-154405091530") : secret "plugin-serving-cert" not found
Jul 20 18:04:53.541923 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:04:53.541903    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/4d880c15-e2ed-4921-8494-154405091530-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-wpwhf\" (UID: \"4d880c15-e2ed-4921-8494-154405091530\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-wpwhf"
Jul 20 18:04:53.561852 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:04:53.561832    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7wkj\" (UniqueName: \"kubernetes.io/projected/4d880c15-e2ed-4921-8494-154405091530-kube-api-access-q7wkj\") pod \"kuadrant-console-plugin-66b5774985-wpwhf\" (UID: \"4d880c15-e2ed-4921-8494-154405091530\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-wpwhf"
Jul 20 18:04:54.045606 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:04:54.045575    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/4d880c15-e2ed-4921-8494-154405091530-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-wpwhf\" (UID: \"4d880c15-e2ed-4921-8494-154405091530\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-wpwhf"
Jul 20 18:04:54.047888 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:04:54.047869    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/4d880c15-e2ed-4921-8494-154405091530-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-wpwhf\" (UID: \"4d880c15-e2ed-4921-8494-154405091530\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-wpwhf"
Jul 20 18:04:54.260054 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:04:54.256014    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-5hqz2\""
Jul 20 18:04:54.264207 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:04:54.264185    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-wpwhf"
Jul 20 18:04:54.390580 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:04:54.390552    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-wpwhf"]
Jul 20 18:04:54.393588 ip-10-0-137-247 kubenswrapper[2569]: W0720 18:04:54.393556    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4d880c15_e2ed_4921_8494_154405091530.slice/crio-3d74f275f2f680f637ebf344bdb2b054817473db7d7c6df56e6b5735b42055b0 WatchSource:0}: Error finding container 3d74f275f2f680f637ebf344bdb2b054817473db7d7c6df56e6b5735b42055b0: Status 404 returned error can't find the container with id 3d74f275f2f680f637ebf344bdb2b054817473db7d7c6df56e6b5735b42055b0
Jul 20 18:04:55.019607 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:04:55.019573    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-wpwhf" event={"ID":"4d880c15-e2ed-4921-8494-154405091530","Type":"ContainerStarted","Data":"3d74f275f2f680f637ebf344bdb2b054817473db7d7c6df56e6b5735b42055b0"}
Jul 20 18:05:18.121736 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:05:18.121687    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-wpwhf" event={"ID":"4d880c15-e2ed-4921-8494-154405091530","Type":"ContainerStarted","Data":"da4158f795cea449dbecc613d66ad1819b683fb200ee44f469ee65107cea4edd"}
Jul 20 18:05:18.137752 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:05:18.137691    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-wpwhf" podStartSLOduration=1.9675304279999999 podStartE2EDuration="25.137676628s" totalImagesPullingTime="23.1701462s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 18:04:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 18:05:18.136249778 +0000 UTC m=+564.382783187" watchObservedRunningTime="2026-07-20 18:05:18.137676628 +0000 UTC m=+564.384210038"
Jul 20 18:05:40.713514 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:05:40.713481    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9thxb"]
Jul 20 18:05:40.985948 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:05:40.985874    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9thxb"]
Jul 20 18:05:40.985948 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:05:40.985911    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9thxb"]
Jul 20 18:05:40.986118 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:05:40.985998    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-9thxb"
Jul 20 18:05:40.988480 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:05:40.988459    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 20 18:05:41.120155 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:05:41.120125    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c9996300-b0a7-43ac-86ca-469038001e1f-config-file\") pod \"limitador-limitador-54d6d988d5-9thxb\" (UID: \"c9996300-b0a7-43ac-86ca-469038001e1f\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-9thxb"
Jul 20 18:05:41.120310 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:05:41.120175    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xbvr\" (UniqueName: \"kubernetes.io/projected/c9996300-b0a7-43ac-86ca-469038001e1f-kube-api-access-5xbvr\") pod \"limitador-limitador-54d6d988d5-9thxb\" (UID: \"c9996300-b0a7-43ac-86ca-469038001e1f\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-9thxb"
Jul 20 18:05:41.220983 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:05:41.220958    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xbvr\" (UniqueName: \"kubernetes.io/projected/c9996300-b0a7-43ac-86ca-469038001e1f-kube-api-access-5xbvr\") pod \"limitador-limitador-54d6d988d5-9thxb\" (UID: \"c9996300-b0a7-43ac-86ca-469038001e1f\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-9thxb"
Jul 20 18:05:41.221144 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:05:41.221013    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c9996300-b0a7-43ac-86ca-469038001e1f-config-file\") pod \"limitador-limitador-54d6d988d5-9thxb\" (UID: \"c9996300-b0a7-43ac-86ca-469038001e1f\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-9thxb"
Jul 20 18:05:41.221561 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:05:41.221542    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c9996300-b0a7-43ac-86ca-469038001e1f-config-file\") pod \"limitador-limitador-54d6d988d5-9thxb\" (UID: \"c9996300-b0a7-43ac-86ca-469038001e1f\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-9thxb"
Jul 20 18:05:41.232440 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:05:41.232415    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xbvr\" (UniqueName: \"kubernetes.io/projected/c9996300-b0a7-43ac-86ca-469038001e1f-kube-api-access-5xbvr\") pod \"limitador-limitador-54d6d988d5-9thxb\" (UID: \"c9996300-b0a7-43ac-86ca-469038001e1f\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-9thxb"
Jul 20 18:05:41.295195 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:05:41.295168    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-9thxb"
Jul 20 18:05:41.443058 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:05:41.443024    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9thxb"]
Jul 20 18:05:41.446203 ip-10-0-137-247 kubenswrapper[2569]: W0720 18:05:41.446172    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9996300_b0a7_43ac_86ca_469038001e1f.slice/crio-5b177095138d1586fbd1fdaed9b0ac14c7bc269ff952dc5dd9898585705d6534 WatchSource:0}: Error finding container 5b177095138d1586fbd1fdaed9b0ac14c7bc269ff952dc5dd9898585705d6534: Status 404 returned error can't find the container with id 5b177095138d1586fbd1fdaed9b0ac14c7bc269ff952dc5dd9898585705d6534
Jul 20 18:05:42.215433 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:05:42.215366    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-9thxb" event={"ID":"c9996300-b0a7-43ac-86ca-469038001e1f","Type":"ContainerStarted","Data":"5b177095138d1586fbd1fdaed9b0ac14c7bc269ff952dc5dd9898585705d6534"}
Jul 20 18:05:47.238889 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:05:47.238848    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-9thxb" event={"ID":"c9996300-b0a7-43ac-86ca-469038001e1f","Type":"ContainerStarted","Data":"062b100d5b2bc2895d2ebe0c28f2934ec22df8b004394a6aec4997ead7dfb0da"}
Jul 20 18:05:47.239253 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:05:47.238981    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-9thxb"
Jul 20 18:05:47.273044 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:05:47.272996    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-9thxb" podStartSLOduration=2.00950711 podStartE2EDuration="7.27298161s" totalImagesPullingTime="5.2634745s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 18:05:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 18:05:47.269606397 +0000 UTC m=+593.516139812" watchObservedRunningTime="2026-07-20 18:05:47.27298161 +0000 UTC m=+593.519515019"
Jul 20 18:05:54.102305 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:05:54.102270    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wjs8g_07587a78-2d3a-40ad-9aae-a4875200263a/ovn-acl-logging/0.log"
Jul 20 18:05:54.103579 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:05:54.103549    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wjs8g_07587a78-2d3a-40ad-9aae-a4875200263a/ovn-acl-logging/0.log"
Jul 20 18:05:58.243868 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:05:58.243834    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-9thxb"
Jul 20 18:06:56.654969 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:06:56.654934    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9thxb"]
Jul 20 18:07:20.590334 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:07:20.590293    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9thxb"]
Jul 20 18:07:31.955429 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:07:31.955390    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9thxb"]
Jul 20 18:07:44.050792 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:07:44.050756    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9thxb"]
Jul 20 18:07:56.545632 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:07:56.545597    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9thxb"]
Jul 20 18:08:14.647206 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:08:14.647169    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9thxb"]
Jul 20 18:08:19.815152 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:08:19.815121    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-tkgh7"]
Jul 20 18:08:19.817922 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:08:19.817904    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tkgh7"
Jul 20 18:08:19.820309 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:08:19.820285    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 20 18:08:19.821002 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:08:19.820978    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 20 18:08:19.821120 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:08:19.821059    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-4dzgb\""
Jul 20 18:08:19.821180 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:08:19.821126    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Jul 20 18:08:19.828256 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:08:19.828233    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-tkgh7"]
Jul 20 18:08:19.841480 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:08:19.841457    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/042bde04-5c1f-4f79-a1d5-eea437821bf4-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tkgh7\" (UID: \"042bde04-5c1f-4f79-a1d5-eea437821bf4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tkgh7"
Jul 20 18:08:19.841634 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:08:19.841508    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/042bde04-5c1f-4f79-a1d5-eea437821bf4-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tkgh7\" (UID: \"042bde04-5c1f-4f79-a1d5-eea437821bf4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tkgh7"
Jul 20 18:08:19.841634 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:08:19.841539    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wccb2\" (UniqueName: \"kubernetes.io/projected/042bde04-5c1f-4f79-a1d5-eea437821bf4-kube-api-access-wccb2\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tkgh7\" (UID: \"042bde04-5c1f-4f79-a1d5-eea437821bf4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tkgh7"
Jul 20 18:08:19.841634 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:08:19.841593    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/042bde04-5c1f-4f79-a1d5-eea437821bf4-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tkgh7\" (UID: \"042bde04-5c1f-4f79-a1d5-eea437821bf4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tkgh7"
Jul 20 18:08:19.842044 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:08:19.841655    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/042bde04-5c1f-4f79-a1d5-eea437821bf4-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tkgh7\" (UID: \"042bde04-5c1f-4f79-a1d5-eea437821bf4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tkgh7"
Jul 20 18:08:19.842044 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:08:19.841687    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/042bde04-5c1f-4f79-a1d5-eea437821bf4-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tkgh7\" (UID: \"042bde04-5c1f-4f79-a1d5-eea437821bf4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tkgh7"
Jul 20 18:08:19.942680 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:08:19.942651    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/042bde04-5c1f-4f79-a1d5-eea437821bf4-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tkgh7\" (UID: \"042bde04-5c1f-4f79-a1d5-eea437821bf4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tkgh7"
Jul 20 18:08:19.942819 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:08:19.942691    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/042bde04-5c1f-4f79-a1d5-eea437821bf4-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tkgh7\" (UID: \"042bde04-5c1f-4f79-a1d5-eea437821bf4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tkgh7"
Jul 20 18:08:19.942819 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:08:19.942709    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/042bde04-5c1f-4f79-a1d5-eea437821bf4-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tkgh7\" (UID: \"042bde04-5c1f-4f79-a1d5-eea437821bf4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tkgh7"
Jul 20 18:08:19.942819 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:08:19.942758    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/042bde04-5c1f-4f79-a1d5-eea437821bf4-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tkgh7\" (UID: \"042bde04-5c1f-4f79-a1d5-eea437821bf4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tkgh7"
Jul 20 18:08:19.942819 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:08:19.942794    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/042bde04-5c1f-4f79-a1d5-eea437821bf4-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tkgh7\" (UID: \"042bde04-5c1f-4f79-a1d5-eea437821bf4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tkgh7"
Jul 20 18:08:19.943043 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:08:19.942824    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wccb2\" (UniqueName: \"kubernetes.io/projected/042bde04-5c1f-4f79-a1d5-eea437821bf4-kube-api-access-wccb2\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tkgh7\" (UID: \"042bde04-5c1f-4f79-a1d5-eea437821bf4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tkgh7"
Jul 20 18:08:19.943180 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:08:19.943144    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/042bde04-5c1f-4f79-a1d5-eea437821bf4-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tkgh7\" (UID: \"042bde04-5c1f-4f79-a1d5-eea437821bf4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tkgh7"
Jul 20 18:08:19.943180 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:08:19.943165    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/042bde04-5c1f-4f79-a1d5-eea437821bf4-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tkgh7\" (UID: \"042bde04-5c1f-4f79-a1d5-eea437821bf4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tkgh7"
Jul 20 18:08:19.943325 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:08:19.943200    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/042bde04-5c1f-4f79-a1d5-eea437821bf4-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tkgh7\" (UID: \"042bde04-5c1f-4f79-a1d5-eea437821bf4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tkgh7"
Jul 20 18:08:19.944882 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:08:19.944866    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/042bde04-5c1f-4f79-a1d5-eea437821bf4-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tkgh7\" (UID: \"042bde04-5c1f-4f79-a1d5-eea437821bf4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tkgh7"
Jul 20 18:08:19.945063 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:08:19.945045    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/042bde04-5c1f-4f79-a1d5-eea437821bf4-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tkgh7\" (UID: \"042bde04-5c1f-4f79-a1d5-eea437821bf4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tkgh7"
Jul 20 18:08:19.952191 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:08:19.952165    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wccb2\" (UniqueName: \"kubernetes.io/projected/042bde04-5c1f-4f79-a1d5-eea437821bf4-kube-api-access-wccb2\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tkgh7\" (UID: \"042bde04-5c1f-4f79-a1d5-eea437821bf4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tkgh7"
Jul 20 18:08:20.126285 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:08:20.126202    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tkgh7"
Jul 20 18:08:20.254345 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:08:20.254322    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-tkgh7"]
Jul 20 18:08:20.256929 ip-10-0-137-247 kubenswrapper[2569]: W0720 18:08:20.256875    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod042bde04_5c1f_4f79_a1d5_eea437821bf4.slice/crio-1c637ad5824a65c5a4c7388c1b7c8948d3e0a9242a7890c82c01522795a00075 WatchSource:0}: Error finding container 1c637ad5824a65c5a4c7388c1b7c8948d3e0a9242a7890c82c01522795a00075: Status 404 returned error can't find the container with id 1c637ad5824a65c5a4c7388c1b7c8948d3e0a9242a7890c82c01522795a00075
Jul 20 18:08:20.258144 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:08:20.258122    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 20 18:08:20.836346 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:08:20.836304    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tkgh7" event={"ID":"042bde04-5c1f-4f79-a1d5-eea437821bf4","Type":"ContainerStarted","Data":"1c637ad5824a65c5a4c7388c1b7c8948d3e0a9242a7890c82c01522795a00075"}
Jul 20 18:08:21.841630 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:08:21.841597    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tkgh7" event={"ID":"042bde04-5c1f-4f79-a1d5-eea437821bf4","Type":"ContainerStarted","Data":"28418d5c8ea2e87ea49831eda357cd8dd163f2a8956c1f4225a6891a6bde703c"}
Jul 20 18:08:21.861088 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:08:21.861044    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tkgh7" podStartSLOduration=1.80690896 podStartE2EDuration="2.861029718s" totalImagesPullingTime="1.054120758s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 18:08:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 18:08:21.859262943 +0000 UTC m=+748.105796355" watchObservedRunningTime="2026-07-20 18:08:21.861029718 +0000 UTC m=+748.107563127"
Jul 20 18:08:28.057891 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:08:28.057857    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9thxb"]
Jul 20 18:08:30.127096 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:08:30.127051    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tkgh7"
Jul 20 18:08:30.127498 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:08:30.127113    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tkgh7"
Jul 20 18:08:30.139380 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:08:30.139349    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tkgh7"
Jul 20 18:08:30.891570 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:08:30.891544    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tkgh7"
Jul 20 18:09:55.963360 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:09:55.963321    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9thxb"]
Jul 20 18:10:05.491449 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:10:05.491413    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9thxb"]
Jul 20 18:10:17.206997 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:10:17.206960    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9thxb"]
Jul 20 18:10:54.130893 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:10:54.130868    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wjs8g_07587a78-2d3a-40ad-9aae-a4875200263a/ovn-acl-logging/0.log"
Jul 20 18:10:54.131380 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:10:54.131037    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wjs8g_07587a78-2d3a-40ad-9aae-a4875200263a/ovn-acl-logging/0.log"
Jul 20 18:11:15.065016 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:11:15.064980    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7bd97c8594-spbwk"]
Jul 20 18:11:15.068048 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:11:15.068032    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7bd97c8594-spbwk"
Jul 20 18:11:15.071328 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:11:15.071304    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 20 18:11:15.071328 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:11:15.071322    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 20 18:11:15.071484 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:11:15.071357    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-gfll9\""
Jul 20 18:11:15.080610 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:11:15.080589    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7bd97c8594-spbwk"]
Jul 20 18:11:15.157087 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:11:15.157049    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6a0a3499-f4cc-4b3f-83d2-090a6529663b-webhook-cert\") pod \"maas-controller-7bd97c8594-spbwk\" (UID: \"6a0a3499-f4cc-4b3f-83d2-090a6529663b\") " pod="opendatahub/maas-controller-7bd97c8594-spbwk"
Jul 20 18:11:15.157248 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:11:15.157115    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nc9dl\" (UniqueName: \"kubernetes.io/projected/6a0a3499-f4cc-4b3f-83d2-090a6529663b-kube-api-access-nc9dl\") pod \"maas-controller-7bd97c8594-spbwk\" (UID: \"6a0a3499-f4cc-4b3f-83d2-090a6529663b\") " pod="opendatahub/maas-controller-7bd97c8594-spbwk"
Jul 20 18:11:15.257957 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:11:15.257925    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nc9dl\" (UniqueName: \"kubernetes.io/projected/6a0a3499-f4cc-4b3f-83d2-090a6529663b-kube-api-access-nc9dl\") pod \"maas-controller-7bd97c8594-spbwk\" (UID: \"6a0a3499-f4cc-4b3f-83d2-090a6529663b\") " pod="opendatahub/maas-controller-7bd97c8594-spbwk"
Jul 20 18:11:15.258116 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:11:15.257980    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6a0a3499-f4cc-4b3f-83d2-090a6529663b-webhook-cert\") pod \"maas-controller-7bd97c8594-spbwk\" (UID: \"6a0a3499-f4cc-4b3f-83d2-090a6529663b\") " pod="opendatahub/maas-controller-7bd97c8594-spbwk"
Jul 20 18:11:15.260559 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:11:15.260536    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6a0a3499-f4cc-4b3f-83d2-090a6529663b-webhook-cert\") pod \"maas-controller-7bd97c8594-spbwk\" (UID: \"6a0a3499-f4cc-4b3f-83d2-090a6529663b\") " pod="opendatahub/maas-controller-7bd97c8594-spbwk"
Jul 20 18:11:15.267096 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:11:15.267073    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nc9dl\" (UniqueName: \"kubernetes.io/projected/6a0a3499-f4cc-4b3f-83d2-090a6529663b-kube-api-access-nc9dl\") pod \"maas-controller-7bd97c8594-spbwk\" (UID: \"6a0a3499-f4cc-4b3f-83d2-090a6529663b\") " pod="opendatahub/maas-controller-7bd97c8594-spbwk"
Jul 20 18:11:15.377963 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:11:15.377893    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7bd97c8594-spbwk"
Jul 20 18:11:15.506557 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:11:15.506527    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7bd97c8594-spbwk"]
Jul 20 18:11:15.509671 ip-10-0-137-247 kubenswrapper[2569]: W0720 18:11:15.509644    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a0a3499_f4cc_4b3f_83d2_090a6529663b.slice/crio-35d145367213b1b78c2cd3858308f21a57256d3c0848778ec0947d5c05b610fb WatchSource:0}: Error finding container 35d145367213b1b78c2cd3858308f21a57256d3c0848778ec0947d5c05b610fb: Status 404 returned error can't find the container with id 35d145367213b1b78c2cd3858308f21a57256d3c0848778ec0947d5c05b610fb
Jul 20 18:11:15.523279 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:11:15.523254    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7bd97c8594-spbwk" event={"ID":"6a0a3499-f4cc-4b3f-83d2-090a6529663b","Type":"ContainerStarted","Data":"35d145367213b1b78c2cd3858308f21a57256d3c0848778ec0947d5c05b610fb"}
Jul 20 18:11:18.536870 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:11:18.536836    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7bd97c8594-spbwk" event={"ID":"6a0a3499-f4cc-4b3f-83d2-090a6529663b","Type":"ContainerStarted","Data":"36d783096748b3a10f759d8b00b1f1cfc4af263c43c2d12b00af30f5e0c3527e"}
Jul 20 18:11:18.537331 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:11:18.536957    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7bd97c8594-spbwk"
Jul 20 18:11:18.558525 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:11:18.558454    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7bd97c8594-spbwk" podStartSLOduration=1.314584021 podStartE2EDuration="3.558440773s" totalImagesPullingTime="2.243856752s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 18:11:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 18:11:18.557473516 +0000 UTC m=+924.804006963" watchObservedRunningTime="2026-07-20 18:11:18.558440773 +0000 UTC m=+924.804974182"
Jul 20 18:11:29.542452 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:11:29.542421    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7bd97c8594-spbwk"
Jul 20 18:11:34.199955 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:11:34.199917    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7bd97c8594-spbwk"]
Jul 20 18:11:34.200353 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:11:34.200103    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-7bd97c8594-spbwk" podUID="6a0a3499-f4cc-4b3f-83d2-090a6529663b" containerName="manager" containerID="cri-o://36d783096748b3a10f759d8b00b1f1cfc4af263c43c2d12b00af30f5e0c3527e" gracePeriod=30
Jul 20 18:11:34.440082 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:11:34.440057    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7bd97c8594-spbwk"
Jul 20 18:11:34.511792 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:11:34.511764    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nc9dl\" (UniqueName: \"kubernetes.io/projected/6a0a3499-f4cc-4b3f-83d2-090a6529663b-kube-api-access-nc9dl\") pod \"6a0a3499-f4cc-4b3f-83d2-090a6529663b\" (UID: \"6a0a3499-f4cc-4b3f-83d2-090a6529663b\") "
Jul 20 18:11:34.511938 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:11:34.511817    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6a0a3499-f4cc-4b3f-83d2-090a6529663b-webhook-cert\") pod \"6a0a3499-f4cc-4b3f-83d2-090a6529663b\" (UID: \"6a0a3499-f4cc-4b3f-83d2-090a6529663b\") "
Jul 20 18:11:34.513921 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:11:34.513889    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a0a3499-f4cc-4b3f-83d2-090a6529663b-kube-api-access-nc9dl" (OuterVolumeSpecName: "kube-api-access-nc9dl") pod "6a0a3499-f4cc-4b3f-83d2-090a6529663b" (UID: "6a0a3499-f4cc-4b3f-83d2-090a6529663b"). InnerVolumeSpecName "kube-api-access-nc9dl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 18:11:34.514024 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:11:34.513924    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a0a3499-f4cc-4b3f-83d2-090a6529663b-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "6a0a3499-f4cc-4b3f-83d2-090a6529663b" (UID: "6a0a3499-f4cc-4b3f-83d2-090a6529663b"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 18:11:34.605854 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:11:34.605823    2569 generic.go:356] "Generic (PLEG): container finished" podID="6a0a3499-f4cc-4b3f-83d2-090a6529663b" containerID="36d783096748b3a10f759d8b00b1f1cfc4af263c43c2d12b00af30f5e0c3527e" exitCode=0
Jul 20 18:11:34.605992 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:11:34.605860    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7bd97c8594-spbwk" event={"ID":"6a0a3499-f4cc-4b3f-83d2-090a6529663b","Type":"ContainerDied","Data":"36d783096748b3a10f759d8b00b1f1cfc4af263c43c2d12b00af30f5e0c3527e"}
Jul 20 18:11:34.605992 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:11:34.605867    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7bd97c8594-spbwk"
Jul 20 18:11:34.605992 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:11:34.605895    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7bd97c8594-spbwk" event={"ID":"6a0a3499-f4cc-4b3f-83d2-090a6529663b","Type":"ContainerDied","Data":"35d145367213b1b78c2cd3858308f21a57256d3c0848778ec0947d5c05b610fb"}
Jul 20 18:11:34.605992 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:11:34.605925    2569 scope.go:117] "RemoveContainer" containerID="36d783096748b3a10f759d8b00b1f1cfc4af263c43c2d12b00af30f5e0c3527e"
Jul 20 18:11:34.613458 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:11:34.613432    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nc9dl\" (UniqueName: \"kubernetes.io/projected/6a0a3499-f4cc-4b3f-83d2-090a6529663b-kube-api-access-nc9dl\") on node \"ip-10-0-137-247.ec2.internal\" DevicePath \"\""
Jul 20 18:11:34.613768 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:11:34.613750    2569 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6a0a3499-f4cc-4b3f-83d2-090a6529663b-webhook-cert\") on node \"ip-10-0-137-247.ec2.internal\" DevicePath \"\""
Jul 20 18:11:34.615943 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:11:34.615869    2569 scope.go:117] "RemoveContainer" containerID="36d783096748b3a10f759d8b00b1f1cfc4af263c43c2d12b00af30f5e0c3527e"
Jul 20 18:11:34.616252 ip-10-0-137-247 kubenswrapper[2569]: E0720 18:11:34.616188    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36d783096748b3a10f759d8b00b1f1cfc4af263c43c2d12b00af30f5e0c3527e\": container with ID starting with 36d783096748b3a10f759d8b00b1f1cfc4af263c43c2d12b00af30f5e0c3527e not found: ID does not exist" containerID="36d783096748b3a10f759d8b00b1f1cfc4af263c43c2d12b00af30f5e0c3527e"
Jul 20 18:11:34.616252 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:11:34.616222    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36d783096748b3a10f759d8b00b1f1cfc4af263c43c2d12b00af30f5e0c3527e"} err="failed to get container status \"36d783096748b3a10f759d8b00b1f1cfc4af263c43c2d12b00af30f5e0c3527e\": rpc error: code = NotFound desc = could not find container \"36d783096748b3a10f759d8b00b1f1cfc4af263c43c2d12b00af30f5e0c3527e\": container with ID starting with 36d783096748b3a10f759d8b00b1f1cfc4af263c43c2d12b00af30f5e0c3527e not found: ID does not exist"
Jul 20 18:11:34.627358 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:11:34.627335    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7bd97c8594-spbwk"]
Jul 20 18:11:34.630476 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:11:34.630454    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7bd97c8594-spbwk"]
Jul 20 18:11:35.446587 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:11:35.446547    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7bd97c8594-qntzv"]
Jul 20 18:11:35.447069 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:11:35.446973    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a0a3499-f4cc-4b3f-83d2-090a6529663b" containerName="manager"
Jul 20 18:11:35.447069 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:11:35.446991    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a0a3499-f4cc-4b3f-83d2-090a6529663b" containerName="manager"
Jul 20 18:11:35.447069 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:11:35.447069    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a0a3499-f4cc-4b3f-83d2-090a6529663b" containerName="manager"
Jul 20 18:11:35.451243 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:11:35.451222    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7bd97c8594-qntzv"
Jul 20 18:11:35.454258 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:11:35.454242    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-gfll9\""
Jul 20 18:11:35.454343 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:11:35.454249    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 20 18:11:35.454343 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:11:35.454289    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 20 18:11:35.457602 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:11:35.457582    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7bd97c8594-qntzv"]
Jul 20 18:11:35.521905 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:11:35.521870    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tj8lq\" (UniqueName: \"kubernetes.io/projected/9b8f6824-4ca6-4b50-abcd-55073c9dd002-kube-api-access-tj8lq\") pod \"maas-controller-7bd97c8594-qntzv\" (UID: \"9b8f6824-4ca6-4b50-abcd-55073c9dd002\") " pod="opendatahub/maas-controller-7bd97c8594-qntzv"
Jul 20 18:11:35.522062 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:11:35.521937    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9b8f6824-4ca6-4b50-abcd-55073c9dd002-webhook-cert\") pod \"maas-controller-7bd97c8594-qntzv\" (UID: \"9b8f6824-4ca6-4b50-abcd-55073c9dd002\") " pod="opendatahub/maas-controller-7bd97c8594-qntzv"
Jul 20 18:11:35.623226 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:11:35.623203    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tj8lq\" (UniqueName: \"kubernetes.io/projected/9b8f6824-4ca6-4b50-abcd-55073c9dd002-kube-api-access-tj8lq\") pod \"maas-controller-7bd97c8594-qntzv\" (UID: \"9b8f6824-4ca6-4b50-abcd-55073c9dd002\") " pod="opendatahub/maas-controller-7bd97c8594-qntzv"
Jul 20 18:11:35.623327 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:11:35.623262    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9b8f6824-4ca6-4b50-abcd-55073c9dd002-webhook-cert\") pod \"maas-controller-7bd97c8594-qntzv\" (UID: \"9b8f6824-4ca6-4b50-abcd-55073c9dd002\") " pod="opendatahub/maas-controller-7bd97c8594-qntzv"
Jul 20 18:11:35.625738 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:11:35.625696    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9b8f6824-4ca6-4b50-abcd-55073c9dd002-webhook-cert\") pod \"maas-controller-7bd97c8594-qntzv\" (UID: \"9b8f6824-4ca6-4b50-abcd-55073c9dd002\") " pod="opendatahub/maas-controller-7bd97c8594-qntzv"
Jul 20 18:11:35.631322 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:11:35.631303    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tj8lq\" (UniqueName: \"kubernetes.io/projected/9b8f6824-4ca6-4b50-abcd-55073c9dd002-kube-api-access-tj8lq\") pod \"maas-controller-7bd97c8594-qntzv\" (UID: \"9b8f6824-4ca6-4b50-abcd-55073c9dd002\") " pod="opendatahub/maas-controller-7bd97c8594-qntzv"
Jul 20 18:11:35.762502 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:11:35.762471    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7bd97c8594-qntzv"
Jul 20 18:11:35.885062 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:11:35.885038    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7bd97c8594-qntzv"]
Jul 20 18:11:35.887695 ip-10-0-137-247 kubenswrapper[2569]: W0720 18:11:35.887672    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b8f6824_4ca6_4b50_abcd_55073c9dd002.slice/crio-99ecde1e4d7f4da1da4daecf7f36cd01b6833404cc07a4a53ed3775f0920c312 WatchSource:0}: Error finding container 99ecde1e4d7f4da1da4daecf7f36cd01b6833404cc07a4a53ed3775f0920c312: Status 404 returned error can't find the container with id 99ecde1e4d7f4da1da4daecf7f36cd01b6833404cc07a4a53ed3775f0920c312
Jul 20 18:11:36.182481 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:11:36.182396    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a0a3499-f4cc-4b3f-83d2-090a6529663b" path="/var/lib/kubelet/pods/6a0a3499-f4cc-4b3f-83d2-090a6529663b/volumes"
Jul 20 18:11:36.615946 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:11:36.615913    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7bd97c8594-qntzv" event={"ID":"9b8f6824-4ca6-4b50-abcd-55073c9dd002","Type":"ContainerStarted","Data":"f76a2d4c02f46f5f29199d75c02ea816e6aa4fb841f4214d73d1de72203bd080"}
Jul 20 18:11:36.616343 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:11:36.615963    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7bd97c8594-qntzv" event={"ID":"9b8f6824-4ca6-4b50-abcd-55073c9dd002","Type":"ContainerStarted","Data":"99ecde1e4d7f4da1da4daecf7f36cd01b6833404cc07a4a53ed3775f0920c312"}
Jul 20 18:11:36.616343 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:11:36.616032    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7bd97c8594-qntzv"
Jul 20 18:11:36.631390 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:11:36.630951    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7bd97c8594-qntzv" podStartSLOduration=1.226713396 podStartE2EDuration="1.630933099s" totalImagesPullingTime="404.219703ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 18:11:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 18:11:36.630093874 +0000 UTC m=+942.876627283" watchObservedRunningTime="2026-07-20 18:11:36.630933099 +0000 UTC m=+942.877466515"
Jul 20 18:11:47.622074 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:11:47.622036    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7bd97c8594-qntzv"
Jul 20 18:11:55.366095 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:11:55.366058    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9thxb"]
Jul 20 18:12:06.261787 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:12:06.261749    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9thxb"]
Jul 20 18:12:44.847815 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:12:44.847781    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9thxb"]
Jul 20 18:12:54.245323 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:12:54.245289    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9thxb"]
Jul 20 18:13:03.846490 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:13:03.846449    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9thxb"]
Jul 20 18:13:13.642943 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:13:13.642897    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9thxb"]
Jul 20 18:13:24.150839 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:13:24.150790    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9thxb"]
Jul 20 18:13:33.469756 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:13:33.469700    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9thxb"]
Jul 20 18:13:44.447168 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:13:44.447133    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9thxb"]
Jul 20 18:13:54.550861 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:13:54.550824    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9thxb"]
Jul 20 18:14:16.673495 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:14:16.673461    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9thxb"]
Jul 20 18:14:29.446894 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:14:29.446859    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9thxb"]
Jul 20 18:14:52.245517 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:14:52.245484    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9thxb"]
Jul 20 18:15:00.142565 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:15:00.142531    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29742855-gfpqp"]
Jul 20 18:15:00.145741 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:15:00.145711    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29742855-gfpqp"
Jul 20 18:15:00.148132 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:15:00.148104    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fk7rm\""
Jul 20 18:15:00.155291 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:15:00.155272    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29742855-gfpqp"
Jul 20 18:15:00.164252 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:15:00.164229    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29742855-gfpqp"]
Jul 20 18:15:00.282088 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:15:00.282046    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29742855-gfpqp"]
Jul 20 18:15:00.284707 ip-10-0-137-247 kubenswrapper[2569]: W0720 18:15:00.284681    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bf83091_d9ec_48ad_af80_a60a764d3eef.slice/crio-570fe9249e9c18e40a2d3e6625bc238f1970d0d239f2c7356435637cd5dc1f39 WatchSource:0}: Error finding container 570fe9249e9c18e40a2d3e6625bc238f1970d0d239f2c7356435637cd5dc1f39: Status 404 returned error can't find the container with id 570fe9249e9c18e40a2d3e6625bc238f1970d0d239f2c7356435637cd5dc1f39
Jul 20 18:15:00.286546 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:15:00.286530    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 20 18:15:00.418080 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:15:00.418003    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29742855-gfpqp" event={"ID":"6bf83091-d9ec-48ad-af80-a60a764d3eef","Type":"ContainerStarted","Data":"570fe9249e9c18e40a2d3e6625bc238f1970d0d239f2c7356435637cd5dc1f39"}
Jul 20 18:15:11.545277 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:15:11.545243    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9thxb"]
Jul 20 18:15:20.149592 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:15:20.149515    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9thxb"]
Jul 20 18:15:24.521105 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:15:24.521015    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29742855-gfpqp" event={"ID":"6bf83091-d9ec-48ad-af80-a60a764d3eef","Type":"ContainerStarted","Data":"6dfd61b03478cad1165bc3ac1022f46804090a90da0dadf76141f9f08b6d7af8"}
Jul 20 18:15:24.538045 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:15:24.537996    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29742855-gfpqp" podStartSLOduration=0.817988391 podStartE2EDuration="24.537981948s" totalImagesPullingTime="23.719993557s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 18:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 18:15:24.53605235 +0000 UTC m=+1170.782585762" watchObservedRunningTime="2026-07-20 18:15:24.537981948 +0000 UTC m=+1170.784515354"
Jul 20 18:15:25.526333 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:15:25.526299    2569 generic.go:356] "Generic (PLEG): container finished" podID="6bf83091-d9ec-48ad-af80-a60a764d3eef" containerID="6dfd61b03478cad1165bc3ac1022f46804090a90da0dadf76141f9f08b6d7af8" exitCode=0
Jul 20 18:15:25.526779 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:15:25.526380    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29742855-gfpqp" event={"ID":"6bf83091-d9ec-48ad-af80-a60a764d3eef","Type":"ContainerDied","Data":"6dfd61b03478cad1165bc3ac1022f46804090a90da0dadf76141f9f08b6d7af8"}
Jul 20 18:15:26.650605 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:15:26.650582    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29742855-gfpqp"
Jul 20 18:15:27.534939 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:15:27.534908    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29742855-gfpqp"
Jul 20 18:15:27.534939 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:15:27.534913    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29742855-gfpqp" event={"ID":"6bf83091-d9ec-48ad-af80-a60a764d3eef","Type":"ContainerDied","Data":"570fe9249e9c18e40a2d3e6625bc238f1970d0d239f2c7356435637cd5dc1f39"}
Jul 20 18:15:27.535134 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:15:27.534958    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="570fe9249e9c18e40a2d3e6625bc238f1970d0d239f2c7356435637cd5dc1f39"
Jul 20 18:15:33.553002 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:15:33.552957    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9thxb"]
Jul 20 18:15:43.050069 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:15:43.050026    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9thxb"]
Jul 20 18:15:52.353794 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:15:52.353760    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9thxb"]
Jul 20 18:15:54.154264 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:15:54.154240    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wjs8g_07587a78-2d3a-40ad-9aae-a4875200263a/ovn-acl-logging/0.log"
Jul 20 18:15:54.155278 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:15:54.155258    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wjs8g_07587a78-2d3a-40ad-9aae-a4875200263a/ovn-acl-logging/0.log"
Jul 20 18:16:01.449889 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:16:01.449849    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9thxb"]
Jul 20 18:17:51.346373 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:17:51.346338    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9thxb"]
Jul 20 18:18:02.641054 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:18:02.641016    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9thxb"]
Jul 20 18:18:10.954444 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:18:10.954405    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9thxb"]
Jul 20 18:18:23.457377 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:18:23.457293    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9thxb"]
Jul 20 18:18:35.156702 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:18:35.156665    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9thxb"]
Jul 20 18:18:44.450808 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:18:44.450770    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9thxb"]
Jul 20 18:18:53.743209 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:18:53.743169    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9thxb"]
Jul 20 18:19:04.649330 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:19:04.649278    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9thxb"]
Jul 20 18:19:16.040772 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:19:16.040716    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9thxb"]
Jul 20 18:19:25.247295 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:19:25.247251    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9thxb"]
Jul 20 18:19:35.945410 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:19:35.945374    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9thxb"]
Jul 20 18:19:56.746937 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:19:56.746902    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9thxb"]
Jul 20 18:20:06.052209 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:20:06.052174    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9thxb"]
Jul 20 18:20:12.413402 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:20:12.413362    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-0f8f0c98-kserve-765c7cdbc5-rgb92"]
Jul 20 18:20:12.413793 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:20:12.413780    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6bf83091-d9ec-48ad-af80-a60a764d3eef" containerName="cleanup"
Jul 20 18:20:12.413838 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:20:12.413796    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="6bf83091-d9ec-48ad-af80-a60a764d3eef" containerName="cleanup"
Jul 20 18:20:12.413900 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:20:12.413887    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="6bf83091-d9ec-48ad-af80-a60a764d3eef" containerName="cleanup"
Jul 20 18:20:12.417065 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:20:12.417044    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-0f8f0c98-kserve-765c7cdbc5-rgb92"
Jul 20 18:20:12.419626 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:20:12.419602    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-0f8f0c98-kserve-self-signed-certs\""
Jul 20 18:20:12.432063 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:20:12.432027    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-0f8f0c98-kserve-765c7cdbc5-rgb92"]
Jul 20 18:20:12.532637 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:20:12.532594    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wt65q\" (UniqueName: \"kubernetes.io/projected/09d9ec23-42d7-438c-90f8-472749766b05-kube-api-access-wt65q\") pod \"e2e-same-id-a-0f8f0c98-kserve-765c7cdbc5-rgb92\" (UID: \"09d9ec23-42d7-438c-90f8-472749766b05\") " pod="llm/e2e-same-id-a-0f8f0c98-kserve-765c7cdbc5-rgb92"
Jul 20 18:20:12.532844 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:20:12.532681    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09d9ec23-42d7-438c-90f8-472749766b05-model-cache\") pod \"e2e-same-id-a-0f8f0c98-kserve-765c7cdbc5-rgb92\" (UID: \"09d9ec23-42d7-438c-90f8-472749766b05\") " pod="llm/e2e-same-id-a-0f8f0c98-kserve-765c7cdbc5-rgb92"
Jul 20 18:20:12.532844 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:20:12.532715    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09d9ec23-42d7-438c-90f8-472749766b05-tmp-dir\") pod \"e2e-same-id-a-0f8f0c98-kserve-765c7cdbc5-rgb92\" (UID: \"09d9ec23-42d7-438c-90f8-472749766b05\") " pod="llm/e2e-same-id-a-0f8f0c98-kserve-765c7cdbc5-rgb92"
Jul 20 18:20:12.532844 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:20:12.532769    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09d9ec23-42d7-438c-90f8-472749766b05-tls-certs\") pod \"e2e-same-id-a-0f8f0c98-kserve-765c7cdbc5-rgb92\" (UID: \"09d9ec23-42d7-438c-90f8-472749766b05\") " pod="llm/e2e-same-id-a-0f8f0c98-kserve-765c7cdbc5-rgb92"
Jul 20 18:20:12.532844 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:20:12.532793    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09d9ec23-42d7-438c-90f8-472749766b05-home\") pod \"e2e-same-id-a-0f8f0c98-kserve-765c7cdbc5-rgb92\" (UID: \"09d9ec23-42d7-438c-90f8-472749766b05\") " pod="llm/e2e-same-id-a-0f8f0c98-kserve-765c7cdbc5-rgb92"
Jul 20 18:20:12.532844 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:20:12.532811    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09d9ec23-42d7-438c-90f8-472749766b05-dshm\") pod \"e2e-same-id-a-0f8f0c98-kserve-765c7cdbc5-rgb92\" (UID: \"09d9ec23-42d7-438c-90f8-472749766b05\") " pod="llm/e2e-same-id-a-0f8f0c98-kserve-765c7cdbc5-rgb92"
Jul 20 18:20:12.633905 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:20:12.633865    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09d9ec23-42d7-438c-90f8-472749766b05-tmp-dir\") pod \"e2e-same-id-a-0f8f0c98-kserve-765c7cdbc5-rgb92\" (UID: \"09d9ec23-42d7-438c-90f8-472749766b05\") " pod="llm/e2e-same-id-a-0f8f0c98-kserve-765c7cdbc5-rgb92"
Jul 20 18:20:12.634097 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:20:12.633916    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09d9ec23-42d7-438c-90f8-472749766b05-tls-certs\") pod \"e2e-same-id-a-0f8f0c98-kserve-765c7cdbc5-rgb92\" (UID: \"09d9ec23-42d7-438c-90f8-472749766b05\") " pod="llm/e2e-same-id-a-0f8f0c98-kserve-765c7cdbc5-rgb92"
Jul 20 18:20:12.634097 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:20:12.633940    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09d9ec23-42d7-438c-90f8-472749766b05-home\") pod \"e2e-same-id-a-0f8f0c98-kserve-765c7cdbc5-rgb92\" (UID: \"09d9ec23-42d7-438c-90f8-472749766b05\") " pod="llm/e2e-same-id-a-0f8f0c98-kserve-765c7cdbc5-rgb92"
Jul 20 18:20:12.634097 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:20:12.633969    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09d9ec23-42d7-438c-90f8-472749766b05-dshm\") pod \"e2e-same-id-a-0f8f0c98-kserve-765c7cdbc5-rgb92\" (UID: \"09d9ec23-42d7-438c-90f8-472749766b05\") " pod="llm/e2e-same-id-a-0f8f0c98-kserve-765c7cdbc5-rgb92"
Jul 20 18:20:12.634097 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:20:12.633999    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wt65q\" (UniqueName: \"kubernetes.io/projected/09d9ec23-42d7-438c-90f8-472749766b05-kube-api-access-wt65q\") pod \"e2e-same-id-a-0f8f0c98-kserve-765c7cdbc5-rgb92\" (UID: \"09d9ec23-42d7-438c-90f8-472749766b05\") " pod="llm/e2e-same-id-a-0f8f0c98-kserve-765c7cdbc5-rgb92"
Jul 20 18:20:12.634097 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:20:12.634067    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09d9ec23-42d7-438c-90f8-472749766b05-model-cache\") pod \"e2e-same-id-a-0f8f0c98-kserve-765c7cdbc5-rgb92\" (UID: \"09d9ec23-42d7-438c-90f8-472749766b05\") " pod="llm/e2e-same-id-a-0f8f0c98-kserve-765c7cdbc5-rgb92"
Jul 20 18:20:12.634521 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:20:12.634360    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09d9ec23-42d7-438c-90f8-472749766b05-tmp-dir\") pod \"e2e-same-id-a-0f8f0c98-kserve-765c7cdbc5-rgb92\" (UID: \"09d9ec23-42d7-438c-90f8-472749766b05\") " pod="llm/e2e-same-id-a-0f8f0c98-kserve-765c7cdbc5-rgb92"
Jul 20 18:20:12.634521 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:20:12.634387    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09d9ec23-42d7-438c-90f8-472749766b05-home\") pod \"e2e-same-id-a-0f8f0c98-kserve-765c7cdbc5-rgb92\" (UID: \"09d9ec23-42d7-438c-90f8-472749766b05\") " pod="llm/e2e-same-id-a-0f8f0c98-kserve-765c7cdbc5-rgb92"
Jul 20 18:20:12.634521 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:20:12.634492    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09d9ec23-42d7-438c-90f8-472749766b05-model-cache\") pod \"e2e-same-id-a-0f8f0c98-kserve-765c7cdbc5-rgb92\" (UID: \"09d9ec23-42d7-438c-90f8-472749766b05\") " pod="llm/e2e-same-id-a-0f8f0c98-kserve-765c7cdbc5-rgb92"
Jul 20 18:20:12.636904 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:20:12.636876    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09d9ec23-42d7-438c-90f8-472749766b05-dshm\") pod \"e2e-same-id-a-0f8f0c98-kserve-765c7cdbc5-rgb92\" (UID: \"09d9ec23-42d7-438c-90f8-472749766b05\") " pod="llm/e2e-same-id-a-0f8f0c98-kserve-765c7cdbc5-rgb92"
Jul 20 18:20:12.637032 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:20:12.637015    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09d9ec23-42d7-438c-90f8-472749766b05-tls-certs\") pod \"e2e-same-id-a-0f8f0c98-kserve-765c7cdbc5-rgb92\" (UID: \"09d9ec23-42d7-438c-90f8-472749766b05\") " pod="llm/e2e-same-id-a-0f8f0c98-kserve-765c7cdbc5-rgb92"
Jul 20 18:20:12.642675 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:20:12.642651    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wt65q\" (UniqueName: \"kubernetes.io/projected/09d9ec23-42d7-438c-90f8-472749766b05-kube-api-access-wt65q\") pod \"e2e-same-id-a-0f8f0c98-kserve-765c7cdbc5-rgb92\" (UID: \"09d9ec23-42d7-438c-90f8-472749766b05\") " pod="llm/e2e-same-id-a-0f8f0c98-kserve-765c7cdbc5-rgb92"
Jul 20 18:20:12.726574 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:20:12.726489    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-0f8f0c98-kserve-765c7cdbc5-rgb92"
Jul 20 18:20:12.857351 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:20:12.857210    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-0f8f0c98-kserve-765c7cdbc5-rgb92"]
Jul 20 18:20:12.859644 ip-10-0-137-247 kubenswrapper[2569]: W0720 18:20:12.859617    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod09d9ec23_42d7_438c_90f8_472749766b05.slice/crio-f1edd9e126ff3cd05e812e17d4d55dff1574144e05015e7a0a5fa1513ef0adcd WatchSource:0}: Error finding container f1edd9e126ff3cd05e812e17d4d55dff1574144e05015e7a0a5fa1513ef0adcd: Status 404 returned error can't find the container with id f1edd9e126ff3cd05e812e17d4d55dff1574144e05015e7a0a5fa1513ef0adcd
Jul 20 18:20:13.707311 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:20:13.707269    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-0f8f0c98-kserve-765c7cdbc5-rgb92" event={"ID":"09d9ec23-42d7-438c-90f8-472749766b05","Type":"ContainerStarted","Data":"32a795db4b11d35296ea02e8ecb999b7bafcc41572f4dfcd08318c8c379db444"}
Jul 20 18:20:13.707669 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:20:13.707319    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-0f8f0c98-kserve-765c7cdbc5-rgb92" event={"ID":"09d9ec23-42d7-438c-90f8-472749766b05","Type":"ContainerStarted","Data":"f1edd9e126ff3cd05e812e17d4d55dff1574144e05015e7a0a5fa1513ef0adcd"}
Jul 20 18:20:13.726219 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:20:13.726167    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-0f8f0c98-kserve-765c7cdbc5-rgb92" podStartSLOduration=1.726151696 podStartE2EDuration="1.726151696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 18:20:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 18:20:13.724611492 +0000 UTC m=+1459.971144937" watchObservedRunningTime="2026-07-20 18:20:13.726151696 +0000 UTC m=+1459.972685106"
Jul 20 18:20:17.001267 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:20:17.001232    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9thxb"]
Jul 20 18:20:22.727168 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:20:22.727070    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-0f8f0c98-kserve-765c7cdbc5-rgb92"
Jul 20 18:20:22.727168 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:20:22.727130    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-0f8f0c98-kserve-765c7cdbc5-rgb92"
Jul 20 18:20:22.739738 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:20:22.739691    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-0f8f0c98-kserve-765c7cdbc5-rgb92"
Jul 20 18:20:22.755567 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:20:22.755530    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-0f8f0c98-kserve-765c7cdbc5-rgb92"
Jul 20 18:20:24.386498 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:20:24.386461    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-0f8f0c98-kserve-6fc74f9b79-bqwxz"]
Jul 20 18:20:24.389704 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:20:24.389684    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-0f8f0c98-kserve-6fc74f9b79-bqwxz"
Jul 20 18:20:24.391909 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:20:24.391888    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-0f8f0c98-kserve-self-signed-certs\""
Jul 20 18:20:24.400395 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:20:24.400372    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-0f8f0c98-kserve-6fc74f9b79-bqwxz"]
Jul 20 18:20:24.429953 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:20:24.429923    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bda80bf1-f32f-446d-9a94-e057ee0575c6-tls-certs\") pod \"e2e-same-id-b-0f8f0c98-kserve-6fc74f9b79-bqwxz\" (UID: \"bda80bf1-f32f-446d-9a94-e057ee0575c6\") " pod="llm/e2e-same-id-b-0f8f0c98-kserve-6fc74f9b79-bqwxz"
Jul 20 18:20:24.430098 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:20:24.429960    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bda80bf1-f32f-446d-9a94-e057ee0575c6-model-cache\") pod \"e2e-same-id-b-0f8f0c98-kserve-6fc74f9b79-bqwxz\" (UID: \"bda80bf1-f32f-446d-9a94-e057ee0575c6\") " pod="llm/e2e-same-id-b-0f8f0c98-kserve-6fc74f9b79-bqwxz"
Jul 20 18:20:24.430098 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:20:24.429988    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ff5m\" (UniqueName: \"kubernetes.io/projected/bda80bf1-f32f-446d-9a94-e057ee0575c6-kube-api-access-9ff5m\") pod \"e2e-same-id-b-0f8f0c98-kserve-6fc74f9b79-bqwxz\" (UID: \"bda80bf1-f32f-446d-9a94-e057ee0575c6\") " pod="llm/e2e-same-id-b-0f8f0c98-kserve-6fc74f9b79-bqwxz"
Jul 20 18:20:24.430098 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:20:24.430005    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bda80bf1-f32f-446d-9a94-e057ee0575c6-home\") pod \"e2e-same-id-b-0f8f0c98-kserve-6fc74f9b79-bqwxz\" (UID: \"bda80bf1-f32f-446d-9a94-e057ee0575c6\") " pod="llm/e2e-same-id-b-0f8f0c98-kserve-6fc74f9b79-bqwxz"
Jul 20 18:20:24.430246 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:20:24.430102    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bda80bf1-f32f-446d-9a94-e057ee0575c6-dshm\") pod \"e2e-same-id-b-0f8f0c98-kserve-6fc74f9b79-bqwxz\" (UID: \"bda80bf1-f32f-446d-9a94-e057ee0575c6\") " pod="llm/e2e-same-id-b-0f8f0c98-kserve-6fc74f9b79-bqwxz"
Jul 20 18:20:24.430246 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:20:24.430144    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bda80bf1-f32f-446d-9a94-e057ee0575c6-tmp-dir\") pod \"e2e-same-id-b-0f8f0c98-kserve-6fc74f9b79-bqwxz\" (UID: \"bda80bf1-f32f-446d-9a94-e057ee0575c6\") " pod="llm/e2e-same-id-b-0f8f0c98-kserve-6fc74f9b79-bqwxz"
Jul 20 18:20:24.531350 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:20:24.531313    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bda80bf1-f32f-446d-9a94-e057ee0575c6-tls-certs\") pod \"e2e-same-id-b-0f8f0c98-kserve-6fc74f9b79-bqwxz\" (UID: \"bda80bf1-f32f-446d-9a94-e057ee0575c6\") " pod="llm/e2e-same-id-b-0f8f0c98-kserve-6fc74f9b79-bqwxz"
Jul 20 18:20:24.531350 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:20:24.531355    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bda80bf1-f32f-446d-9a94-e057ee0575c6-model-cache\") pod \"e2e-same-id-b-0f8f0c98-kserve-6fc74f9b79-bqwxz\" (UID: \"bda80bf1-f32f-446d-9a94-e057ee0575c6\") " pod="llm/e2e-same-id-b-0f8f0c98-kserve-6fc74f9b79-bqwxz"
Jul 20 18:20:24.531592 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:20:24.531381    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ff5m\" (UniqueName: \"kubernetes.io/projected/bda80bf1-f32f-446d-9a94-e057ee0575c6-kube-api-access-9ff5m\") pod \"e2e-same-id-b-0f8f0c98-kserve-6fc74f9b79-bqwxz\" (UID: \"bda80bf1-f32f-446d-9a94-e057ee0575c6\") " pod="llm/e2e-same-id-b-0f8f0c98-kserve-6fc74f9b79-bqwxz"
Jul 20 18:20:24.531592 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:20:24.531401    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bda80bf1-f32f-446d-9a94-e057ee0575c6-home\") pod \"e2e-same-id-b-0f8f0c98-kserve-6fc74f9b79-bqwxz\" (UID: \"bda80bf1-f32f-446d-9a94-e057ee0575c6\") " pod="llm/e2e-same-id-b-0f8f0c98-kserve-6fc74f9b79-bqwxz"
Jul 20 18:20:24.531592 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:20:24.531434    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bda80bf1-f32f-446d-9a94-e057ee0575c6-dshm\") pod \"e2e-same-id-b-0f8f0c98-kserve-6fc74f9b79-bqwxz\" (UID: \"bda80bf1-f32f-446d-9a94-e057ee0575c6\") " pod="llm/e2e-same-id-b-0f8f0c98-kserve-6fc74f9b79-bqwxz"
Jul 20 18:20:24.531765 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:20:24.531616    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bda80bf1-f32f-446d-9a94-e057ee0575c6-tmp-dir\") pod \"e2e-same-id-b-0f8f0c98-kserve-6fc74f9b79-bqwxz\" (UID: \"bda80bf1-f32f-446d-9a94-e057ee0575c6\") " pod="llm/e2e-same-id-b-0f8f0c98-kserve-6fc74f9b79-bqwxz"
Jul 20 18:20:24.531880 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:20:24.531852    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bda80bf1-f32f-446d-9a94-e057ee0575c6-model-cache\") pod \"e2e-same-id-b-0f8f0c98-kserve-6fc74f9b79-bqwxz\" (UID: \"bda80bf1-f32f-446d-9a94-e057ee0575c6\") " pod="llm/e2e-same-id-b-0f8f0c98-kserve-6fc74f9b79-bqwxz"
Jul 20 18:20:24.531994 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:20:24.531889    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bda80bf1-f32f-446d-9a94-e057ee0575c6-home\") pod \"e2e-same-id-b-0f8f0c98-kserve-6fc74f9b79-bqwxz\" (UID: \"bda80bf1-f32f-446d-9a94-e057ee0575c6\") " pod="llm/e2e-same-id-b-0f8f0c98-kserve-6fc74f9b79-bqwxz"
Jul 20 18:20:24.532038 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:20:24.531995    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bda80bf1-f32f-446d-9a94-e057ee0575c6-tmp-dir\") pod \"e2e-same-id-b-0f8f0c98-kserve-6fc74f9b79-bqwxz\" (UID: \"bda80bf1-f32f-446d-9a94-e057ee0575c6\") " pod="llm/e2e-same-id-b-0f8f0c98-kserve-6fc74f9b79-bqwxz"
Jul 20 18:20:24.533742 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:20:24.533693    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bda80bf1-f32f-446d-9a94-e057ee0575c6-dshm\") pod \"e2e-same-id-b-0f8f0c98-kserve-6fc74f9b79-bqwxz\" (UID: \"bda80bf1-f32f-446d-9a94-e057ee0575c6\") " pod="llm/e2e-same-id-b-0f8f0c98-kserve-6fc74f9b79-bqwxz"
Jul 20 18:20:24.534017 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:20:24.533998    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bda80bf1-f32f-446d-9a94-e057ee0575c6-tls-certs\") pod \"e2e-same-id-b-0f8f0c98-kserve-6fc74f9b79-bqwxz\" (UID: \"bda80bf1-f32f-446d-9a94-e057ee0575c6\") " pod="llm/e2e-same-id-b-0f8f0c98-kserve-6fc74f9b79-bqwxz"
Jul 20 18:20:24.538660 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:20:24.538643    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ff5m\" (UniqueName: \"kubernetes.io/projected/bda80bf1-f32f-446d-9a94-e057ee0575c6-kube-api-access-9ff5m\") pod \"e2e-same-id-b-0f8f0c98-kserve-6fc74f9b79-bqwxz\" (UID: \"bda80bf1-f32f-446d-9a94-e057ee0575c6\") " pod="llm/e2e-same-id-b-0f8f0c98-kserve-6fc74f9b79-bqwxz"
Jul 20 18:20:24.700112 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:20:24.700021    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-0f8f0c98-kserve-6fc74f9b79-bqwxz"
Jul 20 18:20:24.833248 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:20:24.833217    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-0f8f0c98-kserve-6fc74f9b79-bqwxz"]
Jul 20 18:20:24.836267 ip-10-0-137-247 kubenswrapper[2569]: W0720 18:20:24.836240    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbda80bf1_f32f_446d_9a94_e057ee0575c6.slice/crio-2078c49a5a96ef6df76e0e2b1bf123b0522f468e23c6ad0795e9660d8c685566 WatchSource:0}: Error finding container 2078c49a5a96ef6df76e0e2b1bf123b0522f468e23c6ad0795e9660d8c685566: Status 404 returned error can't find the container with id 2078c49a5a96ef6df76e0e2b1bf123b0522f468e23c6ad0795e9660d8c685566
Jul 20 18:20:25.759287 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:20:25.759245    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-0f8f0c98-kserve-6fc74f9b79-bqwxz" event={"ID":"bda80bf1-f32f-446d-9a94-e057ee0575c6","Type":"ContainerStarted","Data":"c850143d1a40e837a6340faccdec029b1d7fd57ce999d1c629a8e80a45c724e8"}
Jul 20 18:20:25.759760 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:20:25.759297    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-0f8f0c98-kserve-6fc74f9b79-bqwxz" event={"ID":"bda80bf1-f32f-446d-9a94-e057ee0575c6","Type":"ContainerStarted","Data":"2078c49a5a96ef6df76e0e2b1bf123b0522f468e23c6ad0795e9660d8c685566"}
Jul 20 18:20:25.777086 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:20:25.777030    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-0f8f0c98-kserve-6fc74f9b79-bqwxz" podStartSLOduration=1.777008935 podStartE2EDuration="1.777008935s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 18:20:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 18:20:25.77653652 +0000 UTC m=+1472.023069930" watchObservedRunningTime="2026-07-20 18:20:25.777008935 +0000 UTC m=+1472.023542346"
Jul 20 18:20:28.451671 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:20:28.451631    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9thxb"]
Jul 20 18:20:34.700938 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:20:34.700900    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-0f8f0c98-kserve-6fc74f9b79-bqwxz"
Jul 20 18:20:34.700938 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:20:34.700951    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-0f8f0c98-kserve-6fc74f9b79-bqwxz"
Jul 20 18:20:34.713526 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:20:34.713497    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-0f8f0c98-kserve-6fc74f9b79-bqwxz"
Jul 20 18:20:34.813773 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:20:34.813742    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-0f8f0c98-kserve-6fc74f9b79-bqwxz"
Jul 20 18:20:54.178062 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:20:54.178029    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wjs8g_07587a78-2d3a-40ad-9aae-a4875200263a/ovn-acl-logging/0.log"
Jul 20 18:20:54.179767 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:20:54.179746    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wjs8g_07587a78-2d3a-40ad-9aae-a4875200263a/ovn-acl-logging/0.log"
Jul 20 18:20:56.517788 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:20:56.517756    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-0f8f0c98-kserve-765c7cdbc5-rgb92"]
Jul 20 18:20:57.665780 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:20:57.665736    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-0f8f0c98-kserve-6fc74f9b79-bqwxz"]
Jul 20 18:21:00.744800 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:21:00.744763    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9thxb"]
Jul 20 18:21:11.537566 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:21:11.537523    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-0f8f0c98-kserve-765c7cdbc5-rgb92" podUID="09d9ec23-42d7-438c-90f8-472749766b05" containerName="main" containerID="cri-o://32a795db4b11d35296ea02e8ecb999b7bafcc41572f4dfcd08318c8c379db444" gracePeriod=45
Jul 20 18:21:11.777258 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:21:11.777235    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-0f8f0c98-kserve-765c7cdbc5-rgb92"
Jul 20 18:21:11.812500 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:21:11.812437    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wt65q\" (UniqueName: \"kubernetes.io/projected/09d9ec23-42d7-438c-90f8-472749766b05-kube-api-access-wt65q\") pod \"09d9ec23-42d7-438c-90f8-472749766b05\" (UID: \"09d9ec23-42d7-438c-90f8-472749766b05\") "
Jul 20 18:21:11.812500 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:21:11.812469    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09d9ec23-42d7-438c-90f8-472749766b05-tls-certs\") pod \"09d9ec23-42d7-438c-90f8-472749766b05\" (UID: \"09d9ec23-42d7-438c-90f8-472749766b05\") "
Jul 20 18:21:11.812676 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:21:11.812510    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09d9ec23-42d7-438c-90f8-472749766b05-model-cache\") pod \"09d9ec23-42d7-438c-90f8-472749766b05\" (UID: \"09d9ec23-42d7-438c-90f8-472749766b05\") "
Jul 20 18:21:11.812676 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:21:11.812537    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09d9ec23-42d7-438c-90f8-472749766b05-home\") pod \"09d9ec23-42d7-438c-90f8-472749766b05\" (UID: \"09d9ec23-42d7-438c-90f8-472749766b05\") "
Jul 20 18:21:11.812676 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:21:11.812557    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09d9ec23-42d7-438c-90f8-472749766b05-tmp-dir\") pod \"09d9ec23-42d7-438c-90f8-472749766b05\" (UID: \"09d9ec23-42d7-438c-90f8-472749766b05\") "
Jul 20 18:21:11.812676 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:21:11.812594    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09d9ec23-42d7-438c-90f8-472749766b05-dshm\") pod \"09d9ec23-42d7-438c-90f8-472749766b05\" (UID: \"09d9ec23-42d7-438c-90f8-472749766b05\") "
Jul 20 18:21:11.812904 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:21:11.812759    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09d9ec23-42d7-438c-90f8-472749766b05-model-cache" (OuterVolumeSpecName: "model-cache") pod "09d9ec23-42d7-438c-90f8-472749766b05" (UID: "09d9ec23-42d7-438c-90f8-472749766b05"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 18:21:11.812904 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:21:11.812861    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09d9ec23-42d7-438c-90f8-472749766b05-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "09d9ec23-42d7-438c-90f8-472749766b05" (UID: "09d9ec23-42d7-438c-90f8-472749766b05"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 18:21:11.812904 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:21:11.812878    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09d9ec23-42d7-438c-90f8-472749766b05-home" (OuterVolumeSpecName: "home") pod "09d9ec23-42d7-438c-90f8-472749766b05" (UID: "09d9ec23-42d7-438c-90f8-472749766b05"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 18:21:11.814643 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:21:11.814614    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09d9ec23-42d7-438c-90f8-472749766b05-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "09d9ec23-42d7-438c-90f8-472749766b05" (UID: "09d9ec23-42d7-438c-90f8-472749766b05"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 18:21:11.814762 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:21:11.814708    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09d9ec23-42d7-438c-90f8-472749766b05-dshm" (OuterVolumeSpecName: "dshm") pod "09d9ec23-42d7-438c-90f8-472749766b05" (UID: "09d9ec23-42d7-438c-90f8-472749766b05"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 18:21:11.814762 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:21:11.814741    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09d9ec23-42d7-438c-90f8-472749766b05-kube-api-access-wt65q" (OuterVolumeSpecName: "kube-api-access-wt65q") pod "09d9ec23-42d7-438c-90f8-472749766b05" (UID: "09d9ec23-42d7-438c-90f8-472749766b05"). InnerVolumeSpecName "kube-api-access-wt65q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 18:21:11.913131 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:21:11.913098    2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09d9ec23-42d7-438c-90f8-472749766b05-dshm\") on node \"ip-10-0-137-247.ec2.internal\" DevicePath \"\""
Jul 20 18:21:11.913131 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:21:11.913127    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wt65q\" (UniqueName: \"kubernetes.io/projected/09d9ec23-42d7-438c-90f8-472749766b05-kube-api-access-wt65q\") on node \"ip-10-0-137-247.ec2.internal\" DevicePath \"\""
Jul 20 18:21:11.913131 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:21:11.913137    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09d9ec23-42d7-438c-90f8-472749766b05-tls-certs\") on node \"ip-10-0-137-247.ec2.internal\" DevicePath \"\""
Jul 20 18:21:11.913341 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:21:11.913146    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09d9ec23-42d7-438c-90f8-472749766b05-model-cache\") on node \"ip-10-0-137-247.ec2.internal\" DevicePath \"\""
Jul 20 18:21:11.913341 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:21:11.913154    2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09d9ec23-42d7-438c-90f8-472749766b05-home\") on node \"ip-10-0-137-247.ec2.internal\" DevicePath \"\""
Jul 20 18:21:11.913341 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:21:11.913160    2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09d9ec23-42d7-438c-90f8-472749766b05-tmp-dir\") on node \"ip-10-0-137-247.ec2.internal\" DevicePath \"\""
Jul 20 18:21:11.957699 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:21:11.957665    2569 generic.go:356] "Generic (PLEG): container finished" podID="09d9ec23-42d7-438c-90f8-472749766b05" containerID="32a795db4b11d35296ea02e8ecb999b7bafcc41572f4dfcd08318c8c379db444" exitCode=0
Jul 20 18:21:11.957846 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:21:11.957708    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-0f8f0c98-kserve-765c7cdbc5-rgb92"
Jul 20 18:21:11.957846 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:21:11.957761    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-0f8f0c98-kserve-765c7cdbc5-rgb92" event={"ID":"09d9ec23-42d7-438c-90f8-472749766b05","Type":"ContainerDied","Data":"32a795db4b11d35296ea02e8ecb999b7bafcc41572f4dfcd08318c8c379db444"}
Jul 20 18:21:11.957846 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:21:11.957807    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-0f8f0c98-kserve-765c7cdbc5-rgb92" event={"ID":"09d9ec23-42d7-438c-90f8-472749766b05","Type":"ContainerDied","Data":"f1edd9e126ff3cd05e812e17d4d55dff1574144e05015e7a0a5fa1513ef0adcd"}
Jul 20 18:21:11.957846 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:21:11.957829    2569 scope.go:117] "RemoveContainer" containerID="32a795db4b11d35296ea02e8ecb999b7bafcc41572f4dfcd08318c8c379db444"
Jul 20 18:21:11.966233 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:21:11.966214    2569 scope.go:117] "RemoveContainer" containerID="32a795db4b11d35296ea02e8ecb999b7bafcc41572f4dfcd08318c8c379db444"
Jul 20 18:21:11.966468 ip-10-0-137-247 kubenswrapper[2569]: E0720 18:21:11.966449    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32a795db4b11d35296ea02e8ecb999b7bafcc41572f4dfcd08318c8c379db444\": container with ID starting with 32a795db4b11d35296ea02e8ecb999b7bafcc41572f4dfcd08318c8c379db444 not found: ID does not exist" containerID="32a795db4b11d35296ea02e8ecb999b7bafcc41572f4dfcd08318c8c379db444"
Jul 20 18:21:11.966519 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:21:11.966476    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32a795db4b11d35296ea02e8ecb999b7bafcc41572f4dfcd08318c8c379db444"} err="failed to get container status \"32a795db4b11d35296ea02e8ecb999b7bafcc41572f4dfcd08318c8c379db444\": rpc error: code = NotFound desc = could not find container \"32a795db4b11d35296ea02e8ecb999b7bafcc41572f4dfcd08318c8c379db444\": container with ID starting with 32a795db4b11d35296ea02e8ecb999b7bafcc41572f4dfcd08318c8c379db444 not found: ID does not exist"
Jul 20 18:21:11.978232 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:21:11.978211    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-0f8f0c98-kserve-765c7cdbc5-rgb92"]
Jul 20 18:21:11.982083 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:21:11.982063    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-0f8f0c98-kserve-765c7cdbc5-rgb92"]
Jul 20 18:21:12.181928 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:21:12.181839    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="09d9ec23-42d7-438c-90f8-472749766b05" path="/var/lib/kubelet/pods/09d9ec23-42d7-438c-90f8-472749766b05/volumes"
Jul 20 18:21:12.685362 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:21:12.685321    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-0f8f0c98-kserve-6fc74f9b79-bqwxz" podUID="bda80bf1-f32f-446d-9a94-e057ee0575c6" containerName="main" containerID="cri-o://c850143d1a40e837a6340faccdec029b1d7fd57ce999d1c629a8e80a45c724e8" gracePeriod=45
Jul 20 18:21:12.932098 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:21:12.932073    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-0f8f0c98-kserve-6fc74f9b79-bqwxz"
Jul 20 18:21:12.964169 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:21:12.964093    2569 generic.go:356] "Generic (PLEG): container finished" podID="bda80bf1-f32f-446d-9a94-e057ee0575c6" containerID="c850143d1a40e837a6340faccdec029b1d7fd57ce999d1c629a8e80a45c724e8" exitCode=0
Jul 20 18:21:12.964169 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:21:12.964128    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-0f8f0c98-kserve-6fc74f9b79-bqwxz" event={"ID":"bda80bf1-f32f-446d-9a94-e057ee0575c6","Type":"ContainerDied","Data":"c850143d1a40e837a6340faccdec029b1d7fd57ce999d1c629a8e80a45c724e8"}
Jul 20 18:21:12.964169 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:21:12.964151    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-0f8f0c98-kserve-6fc74f9b79-bqwxz" event={"ID":"bda80bf1-f32f-446d-9a94-e057ee0575c6","Type":"ContainerDied","Data":"2078c49a5a96ef6df76e0e2b1bf123b0522f468e23c6ad0795e9660d8c685566"}
Jul 20 18:21:12.964169 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:21:12.964154    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-0f8f0c98-kserve-6fc74f9b79-bqwxz"
Jul 20 18:21:12.964169 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:21:12.964166    2569 scope.go:117] "RemoveContainer" containerID="c850143d1a40e837a6340faccdec029b1d7fd57ce999d1c629a8e80a45c724e8"
Jul 20 18:21:12.971992 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:21:12.971975    2569 scope.go:117] "RemoveContainer" containerID="c850143d1a40e837a6340faccdec029b1d7fd57ce999d1c629a8e80a45c724e8"
Jul 20 18:21:12.972225 ip-10-0-137-247 kubenswrapper[2569]: E0720 18:21:12.972207    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c850143d1a40e837a6340faccdec029b1d7fd57ce999d1c629a8e80a45c724e8\": container with ID starting with c850143d1a40e837a6340faccdec029b1d7fd57ce999d1c629a8e80a45c724e8 not found: ID does not exist" containerID="c850143d1a40e837a6340faccdec029b1d7fd57ce999d1c629a8e80a45c724e8"
Jul 20 18:21:12.972294 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:21:12.972230    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c850143d1a40e837a6340faccdec029b1d7fd57ce999d1c629a8e80a45c724e8"} err="failed to get container status \"c850143d1a40e837a6340faccdec029b1d7fd57ce999d1c629a8e80a45c724e8\": rpc error: code = NotFound desc = could not find container \"c850143d1a40e837a6340faccdec029b1d7fd57ce999d1c629a8e80a45c724e8\": container with ID starting with c850143d1a40e837a6340faccdec029b1d7fd57ce999d1c629a8e80a45c724e8 not found: ID does not exist"
Jul 20 18:21:13.019877 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:21:13.019848    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bda80bf1-f32f-446d-9a94-e057ee0575c6-tls-certs\") pod \"bda80bf1-f32f-446d-9a94-e057ee0575c6\" (UID: \"bda80bf1-f32f-446d-9a94-e057ee0575c6\") "
Jul 20 18:21:13.019983 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:21:13.019893    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bda80bf1-f32f-446d-9a94-e057ee0575c6-tmp-dir\") pod \"bda80bf1-f32f-446d-9a94-e057ee0575c6\" (UID: \"bda80bf1-f32f-446d-9a94-e057ee0575c6\") "
Jul 20 18:21:13.019983 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:21:13.019921    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9ff5m\" (UniqueName: \"kubernetes.io/projected/bda80bf1-f32f-446d-9a94-e057ee0575c6-kube-api-access-9ff5m\") pod \"bda80bf1-f32f-446d-9a94-e057ee0575c6\" (UID: \"bda80bf1-f32f-446d-9a94-e057ee0575c6\") "
Jul 20 18:21:13.020053 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:21:13.020025    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bda80bf1-f32f-446d-9a94-e057ee0575c6-home\") pod \"bda80bf1-f32f-446d-9a94-e057ee0575c6\" (UID: \"bda80bf1-f32f-446d-9a94-e057ee0575c6\") "
Jul 20 18:21:13.020130 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:21:13.020114    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bda80bf1-f32f-446d-9a94-e057ee0575c6-dshm\") pod \"bda80bf1-f32f-446d-9a94-e057ee0575c6\" (UID: \"bda80bf1-f32f-446d-9a94-e057ee0575c6\") "
Jul 20 18:21:13.020232 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:21:13.020190    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bda80bf1-f32f-446d-9a94-e057ee0575c6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bda80bf1-f32f-446d-9a94-e057ee0575c6" (UID: "bda80bf1-f32f-446d-9a94-e057ee0575c6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 18:21:13.020366 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:21:13.020257    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bda80bf1-f32f-446d-9a94-e057ee0575c6-home" (OuterVolumeSpecName: "home") pod "bda80bf1-f32f-446d-9a94-e057ee0575c6" (UID: "bda80bf1-f32f-446d-9a94-e057ee0575c6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 18:21:13.021942 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:21:13.021920    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bda80bf1-f32f-446d-9a94-e057ee0575c6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bda80bf1-f32f-446d-9a94-e057ee0575c6" (UID: "bda80bf1-f32f-446d-9a94-e057ee0575c6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 18:21:13.022029 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:21:13.021982    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bda80bf1-f32f-446d-9a94-e057ee0575c6-dshm" (OuterVolumeSpecName: "dshm") pod "bda80bf1-f32f-446d-9a94-e057ee0575c6" (UID: "bda80bf1-f32f-446d-9a94-e057ee0575c6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 18:21:13.022072 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:21:13.022025    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bda80bf1-f32f-446d-9a94-e057ee0575c6-kube-api-access-9ff5m" (OuterVolumeSpecName: "kube-api-access-9ff5m") pod "bda80bf1-f32f-446d-9a94-e057ee0575c6" (UID: "bda80bf1-f32f-446d-9a94-e057ee0575c6"). InnerVolumeSpecName "kube-api-access-9ff5m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 18:21:13.120603 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:21:13.120561    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bda80bf1-f32f-446d-9a94-e057ee0575c6-model-cache\") pod \"bda80bf1-f32f-446d-9a94-e057ee0575c6\" (UID: \"bda80bf1-f32f-446d-9a94-e057ee0575c6\") "
Jul 20 18:21:13.120786 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:21:13.120714    2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bda80bf1-f32f-446d-9a94-e057ee0575c6-tmp-dir\") on node \"ip-10-0-137-247.ec2.internal\" DevicePath \"\""
Jul 20 18:21:13.120786 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:21:13.120756    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ff5m\" (UniqueName: \"kubernetes.io/projected/bda80bf1-f32f-446d-9a94-e057ee0575c6-kube-api-access-9ff5m\") on node \"ip-10-0-137-247.ec2.internal\" DevicePath \"\""
Jul 20 18:21:13.120786 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:21:13.120765    2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bda80bf1-f32f-446d-9a94-e057ee0575c6-home\") on node \"ip-10-0-137-247.ec2.internal\" DevicePath \"\""
Jul 20 18:21:13.120786 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:21:13.120774    2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bda80bf1-f32f-446d-9a94-e057ee0575c6-dshm\") on node \"ip-10-0-137-247.ec2.internal\" DevicePath \"\""
Jul 20 18:21:13.120786 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:21:13.120781    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bda80bf1-f32f-446d-9a94-e057ee0575c6-tls-certs\") on node \"ip-10-0-137-247.ec2.internal\" DevicePath \"\""
Jul 20 18:21:13.120994 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:21:13.120866    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bda80bf1-f32f-446d-9a94-e057ee0575c6-model-cache" (OuterVolumeSpecName: "model-cache") pod "bda80bf1-f32f-446d-9a94-e057ee0575c6" (UID: "bda80bf1-f32f-446d-9a94-e057ee0575c6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 18:21:13.221632 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:21:13.221557    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bda80bf1-f32f-446d-9a94-e057ee0575c6-model-cache\") on node \"ip-10-0-137-247.ec2.internal\" DevicePath \"\""
Jul 20 18:21:13.285435 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:21:13.285405    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-0f8f0c98-kserve-6fc74f9b79-bqwxz"]
Jul 20 18:21:13.287462 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:21:13.287441    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-0f8f0c98-kserve-6fc74f9b79-bqwxz"]
Jul 20 18:21:14.181871 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:21:14.181833    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bda80bf1-f32f-446d-9a94-e057ee0575c6" path="/var/lib/kubelet/pods/bda80bf1-f32f-446d-9a94-e057ee0575c6/volumes"
Jul 20 18:21:14.549908 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:21:14.549874    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9thxb"]
Jul 20 18:21:21.447123 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:21:21.447043    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9thxb"]
Jul 20 18:21:30.747301 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:21:30.747263    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9thxb"]
Jul 20 18:21:39.055852 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:21:39.055815    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9thxb"]
Jul 20 18:21:48.756161 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:21:48.756121    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9thxb"]
Jul 20 18:21:57.844749 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:21:57.844679    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9thxb"]
Jul 20 18:22:08.260005 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:22:08.259971    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9thxb"]
Jul 20 18:22:17.368057 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:22:17.368020    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9thxb"]
Jul 20 18:22:26.742912 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:22:26.742865    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9thxb"]
Jul 20 18:22:41.747770 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:22:41.747717    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9thxb"]
Jul 20 18:22:50.944297 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:22:50.944256    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9thxb"]
Jul 20 18:23:00.282159 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:23:00.282121    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9thxb"]
Jul 20 18:23:11.349614 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:23:11.349577    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9thxb"]
Jul 20 18:23:30.147565 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:23:30.147530    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9thxb"]
Jul 20 18:23:43.040542 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:23:43.040502    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9thxb"]
Jul 20 18:23:55.150102 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:23:55.150066    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9thxb"]
Jul 20 18:24:04.342158 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:24:04.342115    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9thxb"]
Jul 20 18:24:17.958222 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:24:17.958129    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9thxb"]
Jul 20 18:24:26.942886 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:24:26.942840    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9thxb"]
Jul 20 18:24:36.246095 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:24:36.246056    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9thxb"]
Jul 20 18:24:47.105373 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:24:47.105340    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9thxb"]
Jul 20 18:25:05.197492 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:25:05.197461    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7f0c0a5b-wlz7w"]
Jul 20 18:25:05.197881 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:25:05.197756    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09d9ec23-42d7-438c-90f8-472749766b05" containerName="main"
Jul 20 18:25:05.197881 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:25:05.197768    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="09d9ec23-42d7-438c-90f8-472749766b05" containerName="main"
Jul 20 18:25:05.197881 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:25:05.197777    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bda80bf1-f32f-446d-9a94-e057ee0575c6" containerName="main"
Jul 20 18:25:05.197881 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:25:05.197783    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="bda80bf1-f32f-446d-9a94-e057ee0575c6" containerName="main"
Jul 20 18:25:05.197881 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:25:05.197823    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="09d9ec23-42d7-438c-90f8-472749766b05" containerName="main"
Jul 20 18:25:05.197881 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:25:05.197832    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="bda80bf1-f32f-446d-9a94-e057ee0575c6" containerName="main"
Jul 20 18:25:05.200548 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:25:05.200532    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7f0c0a5b-wlz7w"
Jul 20 18:25:05.204271 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:25:05.204243    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fk7rm\""
Jul 20 18:25:05.204423 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:25:05.204408    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 20 18:25:05.218204 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:25:05.218181    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7f0c0a5b-wlz7w"]
Jul 20 18:25:05.288548 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:25:05.288513    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a3c4389-90eb-484f-bb22-cb6d90639135-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7f0c0a5b-wlz7w\" (UID: \"2a3c4389-90eb-484f-bb22-cb6d90639135\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7f0c0a5b-wlz7w"
Jul 20 18:25:05.389594 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:25:05.389560    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a3c4389-90eb-484f-bb22-cb6d90639135-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7f0c0a5b-wlz7w\" (UID: \"2a3c4389-90eb-484f-bb22-cb6d90639135\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7f0c0a5b-wlz7w"
Jul 20 18:25:05.390191 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:25:05.390170    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a3c4389-90eb-484f-bb22-cb6d90639135-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7f0c0a5b-wlz7w\" (UID: \"2a3c4389-90eb-484f-bb22-cb6d90639135\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7f0c0a5b-wlz7w"
Jul 20 18:25:05.509990 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:25:05.509965    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7f0c0a5b-wlz7w"
Jul 20 18:25:05.635751 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:25:05.635713    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7f0c0a5b-wlz7w"]
Jul 20 18:25:05.638388 ip-10-0-137-247 kubenswrapper[2569]: W0720 18:25:05.638360    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a3c4389_90eb_484f_bb22_cb6d90639135.slice/crio-40643d7123aee8734ba21f42e3ab8392a49adbee30fafe20be275a1a9b0ba702 WatchSource:0}: Error finding container 40643d7123aee8734ba21f42e3ab8392a49adbee30fafe20be275a1a9b0ba702: Status 404 returned error can't find the container with id 40643d7123aee8734ba21f42e3ab8392a49adbee30fafe20be275a1a9b0ba702
Jul 20 18:25:05.911121 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:25:05.911086    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7f0c0a5b-wlz7w" event={"ID":"2a3c4389-90eb-484f-bb22-cb6d90639135","Type":"ContainerStarted","Data":"40643d7123aee8734ba21f42e3ab8392a49adbee30fafe20be275a1a9b0ba702"}
Jul 20 18:25:06.916255 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:25:06.916220    2569 generic.go:356] "Generic (PLEG): container finished" podID="2a3c4389-90eb-484f-bb22-cb6d90639135" containerID="a158241984c8524fea8c84155268cef7502264c62afe8704e50187cb4ac7e767" exitCode=0
Jul 20 18:25:06.916745 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:25:06.916309    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7f0c0a5b-wlz7w" event={"ID":"2a3c4389-90eb-484f-bb22-cb6d90639135","Type":"ContainerDied","Data":"a158241984c8524fea8c84155268cef7502264c62afe8704e50187cb4ac7e767"}
Jul 20 18:25:08.040453 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:25:08.040428    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7f0c0a5b-wlz7w"
Jul 20 18:25:08.110074 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:25:08.110042    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a3c4389-90eb-484f-bb22-cb6d90639135-maas-api-service-ca\") pod \"2a3c4389-90eb-484f-bb22-cb6d90639135\" (UID: \"2a3c4389-90eb-484f-bb22-cb6d90639135\") "
Jul 20 18:25:08.110226 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:25:08.110209    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a3c4389-90eb-484f-bb22-cb6d90639135-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2a3c4389-90eb-484f-bb22-cb6d90639135" (UID: "2a3c4389-90eb-484f-bb22-cb6d90639135"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 18:25:08.110279 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:25:08.110267    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a3c4389-90eb-484f-bb22-cb6d90639135-maas-api-service-ca\") on node \"ip-10-0-137-247.ec2.internal\" DevicePath \"\""
Jul 20 18:25:08.925789 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:25:08.925754    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7f0c0a5b-wlz7w" event={"ID":"2a3c4389-90eb-484f-bb22-cb6d90639135","Type":"ContainerDied","Data":"40643d7123aee8734ba21f42e3ab8392a49adbee30fafe20be275a1a9b0ba702"}
Jul 20 18:25:08.925789 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:25:08.925773    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7f0c0a5b-wlz7w"
Jul 20 18:25:08.925984 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:25:08.925807    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="40643d7123aee8734ba21f42e3ab8392a49adbee30fafe20be275a1a9b0ba702"
Jul 20 18:25:17.741579 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:25:17.741544    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9thxb"]
Jul 20 18:25:20.274870 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:25:20.274831    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7f0c0a5b-wlz7w"]
Jul 20 18:25:20.277955 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:25:20.277931    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7f0c0a5b-wlz7w"]
Jul 20 18:25:22.181513 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:25:22.181479    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a3c4389-90eb-484f-bb22-cb6d90639135" path="/var/lib/kubelet/pods/2a3c4389-90eb-484f-bb22-cb6d90639135/volumes"
Jul 20 18:25:29.764492 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:25:29.764457    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9thxb"]
Jul 20 18:25:51.500994 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:25:51.500954    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0ae63acc-7cp4j"]
Jul 20 18:25:51.501405 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:25:51.501241    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a3c4389-90eb-484f-bb22-cb6d90639135" containerName="revoke-keys"
Jul 20 18:25:51.501405 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:25:51.501252    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a3c4389-90eb-484f-bb22-cb6d90639135" containerName="revoke-keys"
Jul 20 18:25:51.501405 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:25:51.501308    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a3c4389-90eb-484f-bb22-cb6d90639135" containerName="revoke-keys"
Jul 20 18:25:51.504197 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:25:51.504180    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0ae63acc-7cp4j"
Jul 20 18:25:51.509404 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:25:51.509364    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 20 18:25:51.509404 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:25:51.509364    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fk7rm\""
Jul 20 18:25:51.529197 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:25:51.529167    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0ae63acc-7cp4j"]
Jul 20 18:25:51.565678 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:25:51.565635    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d35c652f-8e6f-4633-acd6-57952f773850-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0ae63acc-7cp4j\" (UID: \"d35c652f-8e6f-4633-acd6-57952f773850\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0ae63acc-7cp4j"
Jul 20 18:25:51.666833 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:25:51.666785    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d35c652f-8e6f-4633-acd6-57952f773850-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0ae63acc-7cp4j\" (UID: \"d35c652f-8e6f-4633-acd6-57952f773850\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0ae63acc-7cp4j"
Jul 20 18:25:51.667401 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:25:51.667380    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d35c652f-8e6f-4633-acd6-57952f773850-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0ae63acc-7cp4j\" (UID: \"d35c652f-8e6f-4633-acd6-57952f773850\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0ae63acc-7cp4j"
Jul 20 18:25:51.814034 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:25:51.813998    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0ae63acc-7cp4j"
Jul 20 18:25:51.942139 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:25:51.942112    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0ae63acc-7cp4j"]
Jul 20 18:25:51.944470 ip-10-0-137-247 kubenswrapper[2569]: W0720 18:25:51.944441    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd35c652f_8e6f_4633_acd6_57952f773850.slice/crio-af03ec8cc08c2a740322a0e16b815974a497c2a25bd63d693151460722741398 WatchSource:0}: Error finding container af03ec8cc08c2a740322a0e16b815974a497c2a25bd63d693151460722741398: Status 404 returned error can't find the container with id af03ec8cc08c2a740322a0e16b815974a497c2a25bd63d693151460722741398
Jul 20 18:25:52.100420 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:25:52.100383    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0ae63acc-7cp4j" event={"ID":"d35c652f-8e6f-4633-acd6-57952f773850","Type":"ContainerStarted","Data":"af03ec8cc08c2a740322a0e16b815974a497c2a25bd63d693151460722741398"}
Jul 20 18:25:53.105663 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:25:53.105624    2569 generic.go:356] "Generic (PLEG): container finished" podID="d35c652f-8e6f-4633-acd6-57952f773850" containerID="d9f17ca384a8fb7aa5c26961078dface65846975f37cee47a8122a3f998314f9" exitCode=0
Jul 20 18:25:53.106053 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:25:53.105683    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0ae63acc-7cp4j" event={"ID":"d35c652f-8e6f-4633-acd6-57952f773850","Type":"ContainerDied","Data":"d9f17ca384a8fb7aa5c26961078dface65846975f37cee47a8122a3f998314f9"}
Jul 20 18:25:54.207203 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:25:54.207175    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wjs8g_07587a78-2d3a-40ad-9aae-a4875200263a/ovn-acl-logging/0.log"
Jul 20 18:25:54.213932 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:25:54.213911    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wjs8g_07587a78-2d3a-40ad-9aae-a4875200263a/ovn-acl-logging/0.log"
Jul 20 18:25:54.244982 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:25:54.244953    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0ae63acc-7cp4j"
Jul 20 18:25:54.289436 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:25:54.289406    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d35c652f-8e6f-4633-acd6-57952f773850-maas-api-service-ca\") pod \"d35c652f-8e6f-4633-acd6-57952f773850\" (UID: \"d35c652f-8e6f-4633-acd6-57952f773850\") "
Jul 20 18:25:54.289773 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:25:54.289717    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d35c652f-8e6f-4633-acd6-57952f773850-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d35c652f-8e6f-4633-acd6-57952f773850" (UID: "d35c652f-8e6f-4633-acd6-57952f773850"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 18:25:54.391053 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:25:54.390980    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d35c652f-8e6f-4633-acd6-57952f773850-maas-api-service-ca\") on node \"ip-10-0-137-247.ec2.internal\" DevicePath \"\""
Jul 20 18:25:55.120244 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:25:55.120217    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0ae63acc-7cp4j"
Jul 20 18:25:55.120405 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:25:55.120257    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0ae63acc-7cp4j" event={"ID":"d35c652f-8e6f-4633-acd6-57952f773850","Type":"ContainerDied","Data":"af03ec8cc08c2a740322a0e16b815974a497c2a25bd63d693151460722741398"}
Jul 20 18:25:55.120405 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:25:55.120309    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af03ec8cc08c2a740322a0e16b815974a497c2a25bd63d693151460722741398"
Jul 20 18:26:04.851577 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:26:04.851535    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9thxb"]
Jul 20 18:26:06.557276 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:26:06.557237    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0ae63acc-7cp4j"]
Jul 20 18:26:06.560447 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:26:06.560421    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0ae63acc-7cp4j"]
Jul 20 18:26:08.181804 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:26:08.181771    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d35c652f-8e6f-4633-acd6-57952f773850" path="/var/lib/kubelet/pods/d35c652f-8e6f-4633-acd6-57952f773850/volumes"
Jul 20 18:26:15.584793 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:26:15.584756    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9thxb"]
Jul 20 18:26:35.327066 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:26:35.327032    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-674433c6-f9t5f"]
Jul 20 18:26:35.327656 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:26:35.327549    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d35c652f-8e6f-4633-acd6-57952f773850" containerName="revoke-keys"
Jul 20 18:26:35.327656 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:26:35.327570    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="d35c652f-8e6f-4633-acd6-57952f773850" containerName="revoke-keys"
Jul 20 18:26:35.327802 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:26:35.327668    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="d35c652f-8e6f-4633-acd6-57952f773850" containerName="revoke-keys"
Jul 20 18:26:35.330496 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:26:35.330473    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-674433c6-f9t5f"
Jul 20 18:26:35.334673 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:26:35.334655    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fk7rm\""
Jul 20 18:26:35.335683 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:26:35.335662    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 20 18:26:35.347135 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:26:35.347111    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-674433c6-f9t5f"]
Jul 20 18:26:35.428417 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:26:35.428375    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07b55ff6-0e1b-4189-8678-add7eb78a0e3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-674433c6-f9t5f\" (UID: \"07b55ff6-0e1b-4189-8678-add7eb78a0e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-674433c6-f9t5f"
Jul 20 18:26:35.529484 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:26:35.529455    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07b55ff6-0e1b-4189-8678-add7eb78a0e3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-674433c6-f9t5f\" (UID: \"07b55ff6-0e1b-4189-8678-add7eb78a0e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-674433c6-f9t5f"
Jul 20 18:26:35.530075 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:26:35.530057    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07b55ff6-0e1b-4189-8678-add7eb78a0e3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-674433c6-f9t5f\" (UID: \"07b55ff6-0e1b-4189-8678-add7eb78a0e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-674433c6-f9t5f"
Jul 20 18:26:35.640458 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:26:35.640374    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-674433c6-f9t5f"
Jul 20 18:26:35.770577 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:26:35.770549    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-674433c6-f9t5f"]
Jul 20 18:26:35.773664 ip-10-0-137-247 kubenswrapper[2569]: W0720 18:26:35.773639    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07b55ff6_0e1b_4189_8678_add7eb78a0e3.slice/crio-16cb681aef8861f7500965be50b5c2f283484b262a171533d681dfb246e37567 WatchSource:0}: Error finding container 16cb681aef8861f7500965be50b5c2f283484b262a171533d681dfb246e37567: Status 404 returned error can't find the container with id 16cb681aef8861f7500965be50b5c2f283484b262a171533d681dfb246e37567
Jul 20 18:26:36.290515 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:26:36.290476    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-674433c6-f9t5f" event={"ID":"07b55ff6-0e1b-4189-8678-add7eb78a0e3","Type":"ContainerStarted","Data":"d20fe73e884f2e14d8d8bf2281b533621352e1550ee7462f7ec7a75ae4744d69"}
Jul 20 18:26:36.290688 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:26:36.290536    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-674433c6-f9t5f" event={"ID":"07b55ff6-0e1b-4189-8678-add7eb78a0e3","Type":"ContainerStarted","Data":"16cb681aef8861f7500965be50b5c2f283484b262a171533d681dfb246e37567"}
Jul 20 18:26:36.317022 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:26:36.316976    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-674433c6-f9t5f" podStartSLOduration=1.3169588970000001 podStartE2EDuration="1.316958897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 18:26:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 18:26:36.315895306 +0000 UTC m=+1842.562428717" watchObservedRunningTime="2026-07-20 18:26:36.316958897 +0000 UTC m=+1842.563492307"
Jul 20 18:26:37.295850 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:26:37.295818    2569 generic.go:356] "Generic (PLEG): container finished" podID="07b55ff6-0e1b-4189-8678-add7eb78a0e3" containerID="d20fe73e884f2e14d8d8bf2281b533621352e1550ee7462f7ec7a75ae4744d69" exitCode=0
Jul 20 18:26:37.296236 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:26:37.295866    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-674433c6-f9t5f" event={"ID":"07b55ff6-0e1b-4189-8678-add7eb78a0e3","Type":"ContainerDied","Data":"d20fe73e884f2e14d8d8bf2281b533621352e1550ee7462f7ec7a75ae4744d69"}
Jul 20 18:26:38.425332 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:26:38.425303    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-674433c6-f9t5f"
Jul 20 18:26:38.449581 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:26:38.449557    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07b55ff6-0e1b-4189-8678-add7eb78a0e3-maas-api-service-ca\") pod \"07b55ff6-0e1b-4189-8678-add7eb78a0e3\" (UID: \"07b55ff6-0e1b-4189-8678-add7eb78a0e3\") "
Jul 20 18:26:38.449884 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:26:38.449862    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07b55ff6-0e1b-4189-8678-add7eb78a0e3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "07b55ff6-0e1b-4189-8678-add7eb78a0e3" (UID: "07b55ff6-0e1b-4189-8678-add7eb78a0e3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 18:26:38.550935 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:26:38.550832    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07b55ff6-0e1b-4189-8678-add7eb78a0e3-maas-api-service-ca\") on node \"ip-10-0-137-247.ec2.internal\" DevicePath \"\""
Jul 20 18:26:39.305491 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:26:39.305450    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-674433c6-f9t5f" event={"ID":"07b55ff6-0e1b-4189-8678-add7eb78a0e3","Type":"ContainerDied","Data":"16cb681aef8861f7500965be50b5c2f283484b262a171533d681dfb246e37567"}
Jul 20 18:26:39.305655 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:26:39.305508    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="16cb681aef8861f7500965be50b5c2f283484b262a171533d681dfb246e37567"
Jul 20 18:26:39.305655 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:26:39.305470    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-674433c6-f9t5f"
Jul 20 18:26:48.947606 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:26:48.947561    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9thxb"]
Jul 20 18:26:50.421013 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:26:50.420977    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-674433c6-f9t5f"]
Jul 20 18:26:50.423941 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:26:50.423912    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-674433c6-f9t5f"]
Jul 20 18:26:52.181878 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:26:52.181841    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="07b55ff6-0e1b-4189-8678-add7eb78a0e3" path="/var/lib/kubelet/pods/07b55ff6-0e1b-4189-8678-add7eb78a0e3/volumes"
Jul 20 18:26:59.856155 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:26:59.856118    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9thxb"]
Jul 20 18:27:19.339085 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:27:19.339047    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-76c00924-wslvt"]
Jul 20 18:27:19.341442 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:27:19.339332    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07b55ff6-0e1b-4189-8678-add7eb78a0e3" containerName="revoke-keys"
Jul 20 18:27:19.341442 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:27:19.339343    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="07b55ff6-0e1b-4189-8678-add7eb78a0e3" containerName="revoke-keys"
Jul 20 18:27:19.341442 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:27:19.339396    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="07b55ff6-0e1b-4189-8678-add7eb78a0e3" containerName="revoke-keys"
Jul 20 18:27:19.342329 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:27:19.342311    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-76c00924-wslvt"
Jul 20 18:27:19.347197 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:27:19.347169    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 20 18:27:19.347340 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:27:19.347257    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fk7rm\""
Jul 20 18:27:19.367572 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:27:19.367540    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-76c00924-wslvt"]
Jul 20 18:27:19.461549 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:27:19.461509    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71f5cb44-2bc1-4030-a26b-db206cafefa0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-76c00924-wslvt\" (UID: \"71f5cb44-2bc1-4030-a26b-db206cafefa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-76c00924-wslvt"
Jul 20 18:27:19.562792 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:27:19.562753    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71f5cb44-2bc1-4030-a26b-db206cafefa0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-76c00924-wslvt\" (UID: \"71f5cb44-2bc1-4030-a26b-db206cafefa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-76c00924-wslvt"
Jul 20 18:27:19.563355 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:27:19.563336    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71f5cb44-2bc1-4030-a26b-db206cafefa0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-76c00924-wslvt\" (UID: \"71f5cb44-2bc1-4030-a26b-db206cafefa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-76c00924-wslvt"
Jul 20 18:27:19.651983 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:27:19.651896    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-76c00924-wslvt"
Jul 20 18:27:19.804933 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:27:19.804895    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-76c00924-wslvt"]
Jul 20 18:27:19.808118 ip-10-0-137-247 kubenswrapper[2569]: W0720 18:27:19.808087    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71f5cb44_2bc1_4030_a26b_db206cafefa0.slice/crio-bc8bb8ef07933dc66d7193ff132fa9b1e104464f71faa1af0dea8622182f285a WatchSource:0}: Error finding container bc8bb8ef07933dc66d7193ff132fa9b1e104464f71faa1af0dea8622182f285a: Status 404 returned error can't find the container with id bc8bb8ef07933dc66d7193ff132fa9b1e104464f71faa1af0dea8622182f285a
Jul 20 18:27:20.494659 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:27:20.494627    2569 generic.go:356] "Generic (PLEG): container finished" podID="71f5cb44-2bc1-4030-a26b-db206cafefa0" containerID="d61a3ebf2e823bdecee42c4b2924907fd67bdb06298ba65daed35a4c70beedda" exitCode=0
Jul 20 18:27:20.495059 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:27:20.494717    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-76c00924-wslvt" event={"ID":"71f5cb44-2bc1-4030-a26b-db206cafefa0","Type":"ContainerDied","Data":"d61a3ebf2e823bdecee42c4b2924907fd67bdb06298ba65daed35a4c70beedda"}
Jul 20 18:27:20.495059 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:27:20.494782    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-76c00924-wslvt" event={"ID":"71f5cb44-2bc1-4030-a26b-db206cafefa0","Type":"ContainerStarted","Data":"bc8bb8ef07933dc66d7193ff132fa9b1e104464f71faa1af0dea8622182f285a"}
Jul 20 18:27:21.627058 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:27:21.627034    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-76c00924-wslvt"
Jul 20 18:27:21.781861 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:27:21.781832    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71f5cb44-2bc1-4030-a26b-db206cafefa0-maas-api-service-ca\") pod \"71f5cb44-2bc1-4030-a26b-db206cafefa0\" (UID: \"71f5cb44-2bc1-4030-a26b-db206cafefa0\") "
Jul 20 18:27:21.782202 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:27:21.782178    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71f5cb44-2bc1-4030-a26b-db206cafefa0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "71f5cb44-2bc1-4030-a26b-db206cafefa0" (UID: "71f5cb44-2bc1-4030-a26b-db206cafefa0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 18:27:21.883032 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:27:21.882989    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71f5cb44-2bc1-4030-a26b-db206cafefa0-maas-api-service-ca\") on node \"ip-10-0-137-247.ec2.internal\" DevicePath \"\""
Jul 20 18:27:22.504543 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:27:22.504458    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-76c00924-wslvt"
Jul 20 18:27:22.504543 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:27:22.504482    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-76c00924-wslvt" event={"ID":"71f5cb44-2bc1-4030-a26b-db206cafefa0","Type":"ContainerDied","Data":"bc8bb8ef07933dc66d7193ff132fa9b1e104464f71faa1af0dea8622182f285a"}
Jul 20 18:27:22.504543 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:27:22.504526    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bc8bb8ef07933dc66d7193ff132fa9b1e104464f71faa1af0dea8622182f285a"
Jul 20 18:27:33.174373 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:27:33.174336    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9thxb"]
Jul 20 18:27:34.432903 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:27:34.432865    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-76c00924-wslvt"]
Jul 20 18:27:34.437220 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:27:34.437190    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-76c00924-wslvt"]
Jul 20 18:27:36.181685 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:27:36.181646    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71f5cb44-2bc1-4030-a26b-db206cafefa0" path="/var/lib/kubelet/pods/71f5cb44-2bc1-4030-a26b-db206cafefa0/volumes"
Jul 20 18:29:12.957833 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:29:12.957792    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9thxb"]
Jul 20 18:29:25.875364 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:29:25.875328    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9thxb"]
Jul 20 18:29:40.054449 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:29:40.054410    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9thxb"]
Jul 20 18:29:41.512277 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:29:41.512238    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e68a9f8e-88phg"]
Jul 20 18:29:41.512653 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:29:41.512549    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71f5cb44-2bc1-4030-a26b-db206cafefa0" containerName="revoke-keys"
Jul 20 18:29:41.512653 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:29:41.512560    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="71f5cb44-2bc1-4030-a26b-db206cafefa0" containerName="revoke-keys"
Jul 20 18:29:41.512653 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:29:41.512610    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="71f5cb44-2bc1-4030-a26b-db206cafefa0" containerName="revoke-keys"
Jul 20 18:29:41.515491 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:29:41.515473    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e68a9f8e-88phg"
Jul 20 18:29:41.518118 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:29:41.518088    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fk7rm\""
Jul 20 18:29:41.518118 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:29:41.518115    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 20 18:29:41.523397 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:29:41.523366    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e68a9f8e-88phg"]
Jul 20 18:29:41.541706 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:29:41.541674    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f800eb26-e9ec-41f2-9b0c-8e3c7edb2a8b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-e68a9f8e-88phg\" (UID: \"f800eb26-e9ec-41f2-9b0c-8e3c7edb2a8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e68a9f8e-88phg"
Jul 20 18:29:41.642524 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:29:41.642488    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f800eb26-e9ec-41f2-9b0c-8e3c7edb2a8b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-e68a9f8e-88phg\" (UID: \"f800eb26-e9ec-41f2-9b0c-8e3c7edb2a8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e68a9f8e-88phg"
Jul 20 18:29:41.643154 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:29:41.643133    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f800eb26-e9ec-41f2-9b0c-8e3c7edb2a8b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-e68a9f8e-88phg\" (UID: \"f800eb26-e9ec-41f2-9b0c-8e3c7edb2a8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e68a9f8e-88phg"
Jul 20 18:29:41.825930 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:29:41.825841    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e68a9f8e-88phg"
Jul 20 18:29:41.959418 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:29:41.959380    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e68a9f8e-88phg"]
Jul 20 18:29:41.962552 ip-10-0-137-247 kubenswrapper[2569]: W0720 18:29:41.962526    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf800eb26_e9ec_41f2_9b0c_8e3c7edb2a8b.slice/crio-2c675f4728150153a741884e6a5c969fe9b9a78df2877b9eeedbf327d7870027 WatchSource:0}: Error finding container 2c675f4728150153a741884e6a5c969fe9b9a78df2877b9eeedbf327d7870027: Status 404 returned error can't find the container with id 2c675f4728150153a741884e6a5c969fe9b9a78df2877b9eeedbf327d7870027
Jul 20 18:29:42.107124 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:29:42.107042    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e68a9f8e-88phg" event={"ID":"f800eb26-e9ec-41f2-9b0c-8e3c7edb2a8b","Type":"ContainerStarted","Data":"2c675f4728150153a741884e6a5c969fe9b9a78df2877b9eeedbf327d7870027"}
Jul 20 18:29:43.112572 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:29:43.112538    2569 generic.go:356] "Generic (PLEG): container finished" podID="f800eb26-e9ec-41f2-9b0c-8e3c7edb2a8b" containerID="dfd1212dc727ada3b78001acec980010b92caa96627725eed975c09fdb02dea2" exitCode=0
Jul 20 18:29:43.112986 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:29:43.112599    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e68a9f8e-88phg" event={"ID":"f800eb26-e9ec-41f2-9b0c-8e3c7edb2a8b","Type":"ContainerDied","Data":"dfd1212dc727ada3b78001acec980010b92caa96627725eed975c09fdb02dea2"}
Jul 20 18:29:44.244321 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:29:44.244296    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e68a9f8e-88phg"
Jul 20 18:29:44.361598 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:29:44.361559    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f800eb26-e9ec-41f2-9b0c-8e3c7edb2a8b-maas-api-service-ca\") pod \"f800eb26-e9ec-41f2-9b0c-8e3c7edb2a8b\" (UID: \"f800eb26-e9ec-41f2-9b0c-8e3c7edb2a8b\") "
Jul 20 18:29:44.361975 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:29:44.361948    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f800eb26-e9ec-41f2-9b0c-8e3c7edb2a8b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f800eb26-e9ec-41f2-9b0c-8e3c7edb2a8b" (UID: "f800eb26-e9ec-41f2-9b0c-8e3c7edb2a8b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 18:29:44.462572 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:29:44.462488    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f800eb26-e9ec-41f2-9b0c-8e3c7edb2a8b-maas-api-service-ca\") on node \"ip-10-0-137-247.ec2.internal\" DevicePath \"\""
Jul 20 18:29:45.122714 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:29:45.122682    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e68a9f8e-88phg"
Jul 20 18:29:45.122714 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:29:45.122691    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e68a9f8e-88phg" event={"ID":"f800eb26-e9ec-41f2-9b0c-8e3c7edb2a8b","Type":"ContainerDied","Data":"2c675f4728150153a741884e6a5c969fe9b9a78df2877b9eeedbf327d7870027"}
Jul 20 18:29:45.122954 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:29:45.122760    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2c675f4728150153a741884e6a5c969fe9b9a78df2877b9eeedbf327d7870027"
Jul 20 18:29:54.553234 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:29:54.553200    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9thxb"]
Jul 20 18:29:56.591087 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:29:56.591044    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e68a9f8e-88phg"]
Jul 20 18:29:56.593143 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:29:56.593118    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e68a9f8e-88phg"]
Jul 20 18:29:58.182207 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:29:58.182171    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f800eb26-e9ec-41f2-9b0c-8e3c7edb2a8b" path="/var/lib/kubelet/pods/f800eb26-e9ec-41f2-9b0c-8e3c7edb2a8b/volumes"
Jul 20 18:30:00.142024 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:30:00.141992    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29742870-nbv4c"]
Jul 20 18:30:00.142402 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:30:00.142277    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f800eb26-e9ec-41f2-9b0c-8e3c7edb2a8b" containerName="revoke-keys"
Jul 20 18:30:00.142402 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:30:00.142287    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="f800eb26-e9ec-41f2-9b0c-8e3c7edb2a8b" containerName="revoke-keys"
Jul 20 18:30:00.142402 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:30:00.142338    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="f800eb26-e9ec-41f2-9b0c-8e3c7edb2a8b" containerName="revoke-keys"
Jul 20 18:30:00.145280 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:30:00.145264    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29742870-nbv4c"
Jul 20 18:30:00.147971 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:30:00.147952    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fk7rm\""
Jul 20 18:30:00.153661 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:30:00.153636    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29742870-nbv4c"]
Jul 20 18:30:00.154979 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:30:00.154958    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29742870-nbv4c"
Jul 20 18:30:00.290111 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:30:00.289979    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29742870-nbv4c"]
Jul 20 18:30:00.292883 ip-10-0-137-247 kubenswrapper[2569]: W0720 18:30:00.292858    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74615b62_982c_4311_aa1e_8740a61182aa.slice/crio-f7fcb42a9b959f0dfe5ffb3f3812953b95aae69774dd4aaf0237a9042674e2eb WatchSource:0}: Error finding container f7fcb42a9b959f0dfe5ffb3f3812953b95aae69774dd4aaf0237a9042674e2eb: Status 404 returned error can't find the container with id f7fcb42a9b959f0dfe5ffb3f3812953b95aae69774dd4aaf0237a9042674e2eb
Jul 20 18:30:01.194410 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:30:01.194381    2569 generic.go:356] "Generic (PLEG): container finished" podID="74615b62-982c-4311-aa1e-8740a61182aa" containerID="c9b27a029d883459f3afaa03008a38c69e36839fba177fce4a578064e3ebf534" exitCode=0
Jul 20 18:30:01.194713 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:30:01.194461    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29742870-nbv4c" event={"ID":"74615b62-982c-4311-aa1e-8740a61182aa","Type":"ContainerDied","Data":"c9b27a029d883459f3afaa03008a38c69e36839fba177fce4a578064e3ebf534"}
Jul 20 18:30:01.194713 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:30:01.194510    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29742870-nbv4c" event={"ID":"74615b62-982c-4311-aa1e-8740a61182aa","Type":"ContainerStarted","Data":"f7fcb42a9b959f0dfe5ffb3f3812953b95aae69774dd4aaf0237a9042674e2eb"}
Jul 20 18:30:02.325062 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:30:02.325038    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29742870-nbv4c"
Jul 20 18:30:03.205231 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:30:03.205194    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29742870-nbv4c" event={"ID":"74615b62-982c-4311-aa1e-8740a61182aa","Type":"ContainerDied","Data":"f7fcb42a9b959f0dfe5ffb3f3812953b95aae69774dd4aaf0237a9042674e2eb"}
Jul 20 18:30:03.205231 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:30:03.205215    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29742870-nbv4c"
Jul 20 18:30:03.205498 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:30:03.205243    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f7fcb42a9b959f0dfe5ffb3f3812953b95aae69774dd4aaf0237a9042674e2eb"
Jul 20 18:30:35.554304 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:30:35.554218    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9thxb"]
Jul 20 18:30:37.689652 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:30:37.689608    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-da2296d7/e2e-lifecycle-model-da2296d7-kserve-6cc4679c55-cn8mc"]
Jul 20 18:30:37.690056 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:30:37.690038    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74615b62-982c-4311-aa1e-8740a61182aa" containerName="cleanup"
Jul 20 18:30:37.690108 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:30:37.690058    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="74615b62-982c-4311-aa1e-8740a61182aa" containerName="cleanup"
Jul 20 18:30:37.690143 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:30:37.690126    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="74615b62-982c-4311-aa1e-8740a61182aa" containerName="cleanup"
Jul 20 18:30:37.693198 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:30:37.693181    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-da2296d7/e2e-lifecycle-model-da2296d7-kserve-6cc4679c55-cn8mc"
Jul 20 18:30:37.697426 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:30:37.697403    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-da2296d7\"/\"kube-root-ca.crt\""
Jul 20 18:30:37.697426 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:30:37.697410    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-da2296d7\"/\"e2e-lifecycle-model-da2296d7-kserve-self-signed-certs\""
Jul 20 18:30:37.697426 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:30:37.697404    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-da2296d7\"/\"openshift-service-ca.crt\""
Jul 20 18:30:37.697646 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:30:37.697404    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-da2296d7\"/\"default-dockercfg-mk7r8\""
Jul 20 18:30:37.702693 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:30:37.702669    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-da2296d7/e2e-lifecycle-model-da2296d7-kserve-6cc4679c55-cn8mc"]
Jul 20 18:30:37.784512 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:30:37.784476    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8znb\" (UniqueName: \"kubernetes.io/projected/240a0ec7-1b4a-48aa-a5f4-9867f0fcf073-kube-api-access-d8znb\") pod \"e2e-lifecycle-model-da2296d7-kserve-6cc4679c55-cn8mc\" (UID: \"240a0ec7-1b4a-48aa-a5f4-9867f0fcf073\") " pod="ai-tenant-e2e-mt-da2296d7/e2e-lifecycle-model-da2296d7-kserve-6cc4679c55-cn8mc"
Jul 20 18:30:37.784771 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:30:37.784521    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/240a0ec7-1b4a-48aa-a5f4-9867f0fcf073-home\") pod \"e2e-lifecycle-model-da2296d7-kserve-6cc4679c55-cn8mc\" (UID: \"240a0ec7-1b4a-48aa-a5f4-9867f0fcf073\") " pod="ai-tenant-e2e-mt-da2296d7/e2e-lifecycle-model-da2296d7-kserve-6cc4679c55-cn8mc"
Jul 20 18:30:37.784771 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:30:37.784541    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/240a0ec7-1b4a-48aa-a5f4-9867f0fcf073-model-cache\") pod \"e2e-lifecycle-model-da2296d7-kserve-6cc4679c55-cn8mc\" (UID: \"240a0ec7-1b4a-48aa-a5f4-9867f0fcf073\") " pod="ai-tenant-e2e-mt-da2296d7/e2e-lifecycle-model-da2296d7-kserve-6cc4679c55-cn8mc"
Jul 20 18:30:37.784771 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:30:37.784630    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/240a0ec7-1b4a-48aa-a5f4-9867f0fcf073-dshm\") pod \"e2e-lifecycle-model-da2296d7-kserve-6cc4679c55-cn8mc\" (UID: \"240a0ec7-1b4a-48aa-a5f4-9867f0fcf073\") " pod="ai-tenant-e2e-mt-da2296d7/e2e-lifecycle-model-da2296d7-kserve-6cc4679c55-cn8mc"
Jul 20 18:30:37.784771 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:30:37.784671    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/240a0ec7-1b4a-48aa-a5f4-9867f0fcf073-tls-certs\") pod \"e2e-lifecycle-model-da2296d7-kserve-6cc4679c55-cn8mc\" (UID: \"240a0ec7-1b4a-48aa-a5f4-9867f0fcf073\") " pod="ai-tenant-e2e-mt-da2296d7/e2e-lifecycle-model-da2296d7-kserve-6cc4679c55-cn8mc"
Jul 20 18:30:37.784771 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:30:37.784716    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/240a0ec7-1b4a-48aa-a5f4-9867f0fcf073-tmp-dir\") pod \"e2e-lifecycle-model-da2296d7-kserve-6cc4679c55-cn8mc\" (UID: \"240a0ec7-1b4a-48aa-a5f4-9867f0fcf073\") " pod="ai-tenant-e2e-mt-da2296d7/e2e-lifecycle-model-da2296d7-kserve-6cc4679c55-cn8mc"
Jul 20 18:30:37.885895 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:30:37.885836    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8znb\" (UniqueName: \"kubernetes.io/projected/240a0ec7-1b4a-48aa-a5f4-9867f0fcf073-kube-api-access-d8znb\") pod \"e2e-lifecycle-model-da2296d7-kserve-6cc4679c55-cn8mc\" (UID: \"240a0ec7-1b4a-48aa-a5f4-9867f0fcf073\") " pod="ai-tenant-e2e-mt-da2296d7/e2e-lifecycle-model-da2296d7-kserve-6cc4679c55-cn8mc"
Jul 20 18:30:37.885895 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:30:37.885903    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/240a0ec7-1b4a-48aa-a5f4-9867f0fcf073-home\") pod \"e2e-lifecycle-model-da2296d7-kserve-6cc4679c55-cn8mc\" (UID: \"240a0ec7-1b4a-48aa-a5f4-9867f0fcf073\") " pod="ai-tenant-e2e-mt-da2296d7/e2e-lifecycle-model-da2296d7-kserve-6cc4679c55-cn8mc"
Jul 20 18:30:37.886140 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:30:37.885931    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/240a0ec7-1b4a-48aa-a5f4-9867f0fcf073-model-cache\") pod \"e2e-lifecycle-model-da2296d7-kserve-6cc4679c55-cn8mc\" (UID: \"240a0ec7-1b4a-48aa-a5f4-9867f0fcf073\") " pod="ai-tenant-e2e-mt-da2296d7/e2e-lifecycle-model-da2296d7-kserve-6cc4679c55-cn8mc"
Jul 20 18:30:37.886140 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:30:37.885963    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/240a0ec7-1b4a-48aa-a5f4-9867f0fcf073-dshm\") pod \"e2e-lifecycle-model-da2296d7-kserve-6cc4679c55-cn8mc\" (UID: \"240a0ec7-1b4a-48aa-a5f4-9867f0fcf073\") " pod="ai-tenant-e2e-mt-da2296d7/e2e-lifecycle-model-da2296d7-kserve-6cc4679c55-cn8mc"
Jul 20 18:30:37.886140 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:30:37.885985    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/240a0ec7-1b4a-48aa-a5f4-9867f0fcf073-tls-certs\") pod \"e2e-lifecycle-model-da2296d7-kserve-6cc4679c55-cn8mc\" (UID: \"240a0ec7-1b4a-48aa-a5f4-9867f0fcf073\") " pod="ai-tenant-e2e-mt-da2296d7/e2e-lifecycle-model-da2296d7-kserve-6cc4679c55-cn8mc"
Jul 20 18:30:37.886140 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:30:37.886011    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/240a0ec7-1b4a-48aa-a5f4-9867f0fcf073-tmp-dir\") pod \"e2e-lifecycle-model-da2296d7-kserve-6cc4679c55-cn8mc\" (UID: \"240a0ec7-1b4a-48aa-a5f4-9867f0fcf073\") " pod="ai-tenant-e2e-mt-da2296d7/e2e-lifecycle-model-da2296d7-kserve-6cc4679c55-cn8mc"
Jul 20 18:30:37.886399 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:30:37.886372    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/240a0ec7-1b4a-48aa-a5f4-9867f0fcf073-home\") pod \"e2e-lifecycle-model-da2296d7-kserve-6cc4679c55-cn8mc\" (UID: \"240a0ec7-1b4a-48aa-a5f4-9867f0fcf073\") " pod="ai-tenant-e2e-mt-da2296d7/e2e-lifecycle-model-da2296d7-kserve-6cc4679c55-cn8mc"
Jul 20 18:30:37.886399 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:30:37.886393    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/240a0ec7-1b4a-48aa-a5f4-9867f0fcf073-model-cache\") pod \"e2e-lifecycle-model-da2296d7-kserve-6cc4679c55-cn8mc\" (UID: \"240a0ec7-1b4a-48aa-a5f4-9867f0fcf073\") " pod="ai-tenant-e2e-mt-da2296d7/e2e-lifecycle-model-da2296d7-kserve-6cc4679c55-cn8mc"
Jul 20 18:30:37.886509 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:30:37.886456    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/240a0ec7-1b4a-48aa-a5f4-9867f0fcf073-tmp-dir\") pod \"e2e-lifecycle-model-da2296d7-kserve-6cc4679c55-cn8mc\" (UID: \"240a0ec7-1b4a-48aa-a5f4-9867f0fcf073\") " pod="ai-tenant-e2e-mt-da2296d7/e2e-lifecycle-model-da2296d7-kserve-6cc4679c55-cn8mc"
Jul 20 18:30:37.888345 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:30:37.888320    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/240a0ec7-1b4a-48aa-a5f4-9867f0fcf073-dshm\") pod \"e2e-lifecycle-model-da2296d7-kserve-6cc4679c55-cn8mc\" (UID: \"240a0ec7-1b4a-48aa-a5f4-9867f0fcf073\") " pod="ai-tenant-e2e-mt-da2296d7/e2e-lifecycle-model-da2296d7-kserve-6cc4679c55-cn8mc"
Jul 20 18:30:37.888615 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:30:37.888596    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/240a0ec7-1b4a-48aa-a5f4-9867f0fcf073-tls-certs\") pod \"e2e-lifecycle-model-da2296d7-kserve-6cc4679c55-cn8mc\" (UID: \"240a0ec7-1b4a-48aa-a5f4-9867f0fcf073\") " pod="ai-tenant-e2e-mt-da2296d7/e2e-lifecycle-model-da2296d7-kserve-6cc4679c55-cn8mc"
Jul 20 18:30:37.897224 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:30:37.897196    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8znb\" (UniqueName: \"kubernetes.io/projected/240a0ec7-1b4a-48aa-a5f4-9867f0fcf073-kube-api-access-d8znb\") pod \"e2e-lifecycle-model-da2296d7-kserve-6cc4679c55-cn8mc\" (UID: \"240a0ec7-1b4a-48aa-a5f4-9867f0fcf073\") " pod="ai-tenant-e2e-mt-da2296d7/e2e-lifecycle-model-da2296d7-kserve-6cc4679c55-cn8mc"
Jul 20 18:30:38.002886 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:30:38.002778    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-da2296d7/e2e-lifecycle-model-da2296d7-kserve-6cc4679c55-cn8mc"
Jul 20 18:30:38.134185 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:30:38.134156    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-da2296d7/e2e-lifecycle-model-da2296d7-kserve-6cc4679c55-cn8mc"]
Jul 20 18:30:38.136741 ip-10-0-137-247 kubenswrapper[2569]: W0720 18:30:38.136695    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod240a0ec7_1b4a_48aa_a5f4_9867f0fcf073.slice/crio-82ddff9b796b0092cdc30dcdf646dcb6c8240be21c2a5f55c04e4d6553f7dbaa WatchSource:0}: Error finding container 82ddff9b796b0092cdc30dcdf646dcb6c8240be21c2a5f55c04e4d6553f7dbaa: Status 404 returned error can't find the container with id 82ddff9b796b0092cdc30dcdf646dcb6c8240be21c2a5f55c04e4d6553f7dbaa
Jul 20 18:30:38.355159 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:30:38.355126    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-da2296d7/e2e-lifecycle-model-da2296d7-kserve-6cc4679c55-cn8mc" event={"ID":"240a0ec7-1b4a-48aa-a5f4-9867f0fcf073","Type":"ContainerStarted","Data":"1e84ddfdccbf8395351e20fedf935996c57270182272a488d9472d61007ed50f"}
Jul 20 18:30:38.355159 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:30:38.355165    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-da2296d7/e2e-lifecycle-model-da2296d7-kserve-6cc4679c55-cn8mc" event={"ID":"240a0ec7-1b4a-48aa-a5f4-9867f0fcf073","Type":"ContainerStarted","Data":"82ddff9b796b0092cdc30dcdf646dcb6c8240be21c2a5f55c04e4d6553f7dbaa"}
Jul 20 18:30:48.003883 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:30:48.003831    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-da2296d7/e2e-lifecycle-model-da2296d7-kserve-6cc4679c55-cn8mc"
Jul 20 18:30:48.004333 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:30:48.003895    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-da2296d7/e2e-lifecycle-model-da2296d7-kserve-6cc4679c55-cn8mc"
Jul 20 18:30:48.016149 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:30:48.016120    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-da2296d7/e2e-lifecycle-model-da2296d7-kserve-6cc4679c55-cn8mc"
Jul 20 18:30:48.037867 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:30:48.037814    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-da2296d7/e2e-lifecycle-model-da2296d7-kserve-6cc4679c55-cn8mc" podStartSLOduration=11.037798649 podStartE2EDuration="11.037798649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 18:30:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 18:30:38.378371144 +0000 UTC m=+2084.624904554" watchObservedRunningTime="2026-07-20 18:30:48.037798649 +0000 UTC m=+2094.284332059"
Jul 20 18:30:48.413054 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:30:48.413022    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-da2296d7/e2e-lifecycle-model-da2296d7-kserve-6cc4679c55-cn8mc"
Jul 20 18:30:49.556259 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:30:49.556219    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9thxb"]
Jul 20 18:30:54.234652 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:30:54.234624    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wjs8g_07587a78-2d3a-40ad-9aae-a4875200263a/ovn-acl-logging/0.log"
Jul 20 18:30:54.241916 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:30:54.241896    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wjs8g_07587a78-2d3a-40ad-9aae-a4875200263a/ovn-acl-logging/0.log"
Jul 20 18:31:02.756345 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:31:02.756308    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9thxb"]
Jul 20 18:31:06.457893 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:31:06.457860    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-da2296d7-lgn5l"]
Jul 20 18:31:06.461083 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:31:06.461064    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-da2296d7-lgn5l"
Jul 20 18:31:06.463781 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:31:06.463754    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 20 18:31:06.463905 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:31:06.463848    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fk7rm\""
Jul 20 18:31:06.469575 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:31:06.469554    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-da2296d7-lgn5l"]
Jul 20 18:31:06.524214 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:31:06.524173    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/030bad62-3fd2-48d8-90ff-fc1032ed9b09-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-da2296d7-lgn5l\" (UID: \"030bad62-3fd2-48d8-90ff-fc1032ed9b09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-da2296d7-lgn5l"
Jul 20 18:31:06.625237 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:31:06.625198    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/030bad62-3fd2-48d8-90ff-fc1032ed9b09-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-da2296d7-lgn5l\" (UID: \"030bad62-3fd2-48d8-90ff-fc1032ed9b09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-da2296d7-lgn5l"
Jul 20 18:31:06.625897 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:31:06.625879    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/030bad62-3fd2-48d8-90ff-fc1032ed9b09-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-da2296d7-lgn5l\" (UID: \"030bad62-3fd2-48d8-90ff-fc1032ed9b09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-da2296d7-lgn5l"
Jul 20 18:31:06.773054 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:31:06.773022    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-da2296d7-lgn5l"
Jul 20 18:31:06.904883 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:31:06.904857    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-da2296d7-lgn5l"]
Jul 20 18:31:06.907346 ip-10-0-137-247 kubenswrapper[2569]: W0720 18:31:06.907322    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod030bad62_3fd2_48d8_90ff_fc1032ed9b09.slice/crio-3e6f0dfa6887f54d636f50d263a48b973d9e5b5768342ad97d6bd70445542bda WatchSource:0}: Error finding container 3e6f0dfa6887f54d636f50d263a48b973d9e5b5768342ad97d6bd70445542bda: Status 404 returned error can't find the container with id 3e6f0dfa6887f54d636f50d263a48b973d9e5b5768342ad97d6bd70445542bda
Jul 20 18:31:07.488026 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:31:07.487983    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-da2296d7-lgn5l" event={"ID":"030bad62-3fd2-48d8-90ff-fc1032ed9b09","Type":"ContainerStarted","Data":"7b818103cb6754887bf94fdb95b6a62aa3b58b38fc00c90afae4960efb73ff6e"}
Jul 20 18:31:07.488367 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:31:07.488048    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-da2296d7-lgn5l" event={"ID":"030bad62-3fd2-48d8-90ff-fc1032ed9b09","Type":"ContainerStarted","Data":"3e6f0dfa6887f54d636f50d263a48b973d9e5b5768342ad97d6bd70445542bda"}
Jul 20 18:31:07.507977 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:31:07.507927    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-da2296d7-lgn5l" podStartSLOduration=1.5079115060000001 podStartE2EDuration="1.507911506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 18:31:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 18:31:07.507378852 +0000 UTC m=+2113.753912263" watchObservedRunningTime="2026-07-20 18:31:07.507911506 +0000 UTC m=+2113.754444956"
Jul 20 18:31:08.494246 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:31:08.494206    2569 generic.go:356] "Generic (PLEG): container finished" podID="030bad62-3fd2-48d8-90ff-fc1032ed9b09" containerID="7b818103cb6754887bf94fdb95b6a62aa3b58b38fc00c90afae4960efb73ff6e" exitCode=0
Jul 20 18:31:08.494665 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:31:08.494287    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-da2296d7-lgn5l" event={"ID":"030bad62-3fd2-48d8-90ff-fc1032ed9b09","Type":"ContainerDied","Data":"7b818103cb6754887bf94fdb95b6a62aa3b58b38fc00c90afae4960efb73ff6e"}
Jul 20 18:31:09.623600 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:31:09.623574    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-da2296d7-lgn5l"
Jul 20 18:31:09.753130 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:31:09.753028    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/030bad62-3fd2-48d8-90ff-fc1032ed9b09-maas-api-service-ca\") pod \"030bad62-3fd2-48d8-90ff-fc1032ed9b09\" (UID: \"030bad62-3fd2-48d8-90ff-fc1032ed9b09\") "
Jul 20 18:31:09.753418 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:31:09.753396    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/030bad62-3fd2-48d8-90ff-fc1032ed9b09-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "030bad62-3fd2-48d8-90ff-fc1032ed9b09" (UID: "030bad62-3fd2-48d8-90ff-fc1032ed9b09"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 18:31:09.854614 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:31:09.854574    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/030bad62-3fd2-48d8-90ff-fc1032ed9b09-maas-api-service-ca\") on node \"ip-10-0-137-247.ec2.internal\" DevicePath \"\""
Jul 20 18:31:10.503612 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:31:10.503516    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-da2296d7-lgn5l" event={"ID":"030bad62-3fd2-48d8-90ff-fc1032ed9b09","Type":"ContainerDied","Data":"3e6f0dfa6887f54d636f50d263a48b973d9e5b5768342ad97d6bd70445542bda"}
Jul 20 18:31:10.503612 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:31:10.503549    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-da2296d7-lgn5l"
Jul 20 18:31:10.503612 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:31:10.503575    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3e6f0dfa6887f54d636f50d263a48b973d9e5b5768342ad97d6bd70445542bda"
Jul 20 18:31:20.680939 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:31:20.680892    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9thxb"]
Jul 20 18:31:26.558955 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:31:26.558918    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-da2296d7-lgn5l"]
Jul 20 18:31:26.561576 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:31:26.561545    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-da2296d7-lgn5l"]
Jul 20 18:31:28.181973 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:31:28.181942    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="030bad62-3fd2-48d8-90ff-fc1032ed9b09" path="/var/lib/kubelet/pods/030bad62-3fd2-48d8-90ff-fc1032ed9b09/volumes"
Jul 20 18:31:30.862874 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:31:30.862839    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9thxb"]
Jul 20 18:31:41.167872 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:31:41.167832    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-da2296d7/e2e-lifecycle-model-da2296d7-kserve-6cc4679c55-cn8mc"]
Jul 20 18:31:54.283843 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:31:54.283810    2569 scope.go:117] "RemoveContainer" containerID="d9f17ca384a8fb7aa5c26961078dface65846975f37cee47a8122a3f998314f9"
Jul 20 18:31:54.292178 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:31:54.292159    2569 scope.go:117] "RemoveContainer" containerID="a158241984c8524fea8c84155268cef7502264c62afe8704e50187cb4ac7e767"
Jul 20 18:31:56.187968 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:31:56.187904    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-da2296d7/e2e-lifecycle-model-da2296d7-kserve-6cc4679c55-cn8mc" podUID="240a0ec7-1b4a-48aa-a5f4-9867f0fcf073" containerName="main" containerID="cri-o://1e84ddfdccbf8395351e20fedf935996c57270182272a488d9472d61007ed50f" gracePeriod=45
Jul 20 18:31:56.427311 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:31:56.427287    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-da2296d7/e2e-lifecycle-model-da2296d7-kserve-6cc4679c55-cn8mc"
Jul 20 18:31:56.446418 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:31:56.446333    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d8znb\" (UniqueName: \"kubernetes.io/projected/240a0ec7-1b4a-48aa-a5f4-9867f0fcf073-kube-api-access-d8znb\") pod \"240a0ec7-1b4a-48aa-a5f4-9867f0fcf073\" (UID: \"240a0ec7-1b4a-48aa-a5f4-9867f0fcf073\") "
Jul 20 18:31:56.446418 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:31:56.446382    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/240a0ec7-1b4a-48aa-a5f4-9867f0fcf073-home\") pod \"240a0ec7-1b4a-48aa-a5f4-9867f0fcf073\" (UID: \"240a0ec7-1b4a-48aa-a5f4-9867f0fcf073\") "
Jul 20 18:31:56.446418 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:31:56.446408    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/240a0ec7-1b4a-48aa-a5f4-9867f0fcf073-dshm\") pod \"240a0ec7-1b4a-48aa-a5f4-9867f0fcf073\" (UID: \"240a0ec7-1b4a-48aa-a5f4-9867f0fcf073\") "
Jul 20 18:31:56.446670 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:31:56.446437    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/240a0ec7-1b4a-48aa-a5f4-9867f0fcf073-model-cache\") pod \"240a0ec7-1b4a-48aa-a5f4-9867f0fcf073\" (UID: \"240a0ec7-1b4a-48aa-a5f4-9867f0fcf073\") "
Jul 20 18:31:56.446670 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:31:56.446462    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/240a0ec7-1b4a-48aa-a5f4-9867f0fcf073-tls-certs\") pod \"240a0ec7-1b4a-48aa-a5f4-9867f0fcf073\" (UID: \"240a0ec7-1b4a-48aa-a5f4-9867f0fcf073\") "
Jul 20 18:31:56.446670 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:31:56.446517    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/240a0ec7-1b4a-48aa-a5f4-9867f0fcf073-tmp-dir\") pod \"240a0ec7-1b4a-48aa-a5f4-9867f0fcf073\" (UID: \"240a0ec7-1b4a-48aa-a5f4-9867f0fcf073\") "
Jul 20 18:31:56.446854 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:31:56.446680    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/240a0ec7-1b4a-48aa-a5f4-9867f0fcf073-home" (OuterVolumeSpecName: "home") pod "240a0ec7-1b4a-48aa-a5f4-9867f0fcf073" (UID: "240a0ec7-1b4a-48aa-a5f4-9867f0fcf073"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 18:31:56.446942 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:31:56.446913    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/240a0ec7-1b4a-48aa-a5f4-9867f0fcf073-model-cache" (OuterVolumeSpecName: "model-cache") pod "240a0ec7-1b4a-48aa-a5f4-9867f0fcf073" (UID: "240a0ec7-1b4a-48aa-a5f4-9867f0fcf073"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 18:31:56.447326 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:31:56.447113    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/240a0ec7-1b4a-48aa-a5f4-9867f0fcf073-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "240a0ec7-1b4a-48aa-a5f4-9867f0fcf073" (UID: "240a0ec7-1b4a-48aa-a5f4-9867f0fcf073"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 18:31:56.449163 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:31:56.449136    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/240a0ec7-1b4a-48aa-a5f4-9867f0fcf073-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "240a0ec7-1b4a-48aa-a5f4-9867f0fcf073" (UID: "240a0ec7-1b4a-48aa-a5f4-9867f0fcf073"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 18:31:56.452253 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:31:56.452227    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/240a0ec7-1b4a-48aa-a5f4-9867f0fcf073-kube-api-access-d8znb" (OuterVolumeSpecName: "kube-api-access-d8znb") pod "240a0ec7-1b4a-48aa-a5f4-9867f0fcf073" (UID: "240a0ec7-1b4a-48aa-a5f4-9867f0fcf073"). InnerVolumeSpecName "kube-api-access-d8znb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 18:31:56.452346 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:31:56.452263    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/240a0ec7-1b4a-48aa-a5f4-9867f0fcf073-dshm" (OuterVolumeSpecName: "dshm") pod "240a0ec7-1b4a-48aa-a5f4-9867f0fcf073" (UID: "240a0ec7-1b4a-48aa-a5f4-9867f0fcf073"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 18:31:56.547897 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:31:56.547857    2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/240a0ec7-1b4a-48aa-a5f4-9867f0fcf073-tmp-dir\") on node \"ip-10-0-137-247.ec2.internal\" DevicePath \"\""
Jul 20 18:31:56.547897 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:31:56.547892    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8znb\" (UniqueName: \"kubernetes.io/projected/240a0ec7-1b4a-48aa-a5f4-9867f0fcf073-kube-api-access-d8znb\") on node \"ip-10-0-137-247.ec2.internal\" DevicePath \"\""
Jul 20 18:31:56.547897 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:31:56.547902    2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/240a0ec7-1b4a-48aa-a5f4-9867f0fcf073-home\") on node \"ip-10-0-137-247.ec2.internal\" DevicePath \"\""
Jul 20 18:31:56.547897 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:31:56.547911    2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/240a0ec7-1b4a-48aa-a5f4-9867f0fcf073-dshm\") on node \"ip-10-0-137-247.ec2.internal\" DevicePath \"\""
Jul 20 18:31:56.548156 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:31:56.547919    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/240a0ec7-1b4a-48aa-a5f4-9867f0fcf073-model-cache\") on node \"ip-10-0-137-247.ec2.internal\" DevicePath \"\""
Jul 20 18:31:56.548156 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:31:56.547927    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/240a0ec7-1b4a-48aa-a5f4-9867f0fcf073-tls-certs\") on node \"ip-10-0-137-247.ec2.internal\" DevicePath \"\""
Jul 20 18:31:56.711137 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:31:56.711050    2569 generic.go:356] "Generic (PLEG): container finished" podID="240a0ec7-1b4a-48aa-a5f4-9867f0fcf073" containerID="1e84ddfdccbf8395351e20fedf935996c57270182272a488d9472d61007ed50f" exitCode=0
Jul 20 18:31:56.711137 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:31:56.711098    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-da2296d7/e2e-lifecycle-model-da2296d7-kserve-6cc4679c55-cn8mc"
Jul 20 18:31:56.711306 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:31:56.711139    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-da2296d7/e2e-lifecycle-model-da2296d7-kserve-6cc4679c55-cn8mc" event={"ID":"240a0ec7-1b4a-48aa-a5f4-9867f0fcf073","Type":"ContainerDied","Data":"1e84ddfdccbf8395351e20fedf935996c57270182272a488d9472d61007ed50f"}
Jul 20 18:31:56.711306 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:31:56.711179    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-da2296d7/e2e-lifecycle-model-da2296d7-kserve-6cc4679c55-cn8mc" event={"ID":"240a0ec7-1b4a-48aa-a5f4-9867f0fcf073","Type":"ContainerDied","Data":"82ddff9b796b0092cdc30dcdf646dcb6c8240be21c2a5f55c04e4d6553f7dbaa"}
Jul 20 18:31:56.711306 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:31:56.711201    2569 scope.go:117] "RemoveContainer" containerID="1e84ddfdccbf8395351e20fedf935996c57270182272a488d9472d61007ed50f"
Jul 20 18:31:56.720267 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:31:56.720250    2569 scope.go:117] "RemoveContainer" containerID="1e84ddfdccbf8395351e20fedf935996c57270182272a488d9472d61007ed50f"
Jul 20 18:31:56.720541 ip-10-0-137-247 kubenswrapper[2569]: E0720 18:31:56.720520    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e84ddfdccbf8395351e20fedf935996c57270182272a488d9472d61007ed50f\": container with ID starting with 1e84ddfdccbf8395351e20fedf935996c57270182272a488d9472d61007ed50f not found: ID does not exist" containerID="1e84ddfdccbf8395351e20fedf935996c57270182272a488d9472d61007ed50f"
Jul 20 18:31:56.720661 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:31:56.720553    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e84ddfdccbf8395351e20fedf935996c57270182272a488d9472d61007ed50f"} err="failed to get container status \"1e84ddfdccbf8395351e20fedf935996c57270182272a488d9472d61007ed50f\": rpc error: code = NotFound desc = could not find container \"1e84ddfdccbf8395351e20fedf935996c57270182272a488d9472d61007ed50f\": container with ID starting with 1e84ddfdccbf8395351e20fedf935996c57270182272a488d9472d61007ed50f not found: ID does not exist"
Jul 20 18:31:56.733551 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:31:56.733528    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-da2296d7/e2e-lifecycle-model-da2296d7-kserve-6cc4679c55-cn8mc"]
Jul 20 18:31:56.740388 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:31:56.740364    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-da2296d7/e2e-lifecycle-model-da2296d7-kserve-6cc4679c55-cn8mc"]
Jul 20 18:31:58.182365 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:31:58.182333    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="240a0ec7-1b4a-48aa-a5f4-9867f0fcf073" path="/var/lib/kubelet/pods/240a0ec7-1b4a-48aa-a5f4-9867f0fcf073/volumes"
Jul 20 18:32:28.549193 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:32:28.549149    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9thxb"]
Jul 20 18:32:38.451677 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:32:38.451645    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9thxb"]
Jul 20 18:32:54.312432 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:32:54.312402    2569 scope.go:117] "RemoveContainer" containerID="d20fe73e884f2e14d8d8bf2281b533621352e1550ee7462f7ec7a75ae4744d69"
Jul 20 18:33:29.754171 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:33:29.754080    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9thxb"]
Jul 20 18:33:43.043779 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:33:43.043742    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9thxb"]
Jul 20 18:33:54.332352 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:33:54.332319    2569 scope.go:117] "RemoveContainer" containerID="d61a3ebf2e823bdecee42c4b2924907fd67bdb06298ba65daed35a4c70beedda"
Jul 20 18:34:05.028672 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:34:05.028632    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-adca0d/test-model-adca0d-kserve-f5c4445bc-k8z2w"]
Jul 20 18:34:05.029182 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:34:05.028925    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="240a0ec7-1b4a-48aa-a5f4-9867f0fcf073" containerName="main"
Jul 20 18:34:05.029182 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:34:05.028936    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="240a0ec7-1b4a-48aa-a5f4-9867f0fcf073" containerName="main"
Jul 20 18:34:05.029182 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:34:05.028945    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="030bad62-3fd2-48d8-90ff-fc1032ed9b09" containerName="revoke-keys"
Jul 20 18:34:05.029182 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:34:05.028951    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="030bad62-3fd2-48d8-90ff-fc1032ed9b09" containerName="revoke-keys"
Jul 20 18:34:05.029182 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:34:05.029007    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="030bad62-3fd2-48d8-90ff-fc1032ed9b09" containerName="revoke-keys"
Jul 20 18:34:05.029182 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:34:05.029015    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="240a0ec7-1b4a-48aa-a5f4-9867f0fcf073" containerName="main"
Jul 20 18:34:05.031763 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:34:05.031743    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-adca0d/test-model-adca0d-kserve-f5c4445bc-k8z2w"
Jul 20 18:34:05.034418 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:34:05.034393    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-adca0d\"/\"openshift-service-ca.crt\""
Jul 20 18:34:05.034533 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:34:05.034504    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-adca0d\"/\"kube-root-ca.crt\""
Jul 20 18:34:05.035253 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:34:05.035238    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-adca0d\"/\"test-model-adca0d-kserve-self-signed-certs\""
Jul 20 18:34:05.035408 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:34:05.035393    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-adca0d\"/\"default-dockercfg-hnfrr\""
Jul 20 18:34:05.044425 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:34:05.044401    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-adca0d/test-model-adca0d-kserve-f5c4445bc-k8z2w"]
Jul 20 18:34:05.062163 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:34:05.062129    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46f7c80c-8831-474a-b5eb-fc82307638d9-tmp-dir\") pod \"test-model-adca0d-kserve-f5c4445bc-k8z2w\" (UID: \"46f7c80c-8831-474a-b5eb-fc82307638d9\") " pod="ai-tenant-e2e-inf-a-adca0d/test-model-adca0d-kserve-f5c4445bc-k8z2w"
Jul 20 18:34:05.062342 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:34:05.062181    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46f7c80c-8831-474a-b5eb-fc82307638d9-model-cache\") pod \"test-model-adca0d-kserve-f5c4445bc-k8z2w\" (UID: \"46f7c80c-8831-474a-b5eb-fc82307638d9\") " pod="ai-tenant-e2e-inf-a-adca0d/test-model-adca0d-kserve-f5c4445bc-k8z2w"
Jul 20 18:34:05.062342 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:34:05.062251    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46f7c80c-8831-474a-b5eb-fc82307638d9-dshm\") pod \"test-model-adca0d-kserve-f5c4445bc-k8z2w\" (UID: \"46f7c80c-8831-474a-b5eb-fc82307638d9\") " pod="ai-tenant-e2e-inf-a-adca0d/test-model-adca0d-kserve-f5c4445bc-k8z2w"
Jul 20 18:34:05.062342 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:34:05.062302    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46f7c80c-8831-474a-b5eb-fc82307638d9-tls-certs\") pod \"test-model-adca0d-kserve-f5c4445bc-k8z2w\" (UID: \"46f7c80c-8831-474a-b5eb-fc82307638d9\") " pod="ai-tenant-e2e-inf-a-adca0d/test-model-adca0d-kserve-f5c4445bc-k8z2w"
Jul 20 18:34:05.062485 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:34:05.062345    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sh4tr\" (UniqueName: \"kubernetes.io/projected/46f7c80c-8831-474a-b5eb-fc82307638d9-kube-api-access-sh4tr\") pod \"test-model-adca0d-kserve-f5c4445bc-k8z2w\" (UID: \"46f7c80c-8831-474a-b5eb-fc82307638d9\") " pod="ai-tenant-e2e-inf-a-adca0d/test-model-adca0d-kserve-f5c4445bc-k8z2w"
Jul 20 18:34:05.062485 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:34:05.062388    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46f7c80c-8831-474a-b5eb-fc82307638d9-home\") pod \"test-model-adca0d-kserve-f5c4445bc-k8z2w\" (UID: \"46f7c80c-8831-474a-b5eb-fc82307638d9\") " pod="ai-tenant-e2e-inf-a-adca0d/test-model-adca0d-kserve-f5c4445bc-k8z2w"
Jul 20 18:34:05.162866 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:34:05.162818    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46f7c80c-8831-474a-b5eb-fc82307638d9-tmp-dir\") pod \"test-model-adca0d-kserve-f5c4445bc-k8z2w\" (UID: \"46f7c80c-8831-474a-b5eb-fc82307638d9\") " pod="ai-tenant-e2e-inf-a-adca0d/test-model-adca0d-kserve-f5c4445bc-k8z2w"
Jul 20 18:34:05.163064 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:34:05.162876    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46f7c80c-8831-474a-b5eb-fc82307638d9-model-cache\") pod \"test-model-adca0d-kserve-f5c4445bc-k8z2w\" (UID: \"46f7c80c-8831-474a-b5eb-fc82307638d9\") " pod="ai-tenant-e2e-inf-a-adca0d/test-model-adca0d-kserve-f5c4445bc-k8z2w"
Jul 20 18:34:05.163064 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:34:05.162913    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46f7c80c-8831-474a-b5eb-fc82307638d9-dshm\") pod \"test-model-adca0d-kserve-f5c4445bc-k8z2w\" (UID: \"46f7c80c-8831-474a-b5eb-fc82307638d9\") " pod="ai-tenant-e2e-inf-a-adca0d/test-model-adca0d-kserve-f5c4445bc-k8z2w"
Jul 20 18:34:05.163064 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:34:05.162937    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46f7c80c-8831-474a-b5eb-fc82307638d9-tls-certs\") pod \"test-model-adca0d-kserve-f5c4445bc-k8z2w\" (UID: \"46f7c80c-8831-474a-b5eb-fc82307638d9\") " pod="ai-tenant-e2e-inf-a-adca0d/test-model-adca0d-kserve-f5c4445bc-k8z2w"
Jul 20 18:34:05.163064 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:34:05.162965    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sh4tr\" (UniqueName: \"kubernetes.io/projected/46f7c80c-8831-474a-b5eb-fc82307638d9-kube-api-access-sh4tr\") pod \"test-model-adca0d-kserve-f5c4445bc-k8z2w\" (UID: \"46f7c80c-8831-474a-b5eb-fc82307638d9\") " pod="ai-tenant-e2e-inf-a-adca0d/test-model-adca0d-kserve-f5c4445bc-k8z2w"
Jul 20 18:34:05.163064 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:34:05.163005    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46f7c80c-8831-474a-b5eb-fc82307638d9-home\") pod \"test-model-adca0d-kserve-f5c4445bc-k8z2w\" (UID: \"46f7c80c-8831-474a-b5eb-fc82307638d9\") " pod="ai-tenant-e2e-inf-a-adca0d/test-model-adca0d-kserve-f5c4445bc-k8z2w"
Jul 20 18:34:05.163319 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:34:05.163219    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46f7c80c-8831-474a-b5eb-fc82307638d9-tmp-dir\") pod \"test-model-adca0d-kserve-f5c4445bc-k8z2w\" (UID: \"46f7c80c-8831-474a-b5eb-fc82307638d9\") " pod="ai-tenant-e2e-inf-a-adca0d/test-model-adca0d-kserve-f5c4445bc-k8z2w"
Jul 20 18:34:05.163413 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:34:05.163395    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46f7c80c-8831-474a-b5eb-fc82307638d9-home\") pod \"test-model-adca0d-kserve-f5c4445bc-k8z2w\" (UID: \"46f7c80c-8831-474a-b5eb-fc82307638d9\") " pod="ai-tenant-e2e-inf-a-adca0d/test-model-adca0d-kserve-f5c4445bc-k8z2w"
Jul 20 18:34:05.163413 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:34:05.163402    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46f7c80c-8831-474a-b5eb-fc82307638d9-model-cache\") pod \"test-model-adca0d-kserve-f5c4445bc-k8z2w\" (UID: \"46f7c80c-8831-474a-b5eb-fc82307638d9\") " pod="ai-tenant-e2e-inf-a-adca0d/test-model-adca0d-kserve-f5c4445bc-k8z2w"
Jul 20 18:34:05.165166 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:34:05.165148    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46f7c80c-8831-474a-b5eb-fc82307638d9-dshm\") pod \"test-model-adca0d-kserve-f5c4445bc-k8z2w\" (UID: \"46f7c80c-8831-474a-b5eb-fc82307638d9\") " pod="ai-tenant-e2e-inf-a-adca0d/test-model-adca0d-kserve-f5c4445bc-k8z2w"
Jul 20 18:34:05.165439 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:34:05.165423    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46f7c80c-8831-474a-b5eb-fc82307638d9-tls-certs\") pod \"test-model-adca0d-kserve-f5c4445bc-k8z2w\" (UID: \"46f7c80c-8831-474a-b5eb-fc82307638d9\") " pod="ai-tenant-e2e-inf-a-adca0d/test-model-adca0d-kserve-f5c4445bc-k8z2w"
Jul 20 18:34:05.172641 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:34:05.172605    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sh4tr\" (UniqueName: \"kubernetes.io/projected/46f7c80c-8831-474a-b5eb-fc82307638d9-kube-api-access-sh4tr\") pod \"test-model-adca0d-kserve-f5c4445bc-k8z2w\" (UID: \"46f7c80c-8831-474a-b5eb-fc82307638d9\") " pod="ai-tenant-e2e-inf-a-adca0d/test-model-adca0d-kserve-f5c4445bc-k8z2w"
Jul 20 18:34:05.341034 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:34:05.340944    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-adca0d/test-model-adca0d-kserve-f5c4445bc-k8z2w"
Jul 20 18:34:05.472174 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:34:05.472147    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-adca0d/test-model-adca0d-kserve-f5c4445bc-k8z2w"]
Jul 20 18:34:05.474630 ip-10-0-137-247 kubenswrapper[2569]: W0720 18:34:05.474607    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod46f7c80c_8831_474a_b5eb_fc82307638d9.slice/crio-7b38188bf33ee3ca275777c18ec2661310f138ed30fc532b28b74eec3b34680e WatchSource:0}: Error finding container 7b38188bf33ee3ca275777c18ec2661310f138ed30fc532b28b74eec3b34680e: Status 404 returned error can't find the container with id 7b38188bf33ee3ca275777c18ec2661310f138ed30fc532b28b74eec3b34680e
Jul 20 18:34:06.263606 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:34:06.263568    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-adca0d/test-model-adca0d-kserve-f5c4445bc-k8z2w" event={"ID":"46f7c80c-8831-474a-b5eb-fc82307638d9","Type":"ContainerStarted","Data":"538bbc9eddcb51d60e02dcd2fb4a1c630e29444e9c611cbf96fbd946a5016080"}
Jul 20 18:34:06.264013 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:34:06.263616    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-adca0d/test-model-adca0d-kserve-f5c4445bc-k8z2w" event={"ID":"46f7c80c-8831-474a-b5eb-fc82307638d9","Type":"ContainerStarted","Data":"7b38188bf33ee3ca275777c18ec2661310f138ed30fc532b28b74eec3b34680e"}
Jul 20 18:34:06.281463 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:34:06.281414    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-adca0d/test-model-adca0d-kserve-f5c4445bc-k8z2w" podStartSLOduration=1.281397575 podStartE2EDuration="1.281397575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 18:34:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 18:34:06.279862753 +0000 UTC m=+2292.526396164" watchObservedRunningTime="2026-07-20 18:34:06.281397575 +0000 UTC m=+2292.527930985"
Jul 20 18:34:09.553516 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:34:09.553476    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9thxb"]
Jul 20 18:34:15.341880 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:34:15.341839    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-adca0d/test-model-adca0d-kserve-f5c4445bc-k8z2w"
Jul 20 18:34:15.341880 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:34:15.341886    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-adca0d/test-model-adca0d-kserve-f5c4445bc-k8z2w"
Jul 20 18:34:15.354796 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:34:15.354767    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-adca0d/test-model-adca0d-kserve-f5c4445bc-k8z2w"
Jul 20 18:34:16.323667 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:34:16.323636    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-adca0d/test-model-adca0d-kserve-f5c4445bc-k8z2w"
Jul 20 18:34:30.410620 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:34:30.410585    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-02ddb5/test-model-02ddb5-kserve-b594c9655-4lhnt"]
Jul 20 18:34:30.413613 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:34:30.413597    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-02ddb5/test-model-02ddb5-kserve-b594c9655-4lhnt"
Jul 20 18:34:30.417192 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:34:30.417169    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-02ddb5\"/\"kube-root-ca.crt\""
Jul 20 18:34:30.417320 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:34:30.417173    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-02ddb5\"/\"default-dockercfg-dmqgx\""
Jul 20 18:34:30.417320 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:34:30.417219    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-02ddb5\"/\"openshift-service-ca.crt\""
Jul 20 18:34:30.417320 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:34:30.417259    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-02ddb5\"/\"test-model-02ddb5-kserve-self-signed-certs\""
Jul 20 18:34:30.423434 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:34:30.423412    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-02ddb5/test-model-02ddb5-kserve-b594c9655-4lhnt"]
Jul 20 18:34:30.476741 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:34:30.476700    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52caccec-ec69-4e75-97a2-ab9c96d05a79-model-cache\") pod \"test-model-02ddb5-kserve-b594c9655-4lhnt\" (UID: \"52caccec-ec69-4e75-97a2-ab9c96d05a79\") " pod="ai-tenant-e2e-inf-b-02ddb5/test-model-02ddb5-kserve-b594c9655-4lhnt"
Jul 20 18:34:30.476920 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:34:30.476811    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52caccec-ec69-4e75-97a2-ab9c96d05a79-home\") pod \"test-model-02ddb5-kserve-b594c9655-4lhnt\" (UID: \"52caccec-ec69-4e75-97a2-ab9c96d05a79\") " pod="ai-tenant-e2e-inf-b-02ddb5/test-model-02ddb5-kserve-b594c9655-4lhnt"
Jul 20 18:34:30.476920 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:34:30.476850    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52caccec-ec69-4e75-97a2-ab9c96d05a79-dshm\") pod \"test-model-02ddb5-kserve-b594c9655-4lhnt\" (UID: \"52caccec-ec69-4e75-97a2-ab9c96d05a79\") " pod="ai-tenant-e2e-inf-b-02ddb5/test-model-02ddb5-kserve-b594c9655-4lhnt"
Jul 20 18:34:30.477005 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:34:30.476922    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmsgh\" (UniqueName: \"kubernetes.io/projected/52caccec-ec69-4e75-97a2-ab9c96d05a79-kube-api-access-qmsgh\") pod \"test-model-02ddb5-kserve-b594c9655-4lhnt\" (UID: \"52caccec-ec69-4e75-97a2-ab9c96d05a79\") " pod="ai-tenant-e2e-inf-b-02ddb5/test-model-02ddb5-kserve-b594c9655-4lhnt"
Jul 20 18:34:30.477005 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:34:30.476996    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52caccec-ec69-4e75-97a2-ab9c96d05a79-tmp-dir\") pod \"test-model-02ddb5-kserve-b594c9655-4lhnt\" (UID: \"52caccec-ec69-4e75-97a2-ab9c96d05a79\") " pod="ai-tenant-e2e-inf-b-02ddb5/test-model-02ddb5-kserve-b594c9655-4lhnt"
Jul 20 18:34:30.477076 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:34:30.477029    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52caccec-ec69-4e75-97a2-ab9c96d05a79-tls-certs\") pod \"test-model-02ddb5-kserve-b594c9655-4lhnt\" (UID: \"52caccec-ec69-4e75-97a2-ab9c96d05a79\") " pod="ai-tenant-e2e-inf-b-02ddb5/test-model-02ddb5-kserve-b594c9655-4lhnt"
Jul 20 18:34:30.577963 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:34:30.577924    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52caccec-ec69-4e75-97a2-ab9c96d05a79-tmp-dir\") pod \"test-model-02ddb5-kserve-b594c9655-4lhnt\" (UID: \"52caccec-ec69-4e75-97a2-ab9c96d05a79\") " pod="ai-tenant-e2e-inf-b-02ddb5/test-model-02ddb5-kserve-b594c9655-4lhnt"
Jul 20 18:34:30.577963 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:34:30.577966    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52caccec-ec69-4e75-97a2-ab9c96d05a79-tls-certs\") pod \"test-model-02ddb5-kserve-b594c9655-4lhnt\" (UID: \"52caccec-ec69-4e75-97a2-ab9c96d05a79\") " pod="ai-tenant-e2e-inf-b-02ddb5/test-model-02ddb5-kserve-b594c9655-4lhnt"
Jul 20 18:34:30.578169 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:34:30.577990    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52caccec-ec69-4e75-97a2-ab9c96d05a79-model-cache\") pod \"test-model-02ddb5-kserve-b594c9655-4lhnt\" (UID: \"52caccec-ec69-4e75-97a2-ab9c96d05a79\") " pod="ai-tenant-e2e-inf-b-02ddb5/test-model-02ddb5-kserve-b594c9655-4lhnt"
Jul 20 18:34:30.578169 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:34:30.578025    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52caccec-ec69-4e75-97a2-ab9c96d05a79-home\") pod \"test-model-02ddb5-kserve-b594c9655-4lhnt\" (UID: \"52caccec-ec69-4e75-97a2-ab9c96d05a79\") " pod="ai-tenant-e2e-inf-b-02ddb5/test-model-02ddb5-kserve-b594c9655-4lhnt"
Jul 20 18:34:30.578169 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:34:30.578049    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52caccec-ec69-4e75-97a2-ab9c96d05a79-dshm\") pod \"test-model-02ddb5-kserve-b594c9655-4lhnt\" (UID: \"52caccec-ec69-4e75-97a2-ab9c96d05a79\") " pod="ai-tenant-e2e-inf-b-02ddb5/test-model-02ddb5-kserve-b594c9655-4lhnt"
Jul 20 18:34:30.578169 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:34:30.578081    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmsgh\" (UniqueName: \"kubernetes.io/projected/52caccec-ec69-4e75-97a2-ab9c96d05a79-kube-api-access-qmsgh\") pod \"test-model-02ddb5-kserve-b594c9655-4lhnt\" (UID: \"52caccec-ec69-4e75-97a2-ab9c96d05a79\") " pod="ai-tenant-e2e-inf-b-02ddb5/test-model-02ddb5-kserve-b594c9655-4lhnt"
Jul 20 18:34:30.578418 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:34:30.578392    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52caccec-ec69-4e75-97a2-ab9c96d05a79-tmp-dir\") pod \"test-model-02ddb5-kserve-b594c9655-4lhnt\" (UID: \"52caccec-ec69-4e75-97a2-ab9c96d05a79\") " pod="ai-tenant-e2e-inf-b-02ddb5/test-model-02ddb5-kserve-b594c9655-4lhnt"
Jul 20 18:34:30.578496 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:34:30.578439    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52caccec-ec69-4e75-97a2-ab9c96d05a79-model-cache\") pod \"test-model-02ddb5-kserve-b594c9655-4lhnt\" (UID: \"52caccec-ec69-4e75-97a2-ab9c96d05a79\") " pod="ai-tenant-e2e-inf-b-02ddb5/test-model-02ddb5-kserve-b594c9655-4lhnt"
Jul 20 18:34:30.578496 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:34:30.578479    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52caccec-ec69-4e75-97a2-ab9c96d05a79-home\") pod \"test-model-02ddb5-kserve-b594c9655-4lhnt\" (UID: \"52caccec-ec69-4e75-97a2-ab9c96d05a79\") " pod="ai-tenant-e2e-inf-b-02ddb5/test-model-02ddb5-kserve-b594c9655-4lhnt"
Jul 20 18:34:30.580374 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:34:30.580347    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52caccec-ec69-4e75-97a2-ab9c96d05a79-dshm\") pod \"test-model-02ddb5-kserve-b594c9655-4lhnt\" (UID: \"52caccec-ec69-4e75-97a2-ab9c96d05a79\") " pod="ai-tenant-e2e-inf-b-02ddb5/test-model-02ddb5-kserve-b594c9655-4lhnt"
Jul 20 18:34:30.580503 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:34:30.580488    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52caccec-ec69-4e75-97a2-ab9c96d05a79-tls-certs\") pod \"test-model-02ddb5-kserve-b594c9655-4lhnt\" (UID: \"52caccec-ec69-4e75-97a2-ab9c96d05a79\") " pod="ai-tenant-e2e-inf-b-02ddb5/test-model-02ddb5-kserve-b594c9655-4lhnt"
Jul 20 18:34:30.587259 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:34:30.587240    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmsgh\" (UniqueName: \"kubernetes.io/projected/52caccec-ec69-4e75-97a2-ab9c96d05a79-kube-api-access-qmsgh\") pod \"test-model-02ddb5-kserve-b594c9655-4lhnt\" (UID: \"52caccec-ec69-4e75-97a2-ab9c96d05a79\") " pod="ai-tenant-e2e-inf-b-02ddb5/test-model-02ddb5-kserve-b594c9655-4lhnt"
Jul 20 18:34:30.723062 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:34:30.722966    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-02ddb5/test-model-02ddb5-kserve-b594c9655-4lhnt"
Jul 20 18:34:30.879863 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:34:30.879826    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-02ddb5/test-model-02ddb5-kserve-b594c9655-4lhnt"]
Jul 20 18:34:30.883184 ip-10-0-137-247 kubenswrapper[2569]: W0720 18:34:30.883152    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod52caccec_ec69_4e75_97a2_ab9c96d05a79.slice/crio-ccc28b81dba0408cdad9edd470ce1f6dec6b8420a1e565380ca90520a0c6c5a8 WatchSource:0}: Error finding container ccc28b81dba0408cdad9edd470ce1f6dec6b8420a1e565380ca90520a0c6c5a8: Status 404 returned error can't find the container with id ccc28b81dba0408cdad9edd470ce1f6dec6b8420a1e565380ca90520a0c6c5a8
Jul 20 18:34:31.376589 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:34:31.376543    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-02ddb5/test-model-02ddb5-kserve-b594c9655-4lhnt" event={"ID":"52caccec-ec69-4e75-97a2-ab9c96d05a79","Type":"ContainerStarted","Data":"c4d83a571675f366d57928585dc0ddfd18a52e5431b0cb7c888e13946661571e"}
Jul 20 18:34:31.376772 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:34:31.376601    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-02ddb5/test-model-02ddb5-kserve-b594c9655-4lhnt" event={"ID":"52caccec-ec69-4e75-97a2-ab9c96d05a79","Type":"ContainerStarted","Data":"ccc28b81dba0408cdad9edd470ce1f6dec6b8420a1e565380ca90520a0c6c5a8"}
Jul 20 18:34:31.396959 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:34:31.396912    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-02ddb5/test-model-02ddb5-kserve-b594c9655-4lhnt" podStartSLOduration=1.396896825 podStartE2EDuration="1.396896825s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 18:34:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 18:34:31.395356919 +0000 UTC m=+2317.641890329" watchObservedRunningTime="2026-07-20 18:34:31.396896825 +0000 UTC m=+2317.643430235"
Jul 20 18:34:34.050096 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:34:34.050061    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9thxb"]
Jul 20 18:34:40.723775 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:34:40.723715    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-02ddb5/test-model-02ddb5-kserve-b594c9655-4lhnt"
Jul 20 18:34:40.724330 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:34:40.723799    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-02ddb5/test-model-02ddb5-kserve-b594c9655-4lhnt"
Jul 20 18:34:40.736541 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:34:40.736504    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-02ddb5/test-model-02ddb5-kserve-b594c9655-4lhnt"
Jul 20 18:34:41.430087 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:34:41.430051    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-02ddb5/test-model-02ddb5-kserve-b594c9655-4lhnt"
Jul 20 18:34:49.148965 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:34:49.148873    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9thxb"]
Jul 20 18:35:10.540300 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:35:10.540267    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-adca0d/test-model-adca0d-kserve-f5c4445bc-k8z2w"]
Jul 20 18:35:11.416529 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:35:11.416491    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-02ddb5/test-model-02ddb5-kserve-b594c9655-4lhnt"]
Jul 20 18:35:15.463918 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:35:15.463884    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9thxb"]
Jul 20 18:35:25.560360 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:35:25.560300    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-adca0d/test-model-adca0d-kserve-f5c4445bc-k8z2w" podUID="46f7c80c-8831-474a-b5eb-fc82307638d9" containerName="main" containerID="cri-o://538bbc9eddcb51d60e02dcd2fb4a1c630e29444e9c611cbf96fbd946a5016080" gracePeriod=45
Jul 20 18:35:25.804660 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:35:25.804629    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-adca0d/test-model-adca0d-kserve-f5c4445bc-k8z2w"
Jul 20 18:35:25.933307 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:35:25.933212    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sh4tr\" (UniqueName: \"kubernetes.io/projected/46f7c80c-8831-474a-b5eb-fc82307638d9-kube-api-access-sh4tr\") pod \"46f7c80c-8831-474a-b5eb-fc82307638d9\" (UID: \"46f7c80c-8831-474a-b5eb-fc82307638d9\") "
Jul 20 18:35:25.933307 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:35:25.933269    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46f7c80c-8831-474a-b5eb-fc82307638d9-tls-certs\") pod \"46f7c80c-8831-474a-b5eb-fc82307638d9\" (UID: \"46f7c80c-8831-474a-b5eb-fc82307638d9\") "
Jul 20 18:35:25.933537 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:35:25.933322    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46f7c80c-8831-474a-b5eb-fc82307638d9-model-cache\") pod \"46f7c80c-8831-474a-b5eb-fc82307638d9\" (UID: \"46f7c80c-8831-474a-b5eb-fc82307638d9\") "
Jul 20 18:35:25.933537 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:35:25.933368    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46f7c80c-8831-474a-b5eb-fc82307638d9-dshm\") pod \"46f7c80c-8831-474a-b5eb-fc82307638d9\" (UID: \"46f7c80c-8831-474a-b5eb-fc82307638d9\") "
Jul 20 18:35:25.933537 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:35:25.933435    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46f7c80c-8831-474a-b5eb-fc82307638d9-home\") pod \"46f7c80c-8831-474a-b5eb-fc82307638d9\" (UID: \"46f7c80c-8831-474a-b5eb-fc82307638d9\") "
Jul 20 18:35:25.933537 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:35:25.933461    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46f7c80c-8831-474a-b5eb-fc82307638d9-tmp-dir\") pod \"46f7c80c-8831-474a-b5eb-fc82307638d9\" (UID: \"46f7c80c-8831-474a-b5eb-fc82307638d9\") "
Jul 20 18:35:25.933752 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:35:25.933601    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46f7c80c-8831-474a-b5eb-fc82307638d9-model-cache" (OuterVolumeSpecName: "model-cache") pod "46f7c80c-8831-474a-b5eb-fc82307638d9" (UID: "46f7c80c-8831-474a-b5eb-fc82307638d9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 18:35:25.933752 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:35:25.933678    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46f7c80c-8831-474a-b5eb-fc82307638d9-home" (OuterVolumeSpecName: "home") pod "46f7c80c-8831-474a-b5eb-fc82307638d9" (UID: "46f7c80c-8831-474a-b5eb-fc82307638d9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 18:35:25.933860 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:35:25.933712    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46f7c80c-8831-474a-b5eb-fc82307638d9-model-cache\") on node \"ip-10-0-137-247.ec2.internal\" DevicePath \"\""
Jul 20 18:35:25.933860 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:35:25.933774    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46f7c80c-8831-474a-b5eb-fc82307638d9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "46f7c80c-8831-474a-b5eb-fc82307638d9" (UID: "46f7c80c-8831-474a-b5eb-fc82307638d9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 18:35:25.935573 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:35:25.935542    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46f7c80c-8831-474a-b5eb-fc82307638d9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "46f7c80c-8831-474a-b5eb-fc82307638d9" (UID: "46f7c80c-8831-474a-b5eb-fc82307638d9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 18:35:25.935573 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:35:25.935542    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46f7c80c-8831-474a-b5eb-fc82307638d9-dshm" (OuterVolumeSpecName: "dshm") pod "46f7c80c-8831-474a-b5eb-fc82307638d9" (UID: "46f7c80c-8831-474a-b5eb-fc82307638d9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 18:35:25.935711 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:35:25.935604    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46f7c80c-8831-474a-b5eb-fc82307638d9-kube-api-access-sh4tr" (OuterVolumeSpecName: "kube-api-access-sh4tr") pod "46f7c80c-8831-474a-b5eb-fc82307638d9" (UID: "46f7c80c-8831-474a-b5eb-fc82307638d9"). InnerVolumeSpecName "kube-api-access-sh4tr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 18:35:26.034397 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:35:26.034342    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46f7c80c-8831-474a-b5eb-fc82307638d9-tls-certs\") on node \"ip-10-0-137-247.ec2.internal\" DevicePath \"\""
Jul 20 18:35:26.034397 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:35:26.034391    2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46f7c80c-8831-474a-b5eb-fc82307638d9-dshm\") on node \"ip-10-0-137-247.ec2.internal\" DevicePath \"\""
Jul 20 18:35:26.034397 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:35:26.034399    2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46f7c80c-8831-474a-b5eb-fc82307638d9-home\") on node \"ip-10-0-137-247.ec2.internal\" DevicePath \"\""
Jul 20 18:35:26.034397 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:35:26.034407    2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46f7c80c-8831-474a-b5eb-fc82307638d9-tmp-dir\") on node \"ip-10-0-137-247.ec2.internal\" DevicePath \"\""
Jul 20 18:35:26.034397 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:35:26.034415    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sh4tr\" (UniqueName: \"kubernetes.io/projected/46f7c80c-8831-474a-b5eb-fc82307638d9-kube-api-access-sh4tr\") on node \"ip-10-0-137-247.ec2.internal\" DevicePath \"\""
Jul 20 18:35:26.436392 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:35:26.436353    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-02ddb5/test-model-02ddb5-kserve-b594c9655-4lhnt" podUID="52caccec-ec69-4e75-97a2-ab9c96d05a79" containerName="main" containerID="cri-o://c4d83a571675f366d57928585dc0ddfd18a52e5431b0cb7c888e13946661571e" gracePeriod=45
Jul 20 18:35:26.610995 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:35:26.610961    2569 generic.go:356] "Generic (PLEG): container finished" podID="52caccec-ec69-4e75-97a2-ab9c96d05a79" containerID="c4d83a571675f366d57928585dc0ddfd18a52e5431b0cb7c888e13946661571e" exitCode=0
Jul 20 18:35:26.611418 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:35:26.611030    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-02ddb5/test-model-02ddb5-kserve-b594c9655-4lhnt" event={"ID":"52caccec-ec69-4e75-97a2-ab9c96d05a79","Type":"ContainerDied","Data":"c4d83a571675f366d57928585dc0ddfd18a52e5431b0cb7c888e13946661571e"}
Jul 20 18:35:26.612439 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:35:26.612409    2569 generic.go:356] "Generic (PLEG): container finished" podID="46f7c80c-8831-474a-b5eb-fc82307638d9" containerID="538bbc9eddcb51d60e02dcd2fb4a1c630e29444e9c611cbf96fbd946a5016080" exitCode=0
Jul 20 18:35:26.612560 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:35:26.612458    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-adca0d/test-model-adca0d-kserve-f5c4445bc-k8z2w"
Jul 20 18:35:26.612560 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:35:26.612488    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-adca0d/test-model-adca0d-kserve-f5c4445bc-k8z2w" event={"ID":"46f7c80c-8831-474a-b5eb-fc82307638d9","Type":"ContainerDied","Data":"538bbc9eddcb51d60e02dcd2fb4a1c630e29444e9c611cbf96fbd946a5016080"}
Jul 20 18:35:26.612645 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:35:26.612553    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-adca0d/test-model-adca0d-kserve-f5c4445bc-k8z2w" event={"ID":"46f7c80c-8831-474a-b5eb-fc82307638d9","Type":"ContainerDied","Data":"7b38188bf33ee3ca275777c18ec2661310f138ed30fc532b28b74eec3b34680e"}
Jul 20 18:35:26.612645 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:35:26.612584    2569 scope.go:117] "RemoveContainer" containerID="538bbc9eddcb51d60e02dcd2fb4a1c630e29444e9c611cbf96fbd946a5016080"
Jul 20 18:35:26.621668 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:35:26.621431    2569 scope.go:117] "RemoveContainer" containerID="538bbc9eddcb51d60e02dcd2fb4a1c630e29444e9c611cbf96fbd946a5016080"
Jul 20 18:35:26.621819 ip-10-0-137-247 kubenswrapper[2569]: E0720 18:35:26.621795    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"538bbc9eddcb51d60e02dcd2fb4a1c630e29444e9c611cbf96fbd946a5016080\": container with ID starting with 538bbc9eddcb51d60e02dcd2fb4a1c630e29444e9c611cbf96fbd946a5016080 not found: ID does not exist" containerID="538bbc9eddcb51d60e02dcd2fb4a1c630e29444e9c611cbf96fbd946a5016080"
Jul 20 18:35:26.621957 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:35:26.621831    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"538bbc9eddcb51d60e02dcd2fb4a1c630e29444e9c611cbf96fbd946a5016080"} err="failed to get container status \"538bbc9eddcb51d60e02dcd2fb4a1c630e29444e9c611cbf96fbd946a5016080\": rpc error: code = NotFound desc = could not find container \"538bbc9eddcb51d60e02dcd2fb4a1c630e29444e9c611cbf96fbd946a5016080\": container with ID starting with 538bbc9eddcb51d60e02dcd2fb4a1c630e29444e9c611cbf96fbd946a5016080 not found: ID does not exist"
Jul 20 18:35:26.630019 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:35:26.629984    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-adca0d/test-model-adca0d-kserve-f5c4445bc-k8z2w"]
Jul 20 18:35:26.634768 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:35:26.634742    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-adca0d/test-model-adca0d-kserve-f5c4445bc-k8z2w"]
Jul 20 18:35:26.682740 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:35:26.682703    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-02ddb5/test-model-02ddb5-kserve-b594c9655-4lhnt"
Jul 20 18:35:26.740232 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:35:26.740152    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52caccec-ec69-4e75-97a2-ab9c96d05a79-tls-certs\") pod \"52caccec-ec69-4e75-97a2-ab9c96d05a79\" (UID: \"52caccec-ec69-4e75-97a2-ab9c96d05a79\") "
Jul 20 18:35:26.740232 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:35:26.740191    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52caccec-ec69-4e75-97a2-ab9c96d05a79-model-cache\") pod \"52caccec-ec69-4e75-97a2-ab9c96d05a79\" (UID: \"52caccec-ec69-4e75-97a2-ab9c96d05a79\") "
Jul 20 18:35:26.740232 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:35:26.740228    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qmsgh\" (UniqueName: \"kubernetes.io/projected/52caccec-ec69-4e75-97a2-ab9c96d05a79-kube-api-access-qmsgh\") pod \"52caccec-ec69-4e75-97a2-ab9c96d05a79\" (UID: \"52caccec-ec69-4e75-97a2-ab9c96d05a79\") "
Jul 20 18:35:26.740479 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:35:26.740258    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52caccec-ec69-4e75-97a2-ab9c96d05a79-home\") pod \"52caccec-ec69-4e75-97a2-ab9c96d05a79\" (UID: \"52caccec-ec69-4e75-97a2-ab9c96d05a79\") "
Jul 20 18:35:26.740479 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:35:26.740289    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52caccec-ec69-4e75-97a2-ab9c96d05a79-tmp-dir\") pod \"52caccec-ec69-4e75-97a2-ab9c96d05a79\" (UID: \"52caccec-ec69-4e75-97a2-ab9c96d05a79\") "
Jul 20 18:35:26.740479 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:35:26.740314    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52caccec-ec69-4e75-97a2-ab9c96d05a79-dshm\") pod \"52caccec-ec69-4e75-97a2-ab9c96d05a79\" (UID: \"52caccec-ec69-4e75-97a2-ab9c96d05a79\") "
Jul 20 18:35:26.740479 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:35:26.740441    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52caccec-ec69-4e75-97a2-ab9c96d05a79-model-cache" (OuterVolumeSpecName: "model-cache") pod "52caccec-ec69-4e75-97a2-ab9c96d05a79" (UID: "52caccec-ec69-4e75-97a2-ab9c96d05a79"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 18:35:26.740670 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:35:26.740553    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52caccec-ec69-4e75-97a2-ab9c96d05a79-model-cache\") on node \"ip-10-0-137-247.ec2.internal\" DevicePath \"\""
Jul 20 18:35:26.740670 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:35:26.740553    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52caccec-ec69-4e75-97a2-ab9c96d05a79-home" (OuterVolumeSpecName: "home") pod "52caccec-ec69-4e75-97a2-ab9c96d05a79" (UID: "52caccec-ec69-4e75-97a2-ab9c96d05a79"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 18:35:26.740670 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:35:26.740617    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52caccec-ec69-4e75-97a2-ab9c96d05a79-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "52caccec-ec69-4e75-97a2-ab9c96d05a79" (UID: "52caccec-ec69-4e75-97a2-ab9c96d05a79"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 18:35:26.742391 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:35:26.742372    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52caccec-ec69-4e75-97a2-ab9c96d05a79-dshm" (OuterVolumeSpecName: "dshm") pod "52caccec-ec69-4e75-97a2-ab9c96d05a79" (UID: "52caccec-ec69-4e75-97a2-ab9c96d05a79"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 18:35:26.742485 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:35:26.742464    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52caccec-ec69-4e75-97a2-ab9c96d05a79-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "52caccec-ec69-4e75-97a2-ab9c96d05a79" (UID: "52caccec-ec69-4e75-97a2-ab9c96d05a79"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 18:35:26.742532 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:35:26.742501    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52caccec-ec69-4e75-97a2-ab9c96d05a79-kube-api-access-qmsgh" (OuterVolumeSpecName: "kube-api-access-qmsgh") pod "52caccec-ec69-4e75-97a2-ab9c96d05a79" (UID: "52caccec-ec69-4e75-97a2-ab9c96d05a79"). InnerVolumeSpecName "kube-api-access-qmsgh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 18:35:26.841775 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:35:26.841743    2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52caccec-ec69-4e75-97a2-ab9c96d05a79-home\") on node \"ip-10-0-137-247.ec2.internal\" DevicePath \"\""
Jul 20 18:35:26.841775 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:35:26.841771    2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52caccec-ec69-4e75-97a2-ab9c96d05a79-tmp-dir\") on node \"ip-10-0-137-247.ec2.internal\" DevicePath \"\""
Jul 20 18:35:26.841775 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:35:26.841781    2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52caccec-ec69-4e75-97a2-ab9c96d05a79-dshm\") on node \"ip-10-0-137-247.ec2.internal\" DevicePath \"\""
Jul 20 18:35:26.842001 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:35:26.841788    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52caccec-ec69-4e75-97a2-ab9c96d05a79-tls-certs\") on node \"ip-10-0-137-247.ec2.internal\" DevicePath \"\""
Jul 20 18:35:26.842001 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:35:26.841796    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmsgh\" (UniqueName: \"kubernetes.io/projected/52caccec-ec69-4e75-97a2-ab9c96d05a79-kube-api-access-qmsgh\") on node \"ip-10-0-137-247.ec2.internal\" DevicePath \"\""
Jul 20 18:35:27.618237 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:35:27.618206    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-02ddb5/test-model-02ddb5-kserve-b594c9655-4lhnt"
Jul 20 18:35:27.618687 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:35:27.618239    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-02ddb5/test-model-02ddb5-kserve-b594c9655-4lhnt" event={"ID":"52caccec-ec69-4e75-97a2-ab9c96d05a79","Type":"ContainerDied","Data":"ccc28b81dba0408cdad9edd470ce1f6dec6b8420a1e565380ca90520a0c6c5a8"}
Jul 20 18:35:27.618687 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:35:27.618298    2569 scope.go:117] "RemoveContainer" containerID="c4d83a571675f366d57928585dc0ddfd18a52e5431b0cb7c888e13946661571e"
Jul 20 18:35:27.641004 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:35:27.640971    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-02ddb5/test-model-02ddb5-kserve-b594c9655-4lhnt"]
Jul 20 18:35:27.642756 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:35:27.642734    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-02ddb5/test-model-02ddb5-kserve-b594c9655-4lhnt"]
Jul 20 18:35:28.181973 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:35:28.181945    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="46f7c80c-8831-474a-b5eb-fc82307638d9" path="/var/lib/kubelet/pods/46f7c80c-8831-474a-b5eb-fc82307638d9/volumes"
Jul 20 18:35:28.182297 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:35:28.182283    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="52caccec-ec69-4e75-97a2-ab9c96d05a79" path="/var/lib/kubelet/pods/52caccec-ec69-4e75-97a2-ab9c96d05a79/volumes"
Jul 20 18:35:29.949848 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:35:29.949806    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9thxb"]
Jul 20 18:35:33.711868 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:35:33.711834    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-02ddb5-p7vqv"]
Jul 20 18:35:33.712242 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:35:33.712107    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46f7c80c-8831-474a-b5eb-fc82307638d9" containerName="main"
Jul 20 18:35:33.712242 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:35:33.712117    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="46f7c80c-8831-474a-b5eb-fc82307638d9" containerName="main"
Jul 20 18:35:33.712242 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:35:33.712131    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52caccec-ec69-4e75-97a2-ab9c96d05a79" containerName="main"
Jul 20 18:35:33.712242 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:35:33.712136    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="52caccec-ec69-4e75-97a2-ab9c96d05a79" containerName="main"
Jul 20 18:35:33.712242 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:35:33.712187    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="52caccec-ec69-4e75-97a2-ab9c96d05a79" containerName="main"
Jul 20 18:35:33.712242 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:35:33.712195    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="46f7c80c-8831-474a-b5eb-fc82307638d9" containerName="main"
Jul 20 18:35:33.714993 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:35:33.714972    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-02ddb5-p7vqv"
Jul 20 18:35:33.718354 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:35:33.718335    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fk7rm\""
Jul 20 18:35:33.718876 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:35:33.718851    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 20 18:35:33.728233 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:35:33.728206    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-02ddb5-p7vqv"]
Jul 20 18:35:33.802284 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:35:33.802247    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b811d45-dfae-42fe-add8-3268fbe968f5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-02ddb5-p7vqv\" (UID: \"3b811d45-dfae-42fe-add8-3268fbe968f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-02ddb5-p7vqv"
Jul 20 18:35:33.903525 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:35:33.903487    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b811d45-dfae-42fe-add8-3268fbe968f5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-02ddb5-p7vqv\" (UID: \"3b811d45-dfae-42fe-add8-3268fbe968f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-02ddb5-p7vqv"
Jul 20 18:35:33.904123 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:35:33.904105    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b811d45-dfae-42fe-add8-3268fbe968f5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-02ddb5-p7vqv\" (UID: \"3b811d45-dfae-42fe-add8-3268fbe968f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-02ddb5-p7vqv"
Jul 20 18:35:34.029625 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:35:34.029586    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-02ddb5-p7vqv"
Jul 20 18:35:34.158384 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:35:34.158355    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-02ddb5-p7vqv"]
Jul 20 18:35:34.160525 ip-10-0-137-247 kubenswrapper[2569]: W0720 18:35:34.160497    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b811d45_dfae_42fe_add8_3268fbe968f5.slice/crio-a57a7f33e526f002bd9e45f981ca1ec4cf82085b37bd9a3b9f38d026ed9d5137 WatchSource:0}: Error finding container a57a7f33e526f002bd9e45f981ca1ec4cf82085b37bd9a3b9f38d026ed9d5137: Status 404 returned error can't find the container with id a57a7f33e526f002bd9e45f981ca1ec4cf82085b37bd9a3b9f38d026ed9d5137
Jul 20 18:35:34.653592 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:35:34.653507    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-02ddb5-p7vqv" event={"ID":"3b811d45-dfae-42fe-add8-3268fbe968f5","Type":"ContainerStarted","Data":"a938107ff8f5e5acfb2999b5957238874e56e8cb6e7baba8f7d57e24e060b1e5"}
Jul 20 18:35:34.653592 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:35:34.653559    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-02ddb5-p7vqv" event={"ID":"3b811d45-dfae-42fe-add8-3268fbe968f5","Type":"ContainerStarted","Data":"a57a7f33e526f002bd9e45f981ca1ec4cf82085b37bd9a3b9f38d026ed9d5137"}
Jul 20 18:35:34.671672 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:35:34.671611    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-02ddb5-p7vqv" podStartSLOduration=1.671591978 podStartE2EDuration="1.671591978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 18:35:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 18:35:34.669299541 +0000 UTC m=+2380.915832988" watchObservedRunningTime="2026-07-20 18:35:34.671591978 +0000 UTC m=+2380.918125391"
Jul 20 18:35:35.659429 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:35:35.659394    2569 generic.go:356] "Generic (PLEG): container finished" podID="3b811d45-dfae-42fe-add8-3268fbe968f5" containerID="a938107ff8f5e5acfb2999b5957238874e56e8cb6e7baba8f7d57e24e060b1e5" exitCode=0
Jul 20 18:35:35.659834 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:35:35.659475    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-02ddb5-p7vqv" event={"ID":"3b811d45-dfae-42fe-add8-3268fbe968f5","Type":"ContainerDied","Data":"a938107ff8f5e5acfb2999b5957238874e56e8cb6e7baba8f7d57e24e060b1e5"}
Jul 20 18:35:36.786894 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:35:36.786858    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-02ddb5-p7vqv"
Jul 20 18:35:36.930162 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:35:36.930067    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b811d45-dfae-42fe-add8-3268fbe968f5-maas-api-service-ca\") pod \"3b811d45-dfae-42fe-add8-3268fbe968f5\" (UID: \"3b811d45-dfae-42fe-add8-3268fbe968f5\") "
Jul 20 18:35:36.930441 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:35:36.930417    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b811d45-dfae-42fe-add8-3268fbe968f5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3b811d45-dfae-42fe-add8-3268fbe968f5" (UID: "3b811d45-dfae-42fe-add8-3268fbe968f5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 18:35:37.031241 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:35:37.031206    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b811d45-dfae-42fe-add8-3268fbe968f5-maas-api-service-ca\") on node \"ip-10-0-137-247.ec2.internal\" DevicePath \"\""
Jul 20 18:35:37.668884 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:35:37.668843    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-02ddb5-p7vqv" event={"ID":"3b811d45-dfae-42fe-add8-3268fbe968f5","Type":"ContainerDied","Data":"a57a7f33e526f002bd9e45f981ca1ec4cf82085b37bd9a3b9f38d026ed9d5137"}
Jul 20 18:35:37.669062 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:35:37.668892    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a57a7f33e526f002bd9e45f981ca1ec4cf82085b37bd9a3b9f38d026ed9d5137"
Jul 20 18:35:37.669062 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:35:37.668850    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-02ddb5-p7vqv"
Jul 20 18:35:47.393680 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:35:47.393640    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9thxb"]
Jul 20 18:35:48.777364 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:35:48.777330    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-02ddb5-p7vqv"]
Jul 20 18:35:48.782310 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:35:48.782273    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-02ddb5-p7vqv"]
Jul 20 18:35:50.184582 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:35:50.184542    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b811d45-dfae-42fe-add8-3268fbe968f5" path="/var/lib/kubelet/pods/3b811d45-dfae-42fe-add8-3268fbe968f5/volumes"
Jul 20 18:35:54.264007 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:35:54.263975    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wjs8g_07587a78-2d3a-40ad-9aae-a4875200263a/ovn-acl-logging/0.log"
Jul 20 18:35:54.269450 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:35:54.269426    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wjs8g_07587a78-2d3a-40ad-9aae-a4875200263a/ovn-acl-logging/0.log"
Jul 20 18:35:54.356865 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:35:54.356835    2569 scope.go:117] "RemoveContainer" containerID="dfd1212dc727ada3b78001acec980010b92caa96627725eed975c09fdb02dea2"
Jul 20 18:36:02.248009 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:36:02.247976    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9thxb"]
Jul 20 18:36:07.164499 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:36:07.164458    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-09a1ba-8c8997646-92clk"]
Jul 20 18:36:07.165017 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:36:07.164964    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b811d45-dfae-42fe-add8-3268fbe968f5" containerName="revoke-keys"
Jul 20 18:36:07.165017 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:36:07.164985    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b811d45-dfae-42fe-add8-3268fbe968f5" containerName="revoke-keys"
Jul 20 18:36:07.165141 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:36:07.165092    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b811d45-dfae-42fe-add8-3268fbe968f5" containerName="revoke-keys"
Jul 20 18:36:07.168559 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:36:07.168534    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-09a1ba-8c8997646-92clk"
Jul 20 18:36:07.171504 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:36:07.171483    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-h2tfv\""
Jul 20 18:36:07.171856 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:36:07.171838    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-09a1ba-serving-cert\""
Jul 20 18:36:07.172361 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:36:07.172342    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 20 18:36:07.172463 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:36:07.172365    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 20 18:36:07.181761 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:36:07.181739    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-09a1ba-8c8997646-92clk"]
Jul 20 18:36:07.286794 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:36:07.286750    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79e938cf-267c-43ce-bef1-7db1d9a0aa74-maas-api-tls\") pod \"maas-api-e2e-api-b-09a1ba-8c8997646-92clk\" (UID: \"79e938cf-267c-43ce-bef1-7db1d9a0aa74\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-09a1ba-8c8997646-92clk"
Jul 20 18:36:07.286964 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:36:07.286811    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9fdh\" (UniqueName: \"kubernetes.io/projected/79e938cf-267c-43ce-bef1-7db1d9a0aa74-kube-api-access-m9fdh\") pod \"maas-api-e2e-api-b-09a1ba-8c8997646-92clk\" (UID: \"79e938cf-267c-43ce-bef1-7db1d9a0aa74\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-09a1ba-8c8997646-92clk"
Jul 20 18:36:07.286964 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:36:07.286851    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79e938cf-267c-43ce-bef1-7db1d9a0aa74-openshift-service-ca\") pod \"maas-api-e2e-api-b-09a1ba-8c8997646-92clk\" (UID: \"79e938cf-267c-43ce-bef1-7db1d9a0aa74\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-09a1ba-8c8997646-92clk"
Jul 20 18:36:07.388098 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:36:07.388061    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79e938cf-267c-43ce-bef1-7db1d9a0aa74-openshift-service-ca\") pod \"maas-api-e2e-api-b-09a1ba-8c8997646-92clk\" (UID: \"79e938cf-267c-43ce-bef1-7db1d9a0aa74\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-09a1ba-8c8997646-92clk"
Jul 20 18:36:07.388271 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:36:07.388114    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79e938cf-267c-43ce-bef1-7db1d9a0aa74-maas-api-tls\") pod \"maas-api-e2e-api-b-09a1ba-8c8997646-92clk\" (UID: \"79e938cf-267c-43ce-bef1-7db1d9a0aa74\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-09a1ba-8c8997646-92clk"
Jul 20 18:36:07.388271 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:36:07.388154    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9fdh\" (UniqueName: \"kubernetes.io/projected/79e938cf-267c-43ce-bef1-7db1d9a0aa74-kube-api-access-m9fdh\") pod \"maas-api-e2e-api-b-09a1ba-8c8997646-92clk\" (UID: \"79e938cf-267c-43ce-bef1-7db1d9a0aa74\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-09a1ba-8c8997646-92clk"
Jul 20 18:36:07.388349 ip-10-0-137-247 kubenswrapper[2569]: E0720 18:36:07.388271    2569 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-09a1ba-serving-cert: secret "maas-api-e2e-api-b-09a1ba-serving-cert" not found
Jul 20 18:36:07.388349 ip-10-0-137-247 kubenswrapper[2569]: E0720 18:36:07.388340    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79e938cf-267c-43ce-bef1-7db1d9a0aa74-maas-api-tls podName:79e938cf-267c-43ce-bef1-7db1d9a0aa74 nodeName:}" failed. No retries permitted until 2026-07-20 18:36:07.888319465 +0000 UTC m=+2414.134852859 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/79e938cf-267c-43ce-bef1-7db1d9a0aa74-maas-api-tls") pod "maas-api-e2e-api-b-09a1ba-8c8997646-92clk" (UID: "79e938cf-267c-43ce-bef1-7db1d9a0aa74") : secret "maas-api-e2e-api-b-09a1ba-serving-cert" not found
Jul 20 18:36:07.388764 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:36:07.388745    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79e938cf-267c-43ce-bef1-7db1d9a0aa74-openshift-service-ca\") pod \"maas-api-e2e-api-b-09a1ba-8c8997646-92clk\" (UID: \"79e938cf-267c-43ce-bef1-7db1d9a0aa74\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-09a1ba-8c8997646-92clk"
Jul 20 18:36:07.400124 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:36:07.400093    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9fdh\" (UniqueName: \"kubernetes.io/projected/79e938cf-267c-43ce-bef1-7db1d9a0aa74-kube-api-access-m9fdh\") pod \"maas-api-e2e-api-b-09a1ba-8c8997646-92clk\" (UID: \"79e938cf-267c-43ce-bef1-7db1d9a0aa74\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-09a1ba-8c8997646-92clk"
Jul 20 18:36:07.892912 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:36:07.892882    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79e938cf-267c-43ce-bef1-7db1d9a0aa74-maas-api-tls\") pod \"maas-api-e2e-api-b-09a1ba-8c8997646-92clk\" (UID: \"79e938cf-267c-43ce-bef1-7db1d9a0aa74\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-09a1ba-8c8997646-92clk"
Jul 20 18:36:07.895527 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:36:07.895502    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79e938cf-267c-43ce-bef1-7db1d9a0aa74-maas-api-tls\") pod \"maas-api-e2e-api-b-09a1ba-8c8997646-92clk\" (UID: \"79e938cf-267c-43ce-bef1-7db1d9a0aa74\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-09a1ba-8c8997646-92clk"
Jul 20 18:36:08.080983 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:36:08.080940    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-09a1ba-8c8997646-92clk"
Jul 20 18:36:08.226493 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:36:08.226459    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-09a1ba-8c8997646-92clk"]
Jul 20 18:36:08.228782 ip-10-0-137-247 kubenswrapper[2569]: W0720 18:36:08.228749    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79e938cf_267c_43ce_bef1_7db1d9a0aa74.slice/crio-67e00f4ddac0f86b1d79d855de72996546af9ad0afdd536c11eb9dd8caea0eed WatchSource:0}: Error finding container 67e00f4ddac0f86b1d79d855de72996546af9ad0afdd536c11eb9dd8caea0eed: Status 404 returned error can't find the container with id 67e00f4ddac0f86b1d79d855de72996546af9ad0afdd536c11eb9dd8caea0eed
Jul 20 18:36:08.230300 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:36:08.230279    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 20 18:36:08.801487 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:36:08.801455    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-09a1ba-8c8997646-92clk" event={"ID":"79e938cf-267c-43ce-bef1-7db1d9a0aa74","Type":"ContainerStarted","Data":"67e00f4ddac0f86b1d79d855de72996546af9ad0afdd536c11eb9dd8caea0eed"}
Jul 20 18:36:10.811288 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:36:10.811250    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-09a1ba-8c8997646-92clk" event={"ID":"79e938cf-267c-43ce-bef1-7db1d9a0aa74","Type":"ContainerStarted","Data":"2487893ea85a0571b4848452d4d42435ecbf0ff7df67f99584a4a805f14f9bbf"}
Jul 20 18:36:10.811745 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:36:10.811375    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-09a1ba-8c8997646-92clk"
Jul 20 18:36:10.830027 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:36:10.829971    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-09a1ba-8c8997646-92clk" podStartSLOduration=2.23872251 podStartE2EDuration="3.829950397s" totalImagesPullingTime="1.591227887s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 18:36:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 18:36:10.828537113 +0000 UTC m=+2417.075070561" watchObservedRunningTime="2026-07-20 18:36:10.829950397 +0000 UTC m=+2417.076483801"
Jul 20 18:36:12.450478 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:36:12.450440    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9thxb"]
Jul 20 18:36:16.822595 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:36:16.822518    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-09a1ba-8c8997646-92clk"
Jul 20 18:36:19.591369 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:36:19.591335    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3054a0-9f4fz"]
Jul 20 18:36:19.594527 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:36:19.594507    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3054a0-9f4fz"
Jul 20 18:36:19.597017 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:36:19.596998    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fk7rm\""
Jul 20 18:36:19.602178 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:36:19.602117    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3054a0-9f4fz"]
Jul 20 18:36:19.692101 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:36:19.692068    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b7563b4-0849-405f-9b04-ae132aa35731-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-3054a0-9f4fz\" (UID: \"9b7563b4-0849-405f-9b04-ae132aa35731\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3054a0-9f4fz"
Jul 20 18:36:19.792823 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:36:19.792786    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b7563b4-0849-405f-9b04-ae132aa35731-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-3054a0-9f4fz\" (UID: \"9b7563b4-0849-405f-9b04-ae132aa35731\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3054a0-9f4fz"
Jul 20 18:36:19.793408 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:36:19.793388    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b7563b4-0849-405f-9b04-ae132aa35731-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-3054a0-9f4fz\" (UID: \"9b7563b4-0849-405f-9b04-ae132aa35731\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3054a0-9f4fz"
Jul 20 18:36:19.907874 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:36:19.907780    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3054a0-9f4fz"
Jul 20 18:36:20.037919 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:36:20.037885    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3054a0-9f4fz"]
Jul 20 18:36:20.041147 ip-10-0-137-247 kubenswrapper[2569]: W0720 18:36:20.041119    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b7563b4_0849_405f_9b04_ae132aa35731.slice/crio-b12deba3bff30c1da6001f7610684c7c16b015ff44633d05976cbbfe1a4dfb75 WatchSource:0}: Error finding container b12deba3bff30c1da6001f7610684c7c16b015ff44633d05976cbbfe1a4dfb75: Status 404 returned error can't find the container with id b12deba3bff30c1da6001f7610684c7c16b015ff44633d05976cbbfe1a4dfb75
Jul 20 18:36:20.857601 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:36:20.857563    2569 generic.go:356] "Generic (PLEG): container finished" podID="9b7563b4-0849-405f-9b04-ae132aa35731" containerID="4b604dbc8465f47e413a71151bf45176f1bbd7042177f7e63cc033b5069220db" exitCode=0
Jul 20 18:36:20.858009 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:36:20.857655    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3054a0-9f4fz" event={"ID":"9b7563b4-0849-405f-9b04-ae132aa35731","Type":"ContainerDied","Data":"4b604dbc8465f47e413a71151bf45176f1bbd7042177f7e63cc033b5069220db"}
Jul 20 18:36:20.858009 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:36:20.857712    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3054a0-9f4fz" event={"ID":"9b7563b4-0849-405f-9b04-ae132aa35731","Type":"ContainerStarted","Data":"b12deba3bff30c1da6001f7610684c7c16b015ff44633d05976cbbfe1a4dfb75"}
Jul 20 18:36:21.988009 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:36:21.987986    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3054a0-9f4fz"
Jul 20 18:36:22.112611 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:36:22.112524    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b7563b4-0849-405f-9b04-ae132aa35731-maas-api-service-ca\") pod \"9b7563b4-0849-405f-9b04-ae132aa35731\" (UID: \"9b7563b4-0849-405f-9b04-ae132aa35731\") "
Jul 20 18:36:22.112914 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:36:22.112890    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b7563b4-0849-405f-9b04-ae132aa35731-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9b7563b4-0849-405f-9b04-ae132aa35731" (UID: "9b7563b4-0849-405f-9b04-ae132aa35731"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 18:36:22.213166 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:36:22.213136    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b7563b4-0849-405f-9b04-ae132aa35731-maas-api-service-ca\") on node \"ip-10-0-137-247.ec2.internal\" DevicePath \"\""
Jul 20 18:36:22.867374 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:36:22.867346    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3054a0-9f4fz"
Jul 20 18:36:22.867547 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:36:22.867347    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3054a0-9f4fz" event={"ID":"9b7563b4-0849-405f-9b04-ae132aa35731","Type":"ContainerDied","Data":"b12deba3bff30c1da6001f7610684c7c16b015ff44633d05976cbbfe1a4dfb75"}
Jul 20 18:36:22.867547 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:36:22.867469    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b12deba3bff30c1da6001f7610684c7c16b015ff44633d05976cbbfe1a4dfb75"
Jul 20 18:36:38.149743 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:36:38.149693    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9thxb"]
Jul 20 18:36:39.676046 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:36:39.676009    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3054a0-9f4fz"]
Jul 20 18:36:39.678950 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:36:39.678926    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3054a0-9f4fz"]
Jul 20 18:36:40.182244 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:36:40.182209    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b7563b4-0849-405f-9b04-ae132aa35731" path="/var/lib/kubelet/pods/9b7563b4-0849-405f-9b04-ae132aa35731/volumes"
Jul 20 18:36:46.749839 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:36:46.749800    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-09a1ba-m2dl7"]
Jul 20 18:36:46.750305 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:36:46.750202    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b7563b4-0849-405f-9b04-ae132aa35731" containerName="revoke-keys"
Jul 20 18:36:46.750305 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:36:46.750219    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b7563b4-0849-405f-9b04-ae132aa35731" containerName="revoke-keys"
Jul 20 18:36:46.750412 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:36:46.750316    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b7563b4-0849-405f-9b04-ae132aa35731" containerName="revoke-keys"
Jul 20 18:36:46.754806 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:36:46.754784    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-09a1ba-m2dl7"
Jul 20 18:36:46.757530 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:36:46.757506    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fk7rm\""
Jul 20 18:36:46.765061 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:36:46.765036    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-09a1ba-m2dl7"]
Jul 20 18:36:46.820143 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:36:46.820110    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c1c87bc-291b-45da-b96b-daa055a0e7ac-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-09a1ba-m2dl7\" (UID: \"4c1c87bc-291b-45da-b96b-daa055a0e7ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-09a1ba-m2dl7"
Jul 20 18:36:46.921573 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:36:46.921534    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c1c87bc-291b-45da-b96b-daa055a0e7ac-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-09a1ba-m2dl7\" (UID: \"4c1c87bc-291b-45da-b96b-daa055a0e7ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-09a1ba-m2dl7"
Jul 20 18:36:46.922216 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:36:46.922190    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c1c87bc-291b-45da-b96b-daa055a0e7ac-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-09a1ba-m2dl7\" (UID: \"4c1c87bc-291b-45da-b96b-daa055a0e7ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-09a1ba-m2dl7"
Jul 20 18:36:47.064658 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:36:47.064613    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-09a1ba-m2dl7"
Jul 20 18:36:47.237623 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:36:47.237587    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-09a1ba-m2dl7"]
Jul 20 18:36:47.242205 ip-10-0-137-247 kubenswrapper[2569]: W0720 18:36:47.242175    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c1c87bc_291b_45da_b96b_daa055a0e7ac.slice/crio-842ce450706cf76db4c3dac79912d90d44512dcb12af960e2de8b89fcd2ffed2 WatchSource:0}: Error finding container 842ce450706cf76db4c3dac79912d90d44512dcb12af960e2de8b89fcd2ffed2: Status 404 returned error can't find the container with id 842ce450706cf76db4c3dac79912d90d44512dcb12af960e2de8b89fcd2ffed2
Jul 20 18:36:47.977675 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:36:47.977645    2569 generic.go:356] "Generic (PLEG): container finished" podID="4c1c87bc-291b-45da-b96b-daa055a0e7ac" containerID="983945607ee730c5f1e46e88786f76bb679306f9ea1137b220dcfcf999d4ccba" exitCode=0
Jul 20 18:36:47.978037 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:36:47.977768    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-09a1ba-m2dl7" event={"ID":"4c1c87bc-291b-45da-b96b-daa055a0e7ac","Type":"ContainerDied","Data":"983945607ee730c5f1e46e88786f76bb679306f9ea1137b220dcfcf999d4ccba"}
Jul 20 18:36:47.978037 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:36:47.977823    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-09a1ba-m2dl7" event={"ID":"4c1c87bc-291b-45da-b96b-daa055a0e7ac","Type":"ContainerStarted","Data":"842ce450706cf76db4c3dac79912d90d44512dcb12af960e2de8b89fcd2ffed2"}
Jul 20 18:36:49.110527 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:36:49.110504    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-09a1ba-m2dl7"
Jul 20 18:36:49.136474 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:36:49.136446    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c1c87bc-291b-45da-b96b-daa055a0e7ac-maas-api-service-ca\") pod \"4c1c87bc-291b-45da-b96b-daa055a0e7ac\" (UID: \"4c1c87bc-291b-45da-b96b-daa055a0e7ac\") "
Jul 20 18:36:49.136821 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:36:49.136799    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c1c87bc-291b-45da-b96b-daa055a0e7ac-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4c1c87bc-291b-45da-b96b-daa055a0e7ac" (UID: "4c1c87bc-291b-45da-b96b-daa055a0e7ac"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 18:36:49.237793 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:36:49.237700    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c1c87bc-291b-45da-b96b-daa055a0e7ac-maas-api-service-ca\") on node \"ip-10-0-137-247.ec2.internal\" DevicePath \"\""
Jul 20 18:36:49.986796 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:36:49.986758    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-09a1ba-m2dl7" event={"ID":"4c1c87bc-291b-45da-b96b-daa055a0e7ac","Type":"ContainerDied","Data":"842ce450706cf76db4c3dac79912d90d44512dcb12af960e2de8b89fcd2ffed2"}
Jul 20 18:36:49.986796 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:36:49.986782    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-09a1ba-m2dl7"
Jul 20 18:36:49.987029 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:36:49.986810    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="842ce450706cf76db4c3dac79912d90d44512dcb12af960e2de8b89fcd2ffed2"
Jul 20 18:37:02.229904 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:37:02.229867    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-09a1ba-8c8997646-92clk"]
Jul 20 18:37:02.230391 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:37:02.230095    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-09a1ba-8c8997646-92clk" podUID="79e938cf-267c-43ce-bef1-7db1d9a0aa74" containerName="maas-api" containerID="cri-o://2487893ea85a0571b4848452d4d42435ecbf0ff7df67f99584a4a805f14f9bbf" gracePeriod=30
Jul 20 18:37:02.465149 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:37:02.465125    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-09a1ba-8c8997646-92clk"
Jul 20 18:37:02.529742 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:37:02.529683    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m9fdh\" (UniqueName: \"kubernetes.io/projected/79e938cf-267c-43ce-bef1-7db1d9a0aa74-kube-api-access-m9fdh\") pod \"79e938cf-267c-43ce-bef1-7db1d9a0aa74\" (UID: \"79e938cf-267c-43ce-bef1-7db1d9a0aa74\") "
Jul 20 18:37:02.529918 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:37:02.529765    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79e938cf-267c-43ce-bef1-7db1d9a0aa74-maas-api-tls\") pod \"79e938cf-267c-43ce-bef1-7db1d9a0aa74\" (UID: \"79e938cf-267c-43ce-bef1-7db1d9a0aa74\") "
Jul 20 18:37:02.529918 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:37:02.529822    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79e938cf-267c-43ce-bef1-7db1d9a0aa74-openshift-service-ca\") pod \"79e938cf-267c-43ce-bef1-7db1d9a0aa74\" (UID: \"79e938cf-267c-43ce-bef1-7db1d9a0aa74\") "
Jul 20 18:37:02.530167 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:37:02.530141    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79e938cf-267c-43ce-bef1-7db1d9a0aa74-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "79e938cf-267c-43ce-bef1-7db1d9a0aa74" (UID: "79e938cf-267c-43ce-bef1-7db1d9a0aa74"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 18:37:02.532019 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:37:02.531999    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79e938cf-267c-43ce-bef1-7db1d9a0aa74-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "79e938cf-267c-43ce-bef1-7db1d9a0aa74" (UID: "79e938cf-267c-43ce-bef1-7db1d9a0aa74"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 18:37:02.532084 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:37:02.532003    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79e938cf-267c-43ce-bef1-7db1d9a0aa74-kube-api-access-m9fdh" (OuterVolumeSpecName: "kube-api-access-m9fdh") pod "79e938cf-267c-43ce-bef1-7db1d9a0aa74" (UID: "79e938cf-267c-43ce-bef1-7db1d9a0aa74"). InnerVolumeSpecName "kube-api-access-m9fdh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 18:37:02.630328 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:37:02.630291    2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79e938cf-267c-43ce-bef1-7db1d9a0aa74-openshift-service-ca\") on node \"ip-10-0-137-247.ec2.internal\" DevicePath \"\""
Jul 20 18:37:02.630328 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:37:02.630322    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9fdh\" (UniqueName: \"kubernetes.io/projected/79e938cf-267c-43ce-bef1-7db1d9a0aa74-kube-api-access-m9fdh\") on node \"ip-10-0-137-247.ec2.internal\" DevicePath \"\""
Jul 20 18:37:02.630328 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:37:02.630332    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79e938cf-267c-43ce-bef1-7db1d9a0aa74-maas-api-tls\") on node \"ip-10-0-137-247.ec2.internal\" DevicePath \"\""
Jul 20 18:37:03.050145 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:37:03.050113    2569 generic.go:356] "Generic (PLEG): container finished" podID="79e938cf-267c-43ce-bef1-7db1d9a0aa74" containerID="2487893ea85a0571b4848452d4d42435ecbf0ff7df67f99584a4a805f14f9bbf" exitCode=0
Jul 20 18:37:03.050349 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:37:03.050166    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-09a1ba-8c8997646-92clk"
Jul 20 18:37:03.050349 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:37:03.050204    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-09a1ba-8c8997646-92clk" event={"ID":"79e938cf-267c-43ce-bef1-7db1d9a0aa74","Type":"ContainerDied","Data":"2487893ea85a0571b4848452d4d42435ecbf0ff7df67f99584a4a805f14f9bbf"}
Jul 20 18:37:03.050349 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:37:03.050257    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-09a1ba-8c8997646-92clk" event={"ID":"79e938cf-267c-43ce-bef1-7db1d9a0aa74","Type":"ContainerDied","Data":"67e00f4ddac0f86b1d79d855de72996546af9ad0afdd536c11eb9dd8caea0eed"}
Jul 20 18:37:03.050349 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:37:03.050281    2569 scope.go:117] "RemoveContainer" containerID="2487893ea85a0571b4848452d4d42435ecbf0ff7df67f99584a4a805f14f9bbf"
Jul 20 18:37:03.060971 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:37:03.060951    2569 scope.go:117] "RemoveContainer" containerID="2487893ea85a0571b4848452d4d42435ecbf0ff7df67f99584a4a805f14f9bbf"
Jul 20 18:37:03.061273 ip-10-0-137-247 kubenswrapper[2569]: E0720 18:37:03.061255    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2487893ea85a0571b4848452d4d42435ecbf0ff7df67f99584a4a805f14f9bbf\": container with ID starting with 2487893ea85a0571b4848452d4d42435ecbf0ff7df67f99584a4a805f14f9bbf not found: ID does not exist" containerID="2487893ea85a0571b4848452d4d42435ecbf0ff7df67f99584a4a805f14f9bbf"
Jul 20 18:37:03.061352 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:37:03.061284    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2487893ea85a0571b4848452d4d42435ecbf0ff7df67f99584a4a805f14f9bbf"} err="failed to get container status \"2487893ea85a0571b4848452d4d42435ecbf0ff7df67f99584a4a805f14f9bbf\": rpc error: code = NotFound desc = could not find container \"2487893ea85a0571b4848452d4d42435ecbf0ff7df67f99584a4a805f14f9bbf\": container with ID starting with 2487893ea85a0571b4848452d4d42435ecbf0ff7df67f99584a4a805f14f9bbf not found: ID does not exist"
Jul 20 18:37:03.074917 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:37:03.074893    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-09a1ba-8c8997646-92clk"]
Jul 20 18:37:03.079085 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:37:03.079062    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-09a1ba-8c8997646-92clk"]
Jul 20 18:37:04.181863 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:37:04.181829    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79e938cf-267c-43ce-bef1-7db1d9a0aa74" path="/var/lib/kubelet/pods/79e938cf-267c-43ce-bef1-7db1d9a0aa74/volumes"
Jul 20 18:37:06.821965 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:37:06.821924    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-09a1ba-m2dl7"]
Jul 20 18:37:06.825924 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:37:06.825895    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-09a1ba-m2dl7"]
Jul 20 18:37:07.148334 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:37:07.148248    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9thxb"]
Jul 20 18:37:08.181576 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:37:08.181540    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4c1c87bc-291b-45da-b96b-daa055a0e7ac" path="/var/lib/kubelet/pods/4c1c87bc-291b-45da-b96b-daa055a0e7ac/volumes"
Jul 20 18:37:17.441529 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:37:17.441494    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-0d90bc-68488d67f6-qqgsn"]
Jul 20 18:37:17.441938 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:37:17.441927    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c1c87bc-291b-45da-b96b-daa055a0e7ac" containerName="revoke-keys"
Jul 20 18:37:17.441985 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:37:17.441944    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c1c87bc-291b-45da-b96b-daa055a0e7ac" containerName="revoke-keys"
Jul 20 18:37:17.441985 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:37:17.441959    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79e938cf-267c-43ce-bef1-7db1d9a0aa74" containerName="maas-api"
Jul 20 18:37:17.441985 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:37:17.441968    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="79e938cf-267c-43ce-bef1-7db1d9a0aa74" containerName="maas-api"
Jul 20 18:37:17.442090 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:37:17.442051    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c1c87bc-291b-45da-b96b-daa055a0e7ac" containerName="revoke-keys"
Jul 20 18:37:17.442090 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:37:17.442063    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="79e938cf-267c-43ce-bef1-7db1d9a0aa74" containerName="maas-api"
Jul 20 18:37:17.446532 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:37:17.446513    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-0d90bc-68488d67f6-qqgsn"
Jul 20 18:37:17.449129 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:37:17.449105    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 20 18:37:17.449265 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:37:17.449133    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-h2tfv\""
Jul 20 18:37:17.449265 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:37:17.449105    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-0d90bc-serving-cert\""
Jul 20 18:37:17.449777 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:37:17.449762    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 20 18:37:17.453951 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:37:17.453929    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-0d90bc-68488d67f6-qqgsn"]
Jul 20 18:37:17.551509 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:37:17.551474    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79a8058a-bf96-44bb-a224-aa87aa6704f6-openshift-service-ca\") pod \"maas-api-e2e-shared-a-0d90bc-68488d67f6-qqgsn\" (UID: \"79a8058a-bf96-44bb-a224-aa87aa6704f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-0d90bc-68488d67f6-qqgsn"
Jul 20 18:37:17.551509 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:37:17.551522    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79a8058a-bf96-44bb-a224-aa87aa6704f6-maas-api-tls\") pod \"maas-api-e2e-shared-a-0d90bc-68488d67f6-qqgsn\" (UID: \"79a8058a-bf96-44bb-a224-aa87aa6704f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-0d90bc-68488d67f6-qqgsn"
Jul 20 18:37:17.551764 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:37:17.551600    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86hrh\" (UniqueName: \"kubernetes.io/projected/79a8058a-bf96-44bb-a224-aa87aa6704f6-kube-api-access-86hrh\") pod \"maas-api-e2e-shared-a-0d90bc-68488d67f6-qqgsn\" (UID: \"79a8058a-bf96-44bb-a224-aa87aa6704f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-0d90bc-68488d67f6-qqgsn"
Jul 20 18:37:17.652896 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:37:17.652863    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86hrh\" (UniqueName: \"kubernetes.io/projected/79a8058a-bf96-44bb-a224-aa87aa6704f6-kube-api-access-86hrh\") pod \"maas-api-e2e-shared-a-0d90bc-68488d67f6-qqgsn\" (UID: \"79a8058a-bf96-44bb-a224-aa87aa6704f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-0d90bc-68488d67f6-qqgsn"
Jul 20 18:37:17.653055 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:37:17.652916    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79a8058a-bf96-44bb-a224-aa87aa6704f6-openshift-service-ca\") pod \"maas-api-e2e-shared-a-0d90bc-68488d67f6-qqgsn\" (UID: \"79a8058a-bf96-44bb-a224-aa87aa6704f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-0d90bc-68488d67f6-qqgsn"
Jul 20 18:37:17.653055 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:37:17.652956    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79a8058a-bf96-44bb-a224-aa87aa6704f6-maas-api-tls\") pod \"maas-api-e2e-shared-a-0d90bc-68488d67f6-qqgsn\" (UID: \"79a8058a-bf96-44bb-a224-aa87aa6704f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-0d90bc-68488d67f6-qqgsn"
Jul 20 18:37:17.653133 ip-10-0-137-247 kubenswrapper[2569]: E0720 18:37:17.653073    2569 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-0d90bc-serving-cert: secret "maas-api-e2e-shared-a-0d90bc-serving-cert" not found
Jul 20 18:37:17.653167 ip-10-0-137-247 kubenswrapper[2569]: E0720 18:37:17.653141    2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79a8058a-bf96-44bb-a224-aa87aa6704f6-maas-api-tls podName:79a8058a-bf96-44bb-a224-aa87aa6704f6 nodeName:}" failed. No retries permitted until 2026-07-20 18:37:18.153120881 +0000 UTC m=+2484.399654270 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/79a8058a-bf96-44bb-a224-aa87aa6704f6-maas-api-tls") pod "maas-api-e2e-shared-a-0d90bc-68488d67f6-qqgsn" (UID: "79a8058a-bf96-44bb-a224-aa87aa6704f6") : secret "maas-api-e2e-shared-a-0d90bc-serving-cert" not found
Jul 20 18:37:17.653661 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:37:17.653642    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79a8058a-bf96-44bb-a224-aa87aa6704f6-openshift-service-ca\") pod \"maas-api-e2e-shared-a-0d90bc-68488d67f6-qqgsn\" (UID: \"79a8058a-bf96-44bb-a224-aa87aa6704f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-0d90bc-68488d67f6-qqgsn"
Jul 20 18:37:17.666844 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:37:17.666816    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-86hrh\" (UniqueName: \"kubernetes.io/projected/79a8058a-bf96-44bb-a224-aa87aa6704f6-kube-api-access-86hrh\") pod \"maas-api-e2e-shared-a-0d90bc-68488d67f6-qqgsn\" (UID: \"79a8058a-bf96-44bb-a224-aa87aa6704f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-0d90bc-68488d67f6-qqgsn"
Jul 20 18:37:18.157840 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:37:18.157807    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79a8058a-bf96-44bb-a224-aa87aa6704f6-maas-api-tls\") pod \"maas-api-e2e-shared-a-0d90bc-68488d67f6-qqgsn\" (UID: \"79a8058a-bf96-44bb-a224-aa87aa6704f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-0d90bc-68488d67f6-qqgsn"
Jul 20 18:37:18.160264 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:37:18.160240    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79a8058a-bf96-44bb-a224-aa87aa6704f6-maas-api-tls\") pod \"maas-api-e2e-shared-a-0d90bc-68488d67f6-qqgsn\" (UID: \"79a8058a-bf96-44bb-a224-aa87aa6704f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-0d90bc-68488d67f6-qqgsn"
Jul 20 18:37:18.358026 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:37:18.357985    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-0d90bc-68488d67f6-qqgsn"
Jul 20 18:37:18.490073 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:37:18.490003    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-0d90bc-68488d67f6-qqgsn"]
Jul 20 18:37:18.492379 ip-10-0-137-247 kubenswrapper[2569]: W0720 18:37:18.492352    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79a8058a_bf96_44bb_a224_aa87aa6704f6.slice/crio-97dd289789533ec2d20ad1e8adad0c506e11426db8b5495b153707a38e313ea0 WatchSource:0}: Error finding container 97dd289789533ec2d20ad1e8adad0c506e11426db8b5495b153707a38e313ea0: Status 404 returned error can't find the container with id 97dd289789533ec2d20ad1e8adad0c506e11426db8b5495b153707a38e313ea0
Jul 20 18:37:19.125787 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:37:19.125749    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-0d90bc-68488d67f6-qqgsn" event={"ID":"79a8058a-bf96-44bb-a224-aa87aa6704f6","Type":"ContainerStarted","Data":"97dd289789533ec2d20ad1e8adad0c506e11426db8b5495b153707a38e313ea0"}
Jul 20 18:37:20.131626 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:37:20.131591    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-0d90bc-68488d67f6-qqgsn" event={"ID":"79a8058a-bf96-44bb-a224-aa87aa6704f6","Type":"ContainerStarted","Data":"9942b5d433682ec540521a26d8c22a22b42ec4b0c418b340a49bd5235bee7ed6"}
Jul 20 18:37:20.132130 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:37:20.131674    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-0d90bc-68488d67f6-qqgsn"
Jul 20 18:37:20.150656 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:37:20.150609    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-0d90bc-68488d67f6-qqgsn" podStartSLOduration=2.526040615 podStartE2EDuration="3.15059532s" totalImagesPullingTime="624.554705ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 18:37:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 18:37:20.149187237 +0000 UTC m=+2486.395720682" watchObservedRunningTime="2026-07-20 18:37:20.15059532 +0000 UTC m=+2486.397128730"
Jul 20 18:37:22.149910 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:37:22.149872    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9thxb"]
Jul 20 18:37:25.512863 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:37:25.512813    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-94313f-757bb7777-x4ldz"]
Jul 20 18:37:25.516662 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:37:25.516640    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-94313f-757bb7777-x4ldz"
Jul 20 18:37:25.519378 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:37:25.519357    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-94313f-serving-cert\""
Jul 20 18:37:25.529398 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:37:25.529371    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-94313f-757bb7777-x4ldz"]
Jul 20 18:37:25.620767 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:37:25.620693    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98rrl\" (UniqueName: \"kubernetes.io/projected/54d351ad-78f0-4f91-a2a2-005d9f60030c-kube-api-access-98rrl\") pod \"maas-api-e2e-shared-b-94313f-757bb7777-x4ldz\" (UID: \"54d351ad-78f0-4f91-a2a2-005d9f60030c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-94313f-757bb7777-x4ldz"
Jul 20 18:37:25.620767 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:37:25.620771    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54d351ad-78f0-4f91-a2a2-005d9f60030c-openshift-service-ca\") pod \"maas-api-e2e-shared-b-94313f-757bb7777-x4ldz\" (UID: \"54d351ad-78f0-4f91-a2a2-005d9f60030c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-94313f-757bb7777-x4ldz"
Jul 20 18:37:25.620977 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:37:25.620821    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54d351ad-78f0-4f91-a2a2-005d9f60030c-maas-api-tls\") pod \"maas-api-e2e-shared-b-94313f-757bb7777-x4ldz\" (UID: \"54d351ad-78f0-4f91-a2a2-005d9f60030c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-94313f-757bb7777-x4ldz"
Jul 20 18:37:25.722294 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:37:25.722258    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54d351ad-78f0-4f91-a2a2-005d9f60030c-maas-api-tls\") pod \"maas-api-e2e-shared-b-94313f-757bb7777-x4ldz\" (UID: \"54d351ad-78f0-4f91-a2a2-005d9f60030c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-94313f-757bb7777-x4ldz"
Jul 20 18:37:25.722554 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:37:25.722322    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98rrl\" (UniqueName: \"kubernetes.io/projected/54d351ad-78f0-4f91-a2a2-005d9f60030c-kube-api-access-98rrl\") pod \"maas-api-e2e-shared-b-94313f-757bb7777-x4ldz\" (UID: \"54d351ad-78f0-4f91-a2a2-005d9f60030c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-94313f-757bb7777-x4ldz"
Jul 20 18:37:25.722554 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:37:25.722343    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54d351ad-78f0-4f91-a2a2-005d9f60030c-openshift-service-ca\") pod \"maas-api-e2e-shared-b-94313f-757bb7777-x4ldz\" (UID: \"54d351ad-78f0-4f91-a2a2-005d9f60030c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-94313f-757bb7777-x4ldz"
Jul 20 18:37:25.723035 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:37:25.723016    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54d351ad-78f0-4f91-a2a2-005d9f60030c-openshift-service-ca\") pod \"maas-api-e2e-shared-b-94313f-757bb7777-x4ldz\" (UID: \"54d351ad-78f0-4f91-a2a2-005d9f60030c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-94313f-757bb7777-x4ldz"
Jul 20 18:37:25.724816 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:37:25.724789    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54d351ad-78f0-4f91-a2a2-005d9f60030c-maas-api-tls\") pod \"maas-api-e2e-shared-b-94313f-757bb7777-x4ldz\" (UID: \"54d351ad-78f0-4f91-a2a2-005d9f60030c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-94313f-757bb7777-x4ldz"
Jul 20 18:37:25.739783 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:37:25.739762    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-98rrl\" (UniqueName: \"kubernetes.io/projected/54d351ad-78f0-4f91-a2a2-005d9f60030c-kube-api-access-98rrl\") pod \"maas-api-e2e-shared-b-94313f-757bb7777-x4ldz\" (UID: \"54d351ad-78f0-4f91-a2a2-005d9f60030c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-94313f-757bb7777-x4ldz"
Jul 20 18:37:25.826983 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:37:25.826885    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-94313f-757bb7777-x4ldz"
Jul 20 18:37:25.960156 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:37:25.960129    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-94313f-757bb7777-x4ldz"]
Jul 20 18:37:25.962602 ip-10-0-137-247 kubenswrapper[2569]: W0720 18:37:25.962573    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54d351ad_78f0_4f91_a2a2_005d9f60030c.slice/crio-9e307afd5e4e93b2bf3b224d590679156d586fd78d70e8489c459a6520b4dd1b WatchSource:0}: Error finding container 9e307afd5e4e93b2bf3b224d590679156d586fd78d70e8489c459a6520b4dd1b: Status 404 returned error can't find the container with id 9e307afd5e4e93b2bf3b224d590679156d586fd78d70e8489c459a6520b4dd1b
Jul 20 18:37:26.142503 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:37:26.142416    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-0d90bc-68488d67f6-qqgsn"
Jul 20 18:37:26.163747 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:37:26.163686    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-94313f-757bb7777-x4ldz" event={"ID":"54d351ad-78f0-4f91-a2a2-005d9f60030c","Type":"ContainerStarted","Data":"9e307afd5e4e93b2bf3b224d590679156d586fd78d70e8489c459a6520b4dd1b"}
Jul 20 18:37:27.169473 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:37:27.169435    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-94313f-757bb7777-x4ldz" event={"ID":"54d351ad-78f0-4f91-a2a2-005d9f60030c","Type":"ContainerStarted","Data":"397fc96245fbbba2c7a03e659b62d15371d3345c3f81708d63ecf662d52de1e5"}
Jul 20 18:37:27.169907 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:37:27.169572    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-94313f-757bb7777-x4ldz"
Jul 20 18:37:27.188948 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:37:27.188898    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-94313f-757bb7777-x4ldz" podStartSLOduration=1.839886211 podStartE2EDuration="2.188884219s" totalImagesPullingTime="348.998008ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 18:37:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 18:37:27.187653527 +0000 UTC m=+2493.434186952" watchObservedRunningTime="2026-07-20 18:37:27.188884219 +0000 UTC m=+2493.435417630"
Jul 20 18:37:33.183835 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:37:33.183801    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-94313f-757bb7777-x4ldz"
Jul 20 18:37:38.545263 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:37:38.545222    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9thxb"]
Jul 20 18:37:50.306866 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:37:50.306834    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-0d90bc/auth-test-model-0d90bc-kserve-5b9f7699f-6ztnb"]
Jul 20 18:37:50.310154 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:37:50.310136    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-0d90bc/auth-test-model-0d90bc-kserve-5b9f7699f-6ztnb"
Jul 20 18:37:50.313236 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:37:50.313211    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-0d90bc\"/\"kube-root-ca.crt\""
Jul 20 18:37:50.313236 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:37:50.313229    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-0d90bc\"/\"openshift-service-ca.crt\""
Jul 20 18:37:50.313451 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:37:50.313219    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-0d90bc\"/\"default-dockercfg-cfnxt\""
Jul 20 18:37:50.313451 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:37:50.313338    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-0d90bc\"/\"auth-test-model-0d90bc-kserve-self-signed-certs\""
Jul 20 18:37:50.319205 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:37:50.319183    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-0d90bc/auth-test-model-0d90bc-kserve-5b9f7699f-6ztnb"]
Jul 20 18:37:50.329901 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:37:50.329877    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9e53ef5-1f07-4df8-91dd-88ae1ba7d7d6-model-cache\") pod \"auth-test-model-0d90bc-kserve-5b9f7699f-6ztnb\" (UID: \"e9e53ef5-1f07-4df8-91dd-88ae1ba7d7d6\") " pod="ai-tenant-e2e-shared-a-0d90bc/auth-test-model-0d90bc-kserve-5b9f7699f-6ztnb"
Jul 20 18:37:50.329996 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:37:50.329916    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9e53ef5-1f07-4df8-91dd-88ae1ba7d7d6-tmp-dir\") pod \"auth-test-model-0d90bc-kserve-5b9f7699f-6ztnb\" (UID: \"e9e53ef5-1f07-4df8-91dd-88ae1ba7d7d6\") " pod="ai-tenant-e2e-shared-a-0d90bc/auth-test-model-0d90bc-kserve-5b9f7699f-6ztnb"
Jul 20 18:37:50.329996 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:37:50.329941    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9e53ef5-1f07-4df8-91dd-88ae1ba7d7d6-tls-certs\") pod \"auth-test-model-0d90bc-kserve-5b9f7699f-6ztnb\" (UID: \"e9e53ef5-1f07-4df8-91dd-88ae1ba7d7d6\") " pod="ai-tenant-e2e-shared-a-0d90bc/auth-test-model-0d90bc-kserve-5b9f7699f-6ztnb"
Jul 20 18:37:50.329996 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:37:50.329988    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lr2ls\" (UniqueName: \"kubernetes.io/projected/e9e53ef5-1f07-4df8-91dd-88ae1ba7d7d6-kube-api-access-lr2ls\") pod \"auth-test-model-0d90bc-kserve-5b9f7699f-6ztnb\" (UID: \"e9e53ef5-1f07-4df8-91dd-88ae1ba7d7d6\") " pod="ai-tenant-e2e-shared-a-0d90bc/auth-test-model-0d90bc-kserve-5b9f7699f-6ztnb"
Jul 20 18:37:50.330100 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:37:50.330027    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9e53ef5-1f07-4df8-91dd-88ae1ba7d7d6-home\") pod \"auth-test-model-0d90bc-kserve-5b9f7699f-6ztnb\" (UID: \"e9e53ef5-1f07-4df8-91dd-88ae1ba7d7d6\") " pod="ai-tenant-e2e-shared-a-0d90bc/auth-test-model-0d90bc-kserve-5b9f7699f-6ztnb"
Jul 20 18:37:50.330100 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:37:50.330068    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9e53ef5-1f07-4df8-91dd-88ae1ba7d7d6-dshm\") pod \"auth-test-model-0d90bc-kserve-5b9f7699f-6ztnb\" (UID: \"e9e53ef5-1f07-4df8-91dd-88ae1ba7d7d6\") " pod="ai-tenant-e2e-shared-a-0d90bc/auth-test-model-0d90bc-kserve-5b9f7699f-6ztnb"
Jul 20 18:37:50.430540 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:37:50.430496    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9e53ef5-1f07-4df8-91dd-88ae1ba7d7d6-tmp-dir\") pod \"auth-test-model-0d90bc-kserve-5b9f7699f-6ztnb\" (UID: \"e9e53ef5-1f07-4df8-91dd-88ae1ba7d7d6\") " pod="ai-tenant-e2e-shared-a-0d90bc/auth-test-model-0d90bc-kserve-5b9f7699f-6ztnb"
Jul 20 18:37:50.430540 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:37:50.430538    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9e53ef5-1f07-4df8-91dd-88ae1ba7d7d6-tls-certs\") pod \"auth-test-model-0d90bc-kserve-5b9f7699f-6ztnb\" (UID: \"e9e53ef5-1f07-4df8-91dd-88ae1ba7d7d6\") " pod="ai-tenant-e2e-shared-a-0d90bc/auth-test-model-0d90bc-kserve-5b9f7699f-6ztnb"
Jul 20 18:37:50.430836 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:37:50.430565    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lr2ls\" (UniqueName: \"kubernetes.io/projected/e9e53ef5-1f07-4df8-91dd-88ae1ba7d7d6-kube-api-access-lr2ls\") pod \"auth-test-model-0d90bc-kserve-5b9f7699f-6ztnb\" (UID: \"e9e53ef5-1f07-4df8-91dd-88ae1ba7d7d6\") " pod="ai-tenant-e2e-shared-a-0d90bc/auth-test-model-0d90bc-kserve-5b9f7699f-6ztnb"
Jul 20 18:37:50.430836 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:37:50.430608    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9e53ef5-1f07-4df8-91dd-88ae1ba7d7d6-home\") pod \"auth-test-model-0d90bc-kserve-5b9f7699f-6ztnb\" (UID: \"e9e53ef5-1f07-4df8-91dd-88ae1ba7d7d6\") " pod="ai-tenant-e2e-shared-a-0d90bc/auth-test-model-0d90bc-kserve-5b9f7699f-6ztnb"
Jul 20 18:37:50.430836 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:37:50.430649    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9e53ef5-1f07-4df8-91dd-88ae1ba7d7d6-dshm\") pod \"auth-test-model-0d90bc-kserve-5b9f7699f-6ztnb\" (UID: \"e9e53ef5-1f07-4df8-91dd-88ae1ba7d7d6\") " pod="ai-tenant-e2e-shared-a-0d90bc/auth-test-model-0d90bc-kserve-5b9f7699f-6ztnb"
Jul 20 18:37:50.430836 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:37:50.430680    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9e53ef5-1f07-4df8-91dd-88ae1ba7d7d6-model-cache\") pod \"auth-test-model-0d90bc-kserve-5b9f7699f-6ztnb\" (UID: \"e9e53ef5-1f07-4df8-91dd-88ae1ba7d7d6\") " pod="ai-tenant-e2e-shared-a-0d90bc/auth-test-model-0d90bc-kserve-5b9f7699f-6ztnb"
Jul 20 18:37:50.431030 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:37:50.430959    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9e53ef5-1f07-4df8-91dd-88ae1ba7d7d6-tmp-dir\") pod \"auth-test-model-0d90bc-kserve-5b9f7699f-6ztnb\" (UID: \"e9e53ef5-1f07-4df8-91dd-88ae1ba7d7d6\") " pod="ai-tenant-e2e-shared-a-0d90bc/auth-test-model-0d90bc-kserve-5b9f7699f-6ztnb"
Jul 20 18:37:50.431030 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:37:50.431009    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9e53ef5-1f07-4df8-91dd-88ae1ba7d7d6-home\") pod \"auth-test-model-0d90bc-kserve-5b9f7699f-6ztnb\" (UID: \"e9e53ef5-1f07-4df8-91dd-88ae1ba7d7d6\") " pod="ai-tenant-e2e-shared-a-0d90bc/auth-test-model-0d90bc-kserve-5b9f7699f-6ztnb"
Jul 20 18:37:50.431143 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:37:50.431036    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9e53ef5-1f07-4df8-91dd-88ae1ba7d7d6-model-cache\") pod \"auth-test-model-0d90bc-kserve-5b9f7699f-6ztnb\" (UID: \"e9e53ef5-1f07-4df8-91dd-88ae1ba7d7d6\") " pod="ai-tenant-e2e-shared-a-0d90bc/auth-test-model-0d90bc-kserve-5b9f7699f-6ztnb"
Jul 20 18:37:50.432978 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:37:50.432955    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9e53ef5-1f07-4df8-91dd-88ae1ba7d7d6-dshm\") pod \"auth-test-model-0d90bc-kserve-5b9f7699f-6ztnb\" (UID: \"e9e53ef5-1f07-4df8-91dd-88ae1ba7d7d6\") " pod="ai-tenant-e2e-shared-a-0d90bc/auth-test-model-0d90bc-kserve-5b9f7699f-6ztnb"
Jul 20 18:37:50.433168 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:37:50.433151    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9e53ef5-1f07-4df8-91dd-88ae1ba7d7d6-tls-certs\") pod \"auth-test-model-0d90bc-kserve-5b9f7699f-6ztnb\" (UID: \"e9e53ef5-1f07-4df8-91dd-88ae1ba7d7d6\") " pod="ai-tenant-e2e-shared-a-0d90bc/auth-test-model-0d90bc-kserve-5b9f7699f-6ztnb"
Jul 20 18:37:50.439343 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:37:50.439324    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lr2ls\" (UniqueName: \"kubernetes.io/projected/e9e53ef5-1f07-4df8-91dd-88ae1ba7d7d6-kube-api-access-lr2ls\") pod \"auth-test-model-0d90bc-kserve-5b9f7699f-6ztnb\" (UID: \"e9e53ef5-1f07-4df8-91dd-88ae1ba7d7d6\") " pod="ai-tenant-e2e-shared-a-0d90bc/auth-test-model-0d90bc-kserve-5b9f7699f-6ztnb"
Jul 20 18:37:50.620060 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:37:50.619964    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-0d90bc/auth-test-model-0d90bc-kserve-5b9f7699f-6ztnb"
Jul 20 18:37:50.750745 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:37:50.750697    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-0d90bc/auth-test-model-0d90bc-kserve-5b9f7699f-6ztnb"]
Jul 20 18:37:50.752929 ip-10-0-137-247 kubenswrapper[2569]: W0720 18:37:50.752894    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode9e53ef5_1f07_4df8_91dd_88ae1ba7d7d6.slice/crio-2da8b9ee38acea69f3cd5bf34d5ed77c1d6abe3688c00e1d1e9f798b5265cdf1 WatchSource:0}: Error finding container 2da8b9ee38acea69f3cd5bf34d5ed77c1d6abe3688c00e1d1e9f798b5265cdf1: Status 404 returned error can't find the container with id 2da8b9ee38acea69f3cd5bf34d5ed77c1d6abe3688c00e1d1e9f798b5265cdf1
Jul 20 18:37:51.275998 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:37:51.275965    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-0d90bc/auth-test-model-0d90bc-kserve-5b9f7699f-6ztnb" event={"ID":"e9e53ef5-1f07-4df8-91dd-88ae1ba7d7d6","Type":"ContainerStarted","Data":"fabfb517bb383705fc363e9c1c46152b8975fbaae9d93fc7944253c858afdf9b"}
Jul 20 18:37:51.275998 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:37:51.276006    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-0d90bc/auth-test-model-0d90bc-kserve-5b9f7699f-6ztnb" event={"ID":"e9e53ef5-1f07-4df8-91dd-88ae1ba7d7d6","Type":"ContainerStarted","Data":"2da8b9ee38acea69f3cd5bf34d5ed77c1d6abe3688c00e1d1e9f798b5265cdf1"}
Jul 20 18:37:51.293380 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:37:51.293332    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-0d90bc/auth-test-model-0d90bc-kserve-5b9f7699f-6ztnb" podStartSLOduration=1.293315467 podStartE2EDuration="1.293315467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 18:37:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 18:37:51.292223549 +0000 UTC m=+2517.538756974" watchObservedRunningTime="2026-07-20 18:37:51.293315467 +0000 UTC m=+2517.539848875"
Jul 20 18:37:53.541534 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:37:53.541492    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9thxb"]
Jul 20 18:37:54.388291 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:37:54.388251    2569 scope.go:117] "RemoveContainer" containerID="7b818103cb6754887bf94fdb95b6a62aa3b58b38fc00c90afae4960efb73ff6e"
Jul 20 18:38:00.620587 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:38:00.620541    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-0d90bc/auth-test-model-0d90bc-kserve-5b9f7699f-6ztnb"
Jul 20 18:38:00.621073 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:38:00.620623    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-0d90bc/auth-test-model-0d90bc-kserve-5b9f7699f-6ztnb"
Jul 20 18:38:00.633516 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:38:00.633491    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-0d90bc/auth-test-model-0d90bc-kserve-5b9f7699f-6ztnb"
Jul 20 18:38:01.336605 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:38:01.336572    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-0d90bc/auth-test-model-0d90bc-kserve-5b9f7699f-6ztnb"
Jul 20 18:38:21.859320 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:38:21.859286    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-94313f/auth-test-model-94313f-kserve-7b99884c67-85sjk"]
Jul 20 18:38:21.862537 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:38:21.862517    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-94313f/auth-test-model-94313f-kserve-7b99884c67-85sjk"
Jul 20 18:38:21.868609 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:38:21.868577    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-94313f\"/\"kube-root-ca.crt\""
Jul 20 18:38:21.868609 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:38:21.868599    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-94313f\"/\"default-dockercfg-xtr2f\""
Jul 20 18:38:21.868839 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:38:21.868615    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-94313f\"/\"auth-test-model-94313f-kserve-self-signed-certs\""
Jul 20 18:38:21.869297 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:38:21.869281    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-94313f\"/\"openshift-service-ca.crt\""
Jul 20 18:38:21.887400 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:38:21.887369    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-94313f/auth-test-model-94313f-kserve-7b99884c67-85sjk"]
Jul 20 18:38:21.962987 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:38:21.962953    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cwnr\" (UniqueName: \"kubernetes.io/projected/dde25e2d-511c-4103-ac11-290825e68fbe-kube-api-access-4cwnr\") pod \"auth-test-model-94313f-kserve-7b99884c67-85sjk\" (UID: \"dde25e2d-511c-4103-ac11-290825e68fbe\") " pod="ai-tenant-e2e-shared-b-94313f/auth-test-model-94313f-kserve-7b99884c67-85sjk"
Jul 20 18:38:21.963200 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:38:21.962992    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dde25e2d-511c-4103-ac11-290825e68fbe-tmp-dir\") pod \"auth-test-model-94313f-kserve-7b99884c67-85sjk\" (UID: \"dde25e2d-511c-4103-ac11-290825e68fbe\") " pod="ai-tenant-e2e-shared-b-94313f/auth-test-model-94313f-kserve-7b99884c67-85sjk"
Jul 20 18:38:21.963200 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:38:21.963012    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dde25e2d-511c-4103-ac11-290825e68fbe-dshm\") pod \"auth-test-model-94313f-kserve-7b99884c67-85sjk\" (UID: \"dde25e2d-511c-4103-ac11-290825e68fbe\") " pod="ai-tenant-e2e-shared-b-94313f/auth-test-model-94313f-kserve-7b99884c67-85sjk"
Jul 20 18:38:21.963200 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:38:21.963078    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dde25e2d-511c-4103-ac11-290825e68fbe-tls-certs\") pod \"auth-test-model-94313f-kserve-7b99884c67-85sjk\" (UID: \"dde25e2d-511c-4103-ac11-290825e68fbe\") " pod="ai-tenant-e2e-shared-b-94313f/auth-test-model-94313f-kserve-7b99884c67-85sjk"
Jul 20 18:38:21.963374 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:38:21.963225    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dde25e2d-511c-4103-ac11-290825e68fbe-model-cache\") pod \"auth-test-model-94313f-kserve-7b99884c67-85sjk\" (UID: \"dde25e2d-511c-4103-ac11-290825e68fbe\") " pod="ai-tenant-e2e-shared-b-94313f/auth-test-model-94313f-kserve-7b99884c67-85sjk"
Jul 20 18:38:21.963374 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:38:21.963274    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dde25e2d-511c-4103-ac11-290825e68fbe-home\") pod \"auth-test-model-94313f-kserve-7b99884c67-85sjk\" (UID: \"dde25e2d-511c-4103-ac11-290825e68fbe\") " pod="ai-tenant-e2e-shared-b-94313f/auth-test-model-94313f-kserve-7b99884c67-85sjk"
Jul 20 18:38:22.064200 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:38:22.064165    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dde25e2d-511c-4103-ac11-290825e68fbe-model-cache\") pod \"auth-test-model-94313f-kserve-7b99884c67-85sjk\" (UID: \"dde25e2d-511c-4103-ac11-290825e68fbe\") " pod="ai-tenant-e2e-shared-b-94313f/auth-test-model-94313f-kserve-7b99884c67-85sjk"
Jul 20 18:38:22.064397 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:38:22.064209    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dde25e2d-511c-4103-ac11-290825e68fbe-home\") pod \"auth-test-model-94313f-kserve-7b99884c67-85sjk\" (UID: \"dde25e2d-511c-4103-ac11-290825e68fbe\") " pod="ai-tenant-e2e-shared-b-94313f/auth-test-model-94313f-kserve-7b99884c67-85sjk"
Jul 20 18:38:22.064397 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:38:22.064242    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cwnr\" (UniqueName: \"kubernetes.io/projected/dde25e2d-511c-4103-ac11-290825e68fbe-kube-api-access-4cwnr\") pod \"auth-test-model-94313f-kserve-7b99884c67-85sjk\" (UID: \"dde25e2d-511c-4103-ac11-290825e68fbe\") " pod="ai-tenant-e2e-shared-b-94313f/auth-test-model-94313f-kserve-7b99884c67-85sjk"
Jul 20 18:38:22.064397 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:38:22.064268    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dde25e2d-511c-4103-ac11-290825e68fbe-tmp-dir\") pod \"auth-test-model-94313f-kserve-7b99884c67-85sjk\" (UID: \"dde25e2d-511c-4103-ac11-290825e68fbe\") " pod="ai-tenant-e2e-shared-b-94313f/auth-test-model-94313f-kserve-7b99884c67-85sjk"
Jul 20 18:38:22.064397 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:38:22.064288    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dde25e2d-511c-4103-ac11-290825e68fbe-dshm\") pod \"auth-test-model-94313f-kserve-7b99884c67-85sjk\" (UID: \"dde25e2d-511c-4103-ac11-290825e68fbe\") " pod="ai-tenant-e2e-shared-b-94313f/auth-test-model-94313f-kserve-7b99884c67-85sjk"
Jul 20 18:38:22.064397 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:38:22.064308    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dde25e2d-511c-4103-ac11-290825e68fbe-tls-certs\") pod \"auth-test-model-94313f-kserve-7b99884c67-85sjk\" (UID: \"dde25e2d-511c-4103-ac11-290825e68fbe\") " pod="ai-tenant-e2e-shared-b-94313f/auth-test-model-94313f-kserve-7b99884c67-85sjk"
Jul 20 18:38:22.064668 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:38:22.064602    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dde25e2d-511c-4103-ac11-290825e68fbe-model-cache\") pod \"auth-test-model-94313f-kserve-7b99884c67-85sjk\" (UID: \"dde25e2d-511c-4103-ac11-290825e68fbe\") " pod="ai-tenant-e2e-shared-b-94313f/auth-test-model-94313f-kserve-7b99884c67-85sjk"
Jul 20 18:38:22.064760 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:38:22.064659    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dde25e2d-511c-4103-ac11-290825e68fbe-home\") pod \"auth-test-model-94313f-kserve-7b99884c67-85sjk\" (UID: \"dde25e2d-511c-4103-ac11-290825e68fbe\") " pod="ai-tenant-e2e-shared-b-94313f/auth-test-model-94313f-kserve-7b99884c67-85sjk"
Jul 20 18:38:22.064760 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:38:22.064695    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dde25e2d-511c-4103-ac11-290825e68fbe-tmp-dir\") pod \"auth-test-model-94313f-kserve-7b99884c67-85sjk\" (UID: \"dde25e2d-511c-4103-ac11-290825e68fbe\") " pod="ai-tenant-e2e-shared-b-94313f/auth-test-model-94313f-kserve-7b99884c67-85sjk"
Jul 20 18:38:22.066657 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:38:22.066636    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dde25e2d-511c-4103-ac11-290825e68fbe-dshm\") pod \"auth-test-model-94313f-kserve-7b99884c67-85sjk\" (UID: \"dde25e2d-511c-4103-ac11-290825e68fbe\") " pod="ai-tenant-e2e-shared-b-94313f/auth-test-model-94313f-kserve-7b99884c67-85sjk"
Jul 20 18:38:22.066818 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:38:22.066802    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dde25e2d-511c-4103-ac11-290825e68fbe-tls-certs\") pod \"auth-test-model-94313f-kserve-7b99884c67-85sjk\" (UID: \"dde25e2d-511c-4103-ac11-290825e68fbe\") " pod="ai-tenant-e2e-shared-b-94313f/auth-test-model-94313f-kserve-7b99884c67-85sjk"
Jul 20 18:38:22.072812 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:38:22.072791    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cwnr\" (UniqueName: \"kubernetes.io/projected/dde25e2d-511c-4103-ac11-290825e68fbe-kube-api-access-4cwnr\") pod \"auth-test-model-94313f-kserve-7b99884c67-85sjk\" (UID: \"dde25e2d-511c-4103-ac11-290825e68fbe\") " pod="ai-tenant-e2e-shared-b-94313f/auth-test-model-94313f-kserve-7b99884c67-85sjk"
Jul 20 18:38:22.171575 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:38:22.171497    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-94313f/auth-test-model-94313f-kserve-7b99884c67-85sjk"
Jul 20 18:38:22.310453 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:38:22.310429    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-94313f/auth-test-model-94313f-kserve-7b99884c67-85sjk"]
Jul 20 18:38:22.312224 ip-10-0-137-247 kubenswrapper[2569]: W0720 18:38:22.312195    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddde25e2d_511c_4103_ac11_290825e68fbe.slice/crio-d320ba89914e410780d60855da3267792f7ba911d17d8c86f3eeb9aab055d095 WatchSource:0}: Error finding container d320ba89914e410780d60855da3267792f7ba911d17d8c86f3eeb9aab055d095: Status 404 returned error can't find the container with id d320ba89914e410780d60855da3267792f7ba911d17d8c86f3eeb9aab055d095
Jul 20 18:38:22.416194 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:38:22.416153    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-94313f/auth-test-model-94313f-kserve-7b99884c67-85sjk" event={"ID":"dde25e2d-511c-4103-ac11-290825e68fbe","Type":"ContainerStarted","Data":"b34714c4bc8b0f3de17c3fbd6bfb4c90a2886a615ad2d38e74defe476f316275"}
Jul 20 18:38:22.416194 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:38:22.416200    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-94313f/auth-test-model-94313f-kserve-7b99884c67-85sjk" event={"ID":"dde25e2d-511c-4103-ac11-290825e68fbe","Type":"ContainerStarted","Data":"d320ba89914e410780d60855da3267792f7ba911d17d8c86f3eeb9aab055d095"}
Jul 20 18:38:22.453410 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:38:22.453312    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-94313f/auth-test-model-94313f-kserve-7b99884c67-85sjk" podStartSLOduration=1.453297461 podStartE2EDuration="1.453297461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 18:38:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 18:38:22.450947862 +0000 UTC m=+2548.697481271" watchObservedRunningTime="2026-07-20 18:38:22.453297461 +0000 UTC m=+2548.699830872"
Jul 20 18:38:25.581468 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:38:25.581432    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9thxb"]
Jul 20 18:38:32.172309 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:38:32.172267    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-94313f/auth-test-model-94313f-kserve-7b99884c67-85sjk"
Jul 20 18:38:32.172851 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:38:32.172335    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-94313f/auth-test-model-94313f-kserve-7b99884c67-85sjk"
Jul 20 18:38:32.185340 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:38:32.185314    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-94313f/auth-test-model-94313f-kserve-7b99884c67-85sjk"
Jul 20 18:38:32.471162 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:38:32.471075    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-94313f/auth-test-model-94313f-kserve-7b99884c67-85sjk"
Jul 20 18:38:41.942316 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:38:41.942281    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9thxb"]
Jul 20 18:38:59.842690 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:38:59.842647    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9thxb"]
Jul 20 18:39:16.857232 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:39:16.857154    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9thxb"]
Jul 20 18:39:34.362357 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:39:34.362319    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9thxb"]
Jul 20 18:39:51.360797 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:39:51.360762    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9thxb"]
Jul 20 18:40:08.158230 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:40:08.158190    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9thxb"]
Jul 20 18:40:28.138848 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:40:28.138801    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-0d90bc/auth-test-model-0d90bc-kserve-5b9f7699f-6ztnb"]
Jul 20 18:40:28.994225 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:40:28.994187    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-94313f/auth-test-model-94313f-kserve-7b99884c67-85sjk"]
Jul 20 18:40:34.064082 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:40:34.064048    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9thxb"]
Jul 20 18:40:43.158350 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:40:43.158308    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-0d90bc/auth-test-model-0d90bc-kserve-5b9f7699f-6ztnb" podUID="e9e53ef5-1f07-4df8-91dd-88ae1ba7d7d6" containerName="main" containerID="cri-o://fabfb517bb383705fc363e9c1c46152b8975fbaae9d93fc7944253c858afdf9b" gracePeriod=45
Jul 20 18:40:43.412387 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:40:43.412323    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-0d90bc/auth-test-model-0d90bc-kserve-5b9f7699f-6ztnb"
Jul 20 18:40:43.533372 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:40:43.533341    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lr2ls\" (UniqueName: \"kubernetes.io/projected/e9e53ef5-1f07-4df8-91dd-88ae1ba7d7d6-kube-api-access-lr2ls\") pod \"e9e53ef5-1f07-4df8-91dd-88ae1ba7d7d6\" (UID: \"e9e53ef5-1f07-4df8-91dd-88ae1ba7d7d6\") "
Jul 20 18:40:43.533558 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:40:43.533398    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9e53ef5-1f07-4df8-91dd-88ae1ba7d7d6-dshm\") pod \"e9e53ef5-1f07-4df8-91dd-88ae1ba7d7d6\" (UID: \"e9e53ef5-1f07-4df8-91dd-88ae1ba7d7d6\") "
Jul 20 18:40:43.533558 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:40:43.533435    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9e53ef5-1f07-4df8-91dd-88ae1ba7d7d6-home\") pod \"e9e53ef5-1f07-4df8-91dd-88ae1ba7d7d6\" (UID: \"e9e53ef5-1f07-4df8-91dd-88ae1ba7d7d6\") "
Jul 20 18:40:43.533558 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:40:43.533455    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9e53ef5-1f07-4df8-91dd-88ae1ba7d7d6-tmp-dir\") pod \"e9e53ef5-1f07-4df8-91dd-88ae1ba7d7d6\" (UID: \"e9e53ef5-1f07-4df8-91dd-88ae1ba7d7d6\") "
Jul 20 18:40:43.533558 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:40:43.533481    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9e53ef5-1f07-4df8-91dd-88ae1ba7d7d6-tls-certs\") pod \"e9e53ef5-1f07-4df8-91dd-88ae1ba7d7d6\" (UID: \"e9e53ef5-1f07-4df8-91dd-88ae1ba7d7d6\") "
Jul 20 18:40:43.533558 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:40:43.533497    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9e53ef5-1f07-4df8-91dd-88ae1ba7d7d6-model-cache\") pod \"e9e53ef5-1f07-4df8-91dd-88ae1ba7d7d6\" (UID: \"e9e53ef5-1f07-4df8-91dd-88ae1ba7d7d6\") "
Jul 20 18:40:43.533870 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:40:43.533684    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9e53ef5-1f07-4df8-91dd-88ae1ba7d7d6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e9e53ef5-1f07-4df8-91dd-88ae1ba7d7d6" (UID: "e9e53ef5-1f07-4df8-91dd-88ae1ba7d7d6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 18:40:43.533870 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:40:43.533700    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9e53ef5-1f07-4df8-91dd-88ae1ba7d7d6-home" (OuterVolumeSpecName: "home") pod "e9e53ef5-1f07-4df8-91dd-88ae1ba7d7d6" (UID: "e9e53ef5-1f07-4df8-91dd-88ae1ba7d7d6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 18:40:43.533980 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:40:43.533879    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9e53ef5-1f07-4df8-91dd-88ae1ba7d7d6-model-cache" (OuterVolumeSpecName: "model-cache") pod "e9e53ef5-1f07-4df8-91dd-88ae1ba7d7d6" (UID: "e9e53ef5-1f07-4df8-91dd-88ae1ba7d7d6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 18:40:43.535694 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:40:43.535671    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9e53ef5-1f07-4df8-91dd-88ae1ba7d7d6-kube-api-access-lr2ls" (OuterVolumeSpecName: "kube-api-access-lr2ls") pod "e9e53ef5-1f07-4df8-91dd-88ae1ba7d7d6" (UID: "e9e53ef5-1f07-4df8-91dd-88ae1ba7d7d6"). InnerVolumeSpecName "kube-api-access-lr2ls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 18:40:43.535694 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:40:43.535680    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9e53ef5-1f07-4df8-91dd-88ae1ba7d7d6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e9e53ef5-1f07-4df8-91dd-88ae1ba7d7d6" (UID: "e9e53ef5-1f07-4df8-91dd-88ae1ba7d7d6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 18:40:43.535828 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:40:43.535753    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9e53ef5-1f07-4df8-91dd-88ae1ba7d7d6-dshm" (OuterVolumeSpecName: "dshm") pod "e9e53ef5-1f07-4df8-91dd-88ae1ba7d7d6" (UID: "e9e53ef5-1f07-4df8-91dd-88ae1ba7d7d6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 18:40:43.634739 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:40:43.634674    2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9e53ef5-1f07-4df8-91dd-88ae1ba7d7d6-dshm\") on node \"ip-10-0-137-247.ec2.internal\" DevicePath \"\""
Jul 20 18:40:43.634739 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:40:43.634712    2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9e53ef5-1f07-4df8-91dd-88ae1ba7d7d6-home\") on node \"ip-10-0-137-247.ec2.internal\" DevicePath \"\""
Jul 20 18:40:43.634739 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:40:43.634739    2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9e53ef5-1f07-4df8-91dd-88ae1ba7d7d6-tmp-dir\") on node \"ip-10-0-137-247.ec2.internal\" DevicePath \"\""
Jul 20 18:40:43.634739 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:40:43.634749    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9e53ef5-1f07-4df8-91dd-88ae1ba7d7d6-tls-certs\") on node \"ip-10-0-137-247.ec2.internal\" DevicePath \"\""
Jul 20 18:40:43.634739 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:40:43.634758    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9e53ef5-1f07-4df8-91dd-88ae1ba7d7d6-model-cache\") on node \"ip-10-0-137-247.ec2.internal\" DevicePath \"\""
Jul 20 18:40:43.635072 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:40:43.634766    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lr2ls\" (UniqueName: \"kubernetes.io/projected/e9e53ef5-1f07-4df8-91dd-88ae1ba7d7d6-kube-api-access-lr2ls\") on node \"ip-10-0-137-247.ec2.internal\" DevicePath \"\""
Jul 20 18:40:44.013675 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:40:44.013616    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-94313f/auth-test-model-94313f-kserve-7b99884c67-85sjk" podUID="dde25e2d-511c-4103-ac11-290825e68fbe" containerName="main" containerID="cri-o://b34714c4bc8b0f3de17c3fbd6bfb4c90a2886a615ad2d38e74defe476f316275" gracePeriod=45
Jul 20 18:40:44.043350 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:40:44.043314    2569 generic.go:356] "Generic (PLEG): container finished" podID="e9e53ef5-1f07-4df8-91dd-88ae1ba7d7d6" containerID="fabfb517bb383705fc363e9c1c46152b8975fbaae9d93fc7944253c858afdf9b" exitCode=0
Jul 20 18:40:44.043505 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:40:44.043363    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-0d90bc/auth-test-model-0d90bc-kserve-5b9f7699f-6ztnb"
Jul 20 18:40:44.043505 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:40:44.043398    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-0d90bc/auth-test-model-0d90bc-kserve-5b9f7699f-6ztnb" event={"ID":"e9e53ef5-1f07-4df8-91dd-88ae1ba7d7d6","Type":"ContainerDied","Data":"fabfb517bb383705fc363e9c1c46152b8975fbaae9d93fc7944253c858afdf9b"}
Jul 20 18:40:44.043505 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:40:44.043454    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-0d90bc/auth-test-model-0d90bc-kserve-5b9f7699f-6ztnb" event={"ID":"e9e53ef5-1f07-4df8-91dd-88ae1ba7d7d6","Type":"ContainerDied","Data":"2da8b9ee38acea69f3cd5bf34d5ed77c1d6abe3688c00e1d1e9f798b5265cdf1"}
Jul 20 18:40:44.043505 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:40:44.043477    2569 scope.go:117] "RemoveContainer" containerID="fabfb517bb383705fc363e9c1c46152b8975fbaae9d93fc7944253c858afdf9b"
Jul 20 18:40:44.090777 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:40:44.090741    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-0d90bc/auth-test-model-0d90bc-kserve-5b9f7699f-6ztnb"]
Jul 20 18:40:44.094546 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:40:44.094520    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-0d90bc/auth-test-model-0d90bc-kserve-5b9f7699f-6ztnb"]
Jul 20 18:40:44.127083 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:40:44.127055    2569 scope.go:117] "RemoveContainer" containerID="fabfb517bb383705fc363e9c1c46152b8975fbaae9d93fc7944253c858afdf9b"
Jul 20 18:40:44.127485 ip-10-0-137-247 kubenswrapper[2569]: E0720 18:40:44.127458    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fabfb517bb383705fc363e9c1c46152b8975fbaae9d93fc7944253c858afdf9b\": container with ID starting with fabfb517bb383705fc363e9c1c46152b8975fbaae9d93fc7944253c858afdf9b not found: ID does not exist" containerID="fabfb517bb383705fc363e9c1c46152b8975fbaae9d93fc7944253c858afdf9b"
Jul 20 18:40:44.127605 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:40:44.127497    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fabfb517bb383705fc363e9c1c46152b8975fbaae9d93fc7944253c858afdf9b"} err="failed to get container status \"fabfb517bb383705fc363e9c1c46152b8975fbaae9d93fc7944253c858afdf9b\": rpc error: code = NotFound desc = could not find container \"fabfb517bb383705fc363e9c1c46152b8975fbaae9d93fc7944253c858afdf9b\": container with ID starting with fabfb517bb383705fc363e9c1c46152b8975fbaae9d93fc7944253c858afdf9b not found: ID does not exist"
Jul 20 18:40:44.182809 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:40:44.182778    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9e53ef5-1f07-4df8-91dd-88ae1ba7d7d6" path="/var/lib/kubelet/pods/e9e53ef5-1f07-4df8-91dd-88ae1ba7d7d6/volumes"
Jul 20 18:40:44.266756 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:40:44.266689    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-94313f/auth-test-model-94313f-kserve-7b99884c67-85sjk"
Jul 20 18:40:44.339873 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:40:44.339837    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4cwnr\" (UniqueName: \"kubernetes.io/projected/dde25e2d-511c-4103-ac11-290825e68fbe-kube-api-access-4cwnr\") pod \"dde25e2d-511c-4103-ac11-290825e68fbe\" (UID: \"dde25e2d-511c-4103-ac11-290825e68fbe\") "
Jul 20 18:40:44.340082 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:40:44.339910    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dde25e2d-511c-4103-ac11-290825e68fbe-tmp-dir\") pod \"dde25e2d-511c-4103-ac11-290825e68fbe\" (UID: \"dde25e2d-511c-4103-ac11-290825e68fbe\") "
Jul 20 18:40:44.340082 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:40:44.339940    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dde25e2d-511c-4103-ac11-290825e68fbe-tls-certs\") pod \"dde25e2d-511c-4103-ac11-290825e68fbe\" (UID: \"dde25e2d-511c-4103-ac11-290825e68fbe\") "
Jul 20 18:40:44.340082 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:40:44.340004    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dde25e2d-511c-4103-ac11-290825e68fbe-dshm\") pod \"dde25e2d-511c-4103-ac11-290825e68fbe\" (UID: \"dde25e2d-511c-4103-ac11-290825e68fbe\") "
Jul 20 18:40:44.340082 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:40:44.340026    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dde25e2d-511c-4103-ac11-290825e68fbe-model-cache\") pod \"dde25e2d-511c-4103-ac11-290825e68fbe\" (UID: \"dde25e2d-511c-4103-ac11-290825e68fbe\") "
Jul 20 18:40:44.340082 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:40:44.340070    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dde25e2d-511c-4103-ac11-290825e68fbe-home\") pod \"dde25e2d-511c-4103-ac11-290825e68fbe\" (UID: \"dde25e2d-511c-4103-ac11-290825e68fbe\") "
Jul 20 18:40:44.340350 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:40:44.340206    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dde25e2d-511c-4103-ac11-290825e68fbe-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "dde25e2d-511c-4103-ac11-290825e68fbe" (UID: "dde25e2d-511c-4103-ac11-290825e68fbe"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 18:40:44.340350 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:40:44.340308    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dde25e2d-511c-4103-ac11-290825e68fbe-model-cache" (OuterVolumeSpecName: "model-cache") pod "dde25e2d-511c-4103-ac11-290825e68fbe" (UID: "dde25e2d-511c-4103-ac11-290825e68fbe"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 18:40:44.340350 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:40:44.340323    2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dde25e2d-511c-4103-ac11-290825e68fbe-tmp-dir\") on node \"ip-10-0-137-247.ec2.internal\" DevicePath \"\""
Jul 20 18:40:44.340350 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:40:44.340334    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dde25e2d-511c-4103-ac11-290825e68fbe-home" (OuterVolumeSpecName: "home") pod "dde25e2d-511c-4103-ac11-290825e68fbe" (UID: "dde25e2d-511c-4103-ac11-290825e68fbe"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 18:40:44.342195 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:40:44.342168    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dde25e2d-511c-4103-ac11-290825e68fbe-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "dde25e2d-511c-4103-ac11-290825e68fbe" (UID: "dde25e2d-511c-4103-ac11-290825e68fbe"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 18:40:44.342195 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:40:44.342181    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dde25e2d-511c-4103-ac11-290825e68fbe-dshm" (OuterVolumeSpecName: "dshm") pod "dde25e2d-511c-4103-ac11-290825e68fbe" (UID: "dde25e2d-511c-4103-ac11-290825e68fbe"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 18:40:44.342337 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:40:44.342174    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dde25e2d-511c-4103-ac11-290825e68fbe-kube-api-access-4cwnr" (OuterVolumeSpecName: "kube-api-access-4cwnr") pod "dde25e2d-511c-4103-ac11-290825e68fbe" (UID: "dde25e2d-511c-4103-ac11-290825e68fbe"). InnerVolumeSpecName "kube-api-access-4cwnr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 18:40:44.441710 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:40:44.441677    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dde25e2d-511c-4103-ac11-290825e68fbe-tls-certs\") on node \"ip-10-0-137-247.ec2.internal\" DevicePath \"\""
Jul 20 18:40:44.441710 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:40:44.441702    2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dde25e2d-511c-4103-ac11-290825e68fbe-dshm\") on node \"ip-10-0-137-247.ec2.internal\" DevicePath \"\""
Jul 20 18:40:44.441710 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:40:44.441712    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dde25e2d-511c-4103-ac11-290825e68fbe-model-cache\") on node \"ip-10-0-137-247.ec2.internal\" DevicePath \"\""
Jul 20 18:40:44.441710 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:40:44.441741    2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dde25e2d-511c-4103-ac11-290825e68fbe-home\") on node \"ip-10-0-137-247.ec2.internal\" DevicePath \"\""
Jul 20 18:40:44.442012 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:40:44.441750    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cwnr\" (UniqueName: \"kubernetes.io/projected/dde25e2d-511c-4103-ac11-290825e68fbe-kube-api-access-4cwnr\") on node \"ip-10-0-137-247.ec2.internal\" DevicePath \"\""
Jul 20 18:40:45.048522 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:40:45.048485    2569 generic.go:356] "Generic (PLEG): container finished" podID="dde25e2d-511c-4103-ac11-290825e68fbe" containerID="b34714c4bc8b0f3de17c3fbd6bfb4c90a2886a615ad2d38e74defe476f316275" exitCode=0
Jul 20 18:40:45.048716 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:40:45.048529    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-94313f/auth-test-model-94313f-kserve-7b99884c67-85sjk"
Jul 20 18:40:45.048716 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:40:45.048567    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-94313f/auth-test-model-94313f-kserve-7b99884c67-85sjk" event={"ID":"dde25e2d-511c-4103-ac11-290825e68fbe","Type":"ContainerDied","Data":"b34714c4bc8b0f3de17c3fbd6bfb4c90a2886a615ad2d38e74defe476f316275"}
Jul 20 18:40:45.048716 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:40:45.048608    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-94313f/auth-test-model-94313f-kserve-7b99884c67-85sjk" event={"ID":"dde25e2d-511c-4103-ac11-290825e68fbe","Type":"ContainerDied","Data":"d320ba89914e410780d60855da3267792f7ba911d17d8c86f3eeb9aab055d095"}
Jul 20 18:40:45.048716 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:40:45.048629    2569 scope.go:117] "RemoveContainer" containerID="b34714c4bc8b0f3de17c3fbd6bfb4c90a2886a615ad2d38e74defe476f316275"
Jul 20 18:40:45.057542 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:40:45.057525    2569 scope.go:117] "RemoveContainer" containerID="b34714c4bc8b0f3de17c3fbd6bfb4c90a2886a615ad2d38e74defe476f316275"
Jul 20 18:40:45.057841 ip-10-0-137-247 kubenswrapper[2569]: E0720 18:40:45.057823    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b34714c4bc8b0f3de17c3fbd6bfb4c90a2886a615ad2d38e74defe476f316275\": container with ID starting with b34714c4bc8b0f3de17c3fbd6bfb4c90a2886a615ad2d38e74defe476f316275 not found: ID does not exist" containerID="b34714c4bc8b0f3de17c3fbd6bfb4c90a2886a615ad2d38e74defe476f316275"
Jul 20 18:40:45.057905 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:40:45.057850    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b34714c4bc8b0f3de17c3fbd6bfb4c90a2886a615ad2d38e74defe476f316275"} err="failed to get container status \"b34714c4bc8b0f3de17c3fbd6bfb4c90a2886a615ad2d38e74defe476f316275\": rpc error: code = NotFound desc = could not find container \"b34714c4bc8b0f3de17c3fbd6bfb4c90a2886a615ad2d38e74defe476f316275\": container with ID starting with b34714c4bc8b0f3de17c3fbd6bfb4c90a2886a615ad2d38e74defe476f316275 not found: ID does not exist"
Jul 20 18:40:45.070271 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:40:45.070249    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-94313f/auth-test-model-94313f-kserve-7b99884c67-85sjk"]
Jul 20 18:40:45.079776 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:40:45.077501    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-94313f/auth-test-model-94313f-kserve-7b99884c67-85sjk"]
Jul 20 18:40:46.182315 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:40:46.182269    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dde25e2d-511c-4103-ac11-290825e68fbe" path="/var/lib/kubelet/pods/dde25e2d-511c-4103-ac11-290825e68fbe/volumes"
Jul 20 18:40:54.214344 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:40:54.214309    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-0d90bc/sub-test-model-0d90bc-kserve-5947686d75-hhxpv"]
Jul 20 18:40:54.214826 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:40:54.214739    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9e53ef5-1f07-4df8-91dd-88ae1ba7d7d6" containerName="main"
Jul 20 18:40:54.214826 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:40:54.214757    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9e53ef5-1f07-4df8-91dd-88ae1ba7d7d6" containerName="main"
Jul 20 18:40:54.214826 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:40:54.214789    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dde25e2d-511c-4103-ac11-290825e68fbe" containerName="main"
Jul 20 18:40:54.214826 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:40:54.214796    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="dde25e2d-511c-4103-ac11-290825e68fbe" containerName="main"
Jul 20 18:40:54.215047 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:40:54.214877    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="dde25e2d-511c-4103-ac11-290825e68fbe" containerName="main"
Jul 20 18:40:54.215047 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:40:54.214891    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9e53ef5-1f07-4df8-91dd-88ae1ba7d7d6" containerName="main"
Jul 20 18:40:54.219391 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:40:54.219369    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-0d90bc/sub-test-model-0d90bc-kserve-5947686d75-hhxpv"
Jul 20 18:40:54.221658 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:40:54.221627    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-0d90bc\"/\"default-dockercfg-cfnxt\""
Jul 20 18:40:54.222485 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:40:54.222466    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-0d90bc\"/\"openshift-service-ca.crt\""
Jul 20 18:40:54.222583 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:40:54.222473    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-0d90bc\"/\"kube-root-ca.crt\""
Jul 20 18:40:54.222783 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:40:54.222768    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-0d90bc\"/\"sub-test-model-0d90bc-kserve-self-signed-certs\""
Jul 20 18:40:54.237644 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:40:54.237621    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-0d90bc/sub-test-model-0d90bc-kserve-5947686d75-hhxpv"]
Jul 20 18:40:54.288637 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:40:54.288610    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wjs8g_07587a78-2d3a-40ad-9aae-a4875200263a/ovn-acl-logging/0.log"
Jul 20 18:40:54.300631 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:40:54.300602    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wjs8g_07587a78-2d3a-40ad-9aae-a4875200263a/ovn-acl-logging/0.log"
Jul 20 18:40:54.325604 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:40:54.325577    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/158f7eaf-fb3d-4dc6-a140-544ef91c8e8c-tmp-dir\") pod \"sub-test-model-0d90bc-kserve-5947686d75-hhxpv\" (UID: \"158f7eaf-fb3d-4dc6-a140-544ef91c8e8c\") " pod="ai-tenant-e2e-shared-a-0d90bc/sub-test-model-0d90bc-kserve-5947686d75-hhxpv"
Jul 20 18:40:54.325769 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:40:54.325613    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/158f7eaf-fb3d-4dc6-a140-544ef91c8e8c-home\") pod \"sub-test-model-0d90bc-kserve-5947686d75-hhxpv\" (UID: \"158f7eaf-fb3d-4dc6-a140-544ef91c8e8c\") " pod="ai-tenant-e2e-shared-a-0d90bc/sub-test-model-0d90bc-kserve-5947686d75-hhxpv"
Jul 20 18:40:54.325769 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:40:54.325635    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/158f7eaf-fb3d-4dc6-a140-544ef91c8e8c-model-cache\") pod \"sub-test-model-0d90bc-kserve-5947686d75-hhxpv\" (UID: \"158f7eaf-fb3d-4dc6-a140-544ef91c8e8c\") " pod="ai-tenant-e2e-shared-a-0d90bc/sub-test-model-0d90bc-kserve-5947686d75-hhxpv"
Jul 20 18:40:54.325769 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:40:54.325697    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/158f7eaf-fb3d-4dc6-a140-544ef91c8e8c-dshm\") pod \"sub-test-model-0d90bc-kserve-5947686d75-hhxpv\" (UID: \"158f7eaf-fb3d-4dc6-a140-544ef91c8e8c\") " pod="ai-tenant-e2e-shared-a-0d90bc/sub-test-model-0d90bc-kserve-5947686d75-hhxpv"
Jul 20 18:40:54.325769 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:40:54.325748    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcqfs\" (UniqueName: \"kubernetes.io/projected/158f7eaf-fb3d-4dc6-a140-544ef91c8e8c-kube-api-access-hcqfs\") pod \"sub-test-model-0d90bc-kserve-5947686d75-hhxpv\" (UID: \"158f7eaf-fb3d-4dc6-a140-544ef91c8e8c\") " pod="ai-tenant-e2e-shared-a-0d90bc/sub-test-model-0d90bc-kserve-5947686d75-hhxpv"
Jul 20 18:40:54.325993 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:40:54.325772    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/158f7eaf-fb3d-4dc6-a140-544ef91c8e8c-tls-certs\") pod \"sub-test-model-0d90bc-kserve-5947686d75-hhxpv\" (UID: \"158f7eaf-fb3d-4dc6-a140-544ef91c8e8c\") " pod="ai-tenant-e2e-shared-a-0d90bc/sub-test-model-0d90bc-kserve-5947686d75-hhxpv"
Jul 20 18:40:54.426316 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:40:54.426180    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/158f7eaf-fb3d-4dc6-a140-544ef91c8e8c-tmp-dir\") pod \"sub-test-model-0d90bc-kserve-5947686d75-hhxpv\" (UID: \"158f7eaf-fb3d-4dc6-a140-544ef91c8e8c\") " pod="ai-tenant-e2e-shared-a-0d90bc/sub-test-model-0d90bc-kserve-5947686d75-hhxpv"
Jul 20 18:40:54.426316 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:40:54.426229    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/158f7eaf-fb3d-4dc6-a140-544ef91c8e8c-home\") pod \"sub-test-model-0d90bc-kserve-5947686d75-hhxpv\" (UID: \"158f7eaf-fb3d-4dc6-a140-544ef91c8e8c\") " pod="ai-tenant-e2e-shared-a-0d90bc/sub-test-model-0d90bc-kserve-5947686d75-hhxpv"
Jul 20 18:40:54.426316 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:40:54.426265    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/158f7eaf-fb3d-4dc6-a140-544ef91c8e8c-model-cache\") pod \"sub-test-model-0d90bc-kserve-5947686d75-hhxpv\" (UID: \"158f7eaf-fb3d-4dc6-a140-544ef91c8e8c\") " pod="ai-tenant-e2e-shared-a-0d90bc/sub-test-model-0d90bc-kserve-5947686d75-hhxpv"
Jul 20 18:40:54.426316 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:40:54.426303    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/158f7eaf-fb3d-4dc6-a140-544ef91c8e8c-dshm\") pod \"sub-test-model-0d90bc-kserve-5947686d75-hhxpv\" (UID: \"158f7eaf-fb3d-4dc6-a140-544ef91c8e8c\") " pod="ai-tenant-e2e-shared-a-0d90bc/sub-test-model-0d90bc-kserve-5947686d75-hhxpv"
Jul 20 18:40:54.426532 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:40:54.426356    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcqfs\" (UniqueName: \"kubernetes.io/projected/158f7eaf-fb3d-4dc6-a140-544ef91c8e8c-kube-api-access-hcqfs\") pod \"sub-test-model-0d90bc-kserve-5947686d75-hhxpv\" (UID: \"158f7eaf-fb3d-4dc6-a140-544ef91c8e8c\") " pod="ai-tenant-e2e-shared-a-0d90bc/sub-test-model-0d90bc-kserve-5947686d75-hhxpv"
Jul 20 18:40:54.426532 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:40:54.426394    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/158f7eaf-fb3d-4dc6-a140-544ef91c8e8c-tls-certs\") pod \"sub-test-model-0d90bc-kserve-5947686d75-hhxpv\" (UID: \"158f7eaf-fb3d-4dc6-a140-544ef91c8e8c\") " pod="ai-tenant-e2e-shared-a-0d90bc/sub-test-model-0d90bc-kserve-5947686d75-hhxpv"
Jul 20 18:40:54.426636 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:40:54.426570    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/158f7eaf-fb3d-4dc6-a140-544ef91c8e8c-tmp-dir\") pod \"sub-test-model-0d90bc-kserve-5947686d75-hhxpv\" (UID: \"158f7eaf-fb3d-4dc6-a140-544ef91c8e8c\") " pod="ai-tenant-e2e-shared-a-0d90bc/sub-test-model-0d90bc-kserve-5947686d75-hhxpv"
Jul 20 18:40:54.426636 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:40:54.426605    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/158f7eaf-fb3d-4dc6-a140-544ef91c8e8c-home\") pod \"sub-test-model-0d90bc-kserve-5947686d75-hhxpv\" (UID: \"158f7eaf-fb3d-4dc6-a140-544ef91c8e8c\") " pod="ai-tenant-e2e-shared-a-0d90bc/sub-test-model-0d90bc-kserve-5947686d75-hhxpv"
Jul 20 18:40:54.426821 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:40:54.426801    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/158f7eaf-fb3d-4dc6-a140-544ef91c8e8c-model-cache\") pod \"sub-test-model-0d90bc-kserve-5947686d75-hhxpv\" (UID: \"158f7eaf-fb3d-4dc6-a140-544ef91c8e8c\") " pod="ai-tenant-e2e-shared-a-0d90bc/sub-test-model-0d90bc-kserve-5947686d75-hhxpv"
Jul 20 18:40:54.428757 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:40:54.428717    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/158f7eaf-fb3d-4dc6-a140-544ef91c8e8c-dshm\") pod \"sub-test-model-0d90bc-kserve-5947686d75-hhxpv\" (UID: \"158f7eaf-fb3d-4dc6-a140-544ef91c8e8c\") " pod="ai-tenant-e2e-shared-a-0d90bc/sub-test-model-0d90bc-kserve-5947686d75-hhxpv"
Jul 20 18:40:54.429005 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:40:54.428984    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/158f7eaf-fb3d-4dc6-a140-544ef91c8e8c-tls-certs\") pod \"sub-test-model-0d90bc-kserve-5947686d75-hhxpv\" (UID: \"158f7eaf-fb3d-4dc6-a140-544ef91c8e8c\") " pod="ai-tenant-e2e-shared-a-0d90bc/sub-test-model-0d90bc-kserve-5947686d75-hhxpv"
Jul 20 18:40:54.434862 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:40:54.434833    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcqfs\" (UniqueName: \"kubernetes.io/projected/158f7eaf-fb3d-4dc6-a140-544ef91c8e8c-kube-api-access-hcqfs\") pod \"sub-test-model-0d90bc-kserve-5947686d75-hhxpv\" (UID: \"158f7eaf-fb3d-4dc6-a140-544ef91c8e8c\") " pod="ai-tenant-e2e-shared-a-0d90bc/sub-test-model-0d90bc-kserve-5947686d75-hhxpv"
Jul 20 18:40:54.528102 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:40:54.528064    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-0d90bc/sub-test-model-0d90bc-kserve-5947686d75-hhxpv"
Jul 20 18:40:54.671460 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:40:54.671426    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-0d90bc/sub-test-model-0d90bc-kserve-5947686d75-hhxpv"]
Jul 20 18:40:54.674425 ip-10-0-137-247 kubenswrapper[2569]: W0720 18:40:54.674396    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod158f7eaf_fb3d_4dc6_a140_544ef91c8e8c.slice/crio-e3c0534af171407e82bdd1375532bdb1816460eed7abddc5e65a64ca2da36c1f WatchSource:0}: Error finding container e3c0534af171407e82bdd1375532bdb1816460eed7abddc5e65a64ca2da36c1f: Status 404 returned error can't find the container with id e3c0534af171407e82bdd1375532bdb1816460eed7abddc5e65a64ca2da36c1f
Jul 20 18:40:55.096158 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:40:55.096119    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-0d90bc/sub-test-model-0d90bc-kserve-5947686d75-hhxpv" event={"ID":"158f7eaf-fb3d-4dc6-a140-544ef91c8e8c","Type":"ContainerStarted","Data":"efe4dc83abc18192a523ec39c8255f43c58718af80d3583ceedb2f8ace686d51"}
Jul 20 18:40:55.096158 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:40:55.096160    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-0d90bc/sub-test-model-0d90bc-kserve-5947686d75-hhxpv" event={"ID":"158f7eaf-fb3d-4dc6-a140-544ef91c8e8c","Type":"ContainerStarted","Data":"e3c0534af171407e82bdd1375532bdb1816460eed7abddc5e65a64ca2da36c1f"}
Jul 20 18:40:55.142895 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:40:55.142836    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-0d90bc/sub-test-model-0d90bc-kserve-5947686d75-hhxpv" podStartSLOduration=1.142820843 podStartE2EDuration="1.142820843s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 18:40:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 18:40:55.139326206 +0000 UTC m=+2701.385859616" watchObservedRunningTime="2026-07-20 18:40:55.142820843 +0000 UTC m=+2701.389354255"
Jul 20 18:40:57.578640 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:40:57.578598    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9thxb"]
Jul 20 18:41:04.528789 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:41:04.528748    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-0d90bc/sub-test-model-0d90bc-kserve-5947686d75-hhxpv"
Jul 20 18:41:04.529169 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:41:04.528815    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-0d90bc/sub-test-model-0d90bc-kserve-5947686d75-hhxpv"
Jul 20 18:41:04.541507 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:41:04.541477    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-0d90bc/sub-test-model-0d90bc-kserve-5947686d75-hhxpv"
Jul 20 18:41:05.152587 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:41:05.152559    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-0d90bc/sub-test-model-0d90bc-kserve-5947686d75-hhxpv"
Jul 20 18:41:19.842261 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:41:19.842227    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-94313f/sub-test-model-94313f-kserve-7bbbb7fd59-xdh64"]
Jul 20 18:41:19.845527 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:41:19.845510    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-94313f/sub-test-model-94313f-kserve-7bbbb7fd59-xdh64"
Jul 20 18:41:19.848387 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:41:19.848364    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-94313f\"/\"sub-test-model-94313f-kserve-self-signed-certs\""
Jul 20 18:41:19.849333 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:41:19.849313    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-94313f\"/\"default-dockercfg-xtr2f\""
Jul 20 18:41:19.849466 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:41:19.849341    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-94313f\"/\"openshift-service-ca.crt\""
Jul 20 18:41:19.849466 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:41:19.849363    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-94313f\"/\"kube-root-ca.crt\""
Jul 20 18:41:19.857796 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:41:19.857771    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-94313f/sub-test-model-94313f-kserve-7bbbb7fd59-xdh64"]
Jul 20 18:41:19.939626 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:41:19.939585    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5ee57dc-1ad3-4c2e-a82f-5df0a0bd8536-tmp-dir\") pod \"sub-test-model-94313f-kserve-7bbbb7fd59-xdh64\" (UID: \"d5ee57dc-1ad3-4c2e-a82f-5df0a0bd8536\") " pod="ai-tenant-e2e-shared-b-94313f/sub-test-model-94313f-kserve-7bbbb7fd59-xdh64"
Jul 20 18:41:19.939626 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:41:19.939632    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hn5zz\" (UniqueName: \"kubernetes.io/projected/d5ee57dc-1ad3-4c2e-a82f-5df0a0bd8536-kube-api-access-hn5zz\") pod \"sub-test-model-94313f-kserve-7bbbb7fd59-xdh64\" (UID: \"d5ee57dc-1ad3-4c2e-a82f-5df0a0bd8536\") " pod="ai-tenant-e2e-shared-b-94313f/sub-test-model-94313f-kserve-7bbbb7fd59-xdh64"
Jul 20 18:41:19.939885 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:41:19.939694    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5ee57dc-1ad3-4c2e-a82f-5df0a0bd8536-tls-certs\") pod \"sub-test-model-94313f-kserve-7bbbb7fd59-xdh64\" (UID: \"d5ee57dc-1ad3-4c2e-a82f-5df0a0bd8536\") " pod="ai-tenant-e2e-shared-b-94313f/sub-test-model-94313f-kserve-7bbbb7fd59-xdh64"
Jul 20 18:41:19.939885 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:41:19.939716    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5ee57dc-1ad3-4c2e-a82f-5df0a0bd8536-dshm\") pod \"sub-test-model-94313f-kserve-7bbbb7fd59-xdh64\" (UID: \"d5ee57dc-1ad3-4c2e-a82f-5df0a0bd8536\") " pod="ai-tenant-e2e-shared-b-94313f/sub-test-model-94313f-kserve-7bbbb7fd59-xdh64"
Jul 20 18:41:19.939885 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:41:19.939797    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5ee57dc-1ad3-4c2e-a82f-5df0a0bd8536-home\") pod \"sub-test-model-94313f-kserve-7bbbb7fd59-xdh64\" (UID: \"d5ee57dc-1ad3-4c2e-a82f-5df0a0bd8536\") " pod="ai-tenant-e2e-shared-b-94313f/sub-test-model-94313f-kserve-7bbbb7fd59-xdh64"
Jul 20 18:41:19.939885 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:41:19.939840    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5ee57dc-1ad3-4c2e-a82f-5df0a0bd8536-model-cache\") pod \"sub-test-model-94313f-kserve-7bbbb7fd59-xdh64\" (UID: \"d5ee57dc-1ad3-4c2e-a82f-5df0a0bd8536\") " pod="ai-tenant-e2e-shared-b-94313f/sub-test-model-94313f-kserve-7bbbb7fd59-xdh64"
Jul 20 18:41:20.040625 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:41:20.040585    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5ee57dc-1ad3-4c2e-a82f-5df0a0bd8536-home\") pod \"sub-test-model-94313f-kserve-7bbbb7fd59-xdh64\" (UID: \"d5ee57dc-1ad3-4c2e-a82f-5df0a0bd8536\") " pod="ai-tenant-e2e-shared-b-94313f/sub-test-model-94313f-kserve-7bbbb7fd59-xdh64"
Jul 20 18:41:20.040824 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:41:20.040632    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5ee57dc-1ad3-4c2e-a82f-5df0a0bd8536-model-cache\") pod \"sub-test-model-94313f-kserve-7bbbb7fd59-xdh64\" (UID: \"d5ee57dc-1ad3-4c2e-a82f-5df0a0bd8536\") " pod="ai-tenant-e2e-shared-b-94313f/sub-test-model-94313f-kserve-7bbbb7fd59-xdh64"
Jul 20 18:41:20.040824 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:41:20.040693    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5ee57dc-1ad3-4c2e-a82f-5df0a0bd8536-tmp-dir\") pod \"sub-test-model-94313f-kserve-7bbbb7fd59-xdh64\" (UID: \"d5ee57dc-1ad3-4c2e-a82f-5df0a0bd8536\") " pod="ai-tenant-e2e-shared-b-94313f/sub-test-model-94313f-kserve-7bbbb7fd59-xdh64"
Jul 20 18:41:20.040824 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:41:20.040759    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hn5zz\" (UniqueName: \"kubernetes.io/projected/d5ee57dc-1ad3-4c2e-a82f-5df0a0bd8536-kube-api-access-hn5zz\") pod \"sub-test-model-94313f-kserve-7bbbb7fd59-xdh64\" (UID: \"d5ee57dc-1ad3-4c2e-a82f-5df0a0bd8536\") " pod="ai-tenant-e2e-shared-b-94313f/sub-test-model-94313f-kserve-7bbbb7fd59-xdh64"
Jul 20 18:41:20.040824 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:41:20.040806    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5ee57dc-1ad3-4c2e-a82f-5df0a0bd8536-tls-certs\") pod \"sub-test-model-94313f-kserve-7bbbb7fd59-xdh64\" (UID: \"d5ee57dc-1ad3-4c2e-a82f-5df0a0bd8536\") " pod="ai-tenant-e2e-shared-b-94313f/sub-test-model-94313f-kserve-7bbbb7fd59-xdh64"
Jul 20 18:41:20.040983 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:41:20.040840    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5ee57dc-1ad3-4c2e-a82f-5df0a0bd8536-dshm\") pod \"sub-test-model-94313f-kserve-7bbbb7fd59-xdh64\" (UID: \"d5ee57dc-1ad3-4c2e-a82f-5df0a0bd8536\") " pod="ai-tenant-e2e-shared-b-94313f/sub-test-model-94313f-kserve-7bbbb7fd59-xdh64"
Jul 20 18:41:20.041110 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:41:20.041088    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5ee57dc-1ad3-4c2e-a82f-5df0a0bd8536-home\") pod \"sub-test-model-94313f-kserve-7bbbb7fd59-xdh64\" (UID: \"d5ee57dc-1ad3-4c2e-a82f-5df0a0bd8536\") " pod="ai-tenant-e2e-shared-b-94313f/sub-test-model-94313f-kserve-7bbbb7fd59-xdh64"
Jul 20 18:41:20.041156 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:41:20.041124    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5ee57dc-1ad3-4c2e-a82f-5df0a0bd8536-model-cache\") pod \"sub-test-model-94313f-kserve-7bbbb7fd59-xdh64\" (UID: \"d5ee57dc-1ad3-4c2e-a82f-5df0a0bd8536\") " pod="ai-tenant-e2e-shared-b-94313f/sub-test-model-94313f-kserve-7bbbb7fd59-xdh64"
Jul 20 18:41:20.041241 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:41:20.041223    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5ee57dc-1ad3-4c2e-a82f-5df0a0bd8536-tmp-dir\") pod \"sub-test-model-94313f-kserve-7bbbb7fd59-xdh64\" (UID: \"d5ee57dc-1ad3-4c2e-a82f-5df0a0bd8536\") " pod="ai-tenant-e2e-shared-b-94313f/sub-test-model-94313f-kserve-7bbbb7fd59-xdh64"
Jul 20 18:41:20.042970 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:41:20.042953    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5ee57dc-1ad3-4c2e-a82f-5df0a0bd8536-dshm\") pod \"sub-test-model-94313f-kserve-7bbbb7fd59-xdh64\" (UID: \"d5ee57dc-1ad3-4c2e-a82f-5df0a0bd8536\") " pod="ai-tenant-e2e-shared-b-94313f/sub-test-model-94313f-kserve-7bbbb7fd59-xdh64"
Jul 20 18:41:20.043279 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:41:20.043262    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5ee57dc-1ad3-4c2e-a82f-5df0a0bd8536-tls-certs\") pod \"sub-test-model-94313f-kserve-7bbbb7fd59-xdh64\" (UID: \"d5ee57dc-1ad3-4c2e-a82f-5df0a0bd8536\") " pod="ai-tenant-e2e-shared-b-94313f/sub-test-model-94313f-kserve-7bbbb7fd59-xdh64"
Jul 20 18:41:20.048929 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:41:20.048904    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hn5zz\" (UniqueName: \"kubernetes.io/projected/d5ee57dc-1ad3-4c2e-a82f-5df0a0bd8536-kube-api-access-hn5zz\") pod \"sub-test-model-94313f-kserve-7bbbb7fd59-xdh64\" (UID: \"d5ee57dc-1ad3-4c2e-a82f-5df0a0bd8536\") " pod="ai-tenant-e2e-shared-b-94313f/sub-test-model-94313f-kserve-7bbbb7fd59-xdh64"
Jul 20 18:41:20.156803 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:41:20.156683    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-94313f/sub-test-model-94313f-kserve-7bbbb7fd59-xdh64"
Jul 20 18:41:20.295085 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:41:20.295011    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-94313f/sub-test-model-94313f-kserve-7bbbb7fd59-xdh64"]
Jul 20 18:41:20.297700 ip-10-0-137-247 kubenswrapper[2569]: W0720 18:41:20.297674    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd5ee57dc_1ad3_4c2e_a82f_5df0a0bd8536.slice/crio-48bd18e24dd9e18cdee38c48d47fe531fc5a421f9ace9b9f928958005dc84a0b WatchSource:0}: Error finding container 48bd18e24dd9e18cdee38c48d47fe531fc5a421f9ace9b9f928958005dc84a0b: Status 404 returned error can't find the container with id 48bd18e24dd9e18cdee38c48d47fe531fc5a421f9ace9b9f928958005dc84a0b
Jul 20 18:41:21.211994 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:41:21.211956    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-94313f/sub-test-model-94313f-kserve-7bbbb7fd59-xdh64" event={"ID":"d5ee57dc-1ad3-4c2e-a82f-5df0a0bd8536","Type":"ContainerStarted","Data":"a9d5c5cc8f22242945d81d119b318c6bc0569c6f0a6d1f2a75773cb3826b3612"}
Jul 20 18:41:21.212424 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:41:21.212003    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-94313f/sub-test-model-94313f-kserve-7bbbb7fd59-xdh64" event={"ID":"d5ee57dc-1ad3-4c2e-a82f-5df0a0bd8536","Type":"ContainerStarted","Data":"48bd18e24dd9e18cdee38c48d47fe531fc5a421f9ace9b9f928958005dc84a0b"}
Jul 20 18:41:21.230736 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:41:21.230681    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-94313f/sub-test-model-94313f-kserve-7bbbb7fd59-xdh64" podStartSLOduration=2.230665207 podStartE2EDuration="2.230665207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 18:41:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 18:41:21.229809133 +0000 UTC m=+2727.476342543" watchObservedRunningTime="2026-07-20 18:41:21.230665207 +0000 UTC m=+2727.477198616"
Jul 20 18:41:23.862645 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:41:23.862604    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9thxb"]
Jul 20 18:41:30.157687 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:41:30.157654    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-94313f/sub-test-model-94313f-kserve-7bbbb7fd59-xdh64"
Jul 20 18:41:30.158091 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:41:30.157746    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-94313f/sub-test-model-94313f-kserve-7bbbb7fd59-xdh64"
Jul 20 18:41:30.170064 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:41:30.170040    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-94313f/sub-test-model-94313f-kserve-7bbbb7fd59-xdh64"
Jul 20 18:41:30.261711 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:41:30.261682    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-94313f/sub-test-model-94313f-kserve-7bbbb7fd59-xdh64"
Jul 20 18:41:37.453186 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:41:37.453148    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9thxb"]
Jul 20 18:41:54.436327 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:41:54.436295    2569 scope.go:117] "RemoveContainer" containerID="a938107ff8f5e5acfb2999b5957238874e56e8cb6e7baba8f7d57e24e060b1e5"
Jul 20 18:41:57.158201 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:41:57.158163    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9thxb"]
Jul 20 18:42:03.614936 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:42:03.614896    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-0d90bc/sub-test-model-0d90bc-kserve-5947686d75-hhxpv"]
Jul 20 18:42:04.464204 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:42:04.464170    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-94313f/sub-test-model-94313f-kserve-7bbbb7fd59-xdh64"]
Jul 20 18:42:13.810299 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:42:13.810256    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-0d90bc/rate-test-model-0d90bc-kserve-6bb7655bc-5ssth"]
Jul 20 18:42:13.817962 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:42:13.817941    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-0d90bc/rate-test-model-0d90bc-kserve-6bb7655bc-5ssth"
Jul 20 18:42:13.820544 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:42:13.820519    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-0d90bc\"/\"rate-test-model-0d90bc-kserve-self-signed-certs\""
Jul 20 18:42:13.825964 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:42:13.825937    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-0d90bc/rate-test-model-0d90bc-kserve-6bb7655bc-5ssth"]
Jul 20 18:42:13.901820 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:42:13.901786    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce58a8db-d6fa-42e8-8682-89c3500fad7f-home\") pod \"rate-test-model-0d90bc-kserve-6bb7655bc-5ssth\" (UID: \"ce58a8db-d6fa-42e8-8682-89c3500fad7f\") " pod="ai-tenant-e2e-shared-a-0d90bc/rate-test-model-0d90bc-kserve-6bb7655bc-5ssth"
Jul 20 18:42:13.902033 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:42:13.901856    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce58a8db-d6fa-42e8-8682-89c3500fad7f-dshm\") pod \"rate-test-model-0d90bc-kserve-6bb7655bc-5ssth\" (UID: \"ce58a8db-d6fa-42e8-8682-89c3500fad7f\") " pod="ai-tenant-e2e-shared-a-0d90bc/rate-test-model-0d90bc-kserve-6bb7655bc-5ssth"
Jul 20 18:42:13.902033 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:42:13.901895    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce58a8db-d6fa-42e8-8682-89c3500fad7f-tls-certs\") pod \"rate-test-model-0d90bc-kserve-6bb7655bc-5ssth\" (UID: \"ce58a8db-d6fa-42e8-8682-89c3500fad7f\") " pod="ai-tenant-e2e-shared-a-0d90bc/rate-test-model-0d90bc-kserve-6bb7655bc-5ssth"
Jul 20 18:42:13.902033 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:42:13.901964    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce58a8db-d6fa-42e8-8682-89c3500fad7f-model-cache\") pod \"rate-test-model-0d90bc-kserve-6bb7655bc-5ssth\" (UID: \"ce58a8db-d6fa-42e8-8682-89c3500fad7f\") " pod="ai-tenant-e2e-shared-a-0d90bc/rate-test-model-0d90bc-kserve-6bb7655bc-5ssth"
Jul 20 18:42:13.902033 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:42:13.901999    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggrhl\" (UniqueName: \"kubernetes.io/projected/ce58a8db-d6fa-42e8-8682-89c3500fad7f-kube-api-access-ggrhl\") pod \"rate-test-model-0d90bc-kserve-6bb7655bc-5ssth\" (UID: \"ce58a8db-d6fa-42e8-8682-89c3500fad7f\") " pod="ai-tenant-e2e-shared-a-0d90bc/rate-test-model-0d90bc-kserve-6bb7655bc-5ssth"
Jul 20 18:42:13.902033 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:42:13.902032    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce58a8db-d6fa-42e8-8682-89c3500fad7f-tmp-dir\") pod \"rate-test-model-0d90bc-kserve-6bb7655bc-5ssth\" (UID: \"ce58a8db-d6fa-42e8-8682-89c3500fad7f\") " pod="ai-tenant-e2e-shared-a-0d90bc/rate-test-model-0d90bc-kserve-6bb7655bc-5ssth"
Jul 20 18:42:14.003253 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:42:14.003212    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce58a8db-d6fa-42e8-8682-89c3500fad7f-tls-certs\") pod \"rate-test-model-0d90bc-kserve-6bb7655bc-5ssth\" (UID: \"ce58a8db-d6fa-42e8-8682-89c3500fad7f\") " pod="ai-tenant-e2e-shared-a-0d90bc/rate-test-model-0d90bc-kserve-6bb7655bc-5ssth"
Jul 20 18:42:14.003421 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:42:14.003268    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce58a8db-d6fa-42e8-8682-89c3500fad7f-model-cache\") pod \"rate-test-model-0d90bc-kserve-6bb7655bc-5ssth\" (UID: \"ce58a8db-d6fa-42e8-8682-89c3500fad7f\") " pod="ai-tenant-e2e-shared-a-0d90bc/rate-test-model-0d90bc-kserve-6bb7655bc-5ssth"
Jul 20 18:42:14.003421 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:42:14.003398    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggrhl\" (UniqueName: \"kubernetes.io/projected/ce58a8db-d6fa-42e8-8682-89c3500fad7f-kube-api-access-ggrhl\") pod \"rate-test-model-0d90bc-kserve-6bb7655bc-5ssth\" (UID: \"ce58a8db-d6fa-42e8-8682-89c3500fad7f\") " pod="ai-tenant-e2e-shared-a-0d90bc/rate-test-model-0d90bc-kserve-6bb7655bc-5ssth"
Jul 20 18:42:14.003545 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:42:14.003438    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce58a8db-d6fa-42e8-8682-89c3500fad7f-tmp-dir\") pod \"rate-test-model-0d90bc-kserve-6bb7655bc-5ssth\" (UID: \"ce58a8db-d6fa-42e8-8682-89c3500fad7f\") " pod="ai-tenant-e2e-shared-a-0d90bc/rate-test-model-0d90bc-kserve-6bb7655bc-5ssth"
Jul 20 18:42:14.003545 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:42:14.003466    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce58a8db-d6fa-42e8-8682-89c3500fad7f-home\") pod \"rate-test-model-0d90bc-kserve-6bb7655bc-5ssth\" (UID: \"ce58a8db-d6fa-42e8-8682-89c3500fad7f\") " pod="ai-tenant-e2e-shared-a-0d90bc/rate-test-model-0d90bc-kserve-6bb7655bc-5ssth"
Jul 20 18:42:14.003650 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:42:14.003564    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce58a8db-d6fa-42e8-8682-89c3500fad7f-dshm\") pod \"rate-test-model-0d90bc-kserve-6bb7655bc-5ssth\" (UID: \"ce58a8db-d6fa-42e8-8682-89c3500fad7f\") " pod="ai-tenant-e2e-shared-a-0d90bc/rate-test-model-0d90bc-kserve-6bb7655bc-5ssth"
Jul 20 18:42:14.003650 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:42:14.003613    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce58a8db-d6fa-42e8-8682-89c3500fad7f-model-cache\") pod \"rate-test-model-0d90bc-kserve-6bb7655bc-5ssth\" (UID: \"ce58a8db-d6fa-42e8-8682-89c3500fad7f\") " pod="ai-tenant-e2e-shared-a-0d90bc/rate-test-model-0d90bc-kserve-6bb7655bc-5ssth"
Jul 20 18:42:14.003789 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:42:14.003765    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce58a8db-d6fa-42e8-8682-89c3500fad7f-tmp-dir\") pod \"rate-test-model-0d90bc-kserve-6bb7655bc-5ssth\" (UID: \"ce58a8db-d6fa-42e8-8682-89c3500fad7f\") " pod="ai-tenant-e2e-shared-a-0d90bc/rate-test-model-0d90bc-kserve-6bb7655bc-5ssth"
Jul 20 18:42:14.003847 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:42:14.003831    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce58a8db-d6fa-42e8-8682-89c3500fad7f-home\") pod \"rate-test-model-0d90bc-kserve-6bb7655bc-5ssth\" (UID: \"ce58a8db-d6fa-42e8-8682-89c3500fad7f\") " pod="ai-tenant-e2e-shared-a-0d90bc/rate-test-model-0d90bc-kserve-6bb7655bc-5ssth"
Jul 20 18:42:14.005707 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:42:14.005679    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce58a8db-d6fa-42e8-8682-89c3500fad7f-dshm\") pod \"rate-test-model-0d90bc-kserve-6bb7655bc-5ssth\" (UID: \"ce58a8db-d6fa-42e8-8682-89c3500fad7f\") " pod="ai-tenant-e2e-shared-a-0d90bc/rate-test-model-0d90bc-kserve-6bb7655bc-5ssth"
Jul 20 18:42:14.005906 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:42:14.005887    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce58a8db-d6fa-42e8-8682-89c3500fad7f-tls-certs\") pod \"rate-test-model-0d90bc-kserve-6bb7655bc-5ssth\" (UID: \"ce58a8db-d6fa-42e8-8682-89c3500fad7f\") " pod="ai-tenant-e2e-shared-a-0d90bc/rate-test-model-0d90bc-kserve-6bb7655bc-5ssth"
Jul 20 18:42:14.020599 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:42:14.020567    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggrhl\" (UniqueName: \"kubernetes.io/projected/ce58a8db-d6fa-42e8-8682-89c3500fad7f-kube-api-access-ggrhl\") pod \"rate-test-model-0d90bc-kserve-6bb7655bc-5ssth\" (UID: \"ce58a8db-d6fa-42e8-8682-89c3500fad7f\") " pod="ai-tenant-e2e-shared-a-0d90bc/rate-test-model-0d90bc-kserve-6bb7655bc-5ssth"
Jul 20 18:42:14.129051 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:42:14.128942    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-0d90bc/rate-test-model-0d90bc-kserve-6bb7655bc-5ssth"
Jul 20 18:42:14.266176 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:42:14.266146    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-0d90bc/rate-test-model-0d90bc-kserve-6bb7655bc-5ssth"]
Jul 20 18:42:14.267938 ip-10-0-137-247 kubenswrapper[2569]: W0720 18:42:14.267915    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podce58a8db_d6fa_42e8_8682_89c3500fad7f.slice/crio-6bf21de87d8dccd157d4c0e34518ef61e96d14523905abd3902ce19efccc39ba WatchSource:0}: Error finding container 6bf21de87d8dccd157d4c0e34518ef61e96d14523905abd3902ce19efccc39ba: Status 404 returned error can't find the container with id 6bf21de87d8dccd157d4c0e34518ef61e96d14523905abd3902ce19efccc39ba
Jul 20 18:42:14.454418 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:42:14.454319    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-0d90bc/rate-test-model-0d90bc-kserve-6bb7655bc-5ssth" event={"ID":"ce58a8db-d6fa-42e8-8682-89c3500fad7f","Type":"ContainerStarted","Data":"f9e03d77b615bf260487e7630cd42bc89c54a9b606f8f353d383b9e1080d1178"}
Jul 20 18:42:14.454418 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:42:14.454364    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-0d90bc/rate-test-model-0d90bc-kserve-6bb7655bc-5ssth" event={"ID":"ce58a8db-d6fa-42e8-8682-89c3500fad7f","Type":"ContainerStarted","Data":"6bf21de87d8dccd157d4c0e34518ef61e96d14523905abd3902ce19efccc39ba"}
Jul 20 18:42:14.473388 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:42:14.473335    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-0d90bc/rate-test-model-0d90bc-kserve-6bb7655bc-5ssth" podStartSLOduration=1.473318466 podStartE2EDuration="1.473318466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 18:42:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 18:42:14.471564478 +0000 UTC m=+2780.718097889" watchObservedRunningTime="2026-07-20 18:42:14.473318466 +0000 UTC m=+2780.719851875"
Jul 20 18:42:17.641390 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:42:17.641310    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9thxb"]
Jul 20 18:42:18.634986 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:42:18.634950    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-0d90bc/sub-test-model-0d90bc-kserve-5947686d75-hhxpv" podUID="158f7eaf-fb3d-4dc6-a140-544ef91c8e8c" containerName="main" containerID="cri-o://efe4dc83abc18192a523ec39c8255f43c58718af80d3583ceedb2f8ace686d51" gracePeriod=45
Jul 20 18:42:18.888110 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:42:18.888053    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-0d90bc/sub-test-model-0d90bc-kserve-5947686d75-hhxpv"
Jul 20 18:42:18.942776 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:42:18.942744    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/158f7eaf-fb3d-4dc6-a140-544ef91c8e8c-home\") pod \"158f7eaf-fb3d-4dc6-a140-544ef91c8e8c\" (UID: \"158f7eaf-fb3d-4dc6-a140-544ef91c8e8c\") "
Jul 20 18:42:18.942949 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:42:18.942803    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/158f7eaf-fb3d-4dc6-a140-544ef91c8e8c-tmp-dir\") pod \"158f7eaf-fb3d-4dc6-a140-544ef91c8e8c\" (UID: \"158f7eaf-fb3d-4dc6-a140-544ef91c8e8c\") "
Jul 20 18:42:18.942949 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:42:18.942853    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/158f7eaf-fb3d-4dc6-a140-544ef91c8e8c-dshm\") pod \"158f7eaf-fb3d-4dc6-a140-544ef91c8e8c\" (UID: \"158f7eaf-fb3d-4dc6-a140-544ef91c8e8c\") "
Jul 20 18:42:18.942949 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:42:18.942915    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/158f7eaf-fb3d-4dc6-a140-544ef91c8e8c-tls-certs\") pod \"158f7eaf-fb3d-4dc6-a140-544ef91c8e8c\" (UID: \"158f7eaf-fb3d-4dc6-a140-544ef91c8e8c\") "
Jul 20 18:42:18.943112 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:42:18.942957    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hcqfs\" (UniqueName: \"kubernetes.io/projected/158f7eaf-fb3d-4dc6-a140-544ef91c8e8c-kube-api-access-hcqfs\") pod \"158f7eaf-fb3d-4dc6-a140-544ef91c8e8c\" (UID: \"158f7eaf-fb3d-4dc6-a140-544ef91c8e8c\") "
Jul 20 18:42:18.943112 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:42:18.943008    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/158f7eaf-fb3d-4dc6-a140-544ef91c8e8c-model-cache\") pod \"158f7eaf-fb3d-4dc6-a140-544ef91c8e8c\" (UID: \"158f7eaf-fb3d-4dc6-a140-544ef91c8e8c\") "
Jul 20 18:42:18.943112 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:42:18.943055    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/158f7eaf-fb3d-4dc6-a140-544ef91c8e8c-home" (OuterVolumeSpecName: "home") pod "158f7eaf-fb3d-4dc6-a140-544ef91c8e8c" (UID: "158f7eaf-fb3d-4dc6-a140-544ef91c8e8c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 18:42:18.943112 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:42:18.943063    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/158f7eaf-fb3d-4dc6-a140-544ef91c8e8c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "158f7eaf-fb3d-4dc6-a140-544ef91c8e8c" (UID: "158f7eaf-fb3d-4dc6-a140-544ef91c8e8c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 18:42:18.943328 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:42:18.943282    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/158f7eaf-fb3d-4dc6-a140-544ef91c8e8c-model-cache" (OuterVolumeSpecName: "model-cache") pod "158f7eaf-fb3d-4dc6-a140-544ef91c8e8c" (UID: "158f7eaf-fb3d-4dc6-a140-544ef91c8e8c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 18:42:18.943328 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:42:18.943290    2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/158f7eaf-fb3d-4dc6-a140-544ef91c8e8c-home\") on node \"ip-10-0-137-247.ec2.internal\" DevicePath \"\""
Jul 20 18:42:18.943328 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:42:18.943312    2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/158f7eaf-fb3d-4dc6-a140-544ef91c8e8c-tmp-dir\") on node \"ip-10-0-137-247.ec2.internal\" DevicePath \"\""
Jul 20 18:42:18.945093 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:42:18.945066    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/158f7eaf-fb3d-4dc6-a140-544ef91c8e8c-dshm" (OuterVolumeSpecName: "dshm") pod "158f7eaf-fb3d-4dc6-a140-544ef91c8e8c" (UID: "158f7eaf-fb3d-4dc6-a140-544ef91c8e8c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 18:42:18.945207 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:42:18.945112    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/158f7eaf-fb3d-4dc6-a140-544ef91c8e8c-kube-api-access-hcqfs" (OuterVolumeSpecName: "kube-api-access-hcqfs") pod "158f7eaf-fb3d-4dc6-a140-544ef91c8e8c" (UID: "158f7eaf-fb3d-4dc6-a140-544ef91c8e8c"). InnerVolumeSpecName "kube-api-access-hcqfs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 18:42:18.945270 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:42:18.945207    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/158f7eaf-fb3d-4dc6-a140-544ef91c8e8c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "158f7eaf-fb3d-4dc6-a140-544ef91c8e8c" (UID: "158f7eaf-fb3d-4dc6-a140-544ef91c8e8c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 18:42:19.044404 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:42:19.044371    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/158f7eaf-fb3d-4dc6-a140-544ef91c8e8c-tls-certs\") on node \"ip-10-0-137-247.ec2.internal\" DevicePath \"\""
Jul 20 18:42:19.044404 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:42:19.044401    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcqfs\" (UniqueName: \"kubernetes.io/projected/158f7eaf-fb3d-4dc6-a140-544ef91c8e8c-kube-api-access-hcqfs\") on node \"ip-10-0-137-247.ec2.internal\" DevicePath \"\""
Jul 20 18:42:19.044404 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:42:19.044412    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/158f7eaf-fb3d-4dc6-a140-544ef91c8e8c-model-cache\") on node \"ip-10-0-137-247.ec2.internal\" DevicePath \"\""
Jul 20 18:42:19.044639 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:42:19.044420    2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/158f7eaf-fb3d-4dc6-a140-544ef91c8e8c-dshm\") on node \"ip-10-0-137-247.ec2.internal\" DevicePath \"\""
Jul 20 18:42:19.479552 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:42:19.479515    2569 generic.go:356] "Generic (PLEG): container finished" podID="158f7eaf-fb3d-4dc6-a140-544ef91c8e8c" containerID="efe4dc83abc18192a523ec39c8255f43c58718af80d3583ceedb2f8ace686d51" exitCode=0
Jul 20 18:42:19.479762 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:42:19.479562    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-0d90bc/sub-test-model-0d90bc-kserve-5947686d75-hhxpv"
Jul 20 18:42:19.479762 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:42:19.479604    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-0d90bc/sub-test-model-0d90bc-kserve-5947686d75-hhxpv" event={"ID":"158f7eaf-fb3d-4dc6-a140-544ef91c8e8c","Type":"ContainerDied","Data":"efe4dc83abc18192a523ec39c8255f43c58718af80d3583ceedb2f8ace686d51"}
Jul 20 18:42:19.479762 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:42:19.479652    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-0d90bc/sub-test-model-0d90bc-kserve-5947686d75-hhxpv" event={"ID":"158f7eaf-fb3d-4dc6-a140-544ef91c8e8c","Type":"ContainerDied","Data":"e3c0534af171407e82bdd1375532bdb1816460eed7abddc5e65a64ca2da36c1f"}
Jul 20 18:42:19.479762 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:42:19.479675    2569 scope.go:117] "RemoveContainer" containerID="efe4dc83abc18192a523ec39c8255f43c58718af80d3583ceedb2f8ace686d51"
Jul 20 18:42:19.483736 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:42:19.483690    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-94313f/sub-test-model-94313f-kserve-7bbbb7fd59-xdh64" podUID="d5ee57dc-1ad3-4c2e-a82f-5df0a0bd8536" containerName="main" containerID="cri-o://a9d5c5cc8f22242945d81d119b318c6bc0569c6f0a6d1f2a75773cb3826b3612" gracePeriod=45
Jul 20 18:42:19.488979 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:42:19.488961    2569 scope.go:117] "RemoveContainer" containerID="efe4dc83abc18192a523ec39c8255f43c58718af80d3583ceedb2f8ace686d51"
Jul 20 18:42:19.489248 ip-10-0-137-247 kubenswrapper[2569]: E0720 18:42:19.489229    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efe4dc83abc18192a523ec39c8255f43c58718af80d3583ceedb2f8ace686d51\": container with ID starting with efe4dc83abc18192a523ec39c8255f43c58718af80d3583ceedb2f8ace686d51 not found: ID does not exist" containerID="efe4dc83abc18192a523ec39c8255f43c58718af80d3583ceedb2f8ace686d51"
Jul 20 18:42:19.489322 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:42:19.489257    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efe4dc83abc18192a523ec39c8255f43c58718af80d3583ceedb2f8ace686d51"} err="failed to get container status \"efe4dc83abc18192a523ec39c8255f43c58718af80d3583ceedb2f8ace686d51\": rpc error: code = NotFound desc = could not find container \"efe4dc83abc18192a523ec39c8255f43c58718af80d3583ceedb2f8ace686d51\": container with ID starting with efe4dc83abc18192a523ec39c8255f43c58718af80d3583ceedb2f8ace686d51 not found: ID does not exist"
Jul 20 18:42:19.501542 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:42:19.501520    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-0d90bc/sub-test-model-0d90bc-kserve-5947686d75-hhxpv"]
Jul 20 18:42:19.505550 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:42:19.505526    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-0d90bc/sub-test-model-0d90bc-kserve-5947686d75-hhxpv"]
Jul 20 18:42:19.731772 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:42:19.731691    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-94313f/sub-test-model-94313f-kserve-7bbbb7fd59-xdh64"
Jul 20 18:42:19.751018 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:42:19.750982    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5ee57dc-1ad3-4c2e-a82f-5df0a0bd8536-tmp-dir\") pod \"d5ee57dc-1ad3-4c2e-a82f-5df0a0bd8536\" (UID: \"d5ee57dc-1ad3-4c2e-a82f-5df0a0bd8536\") "
Jul 20 18:42:19.751204 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:42:19.751082    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5ee57dc-1ad3-4c2e-a82f-5df0a0bd8536-dshm\") pod \"d5ee57dc-1ad3-4c2e-a82f-5df0a0bd8536\" (UID: \"d5ee57dc-1ad3-4c2e-a82f-5df0a0bd8536\") "
Jul 20 18:42:19.751204 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:42:19.751110    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5ee57dc-1ad3-4c2e-a82f-5df0a0bd8536-home\") pod \"d5ee57dc-1ad3-4c2e-a82f-5df0a0bd8536\" (UID: \"d5ee57dc-1ad3-4c2e-a82f-5df0a0bd8536\") "
Jul 20 18:42:19.751204 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:42:19.751143    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hn5zz\" (UniqueName: \"kubernetes.io/projected/d5ee57dc-1ad3-4c2e-a82f-5df0a0bd8536-kube-api-access-hn5zz\") pod \"d5ee57dc-1ad3-4c2e-a82f-5df0a0bd8536\" (UID: \"d5ee57dc-1ad3-4c2e-a82f-5df0a0bd8536\") "
Jul 20 18:42:19.751204 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:42:19.751166    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5ee57dc-1ad3-4c2e-a82f-5df0a0bd8536-model-cache\") pod \"d5ee57dc-1ad3-4c2e-a82f-5df0a0bd8536\" (UID: \"d5ee57dc-1ad3-4c2e-a82f-5df0a0bd8536\") "
Jul 20 18:42:19.751422 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:42:19.751242    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5ee57dc-1ad3-4c2e-a82f-5df0a0bd8536-tls-certs\") pod \"d5ee57dc-1ad3-4c2e-a82f-5df0a0bd8536\" (UID: \"d5ee57dc-1ad3-4c2e-a82f-5df0a0bd8536\") "
Jul 20 18:42:19.751422 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:42:19.751256    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5ee57dc-1ad3-4c2e-a82f-5df0a0bd8536-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d5ee57dc-1ad3-4c2e-a82f-5df0a0bd8536" (UID: "d5ee57dc-1ad3-4c2e-a82f-5df0a0bd8536"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 18:42:19.751422 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:42:19.751386    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5ee57dc-1ad3-4c2e-a82f-5df0a0bd8536-home" (OuterVolumeSpecName: "home") pod "d5ee57dc-1ad3-4c2e-a82f-5df0a0bd8536" (UID: "d5ee57dc-1ad3-4c2e-a82f-5df0a0bd8536"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 18:42:19.751587 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:42:19.751489    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5ee57dc-1ad3-4c2e-a82f-5df0a0bd8536-model-cache" (OuterVolumeSpecName: "model-cache") pod "d5ee57dc-1ad3-4c2e-a82f-5df0a0bd8536" (UID: "d5ee57dc-1ad3-4c2e-a82f-5df0a0bd8536"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 18:42:19.751587 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:42:19.751522    2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5ee57dc-1ad3-4c2e-a82f-5df0a0bd8536-tmp-dir\") on node \"ip-10-0-137-247.ec2.internal\" DevicePath \"\""
Jul 20 18:42:19.751587 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:42:19.751540    2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5ee57dc-1ad3-4c2e-a82f-5df0a0bd8536-home\") on node \"ip-10-0-137-247.ec2.internal\" DevicePath \"\""
Jul 20 18:42:19.753585 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:42:19.753552    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5ee57dc-1ad3-4c2e-a82f-5df0a0bd8536-dshm" (OuterVolumeSpecName: "dshm") pod "d5ee57dc-1ad3-4c2e-a82f-5df0a0bd8536" (UID: "d5ee57dc-1ad3-4c2e-a82f-5df0a0bd8536"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 18:42:19.753848 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:42:19.753823    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5ee57dc-1ad3-4c2e-a82f-5df0a0bd8536-kube-api-access-hn5zz" (OuterVolumeSpecName: "kube-api-access-hn5zz") pod "d5ee57dc-1ad3-4c2e-a82f-5df0a0bd8536" (UID: "d5ee57dc-1ad3-4c2e-a82f-5df0a0bd8536"). InnerVolumeSpecName "kube-api-access-hn5zz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 18:42:19.754048 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:42:19.754029    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5ee57dc-1ad3-4c2e-a82f-5df0a0bd8536-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d5ee57dc-1ad3-4c2e-a82f-5df0a0bd8536" (UID: "d5ee57dc-1ad3-4c2e-a82f-5df0a0bd8536"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 18:42:19.852389 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:42:19.852348    2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5ee57dc-1ad3-4c2e-a82f-5df0a0bd8536-dshm\") on node \"ip-10-0-137-247.ec2.internal\" DevicePath \"\""
Jul 20 18:42:19.852389 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:42:19.852385    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hn5zz\" (UniqueName: \"kubernetes.io/projected/d5ee57dc-1ad3-4c2e-a82f-5df0a0bd8536-kube-api-access-hn5zz\") on node \"ip-10-0-137-247.ec2.internal\" DevicePath \"\""
Jul 20 18:42:19.852389 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:42:19.852397    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5ee57dc-1ad3-4c2e-a82f-5df0a0bd8536-model-cache\") on node \"ip-10-0-137-247.ec2.internal\" DevicePath \"\""
Jul 20 18:42:19.852625 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:42:19.852408    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5ee57dc-1ad3-4c2e-a82f-5df0a0bd8536-tls-certs\") on node \"ip-10-0-137-247.ec2.internal\" DevicePath \"\""
Jul 20 18:42:20.182807 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:42:20.182776    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="158f7eaf-fb3d-4dc6-a140-544ef91c8e8c" path="/var/lib/kubelet/pods/158f7eaf-fb3d-4dc6-a140-544ef91c8e8c/volumes"
Jul 20 18:42:20.486214 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:42:20.486120    2569 generic.go:356] "Generic (PLEG): container finished" podID="d5ee57dc-1ad3-4c2e-a82f-5df0a0bd8536" containerID="a9d5c5cc8f22242945d81d119b318c6bc0569c6f0a6d1f2a75773cb3826b3612" exitCode=0
Jul 20 18:42:20.486214 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:42:20.486169    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-94313f/sub-test-model-94313f-kserve-7bbbb7fd59-xdh64"
Jul 20 18:42:20.486214 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:42:20.486206    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-94313f/sub-test-model-94313f-kserve-7bbbb7fd59-xdh64" event={"ID":"d5ee57dc-1ad3-4c2e-a82f-5df0a0bd8536","Type":"ContainerDied","Data":"a9d5c5cc8f22242945d81d119b318c6bc0569c6f0a6d1f2a75773cb3826b3612"}
Jul 20 18:42:20.486462 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:42:20.486248    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-94313f/sub-test-model-94313f-kserve-7bbbb7fd59-xdh64" event={"ID":"d5ee57dc-1ad3-4c2e-a82f-5df0a0bd8536","Type":"ContainerDied","Data":"48bd18e24dd9e18cdee38c48d47fe531fc5a421f9ace9b9f928958005dc84a0b"}
Jul 20 18:42:20.486462 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:42:20.486275    2569 scope.go:117] "RemoveContainer" containerID="a9d5c5cc8f22242945d81d119b318c6bc0569c6f0a6d1f2a75773cb3826b3612"
Jul 20 18:42:20.494574 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:42:20.494558    2569 scope.go:117] "RemoveContainer" containerID="a9d5c5cc8f22242945d81d119b318c6bc0569c6f0a6d1f2a75773cb3826b3612"
Jul 20 18:42:20.494834 ip-10-0-137-247 kubenswrapper[2569]: E0720 18:42:20.494815    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9d5c5cc8f22242945d81d119b318c6bc0569c6f0a6d1f2a75773cb3826b3612\": container with ID starting with a9d5c5cc8f22242945d81d119b318c6bc0569c6f0a6d1f2a75773cb3826b3612 not found: ID does not exist" containerID="a9d5c5cc8f22242945d81d119b318c6bc0569c6f0a6d1f2a75773cb3826b3612"
Jul 20 18:42:20.494954 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:42:20.494839    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9d5c5cc8f22242945d81d119b318c6bc0569c6f0a6d1f2a75773cb3826b3612"} err="failed to get container status \"a9d5c5cc8f22242945d81d119b318c6bc0569c6f0a6d1f2a75773cb3826b3612\": rpc error: code = NotFound desc = could not find container \"a9d5c5cc8f22242945d81d119b318c6bc0569c6f0a6d1f2a75773cb3826b3612\": container with ID starting with a9d5c5cc8f22242945d81d119b318c6bc0569c6f0a6d1f2a75773cb3826b3612 not found: ID does not exist"
Jul 20 18:42:20.501956 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:42:20.501935    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-94313f/sub-test-model-94313f-kserve-7bbbb7fd59-xdh64"]
Jul 20 18:42:20.505692 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:42:20.505669    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-94313f/sub-test-model-94313f-kserve-7bbbb7fd59-xdh64"]
Jul 20 18:42:22.182571 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:42:22.182539    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5ee57dc-1ad3-4c2e-a82f-5df0a0bd8536" path="/var/lib/kubelet/pods/d5ee57dc-1ad3-4c2e-a82f-5df0a0bd8536/volumes"
Jul 20 18:42:24.129321 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:42:24.129278    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-0d90bc/rate-test-model-0d90bc-kserve-6bb7655bc-5ssth"
Jul 20 18:42:24.129862 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:42:24.129336    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-0d90bc/rate-test-model-0d90bc-kserve-6bb7655bc-5ssth"
Jul 20 18:42:24.141881 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:42:24.141852    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-0d90bc/rate-test-model-0d90bc-kserve-6bb7655bc-5ssth"
Jul 20 18:42:24.515796 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:42:24.515767    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-0d90bc/rate-test-model-0d90bc-kserve-6bb7655bc-5ssth"
Jul 20 18:42:51.197922 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:42:51.197884    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-94313f/rate-test-model-94313f-kserve-857d4b979-thxrz"]
Jul 20 18:42:51.198450 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:42:51.198213    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5ee57dc-1ad3-4c2e-a82f-5df0a0bd8536" containerName="main"
Jul 20 18:42:51.198450 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:42:51.198226    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5ee57dc-1ad3-4c2e-a82f-5df0a0bd8536" containerName="main"
Jul 20 18:42:51.198450 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:42:51.198246    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="158f7eaf-fb3d-4dc6-a140-544ef91c8e8c" containerName="main"
Jul 20 18:42:51.198450 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:42:51.198251    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="158f7eaf-fb3d-4dc6-a140-544ef91c8e8c" containerName="main"
Jul 20 18:42:51.198450 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:42:51.198297    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="158f7eaf-fb3d-4dc6-a140-544ef91c8e8c" containerName="main"
Jul 20 18:42:51.198450 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:42:51.198306    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5ee57dc-1ad3-4c2e-a82f-5df0a0bd8536" containerName="main"
Jul 20 18:42:51.201239 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:42:51.201221    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-94313f/rate-test-model-94313f-kserve-857d4b979-thxrz"
Jul 20 18:42:51.204566 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:42:51.204540    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-94313f\"/\"openshift-service-ca.crt\""
Jul 20 18:42:51.204687 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:42:51.204567    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-94313f\"/\"rate-test-model-94313f-kserve-self-signed-certs\""
Jul 20 18:42:51.204687 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:42:51.204540    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-94313f\"/\"kube-root-ca.crt\""
Jul 20 18:42:51.204687 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:42:51.204575    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-94313f\"/\"default-dockercfg-xtr2f\""
Jul 20 18:42:51.212855 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:42:51.212832    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-94313f/rate-test-model-94313f-kserve-857d4b979-thxrz"]
Jul 20 18:42:51.304851 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:42:51.304813    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28b5d394-c440-48c8-9de8-759500cc96e2-home\") pod \"rate-test-model-94313f-kserve-857d4b979-thxrz\" (UID: \"28b5d394-c440-48c8-9de8-759500cc96e2\") " pod="ai-tenant-e2e-shared-b-94313f/rate-test-model-94313f-kserve-857d4b979-thxrz"
Jul 20 18:42:51.305031 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:42:51.304865    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28b5d394-c440-48c8-9de8-759500cc96e2-tmp-dir\") pod \"rate-test-model-94313f-kserve-857d4b979-thxrz\" (UID: \"28b5d394-c440-48c8-9de8-759500cc96e2\") " pod="ai-tenant-e2e-shared-b-94313f/rate-test-model-94313f-kserve-857d4b979-thxrz"
Jul 20 18:42:51.305031 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:42:51.304892    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28b5d394-c440-48c8-9de8-759500cc96e2-model-cache\") pod \"rate-test-model-94313f-kserve-857d4b979-thxrz\" (UID: \"28b5d394-c440-48c8-9de8-759500cc96e2\") " pod="ai-tenant-e2e-shared-b-94313f/rate-test-model-94313f-kserve-857d4b979-thxrz"
Jul 20 18:42:51.305031 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:42:51.304911    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28b5d394-c440-48c8-9de8-759500cc96e2-dshm\") pod \"rate-test-model-94313f-kserve-857d4b979-thxrz\" (UID: \"28b5d394-c440-48c8-9de8-759500cc96e2\") " pod="ai-tenant-e2e-shared-b-94313f/rate-test-model-94313f-kserve-857d4b979-thxrz"
Jul 20 18:42:51.305031 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:42:51.304933    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28b5d394-c440-48c8-9de8-759500cc96e2-tls-certs\") pod \"rate-test-model-94313f-kserve-857d4b979-thxrz\" (UID: \"28b5d394-c440-48c8-9de8-759500cc96e2\") " pod="ai-tenant-e2e-shared-b-94313f/rate-test-model-94313f-kserve-857d4b979-thxrz"
Jul 20 18:42:51.305031 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:42:51.304967    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srwnx\" (UniqueName: \"kubernetes.io/projected/28b5d394-c440-48c8-9de8-759500cc96e2-kube-api-access-srwnx\") pod \"rate-test-model-94313f-kserve-857d4b979-thxrz\" (UID: \"28b5d394-c440-48c8-9de8-759500cc96e2\") " pod="ai-tenant-e2e-shared-b-94313f/rate-test-model-94313f-kserve-857d4b979-thxrz"
Jul 20 18:42:51.405910 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:42:51.405877    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28b5d394-c440-48c8-9de8-759500cc96e2-home\") pod \"rate-test-model-94313f-kserve-857d4b979-thxrz\" (UID: \"28b5d394-c440-48c8-9de8-759500cc96e2\") " pod="ai-tenant-e2e-shared-b-94313f/rate-test-model-94313f-kserve-857d4b979-thxrz"
Jul 20 18:42:51.406094 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:42:51.405925    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28b5d394-c440-48c8-9de8-759500cc96e2-tmp-dir\") pod \"rate-test-model-94313f-kserve-857d4b979-thxrz\" (UID: \"28b5d394-c440-48c8-9de8-759500cc96e2\") " pod="ai-tenant-e2e-shared-b-94313f/rate-test-model-94313f-kserve-857d4b979-thxrz"
Jul 20 18:42:51.406094 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:42:51.405956    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28b5d394-c440-48c8-9de8-759500cc96e2-model-cache\") pod \"rate-test-model-94313f-kserve-857d4b979-thxrz\" (UID: \"28b5d394-c440-48c8-9de8-759500cc96e2\") " pod="ai-tenant-e2e-shared-b-94313f/rate-test-model-94313f-kserve-857d4b979-thxrz"
Jul 20 18:42:51.406094 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:42:51.405986    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28b5d394-c440-48c8-9de8-759500cc96e2-dshm\") pod \"rate-test-model-94313f-kserve-857d4b979-thxrz\" (UID: \"28b5d394-c440-48c8-9de8-759500cc96e2\") " pod="ai-tenant-e2e-shared-b-94313f/rate-test-model-94313f-kserve-857d4b979-thxrz"
Jul 20 18:42:51.406253 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:42:51.406135    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28b5d394-c440-48c8-9de8-759500cc96e2-tls-certs\") pod \"rate-test-model-94313f-kserve-857d4b979-thxrz\" (UID: \"28b5d394-c440-48c8-9de8-759500cc96e2\") " pod="ai-tenant-e2e-shared-b-94313f/rate-test-model-94313f-kserve-857d4b979-thxrz"
Jul 20 18:42:51.406253 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:42:51.406184    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srwnx\" (UniqueName: \"kubernetes.io/projected/28b5d394-c440-48c8-9de8-759500cc96e2-kube-api-access-srwnx\") pod \"rate-test-model-94313f-kserve-857d4b979-thxrz\" (UID: \"28b5d394-c440-48c8-9de8-759500cc96e2\") " pod="ai-tenant-e2e-shared-b-94313f/rate-test-model-94313f-kserve-857d4b979-thxrz"
Jul 20 18:42:51.406370 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:42:51.406323    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28b5d394-c440-48c8-9de8-759500cc96e2-home\") pod \"rate-test-model-94313f-kserve-857d4b979-thxrz\" (UID: \"28b5d394-c440-48c8-9de8-759500cc96e2\") " pod="ai-tenant-e2e-shared-b-94313f/rate-test-model-94313f-kserve-857d4b979-thxrz"
Jul 20 18:42:51.406427 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:42:51.406366    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28b5d394-c440-48c8-9de8-759500cc96e2-tmp-dir\") pod \"rate-test-model-94313f-kserve-857d4b979-thxrz\" (UID: \"28b5d394-c440-48c8-9de8-759500cc96e2\") " pod="ai-tenant-e2e-shared-b-94313f/rate-test-model-94313f-kserve-857d4b979-thxrz"
Jul 20 18:42:51.406480 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:42:51.406428    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28b5d394-c440-48c8-9de8-759500cc96e2-model-cache\") pod \"rate-test-model-94313f-kserve-857d4b979-thxrz\" (UID: \"28b5d394-c440-48c8-9de8-759500cc96e2\") " pod="ai-tenant-e2e-shared-b-94313f/rate-test-model-94313f-kserve-857d4b979-thxrz"
Jul 20 18:42:51.408180 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:42:51.408151    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28b5d394-c440-48c8-9de8-759500cc96e2-dshm\") pod \"rate-test-model-94313f-kserve-857d4b979-thxrz\" (UID: \"28b5d394-c440-48c8-9de8-759500cc96e2\") " pod="ai-tenant-e2e-shared-b-94313f/rate-test-model-94313f-kserve-857d4b979-thxrz"
Jul 20 18:42:51.408599 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:42:51.408581    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28b5d394-c440-48c8-9de8-759500cc96e2-tls-certs\") pod \"rate-test-model-94313f-kserve-857d4b979-thxrz\" (UID: \"28b5d394-c440-48c8-9de8-759500cc96e2\") " pod="ai-tenant-e2e-shared-b-94313f/rate-test-model-94313f-kserve-857d4b979-thxrz"
Jul 20 18:42:51.416305 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:42:51.416278    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-srwnx\" (UniqueName: \"kubernetes.io/projected/28b5d394-c440-48c8-9de8-759500cc96e2-kube-api-access-srwnx\") pod \"rate-test-model-94313f-kserve-857d4b979-thxrz\" (UID: \"28b5d394-c440-48c8-9de8-759500cc96e2\") " pod="ai-tenant-e2e-shared-b-94313f/rate-test-model-94313f-kserve-857d4b979-thxrz"
Jul 20 18:42:51.510169 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:42:51.510133    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-94313f/rate-test-model-94313f-kserve-857d4b979-thxrz"
Jul 20 18:42:51.640505 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:42:51.640470    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-94313f/rate-test-model-94313f-kserve-857d4b979-thxrz"]
Jul 20 18:42:51.643684 ip-10-0-137-247 kubenswrapper[2569]: W0720 18:42:51.643653    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod28b5d394_c440_48c8_9de8_759500cc96e2.slice/crio-5cd1de33a495c37632211253f95bd48d770a2e5ca67562a3ca14b7fe35f7b054 WatchSource:0}: Error finding container 5cd1de33a495c37632211253f95bd48d770a2e5ca67562a3ca14b7fe35f7b054: Status 404 returned error can't find the container with id 5cd1de33a495c37632211253f95bd48d770a2e5ca67562a3ca14b7fe35f7b054
Jul 20 18:42:52.635013 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:42:52.634974    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-94313f/rate-test-model-94313f-kserve-857d4b979-thxrz" event={"ID":"28b5d394-c440-48c8-9de8-759500cc96e2","Type":"ContainerStarted","Data":"95d60a8e87399e486e469eec5907f4996dffe5b78b8628773493c16da2efa268"}
Jul 20 18:42:52.635013 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:42:52.635019    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-94313f/rate-test-model-94313f-kserve-857d4b979-thxrz" event={"ID":"28b5d394-c440-48c8-9de8-759500cc96e2","Type":"ContainerStarted","Data":"5cd1de33a495c37632211253f95bd48d770a2e5ca67562a3ca14b7fe35f7b054"}
Jul 20 18:42:52.651671 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:42:52.651618    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-94313f/rate-test-model-94313f-kserve-857d4b979-thxrz" podStartSLOduration=1.6516010030000001 podStartE2EDuration="1.651601003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 18:42:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 18:42:52.651084309 +0000 UTC m=+2818.897617723" watchObservedRunningTime="2026-07-20 18:42:52.651601003 +0000 UTC m=+2818.898134403"
Jul 20 18:42:54.454228 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:42:54.454198    2569 scope.go:117] "RemoveContainer" containerID="983945607ee730c5f1e46e88786f76bb679306f9ea1137b220dcfcf999d4ccba"
Jul 20 18:42:54.462159 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:42:54.462140    2569 scope.go:117] "RemoveContainer" containerID="4b604dbc8465f47e413a71151bf45176f1bbd7042177f7e63cc033b5069220db"
Jul 20 18:42:54.945889 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:42:54.945851    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9thxb"]
Jul 20 18:43:01.511116 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:43:01.511072    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-94313f/rate-test-model-94313f-kserve-857d4b979-thxrz"
Jul 20 18:43:01.511116 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:43:01.511120    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-94313f/rate-test-model-94313f-kserve-857d4b979-thxrz"
Jul 20 18:43:01.523312 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:43:01.523280    2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-94313f/rate-test-model-94313f-kserve-857d4b979-thxrz"
Jul 20 18:43:01.689250 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:43:01.689212    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-94313f/rate-test-model-94313f-kserve-857d4b979-thxrz"
Jul 20 18:43:31.641035 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:43:31.641001    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9thxb"]
Jul 20 18:43:53.844784 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:43:53.844741    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9thxb"]
Jul 20 18:43:57.686380 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:43:57.686347    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-0d90bc/rate-test-model-0d90bc-kserve-6bb7655bc-5ssth"]
Jul 20 18:43:58.532361 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:43:58.532329    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-94313f/rate-test-model-94313f-kserve-857d4b979-thxrz"]
Jul 20 18:44:09.853246 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:09.853207    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9thxb"]
Jul 20 18:44:12.706099 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:12.706052    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-0d90bc/rate-test-model-0d90bc-kserve-6bb7655bc-5ssth" podUID="ce58a8db-d6fa-42e8-8682-89c3500fad7f" containerName="main" containerID="cri-o://f9e03d77b615bf260487e7630cd42bc89c54a9b606f8f353d383b9e1080d1178" gracePeriod=45
Jul 20 18:44:12.957262 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:12.957203    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-0d90bc/rate-test-model-0d90bc-kserve-6bb7655bc-5ssth"
Jul 20 18:44:12.982551 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:12.982518    2569 generic.go:356] "Generic (PLEG): container finished" podID="ce58a8db-d6fa-42e8-8682-89c3500fad7f" containerID="f9e03d77b615bf260487e7630cd42bc89c54a9b606f8f353d383b9e1080d1178" exitCode=0
Jul 20 18:44:12.982704 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:12.982578    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-0d90bc/rate-test-model-0d90bc-kserve-6bb7655bc-5ssth"
Jul 20 18:44:12.982704 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:12.982607    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-0d90bc/rate-test-model-0d90bc-kserve-6bb7655bc-5ssth" event={"ID":"ce58a8db-d6fa-42e8-8682-89c3500fad7f","Type":"ContainerDied","Data":"f9e03d77b615bf260487e7630cd42bc89c54a9b606f8f353d383b9e1080d1178"}
Jul 20 18:44:12.982704 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:12.982647    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-0d90bc/rate-test-model-0d90bc-kserve-6bb7655bc-5ssth" event={"ID":"ce58a8db-d6fa-42e8-8682-89c3500fad7f","Type":"ContainerDied","Data":"6bf21de87d8dccd157d4c0e34518ef61e96d14523905abd3902ce19efccc39ba"}
Jul 20 18:44:12.982704 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:12.982667    2569 scope.go:117] "RemoveContainer" containerID="f9e03d77b615bf260487e7630cd42bc89c54a9b606f8f353d383b9e1080d1178"
Jul 20 18:44:12.991758 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:12.991734    2569 scope.go:117] "RemoveContainer" containerID="f9e03d77b615bf260487e7630cd42bc89c54a9b606f8f353d383b9e1080d1178"
Jul 20 18:44:12.992051 ip-10-0-137-247 kubenswrapper[2569]: E0720 18:44:12.992032    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9e03d77b615bf260487e7630cd42bc89c54a9b606f8f353d383b9e1080d1178\": container with ID starting with f9e03d77b615bf260487e7630cd42bc89c54a9b606f8f353d383b9e1080d1178 not found: ID does not exist" containerID="f9e03d77b615bf260487e7630cd42bc89c54a9b606f8f353d383b9e1080d1178"
Jul 20 18:44:12.992126 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:12.992063    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9e03d77b615bf260487e7630cd42bc89c54a9b606f8f353d383b9e1080d1178"} err="failed to get container status \"f9e03d77b615bf260487e7630cd42bc89c54a9b606f8f353d383b9e1080d1178\": rpc error: code = NotFound desc = could not find container \"f9e03d77b615bf260487e7630cd42bc89c54a9b606f8f353d383b9e1080d1178\": container with ID starting with f9e03d77b615bf260487e7630cd42bc89c54a9b606f8f353d383b9e1080d1178 not found: ID does not exist"
Jul 20 18:44:13.002900 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:13.002875    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce58a8db-d6fa-42e8-8682-89c3500fad7f-tls-certs\") pod \"ce58a8db-d6fa-42e8-8682-89c3500fad7f\" (UID: \"ce58a8db-d6fa-42e8-8682-89c3500fad7f\") "
Jul 20 18:44:13.002990 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:13.002939    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce58a8db-d6fa-42e8-8682-89c3500fad7f-dshm\") pod \"ce58a8db-d6fa-42e8-8682-89c3500fad7f\" (UID: \"ce58a8db-d6fa-42e8-8682-89c3500fad7f\") "
Jul 20 18:44:13.002990 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:13.002960    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ggrhl\" (UniqueName: \"kubernetes.io/projected/ce58a8db-d6fa-42e8-8682-89c3500fad7f-kube-api-access-ggrhl\") pod \"ce58a8db-d6fa-42e8-8682-89c3500fad7f\" (UID: \"ce58a8db-d6fa-42e8-8682-89c3500fad7f\") "
Jul 20 18:44:13.003060 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:13.003009    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce58a8db-d6fa-42e8-8682-89c3500fad7f-home\") pod \"ce58a8db-d6fa-42e8-8682-89c3500fad7f\" (UID: \"ce58a8db-d6fa-42e8-8682-89c3500fad7f\") "
Jul 20 18:44:13.003060 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:13.003043    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce58a8db-d6fa-42e8-8682-89c3500fad7f-model-cache\") pod \"ce58a8db-d6fa-42e8-8682-89c3500fad7f\" (UID: \"ce58a8db-d6fa-42e8-8682-89c3500fad7f\") "
Jul 20 18:44:13.003130 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:13.003059    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce58a8db-d6fa-42e8-8682-89c3500fad7f-tmp-dir\") pod \"ce58a8db-d6fa-42e8-8682-89c3500fad7f\" (UID: \"ce58a8db-d6fa-42e8-8682-89c3500fad7f\") "
Jul 20 18:44:13.003321 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:13.003286    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce58a8db-d6fa-42e8-8682-89c3500fad7f-home" (OuterVolumeSpecName: "home") pod "ce58a8db-d6fa-42e8-8682-89c3500fad7f" (UID: "ce58a8db-d6fa-42e8-8682-89c3500fad7f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 18:44:13.003433 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:13.003324    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce58a8db-d6fa-42e8-8682-89c3500fad7f-model-cache" (OuterVolumeSpecName: "model-cache") pod "ce58a8db-d6fa-42e8-8682-89c3500fad7f" (UID: "ce58a8db-d6fa-42e8-8682-89c3500fad7f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 18:44:13.003433 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:13.003397    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce58a8db-d6fa-42e8-8682-89c3500fad7f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ce58a8db-d6fa-42e8-8682-89c3500fad7f" (UID: "ce58a8db-d6fa-42e8-8682-89c3500fad7f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 18:44:13.005085 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:13.005055    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce58a8db-d6fa-42e8-8682-89c3500fad7f-dshm" (OuterVolumeSpecName: "dshm") pod "ce58a8db-d6fa-42e8-8682-89c3500fad7f" (UID: "ce58a8db-d6fa-42e8-8682-89c3500fad7f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 18:44:13.005161 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:13.005104    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce58a8db-d6fa-42e8-8682-89c3500fad7f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ce58a8db-d6fa-42e8-8682-89c3500fad7f" (UID: "ce58a8db-d6fa-42e8-8682-89c3500fad7f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 18:44:13.005161 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:13.005104    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce58a8db-d6fa-42e8-8682-89c3500fad7f-kube-api-access-ggrhl" (OuterVolumeSpecName: "kube-api-access-ggrhl") pod "ce58a8db-d6fa-42e8-8682-89c3500fad7f" (UID: "ce58a8db-d6fa-42e8-8682-89c3500fad7f"). InnerVolumeSpecName "kube-api-access-ggrhl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 18:44:13.104534 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:13.104499    2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce58a8db-d6fa-42e8-8682-89c3500fad7f-dshm\") on node \"ip-10-0-137-247.ec2.internal\" DevicePath \"\""
Jul 20 18:44:13.104534 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:13.104527    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggrhl\" (UniqueName: \"kubernetes.io/projected/ce58a8db-d6fa-42e8-8682-89c3500fad7f-kube-api-access-ggrhl\") on node \"ip-10-0-137-247.ec2.internal\" DevicePath \"\""
Jul 20 18:44:13.104534 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:13.104537    2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce58a8db-d6fa-42e8-8682-89c3500fad7f-home\") on node \"ip-10-0-137-247.ec2.internal\" DevicePath \"\""
Jul 20 18:44:13.104534 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:13.104545    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce58a8db-d6fa-42e8-8682-89c3500fad7f-model-cache\") on node \"ip-10-0-137-247.ec2.internal\" DevicePath \"\""
Jul 20 18:44:13.104820 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:13.104554    2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce58a8db-d6fa-42e8-8682-89c3500fad7f-tmp-dir\") on node \"ip-10-0-137-247.ec2.internal\" DevicePath \"\""
Jul 20 18:44:13.104820 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:13.104562    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce58a8db-d6fa-42e8-8682-89c3500fad7f-tls-certs\") on node \"ip-10-0-137-247.ec2.internal\" DevicePath \"\""
Jul 20 18:44:13.306803 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:13.306770    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-0d90bc/rate-test-model-0d90bc-kserve-6bb7655bc-5ssth"]
Jul 20 18:44:13.311017 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:13.310989    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-0d90bc/rate-test-model-0d90bc-kserve-6bb7655bc-5ssth"]
Jul 20 18:44:13.552150 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:13.552033    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-94313f/rate-test-model-94313f-kserve-857d4b979-thxrz" podUID="28b5d394-c440-48c8-9de8-759500cc96e2" containerName="main" containerID="cri-o://95d60a8e87399e486e469eec5907f4996dffe5b78b8628773493c16da2efa268" gracePeriod=45
Jul 20 18:44:13.801205 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:13.801179    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-94313f/rate-test-model-94313f-kserve-857d4b979-thxrz"
Jul 20 18:44:13.912276 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:13.912188    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28b5d394-c440-48c8-9de8-759500cc96e2-tmp-dir\") pod \"28b5d394-c440-48c8-9de8-759500cc96e2\" (UID: \"28b5d394-c440-48c8-9de8-759500cc96e2\") "
Jul 20 18:44:13.912276 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:13.912236    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28b5d394-c440-48c8-9de8-759500cc96e2-tls-certs\") pod \"28b5d394-c440-48c8-9de8-759500cc96e2\" (UID: \"28b5d394-c440-48c8-9de8-759500cc96e2\") "
Jul 20 18:44:13.912276 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:13.912270    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28b5d394-c440-48c8-9de8-759500cc96e2-model-cache\") pod \"28b5d394-c440-48c8-9de8-759500cc96e2\" (UID: \"28b5d394-c440-48c8-9de8-759500cc96e2\") "
Jul 20 18:44:13.912543 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:13.912288    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28b5d394-c440-48c8-9de8-759500cc96e2-dshm\") pod \"28b5d394-c440-48c8-9de8-759500cc96e2\" (UID: \"28b5d394-c440-48c8-9de8-759500cc96e2\") "
Jul 20 18:44:13.912543 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:13.912325    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-srwnx\" (UniqueName: \"kubernetes.io/projected/28b5d394-c440-48c8-9de8-759500cc96e2-kube-api-access-srwnx\") pod \"28b5d394-c440-48c8-9de8-759500cc96e2\" (UID: \"28b5d394-c440-48c8-9de8-759500cc96e2\") "
Jul 20 18:44:13.912543 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:13.912353    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28b5d394-c440-48c8-9de8-759500cc96e2-home\") pod \"28b5d394-c440-48c8-9de8-759500cc96e2\" (UID: \"28b5d394-c440-48c8-9de8-759500cc96e2\") "
Jul 20 18:44:13.912543 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:13.912450    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28b5d394-c440-48c8-9de8-759500cc96e2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "28b5d394-c440-48c8-9de8-759500cc96e2" (UID: "28b5d394-c440-48c8-9de8-759500cc96e2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 18:44:13.912543 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:13.912514    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28b5d394-c440-48c8-9de8-759500cc96e2-model-cache" (OuterVolumeSpecName: "model-cache") pod "28b5d394-c440-48c8-9de8-759500cc96e2" (UID: "28b5d394-c440-48c8-9de8-759500cc96e2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 18:44:13.912817 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:13.912659    2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28b5d394-c440-48c8-9de8-759500cc96e2-tmp-dir\") on node \"ip-10-0-137-247.ec2.internal\" DevicePath \"\""
Jul 20 18:44:13.912817 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:13.912675    2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28b5d394-c440-48c8-9de8-759500cc96e2-model-cache\") on node \"ip-10-0-137-247.ec2.internal\" DevicePath \"\""
Jul 20 18:44:13.912817 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:13.912691    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28b5d394-c440-48c8-9de8-759500cc96e2-home" (OuterVolumeSpecName: "home") pod "28b5d394-c440-48c8-9de8-759500cc96e2" (UID: "28b5d394-c440-48c8-9de8-759500cc96e2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 18:44:13.914478 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:13.914447    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28b5d394-c440-48c8-9de8-759500cc96e2-dshm" (OuterVolumeSpecName: "dshm") pod "28b5d394-c440-48c8-9de8-759500cc96e2" (UID: "28b5d394-c440-48c8-9de8-759500cc96e2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 18:44:13.914478 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:13.914447    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28b5d394-c440-48c8-9de8-759500cc96e2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "28b5d394-c440-48c8-9de8-759500cc96e2" (UID: "28b5d394-c440-48c8-9de8-759500cc96e2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 18:44:13.914622 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:13.914508    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28b5d394-c440-48c8-9de8-759500cc96e2-kube-api-access-srwnx" (OuterVolumeSpecName: "kube-api-access-srwnx") pod "28b5d394-c440-48c8-9de8-759500cc96e2" (UID: "28b5d394-c440-48c8-9de8-759500cc96e2"). InnerVolumeSpecName "kube-api-access-srwnx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 18:44:13.987605 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:13.987571    2569 generic.go:356] "Generic (PLEG): container finished" podID="28b5d394-c440-48c8-9de8-759500cc96e2" containerID="95d60a8e87399e486e469eec5907f4996dffe5b78b8628773493c16da2efa268" exitCode=0
Jul 20 18:44:13.987808 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:13.987614    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-94313f/rate-test-model-94313f-kserve-857d4b979-thxrz"
Jul 20 18:44:13.987808 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:13.987651    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-94313f/rate-test-model-94313f-kserve-857d4b979-thxrz" event={"ID":"28b5d394-c440-48c8-9de8-759500cc96e2","Type":"ContainerDied","Data":"95d60a8e87399e486e469eec5907f4996dffe5b78b8628773493c16da2efa268"}
Jul 20 18:44:13.987808 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:13.987701    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-94313f/rate-test-model-94313f-kserve-857d4b979-thxrz" event={"ID":"28b5d394-c440-48c8-9de8-759500cc96e2","Type":"ContainerDied","Data":"5cd1de33a495c37632211253f95bd48d770a2e5ca67562a3ca14b7fe35f7b054"}
Jul 20 18:44:13.987808 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:13.987735    2569 scope.go:117] "RemoveContainer" containerID="95d60a8e87399e486e469eec5907f4996dffe5b78b8628773493c16da2efa268"
Jul 20 18:44:13.996496 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:13.996475    2569 scope.go:117] "RemoveContainer" containerID="95d60a8e87399e486e469eec5907f4996dffe5b78b8628773493c16da2efa268"
Jul 20 18:44:13.996780 ip-10-0-137-247 kubenswrapper[2569]: E0720 18:44:13.996753    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95d60a8e87399e486e469eec5907f4996dffe5b78b8628773493c16da2efa268\": container with ID starting with 95d60a8e87399e486e469eec5907f4996dffe5b78b8628773493c16da2efa268 not found: ID does not exist" containerID="95d60a8e87399e486e469eec5907f4996dffe5b78b8628773493c16da2efa268"
Jul 20 18:44:13.996876 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:13.996783    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95d60a8e87399e486e469eec5907f4996dffe5b78b8628773493c16da2efa268"} err="failed to get container status \"95d60a8e87399e486e469eec5907f4996dffe5b78b8628773493c16da2efa268\": rpc error: code = NotFound desc = could not find container \"95d60a8e87399e486e469eec5907f4996dffe5b78b8628773493c16da2efa268\": container with ID starting with 95d60a8e87399e486e469eec5907f4996dffe5b78b8628773493c16da2efa268 not found: ID does not exist"
Jul 20 18:44:14.012548 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:14.012519    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-94313f/rate-test-model-94313f-kserve-857d4b979-thxrz"]
Jul 20 18:44:14.013295 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:14.013273    2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28b5d394-c440-48c8-9de8-759500cc96e2-tls-certs\") on node \"ip-10-0-137-247.ec2.internal\" DevicePath \"\""
Jul 20 18:44:14.013295 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:14.013294    2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28b5d394-c440-48c8-9de8-759500cc96e2-dshm\") on node \"ip-10-0-137-247.ec2.internal\" DevicePath \"\""
Jul 20 18:44:14.013427 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:14.013304    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srwnx\" (UniqueName: \"kubernetes.io/projected/28b5d394-c440-48c8-9de8-759500cc96e2-kube-api-access-srwnx\") on node \"ip-10-0-137-247.ec2.internal\" DevicePath \"\""
Jul 20 18:44:14.013427 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:14.013312    2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28b5d394-c440-48c8-9de8-759500cc96e2-home\") on node \"ip-10-0-137-247.ec2.internal\" DevicePath \"\""
Jul 20 18:44:14.017212 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:14.017188    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-94313f/rate-test-model-94313f-kserve-857d4b979-thxrz"]
Jul 20 18:44:14.187071 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:14.186991    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="28b5d394-c440-48c8-9de8-759500cc96e2" path="/var/lib/kubelet/pods/28b5d394-c440-48c8-9de8-759500cc96e2/volumes"
Jul 20 18:44:14.187380 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:14.187368    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce58a8db-d6fa-42e8-8682-89c3500fad7f" path="/var/lib/kubelet/pods/ce58a8db-d6fa-42e8-8682-89c3500fad7f/volumes"
Jul 20 18:44:28.309406 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:28.309371    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-0d90bc-68488d67f6-qqgsn"]
Jul 20 18:44:28.309934 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:28.309632    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-0d90bc-68488d67f6-qqgsn" podUID="79a8058a-bf96-44bb-a224-aa87aa6704f6" containerName="maas-api" containerID="cri-o://9942b5d433682ec540521a26d8c22a22b42ec4b0c418b340a49bd5235bee7ed6" gracePeriod=30
Jul 20 18:44:28.551154 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:28.551125    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-0d90bc-68488d67f6-qqgsn"
Jul 20 18:44:28.635408 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:28.635313    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79a8058a-bf96-44bb-a224-aa87aa6704f6-maas-api-tls\") pod \"79a8058a-bf96-44bb-a224-aa87aa6704f6\" (UID: \"79a8058a-bf96-44bb-a224-aa87aa6704f6\") "
Jul 20 18:44:28.635408 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:28.635399    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79a8058a-bf96-44bb-a224-aa87aa6704f6-openshift-service-ca\") pod \"79a8058a-bf96-44bb-a224-aa87aa6704f6\" (UID: \"79a8058a-bf96-44bb-a224-aa87aa6704f6\") "
Jul 20 18:44:28.635643 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:28.635428    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-86hrh\" (UniqueName: \"kubernetes.io/projected/79a8058a-bf96-44bb-a224-aa87aa6704f6-kube-api-access-86hrh\") pod \"79a8058a-bf96-44bb-a224-aa87aa6704f6\" (UID: \"79a8058a-bf96-44bb-a224-aa87aa6704f6\") "
Jul 20 18:44:28.635776 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:28.635744    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79a8058a-bf96-44bb-a224-aa87aa6704f6-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "79a8058a-bf96-44bb-a224-aa87aa6704f6" (UID: "79a8058a-bf96-44bb-a224-aa87aa6704f6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 18:44:28.637519 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:28.637486    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79a8058a-bf96-44bb-a224-aa87aa6704f6-kube-api-access-86hrh" (OuterVolumeSpecName: "kube-api-access-86hrh") pod "79a8058a-bf96-44bb-a224-aa87aa6704f6" (UID: "79a8058a-bf96-44bb-a224-aa87aa6704f6"). InnerVolumeSpecName "kube-api-access-86hrh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 18:44:28.637519 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:28.637503    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79a8058a-bf96-44bb-a224-aa87aa6704f6-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "79a8058a-bf96-44bb-a224-aa87aa6704f6" (UID: "79a8058a-bf96-44bb-a224-aa87aa6704f6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 18:44:28.736568 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:28.736533    2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79a8058a-bf96-44bb-a224-aa87aa6704f6-openshift-service-ca\") on node \"ip-10-0-137-247.ec2.internal\" DevicePath \"\""
Jul 20 18:44:28.736568 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:28.736562    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86hrh\" (UniqueName: \"kubernetes.io/projected/79a8058a-bf96-44bb-a224-aa87aa6704f6-kube-api-access-86hrh\") on node \"ip-10-0-137-247.ec2.internal\" DevicePath \"\""
Jul 20 18:44:28.736568 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:28.736572    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79a8058a-bf96-44bb-a224-aa87aa6704f6-maas-api-tls\") on node \"ip-10-0-137-247.ec2.internal\" DevicePath \"\""
Jul 20 18:44:29.053684 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:29.053651    2569 generic.go:356] "Generic (PLEG): container finished" podID="79a8058a-bf96-44bb-a224-aa87aa6704f6" containerID="9942b5d433682ec540521a26d8c22a22b42ec4b0c418b340a49bd5235bee7ed6" exitCode=0
Jul 20 18:44:29.053896 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:29.053696    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-0d90bc-68488d67f6-qqgsn"
Jul 20 18:44:29.053896 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:29.053699    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-0d90bc-68488d67f6-qqgsn" event={"ID":"79a8058a-bf96-44bb-a224-aa87aa6704f6","Type":"ContainerDied","Data":"9942b5d433682ec540521a26d8c22a22b42ec4b0c418b340a49bd5235bee7ed6"}
Jul 20 18:44:29.053896 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:29.053791    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-0d90bc-68488d67f6-qqgsn" event={"ID":"79a8058a-bf96-44bb-a224-aa87aa6704f6","Type":"ContainerDied","Data":"97dd289789533ec2d20ad1e8adad0c506e11426db8b5495b153707a38e313ea0"}
Jul 20 18:44:29.053896 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:29.053821    2569 scope.go:117] "RemoveContainer" containerID="9942b5d433682ec540521a26d8c22a22b42ec4b0c418b340a49bd5235bee7ed6"
Jul 20 18:44:29.064712 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:29.064681    2569 scope.go:117] "RemoveContainer" containerID="9942b5d433682ec540521a26d8c22a22b42ec4b0c418b340a49bd5235bee7ed6"
Jul 20 18:44:29.065465 ip-10-0-137-247 kubenswrapper[2569]: E0720 18:44:29.065173    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9942b5d433682ec540521a26d8c22a22b42ec4b0c418b340a49bd5235bee7ed6\": container with ID starting with 9942b5d433682ec540521a26d8c22a22b42ec4b0c418b340a49bd5235bee7ed6 not found: ID does not exist" containerID="9942b5d433682ec540521a26d8c22a22b42ec4b0c418b340a49bd5235bee7ed6"
Jul 20 18:44:29.065694 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:29.065479    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9942b5d433682ec540521a26d8c22a22b42ec4b0c418b340a49bd5235bee7ed6"} err="failed to get container status \"9942b5d433682ec540521a26d8c22a22b42ec4b0c418b340a49bd5235bee7ed6\": rpc error: code = NotFound desc = could not find container \"9942b5d433682ec540521a26d8c22a22b42ec4b0c418b340a49bd5235bee7ed6\": container with ID starting with 9942b5d433682ec540521a26d8c22a22b42ec4b0c418b340a49bd5235bee7ed6 not found: ID does not exist"
Jul 20 18:44:29.091616 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:29.091572    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-0d90bc-68488d67f6-qqgsn"]
Jul 20 18:44:29.094627 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:29.094597    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-0d90bc-68488d67f6-qqgsn"]
Jul 20 18:44:30.182536 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:30.182502    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79a8058a-bf96-44bb-a224-aa87aa6704f6" path="/var/lib/kubelet/pods/79a8058a-bf96-44bb-a224-aa87aa6704f6/volumes"
Jul 20 18:44:31.560646 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:31.560610    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9thxb"]
Jul 20 18:44:39.864545 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:39.864505    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-94313f-468pf"]
Jul 20 18:44:39.864990 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:39.864815    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce58a8db-d6fa-42e8-8682-89c3500fad7f" containerName="main"
Jul 20 18:44:39.864990 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:39.864828    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce58a8db-d6fa-42e8-8682-89c3500fad7f" containerName="main"
Jul 20 18:44:39.864990 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:39.864839    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79a8058a-bf96-44bb-a224-aa87aa6704f6" containerName="maas-api"
Jul 20 18:44:39.864990 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:39.864845    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="79a8058a-bf96-44bb-a224-aa87aa6704f6" containerName="maas-api"
Jul 20 18:44:39.864990 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:39.864858    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28b5d394-c440-48c8-9de8-759500cc96e2" containerName="main"
Jul 20 18:44:39.864990 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:39.864863    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="28b5d394-c440-48c8-9de8-759500cc96e2" containerName="main"
Jul 20 18:44:39.864990 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:39.864914    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="28b5d394-c440-48c8-9de8-759500cc96e2" containerName="main"
Jul 20 18:44:39.864990 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:39.864922    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="79a8058a-bf96-44bb-a224-aa87aa6704f6" containerName="maas-api"
Jul 20 18:44:39.864990 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:39.864929    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce58a8db-d6fa-42e8-8682-89c3500fad7f" containerName="main"
Jul 20 18:44:39.867717 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:39.867701    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-94313f-468pf"
Jul 20 18:44:39.870288 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:39.870266    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fk7rm\""
Jul 20 18:44:39.874906 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:39.874688    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-94313f-468pf"]
Jul 20 18:44:40.026480 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:40.026434    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3f038bf-333d-4ea0-bad8-a6c590aa54ea-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-94313f-468pf\" (UID: \"b3f038bf-333d-4ea0-bad8-a6c590aa54ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-94313f-468pf"
Jul 20 18:44:40.127018 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:40.126926    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3f038bf-333d-4ea0-bad8-a6c590aa54ea-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-94313f-468pf\" (UID: \"b3f038bf-333d-4ea0-bad8-a6c590aa54ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-94313f-468pf"
Jul 20 18:44:40.127514 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:40.127495    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3f038bf-333d-4ea0-bad8-a6c590aa54ea-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-94313f-468pf\" (UID: \"b3f038bf-333d-4ea0-bad8-a6c590aa54ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-94313f-468pf"
Jul 20 18:44:40.179180 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:40.179148    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-94313f-468pf"
Jul 20 18:44:40.305630 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:40.305598    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-94313f-468pf"]
Jul 20 18:44:40.308620 ip-10-0-137-247 kubenswrapper[2569]: W0720 18:44:40.308596    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3f038bf_333d_4ea0_bad8_a6c590aa54ea.slice/crio-b2fe941f94358ae4876df6cc557fa8b005c43c50ec53c6d3058ff9e036e3a0e5 WatchSource:0}: Error finding container b2fe941f94358ae4876df6cc557fa8b005c43c50ec53c6d3058ff9e036e3a0e5: Status 404 returned error can't find the container with id b2fe941f94358ae4876df6cc557fa8b005c43c50ec53c6d3058ff9e036e3a0e5
Jul 20 18:44:41.109564 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:41.109522    2569 generic.go:356] "Generic (PLEG): container finished" podID="b3f038bf-333d-4ea0-bad8-a6c590aa54ea" containerID="7f36a5e1c6499f18ea592d76ee32dcb7841dede88a63e401f8e98cb48dcd7218" exitCode=0
Jul 20 18:44:41.110049 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:41.109577    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-94313f-468pf" event={"ID":"b3f038bf-333d-4ea0-bad8-a6c590aa54ea","Type":"ContainerDied","Data":"7f36a5e1c6499f18ea592d76ee32dcb7841dede88a63e401f8e98cb48dcd7218"}
Jul 20 18:44:41.110049 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:41.109680    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-94313f-468pf" event={"ID":"b3f038bf-333d-4ea0-bad8-a6c590aa54ea","Type":"ContainerStarted","Data":"b2fe941f94358ae4876df6cc557fa8b005c43c50ec53c6d3058ff9e036e3a0e5"}
Jul 20 18:44:42.238560 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:42.238536    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-94313f-468pf"
Jul 20 18:44:42.346780 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:42.346752    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3f038bf-333d-4ea0-bad8-a6c590aa54ea-maas-api-service-ca\") pod \"b3f038bf-333d-4ea0-bad8-a6c590aa54ea\" (UID: \"b3f038bf-333d-4ea0-bad8-a6c590aa54ea\") "
Jul 20 18:44:42.347106 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:42.347081    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3f038bf-333d-4ea0-bad8-a6c590aa54ea-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b3f038bf-333d-4ea0-bad8-a6c590aa54ea" (UID: "b3f038bf-333d-4ea0-bad8-a6c590aa54ea"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 18:44:42.448278 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:42.448206    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3f038bf-333d-4ea0-bad8-a6c590aa54ea-maas-api-service-ca\") on node \"ip-10-0-137-247.ec2.internal\" DevicePath \"\""
Jul 20 18:44:43.120022 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:43.119989    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-94313f-468pf"
Jul 20 18:44:43.120193 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:43.119993    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-94313f-468pf" event={"ID":"b3f038bf-333d-4ea0-bad8-a6c590aa54ea","Type":"ContainerDied","Data":"b2fe941f94358ae4876df6cc557fa8b005c43c50ec53c6d3058ff9e036e3a0e5"}
Jul 20 18:44:43.120193 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:43.120109    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b2fe941f94358ae4876df6cc557fa8b005c43c50ec53c6d3058ff9e036e3a0e5"
Jul 20 18:44:50.327009 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:50.326973    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-94313f-757bb7777-x4ldz"]
Jul 20 18:44:50.327390 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:50.327215    2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-94313f-757bb7777-x4ldz" podUID="54d351ad-78f0-4f91-a2a2-005d9f60030c" containerName="maas-api" containerID="cri-o://397fc96245fbbba2c7a03e659b62d15371d3345c3f81708d63ecf662d52de1e5" gracePeriod=30
Jul 20 18:44:50.562918 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:50.562891    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-94313f-757bb7777-x4ldz"
Jul 20 18:44:50.715559 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:50.715476    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-98rrl\" (UniqueName: \"kubernetes.io/projected/54d351ad-78f0-4f91-a2a2-005d9f60030c-kube-api-access-98rrl\") pod \"54d351ad-78f0-4f91-a2a2-005d9f60030c\" (UID: \"54d351ad-78f0-4f91-a2a2-005d9f60030c\") "
Jul 20 18:44:50.715559 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:50.715519    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54d351ad-78f0-4f91-a2a2-005d9f60030c-maas-api-tls\") pod \"54d351ad-78f0-4f91-a2a2-005d9f60030c\" (UID: \"54d351ad-78f0-4f91-a2a2-005d9f60030c\") "
Jul 20 18:44:50.715559 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:50.715551    2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54d351ad-78f0-4f91-a2a2-005d9f60030c-openshift-service-ca\") pod \"54d351ad-78f0-4f91-a2a2-005d9f60030c\" (UID: \"54d351ad-78f0-4f91-a2a2-005d9f60030c\") "
Jul 20 18:44:50.715979 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:50.715946    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54d351ad-78f0-4f91-a2a2-005d9f60030c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "54d351ad-78f0-4f91-a2a2-005d9f60030c" (UID: "54d351ad-78f0-4f91-a2a2-005d9f60030c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 18:44:50.717671 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:50.717644    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54d351ad-78f0-4f91-a2a2-005d9f60030c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "54d351ad-78f0-4f91-a2a2-005d9f60030c" (UID: "54d351ad-78f0-4f91-a2a2-005d9f60030c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 18:44:50.717671 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:50.717644    2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54d351ad-78f0-4f91-a2a2-005d9f60030c-kube-api-access-98rrl" (OuterVolumeSpecName: "kube-api-access-98rrl") pod "54d351ad-78f0-4f91-a2a2-005d9f60030c" (UID: "54d351ad-78f0-4f91-a2a2-005d9f60030c"). InnerVolumeSpecName "kube-api-access-98rrl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 18:44:50.816942 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:50.816903    2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98rrl\" (UniqueName: \"kubernetes.io/projected/54d351ad-78f0-4f91-a2a2-005d9f60030c-kube-api-access-98rrl\") on node \"ip-10-0-137-247.ec2.internal\" DevicePath \"\""
Jul 20 18:44:50.816942 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:50.816937    2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54d351ad-78f0-4f91-a2a2-005d9f60030c-maas-api-tls\") on node \"ip-10-0-137-247.ec2.internal\" DevicePath \"\""
Jul 20 18:44:50.816942 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:50.816947    2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54d351ad-78f0-4f91-a2a2-005d9f60030c-openshift-service-ca\") on node \"ip-10-0-137-247.ec2.internal\" DevicePath \"\""
Jul 20 18:44:51.159164 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:51.159124    2569 generic.go:356] "Generic (PLEG): container finished" podID="54d351ad-78f0-4f91-a2a2-005d9f60030c" containerID="397fc96245fbbba2c7a03e659b62d15371d3345c3f81708d63ecf662d52de1e5" exitCode=0
Jul 20 18:44:51.159333 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:51.159172    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-94313f-757bb7777-x4ldz"
Jul 20 18:44:51.159333 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:51.159212    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-94313f-757bb7777-x4ldz" event={"ID":"54d351ad-78f0-4f91-a2a2-005d9f60030c","Type":"ContainerDied","Data":"397fc96245fbbba2c7a03e659b62d15371d3345c3f81708d63ecf662d52de1e5"}
Jul 20 18:44:51.159333 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:51.159267    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-94313f-757bb7777-x4ldz" event={"ID":"54d351ad-78f0-4f91-a2a2-005d9f60030c","Type":"ContainerDied","Data":"9e307afd5e4e93b2bf3b224d590679156d586fd78d70e8489c459a6520b4dd1b"}
Jul 20 18:44:51.159333 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:51.159291    2569 scope.go:117] "RemoveContainer" containerID="397fc96245fbbba2c7a03e659b62d15371d3345c3f81708d63ecf662d52de1e5"
Jul 20 18:44:51.167760 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:51.167717    2569 scope.go:117] "RemoveContainer" containerID="397fc96245fbbba2c7a03e659b62d15371d3345c3f81708d63ecf662d52de1e5"
Jul 20 18:44:51.168019 ip-10-0-137-247 kubenswrapper[2569]: E0720 18:44:51.167997    2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"397fc96245fbbba2c7a03e659b62d15371d3345c3f81708d63ecf662d52de1e5\": container with ID starting with 397fc96245fbbba2c7a03e659b62d15371d3345c3f81708d63ecf662d52de1e5 not found: ID does not exist" containerID="397fc96245fbbba2c7a03e659b62d15371d3345c3f81708d63ecf662d52de1e5"
Jul 20 18:44:51.168093 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:51.168030    2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"397fc96245fbbba2c7a03e659b62d15371d3345c3f81708d63ecf662d52de1e5"} err="failed to get container status \"397fc96245fbbba2c7a03e659b62d15371d3345c3f81708d63ecf662d52de1e5\": rpc error: code = NotFound desc = could not find container \"397fc96245fbbba2c7a03e659b62d15371d3345c3f81708d63ecf662d52de1e5\": container with ID starting with 397fc96245fbbba2c7a03e659b62d15371d3345c3f81708d63ecf662d52de1e5 not found: ID does not exist"
Jul 20 18:44:51.182318 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:51.182296    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-94313f-757bb7777-x4ldz"]
Jul 20 18:44:51.189602 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:51.189583    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-94313f-757bb7777-x4ldz"]
Jul 20 18:44:52.181545 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:52.181510    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="54d351ad-78f0-4f91-a2a2-005d9f60030c" path="/var/lib/kubelet/pods/54d351ad-78f0-4f91-a2a2-005d9f60030c/volumes"
Jul 20 18:44:53.447240 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:53.447199    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-9thxb"]
Jul 20 18:44:54.972068 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:54.972033    2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-94313f-468pf"]
Jul 20 18:44:54.975518 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:54.975491    2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-94313f-468pf"]
Jul 20 18:44:56.183099 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:44:56.183058    2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b3f038bf-333d-4ea0-bad8-a6c590aa54ea" path="/var/lib/kubelet/pods/b3f038bf-333d-4ea0-bad8-a6c590aa54ea/volumes"
Jul 20 18:45:00.164453 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:45:00.164418    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29742885-8c8p4"]
Jul 20 18:45:00.164920 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:45:00.164703    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54d351ad-78f0-4f91-a2a2-005d9f60030c" containerName="maas-api"
Jul 20 18:45:00.164920 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:45:00.164714    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="54d351ad-78f0-4f91-a2a2-005d9f60030c" containerName="maas-api"
Jul 20 18:45:00.164920 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:45:00.164749    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3f038bf-333d-4ea0-bad8-a6c590aa54ea" containerName="revoke-keys"
Jul 20 18:45:00.164920 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:45:00.164755    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3f038bf-333d-4ea0-bad8-a6c590aa54ea" containerName="revoke-keys"
Jul 20 18:45:00.164920 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:45:00.164816    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3f038bf-333d-4ea0-bad8-a6c590aa54ea" containerName="revoke-keys"
Jul 20 18:45:00.164920 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:45:00.164828    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="54d351ad-78f0-4f91-a2a2-005d9f60030c" containerName="maas-api"
Jul 20 18:45:00.169285 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:45:00.169268    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29742885-8c8p4"
Jul 20 18:45:00.172033 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:45:00.172012    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fk7rm\""
Jul 20 18:45:00.179858 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:45:00.179840    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29742885-8c8p4"
Jul 20 18:45:00.186284 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:45:00.186257    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29742885-8c8p4"]
Jul 20 18:45:00.315807 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:45:00.315711    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29742885-8c8p4"]
Jul 20 18:45:00.317986 ip-10-0-137-247 kubenswrapper[2569]: W0720 18:45:00.317961    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a2f6174_6f3a_420d_ba49_679ccccbfb86.slice/crio-52c5644ab3e1442160a422bc92b1ec2143694a3028b999e37bf30d4c95f507c8 WatchSource:0}: Error finding container 52c5644ab3e1442160a422bc92b1ec2143694a3028b999e37bf30d4c95f507c8: Status 404 returned error can't find the container with id 52c5644ab3e1442160a422bc92b1ec2143694a3028b999e37bf30d4c95f507c8
Jul 20 18:45:01.206041 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:45:01.206003    2569 generic.go:356] "Generic (PLEG): container finished" podID="4a2f6174-6f3a-420d-ba49-679ccccbfb86" containerID="7218ef8ccd425dc15cfd9b4561810d6c06542e2d21c53262f64fb4a19e63fea9" exitCode=0
Jul 20 18:45:01.206456 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:45:01.206048    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29742885-8c8p4" event={"ID":"4a2f6174-6f3a-420d-ba49-679ccccbfb86","Type":"ContainerDied","Data":"7218ef8ccd425dc15cfd9b4561810d6c06542e2d21c53262f64fb4a19e63fea9"}
Jul 20 18:45:01.206456 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:45:01.206104    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29742885-8c8p4" event={"ID":"4a2f6174-6f3a-420d-ba49-679ccccbfb86","Type":"ContainerStarted","Data":"52c5644ab3e1442160a422bc92b1ec2143694a3028b999e37bf30d4c95f507c8"}
Jul 20 18:45:02.333287 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:45:02.333263    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29742885-8c8p4"
Jul 20 18:45:03.214645 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:45:03.214614    2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29742885-8c8p4"
Jul 20 18:45:03.214846 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:45:03.214616    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29742885-8c8p4" event={"ID":"4a2f6174-6f3a-420d-ba49-679ccccbfb86","Type":"ContainerDied","Data":"52c5644ab3e1442160a422bc92b1ec2143694a3028b999e37bf30d4c95f507c8"}
Jul 20 18:45:03.214846 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:45:03.214757    2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="52c5644ab3e1442160a422bc92b1ec2143694a3028b999e37bf30d4c95f507c8"
Jul 20 18:45:10.304190 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:45:10.304157    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7bd97c8594-qntzv_9b8f6824-4ca6-4b50-abcd-55073c9dd002/manager/0.log"
Jul 20 18:45:10.685068 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:45:10.684983    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-79747b4d4c-cks45_6caac6ef-ec22-491d-8b82-41665d8ca54c/copy-manifests/0.log"
Jul 20 18:45:10.703820 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:45:10.703794    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-79747b4d4c-cks45_6caac6ef-ec22-491d-8b82-41665d8ca54c/manager/0.log"
Jul 20 18:45:11.556166 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:45:11.556133    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29742855-gfpqp_6bf83091-d9ec-48ad-af80-a60a764d3eef/cleanup/0.log"
Jul 20 18:45:11.690114 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:45:11.690080    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29742870-nbv4c_74615b62-982c-4311-aa1e-8740a61182aa/cleanup/0.log"
Jul 20 18:45:11.813218 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:45:11.813138    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29742885-8c8p4_4a2f6174-6f3a-420d-ba49-679ccccbfb86/cleanup/0.log"
Jul 20 18:45:13.770624 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:45:13.770595    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-wpwhf_4d880c15-e2ed-4921-8494-154405091530/kuadrant-console-plugin/0.log"
Jul 20 18:45:14.205614 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:45:14.205528    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-9thxb_c9996300-b0a7-43ac-86ca-469038001e1f/limitador/0.log"
Jul 20 18:45:17.309477 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:45:17.309398    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-75cc6b94bb-wcbx8_a8cfc3b1-73da-4052-9e03-cb0437933008/kube-auth-proxy/0.log"
Jul 20 18:45:19.387963 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:45:19.387926    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-tkgh7_042bde04-5c1f-4f79-a1d5-eea437821bf4/main/0.log"
Jul 20 18:45:24.950099 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:45:24.950059    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-k5j77/must-gather-vtt8f"]
Jul 20 18:45:24.950471 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:45:24.950383    2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a2f6174-6f3a-420d-ba49-679ccccbfb86" containerName="cleanup"
Jul 20 18:45:24.950471 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:45:24.950394    2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a2f6174-6f3a-420d-ba49-679ccccbfb86" containerName="cleanup"
Jul 20 18:45:24.950471 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:45:24.950454    2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a2f6174-6f3a-420d-ba49-679ccccbfb86" containerName="cleanup"
Jul 20 18:45:24.953524 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:45:24.953508    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-k5j77/must-gather-vtt8f"
Jul 20 18:45:24.955914 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:45:24.955885    2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-k5j77\"/\"default-dockercfg-sk2sz\""
Jul 20 18:45:24.956181 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:45:24.956160    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-k5j77\"/\"openshift-service-ca.crt\""
Jul 20 18:45:24.956682 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:45:24.956663    2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-k5j77\"/\"kube-root-ca.crt\""
Jul 20 18:45:24.965443 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:45:24.965424    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-k5j77/must-gather-vtt8f"]
Jul 20 18:45:25.084962 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:45:25.084931    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pssz7\" (UniqueName: \"kubernetes.io/projected/4f3b988b-2857-472b-b927-9639c8f252e0-kube-api-access-pssz7\") pod \"must-gather-vtt8f\" (UID: \"4f3b988b-2857-472b-b927-9639c8f252e0\") " pod="openshift-must-gather-k5j77/must-gather-vtt8f"
Jul 20 18:45:25.084962 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:45:25.084962    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4f3b988b-2857-472b-b927-9639c8f252e0-must-gather-output\") pod \"must-gather-vtt8f\" (UID: \"4f3b988b-2857-472b-b927-9639c8f252e0\") " pod="openshift-must-gather-k5j77/must-gather-vtt8f"
Jul 20 18:45:25.186267 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:45:25.186234    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pssz7\" (UniqueName: \"kubernetes.io/projected/4f3b988b-2857-472b-b927-9639c8f252e0-kube-api-access-pssz7\") pod \"must-gather-vtt8f\" (UID: \"4f3b988b-2857-472b-b927-9639c8f252e0\") " pod="openshift-must-gather-k5j77/must-gather-vtt8f"
Jul 20 18:45:25.186267 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:45:25.186269    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4f3b988b-2857-472b-b927-9639c8f252e0-must-gather-output\") pod \"must-gather-vtt8f\" (UID: \"4f3b988b-2857-472b-b927-9639c8f252e0\") " pod="openshift-must-gather-k5j77/must-gather-vtt8f"
Jul 20 18:45:25.186620 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:45:25.186598    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4f3b988b-2857-472b-b927-9639c8f252e0-must-gather-output\") pod \"must-gather-vtt8f\" (UID: \"4f3b988b-2857-472b-b927-9639c8f252e0\") " pod="openshift-must-gather-k5j77/must-gather-vtt8f"
Jul 20 18:45:25.196551 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:45:25.196522    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pssz7\" (UniqueName: \"kubernetes.io/projected/4f3b988b-2857-472b-b927-9639c8f252e0-kube-api-access-pssz7\") pod \"must-gather-vtt8f\" (UID: \"4f3b988b-2857-472b-b927-9639c8f252e0\") " pod="openshift-must-gather-k5j77/must-gather-vtt8f"
Jul 20 18:45:25.262551 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:45:25.262528    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-k5j77/must-gather-vtt8f"
Jul 20 18:45:25.404357 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:45:25.404334    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-k5j77/must-gather-vtt8f"]
Jul 20 18:45:25.406219 ip-10-0-137-247 kubenswrapper[2569]: W0720 18:45:25.406186    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4f3b988b_2857_472b_b927_9639c8f252e0.slice/crio-18e6c640e0122414ee357de1ea4bafb0d0dee38deb50274221363a3c2fa609d2 WatchSource:0}: Error finding container 18e6c640e0122414ee357de1ea4bafb0d0dee38deb50274221363a3c2fa609d2: Status 404 returned error can't find the container with id 18e6c640e0122414ee357de1ea4bafb0d0dee38deb50274221363a3c2fa609d2
Jul 20 18:45:25.408200 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:45:25.408181    2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 20 18:45:26.315914 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:45:26.315882    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-k5j77/must-gather-vtt8f" event={"ID":"4f3b988b-2857-472b-b927-9639c8f252e0","Type":"ContainerStarted","Data":"18e6c640e0122414ee357de1ea4bafb0d0dee38deb50274221363a3c2fa609d2"}
Jul 20 18:45:27.323983 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:45:27.323950    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-k5j77/must-gather-vtt8f" event={"ID":"4f3b988b-2857-472b-b927-9639c8f252e0","Type":"ContainerStarted","Data":"00c5c78acb0c242adbe3f2c80592cec66b854fc598cc658cb631efe7f4f5cf18"}
Jul 20 18:45:27.324375 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:45:27.323993    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-k5j77/must-gather-vtt8f" event={"ID":"4f3b988b-2857-472b-b927-9639c8f252e0","Type":"ContainerStarted","Data":"03f3f7bda6fdd11990b2534ef0adb3afe12ba1e5a0dd72afddaadf1e87dce440"}
Jul 20 18:45:27.349249 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:45:27.349177    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-k5j77/must-gather-vtt8f" podStartSLOduration=2.482200844 podStartE2EDuration="3.349155349s" totalImagesPullingTime="866.954505ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 18:45:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 18:45:27.348619126 +0000 UTC m=+2973.595152537" watchObservedRunningTime="2026-07-20 18:45:27.349155349 +0000 UTC m=+2973.595688760"
Jul 20 18:45:28.018580 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:45:28.018552    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-tgg22_eca0024f-2963-4ed1-817e-f1a6febd79aa/global-pull-secret-syncer/0.log"
Jul 20 18:45:28.125219 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:45:28.125182    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-l45qp_76c83a34-e7fe-46a1-a01e-944840b62724/konnectivity-agent/0.log"
Jul 20 18:45:28.220759 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:45:28.220689    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-137-247.ec2.internal_757e30ff139839aada6b39c2403a950d/haproxy/0.log"
Jul 20 18:45:33.907267 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:45:33.907233    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-wpwhf_4d880c15-e2ed-4921-8494-154405091530/kuadrant-console-plugin/0.log"
Jul 20 18:45:34.200910 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:45:34.200831    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-9thxb_c9996300-b0a7-43ac-86ca-469038001e1f/limitador/0.log"
Jul 20 18:45:36.348873 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:45:36.348839    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-664668c958-tt77p_89d7700b-5ef7-4b01-8e3f-e9fd631bd59c/metrics-server/0.log"
Jul 20 18:45:36.418696 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:45:36.418665    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dj4gt_4c02e13a-6eb6-4c56-87d0-ea9258d3912f/node-exporter/0.log"
Jul 20 18:45:36.450048 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:45:36.450016    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dj4gt_4c02e13a-6eb6-4c56-87d0-ea9258d3912f/kube-rbac-proxy/0.log"
Jul 20 18:45:36.484954 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:45:36.484926    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dj4gt_4c02e13a-6eb6-4c56-87d0-ea9258d3912f/init-textfile/0.log"
Jul 20 18:45:37.062935 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:45:37.062904    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-mw5n7_46e0d218-80cf-4517-9265-3acbb2d30631/prometheus-operator/0.log"
Jul 20 18:45:37.085681 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:45:37.085652    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-mw5n7_46e0d218-80cf-4517-9265-3acbb2d30631/kube-rbac-proxy/0.log"
Jul 20 18:45:37.924668 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:45:37.924628    2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-k5j77/perf-node-gather-daemonset-zvq97"]
Jul 20 18:45:37.932440 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:45:37.932412    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-k5j77/perf-node-gather-daemonset-zvq97"
Jul 20 18:45:37.938524 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:45:37.938499    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-k5j77/perf-node-gather-daemonset-zvq97"]
Jul 20 18:45:38.009795 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:45:38.009735    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/24bd2aee-46ef-4db2-bc75-f6853634dc38-sys\") pod \"perf-node-gather-daemonset-zvq97\" (UID: \"24bd2aee-46ef-4db2-bc75-f6853634dc38\") " pod="openshift-must-gather-k5j77/perf-node-gather-daemonset-zvq97"
Jul 20 18:45:38.009795 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:45:38.009790    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/24bd2aee-46ef-4db2-bc75-f6853634dc38-lib-modules\") pod \"perf-node-gather-daemonset-zvq97\" (UID: \"24bd2aee-46ef-4db2-bc75-f6853634dc38\") " pod="openshift-must-gather-k5j77/perf-node-gather-daemonset-zvq97"
Jul 20 18:45:38.010042 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:45:38.009816    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/24bd2aee-46ef-4db2-bc75-f6853634dc38-podres\") pod \"perf-node-gather-daemonset-zvq97\" (UID: \"24bd2aee-46ef-4db2-bc75-f6853634dc38\") " pod="openshift-must-gather-k5j77/perf-node-gather-daemonset-zvq97"
Jul 20 18:45:38.010153 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:45:38.010126    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfnzb\" (UniqueName: \"kubernetes.io/projected/24bd2aee-46ef-4db2-bc75-f6853634dc38-kube-api-access-gfnzb\") pod \"perf-node-gather-daemonset-zvq97\" (UID: \"24bd2aee-46ef-4db2-bc75-f6853634dc38\") " pod="openshift-must-gather-k5j77/perf-node-gather-daemonset-zvq97"
Jul 20 18:45:38.010343 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:45:38.010309    2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/24bd2aee-46ef-4db2-bc75-f6853634dc38-proc\") pod \"perf-node-gather-daemonset-zvq97\" (UID: \"24bd2aee-46ef-4db2-bc75-f6853634dc38\") " pod="openshift-must-gather-k5j77/perf-node-gather-daemonset-zvq97"
Jul 20 18:45:38.112050 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:45:38.112005    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/24bd2aee-46ef-4db2-bc75-f6853634dc38-sys\") pod \"perf-node-gather-daemonset-zvq97\" (UID: \"24bd2aee-46ef-4db2-bc75-f6853634dc38\") " pod="openshift-must-gather-k5j77/perf-node-gather-daemonset-zvq97"
Jul 20 18:45:38.112246 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:45:38.112060    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/24bd2aee-46ef-4db2-bc75-f6853634dc38-lib-modules\") pod \"perf-node-gather-daemonset-zvq97\" (UID: \"24bd2aee-46ef-4db2-bc75-f6853634dc38\") " pod="openshift-must-gather-k5j77/perf-node-gather-daemonset-zvq97"
Jul 20 18:45:38.112246 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:45:38.112089    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/24bd2aee-46ef-4db2-bc75-f6853634dc38-podres\") pod \"perf-node-gather-daemonset-zvq97\" (UID: \"24bd2aee-46ef-4db2-bc75-f6853634dc38\") " pod="openshift-must-gather-k5j77/perf-node-gather-daemonset-zvq97"
Jul 20 18:45:38.112246 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:45:38.112118    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfnzb\" (UniqueName: \"kubernetes.io/projected/24bd2aee-46ef-4db2-bc75-f6853634dc38-kube-api-access-gfnzb\") pod \"perf-node-gather-daemonset-zvq97\" (UID: \"24bd2aee-46ef-4db2-bc75-f6853634dc38\") " pod="openshift-must-gather-k5j77/perf-node-gather-daemonset-zvq97"
Jul 20 18:45:38.112246 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:45:38.112149    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/24bd2aee-46ef-4db2-bc75-f6853634dc38-sys\") pod \"perf-node-gather-daemonset-zvq97\" (UID: \"24bd2aee-46ef-4db2-bc75-f6853634dc38\") " pod="openshift-must-gather-k5j77/perf-node-gather-daemonset-zvq97"
Jul 20 18:45:38.112470 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:45:38.112284    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/24bd2aee-46ef-4db2-bc75-f6853634dc38-podres\") pod \"perf-node-gather-daemonset-zvq97\" (UID: \"24bd2aee-46ef-4db2-bc75-f6853634dc38\") " pod="openshift-must-gather-k5j77/perf-node-gather-daemonset-zvq97"
Jul 20 18:45:38.112470 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:45:38.112338    2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/24bd2aee-46ef-4db2-bc75-f6853634dc38-proc\") pod \"perf-node-gather-daemonset-zvq97\" (UID: \"24bd2aee-46ef-4db2-bc75-f6853634dc38\") " pod="openshift-must-gather-k5j77/perf-node-gather-daemonset-zvq97"
Jul 20 18:45:38.112470 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:45:38.112290    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/24bd2aee-46ef-4db2-bc75-f6853634dc38-lib-modules\") pod \"perf-node-gather-daemonset-zvq97\" (UID: \"24bd2aee-46ef-4db2-bc75-f6853634dc38\") " pod="openshift-must-gather-k5j77/perf-node-gather-daemonset-zvq97"
Jul 20 18:45:38.112470 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:45:38.112439    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/24bd2aee-46ef-4db2-bc75-f6853634dc38-proc\") pod \"perf-node-gather-daemonset-zvq97\" (UID: \"24bd2aee-46ef-4db2-bc75-f6853634dc38\") " pod="openshift-must-gather-k5j77/perf-node-gather-daemonset-zvq97"
Jul 20 18:45:38.121859 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:45:38.121833    2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfnzb\" (UniqueName: \"kubernetes.io/projected/24bd2aee-46ef-4db2-bc75-f6853634dc38-kube-api-access-gfnzb\") pod \"perf-node-gather-daemonset-zvq97\" (UID: \"24bd2aee-46ef-4db2-bc75-f6853634dc38\") " pod="openshift-must-gather-k5j77/perf-node-gather-daemonset-zvq97"
Jul 20 18:45:38.244073 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:45:38.243978    2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-k5j77/perf-node-gather-daemonset-zvq97"
Jul 20 18:45:38.423151 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:45:38.423112    2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-k5j77/perf-node-gather-daemonset-zvq97"]
Jul 20 18:45:38.428624 ip-10-0-137-247 kubenswrapper[2569]: W0720 18:45:38.428594    2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod24bd2aee_46ef_4db2_bc75_f6853634dc38.slice/crio-67f38dbde08cec7ddf5dc9610e0bca911756ad00ed43daa5053694e5488fd360 WatchSource:0}: Error finding container 67f38dbde08cec7ddf5dc9610e0bca911756ad00ed43daa5053694e5488fd360: Status 404 returned error can't find the container with id 67f38dbde08cec7ddf5dc9610e0bca911756ad00ed43daa5053694e5488fd360
Jul 20 18:45:39.397304 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:45:39.397266    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-k5j77/perf-node-gather-daemonset-zvq97" event={"ID":"24bd2aee-46ef-4db2-bc75-f6853634dc38","Type":"ContainerStarted","Data":"5cc7722d8a24360b9766d32cba041254c13c8895ffa5393e8a83d61b713c4f99"}
Jul 20 18:45:39.397774 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:45:39.397328    2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-k5j77/perf-node-gather-daemonset-zvq97" event={"ID":"24bd2aee-46ef-4db2-bc75-f6853634dc38","Type":"ContainerStarted","Data":"67f38dbde08cec7ddf5dc9610e0bca911756ad00ed43daa5053694e5488fd360"}
Jul 20 18:45:39.397774 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:45:39.397408    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-k5j77/perf-node-gather-daemonset-zvq97"
Jul 20 18:45:39.420065 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:45:39.420022    2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-k5j77/perf-node-gather-daemonset-zvq97" podStartSLOduration=2.420007204 podStartE2EDuration="2.420007204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 18:45:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 18:45:39.418755602 +0000 UTC m=+2985.665289010" watchObservedRunningTime="2026-07-20 18:45:39.420007204 +0000 UTC m=+2985.666540634"
Jul 20 18:45:41.411229 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:45:41.411198    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-ktc25_bfe61b3e-4aac-4ddd-bde9-10fb1c7e6e07/dns/0.log"
Jul 20 18:45:41.436430 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:45:41.436379    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-ktc25_bfe61b3e-4aac-4ddd-bde9-10fb1c7e6e07/kube-rbac-proxy/0.log"
Jul 20 18:45:41.610219 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:45:41.610187    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-smcqb_ea740a69-374a-483b-a045-a2c795b99d4c/dns-node-resolver/0.log"
Jul 20 18:45:42.257837 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:45:42.257809    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-q42fv_7a564ed7-d4b4-4f1a-a7da-d1e4d0652601/node-ca/0.log"
Jul 20 18:45:43.428943 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:45:43.428904    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-75cc6b94bb-wcbx8_a8cfc3b1-73da-4052-9e03-cb0437933008/kube-auth-proxy/0.log"
Jul 20 18:45:44.393888 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:45:44.393845    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-67b26_69347ff0-9f35-4334-be7f-b02bca61a64c/serve-healthcheck-canary/0.log"
Jul 20 18:45:45.104043 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:45:45.104009    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kh4zt_24ac2f79-9d88-4cd0-88ef-8fb14fd65674/kube-rbac-proxy/0.log"
Jul 20 18:45:45.131302 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:45:45.131268    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kh4zt_24ac2f79-9d88-4cd0-88ef-8fb14fd65674/exporter/0.log"
Jul 20 18:45:45.159854 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:45:45.159808    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kh4zt_24ac2f79-9d88-4cd0-88ef-8fb14fd65674/extractor/0.log"
Jul 20 18:45:45.413972 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:45:45.413894    2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-k5j77/perf-node-gather-daemonset-zvq97"
Jul 20 18:45:48.353216 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:45:48.353172    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7bd97c8594-qntzv_9b8f6824-4ca6-4b50-abcd-55073c9dd002/manager/0.log"
Jul 20 18:45:48.365265 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:45:48.365234    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-75cc6b94bb-wcbx8_a8cfc3b1-73da-4052-9e03-cb0437933008/kube-auth-proxy/0.log"
Jul 20 18:45:48.556586 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:45:48.556551    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-79747b4d4c-cks45_6caac6ef-ec22-491d-8b82-41665d8ca54c/manager/0.log"
Jul 20 18:45:48.592009 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:45:48.591980    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-79747b4d4c-cks45_6caac6ef-ec22-491d-8b82-41665d8ca54c/copy-manifests/0.log"
Jul 20 18:45:50.010221 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:45:50.010189    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-75cc6b94bb-wcbx8_a8cfc3b1-73da-4052-9e03-cb0437933008/kube-auth-proxy/0.log"
Jul 20 18:45:50.144759 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:45:50.144716    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-57d4489cf8-2q6sw_b6f825b2-5581-4dd1-a340-6eab11b76c3b/manager/0.log"
Jul 20 18:45:50.219168 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:45:50.219134    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-nlcl8_b32cfd04-a0db-4334-9ff6-5136f1ab4f39/openshift-lws-operator/0.log"
Jul 20 18:45:54.319622 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:45:54.319591    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wjs8g_07587a78-2d3a-40ad-9aae-a4875200263a/ovn-acl-logging/0.log"
Jul 20 18:45:54.332370 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:45:54.332347    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wjs8g_07587a78-2d3a-40ad-9aae-a4875200263a/ovn-acl-logging/0.log"
Jul 20 18:45:55.264546 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:45:55.264510    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-fcjsx_8a0b6c5f-6bca-49b3-b6dd-8ba1677ed470/migrator/0.log"
Jul 20 18:45:55.296869 ip-10-0-137-247 kubenswrapper[2569]: I0720 18:45:55.296832    2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-fcjsx_8a0b6c5f-6bca-49b3-b6dd-8ba1677ed470/graceful-termination/0.log"