Jul 27 20:51:23.121397 ip-10-0-141-64 systemd[1]: Starting Kubernetes Kubelet...
Jul 27 20:51:23.566894 ip-10-0-141-64 kubenswrapper[2560]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 27 20:51:23.566894 ip-10-0-141-64 kubenswrapper[2560]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 27 20:51:23.566894 ip-10-0-141-64 kubenswrapper[2560]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 27 20:51:23.566894 ip-10-0-141-64 kubenswrapper[2560]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 27 20:51:23.566894 ip-10-0-141-64 kubenswrapper[2560]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 27 20:51:23.568005 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.567870    2560 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 27 20:51:23.572726 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.572706    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 20:51:23.572726 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.572727    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 20:51:23.572810 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.572732    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 20:51:23.572810 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.572736    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 20:51:23.572810 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.572738    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 20:51:23.572810 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.572741    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 20:51:23.572810 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.572744    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 20:51:23.572810 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.572747    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 20:51:23.572810 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.572750    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 20:51:23.572810 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.572753    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 20:51:23.572810 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.572756    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 20:51:23.572810 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.572758    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 20:51:23.572810 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.572761    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 20:51:23.572810 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.572763    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 20:51:23.572810 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.572766    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 20:51:23.572810 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.572769    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 20:51:23.572810 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.572771    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 20:51:23.572810 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.572774    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 20:51:23.572810 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.572776    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 20:51:23.572810 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.572780    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 20:51:23.572810 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.572782    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 20:51:23.572810 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.572785    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 20:51:23.573314 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.572787    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 20:51:23.573314 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.572790    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 20:51:23.573314 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.572793    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 20:51:23.573314 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.572795    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 20:51:23.573314 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.572797    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 20:51:23.573314 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.572800    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 20:51:23.573314 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.572802    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 20:51:23.573314 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.572804    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 20:51:23.573314 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.572807    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 20:51:23.573314 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.572809    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 20:51:23.573314 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.572812    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 20:51:23.573314 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.572815    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 20:51:23.573314 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.572819    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 20:51:23.573314 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.572824    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 20:51:23.573314 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.572827    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 20:51:23.573314 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.572829    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 20:51:23.573314 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.572832    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 20:51:23.573314 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.572835    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 20:51:23.573314 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.572837    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 20:51:23.573314 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.572840    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 20:51:23.573846 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.572843    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 20:51:23.573846 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.572845    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 20:51:23.573846 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.572850    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 20:51:23.573846 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.572854    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 20:51:23.573846 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.572857    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 20:51:23.573846 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.572859    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 20:51:23.573846 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.572863    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 20:51:23.573846 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.572866    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 20:51:23.573846 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.572868    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 20:51:23.573846 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.572871    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 20:51:23.573846 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.572873    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 20:51:23.573846 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.572876    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 20:51:23.573846 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.572879    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 20:51:23.573846 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.572883    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 20:51:23.573846 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.572885    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 20:51:23.573846 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.572888    2560 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 20:51:23.573846 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.572890    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 20:51:23.573846 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.572893    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 20:51:23.573846 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.572896    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 20:51:23.574336 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.572898    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 20:51:23.574336 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.572901    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 20:51:23.574336 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.572904    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 20:51:23.574336 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.572906    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 20:51:23.574336 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.572909    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 20:51:23.574336 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.572911    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 20:51:23.574336 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.572915    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 20:51:23.574336 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.572917    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 20:51:23.574336 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.572920    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 20:51:23.574336 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.572923    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 20:51:23.574336 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.572926    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 20:51:23.574336 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.572928    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 20:51:23.574336 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.572931    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 20:51:23.574336 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.572933    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 20:51:23.574336 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.572936    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 20:51:23.574336 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.572939    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 20:51:23.574336 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.572941    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 20:51:23.574336 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.572944    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 20:51:23.574336 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.572946    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 20:51:23.574336 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.572949    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 20:51:23.574812 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.572951    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 20:51:23.574812 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.572954    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 20:51:23.574812 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.572957    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 20:51:23.574812 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.572959    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 20:51:23.574812 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.572962    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 20:51:23.574812 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.572965    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 20:51:23.574812 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.572969    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 20:51:23.574812 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.572972    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 20:51:23.574812 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.572974    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 20:51:23.574812 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.572977    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 20:51:23.574812 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.572979    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 20:51:23.574812 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.572983    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 20:51:23.574812 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.572987    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 20:51:23.574812 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.572990    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 20:51:23.574812 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.572993    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 20:51:23.574812 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.572996    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 20:51:23.574812 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.572999    2560 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 20:51:23.574812 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.573001    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 20:51:23.574812 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.573759    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 20:51:23.575297 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.573768    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 20:51:23.575297 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.573772    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 20:51:23.575297 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.573774    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 20:51:23.575297 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.573777    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 20:51:23.575297 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.573781    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 20:51:23.575297 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.573784    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 20:51:23.575297 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.573786    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 20:51:23.575297 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.573789    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 20:51:23.575297 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.573792    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 20:51:23.575297 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.573795    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 20:51:23.575297 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.573797    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 20:51:23.575297 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.573800    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 20:51:23.575297 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.573803    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 20:51:23.575297 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.573805    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 20:51:23.575297 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.573808    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 20:51:23.575297 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.573811    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 20:51:23.575297 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.573813    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 20:51:23.575297 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.573815    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 20:51:23.575297 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.573818    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 20:51:23.575855 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.573822    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 20:51:23.575855 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.573825    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 20:51:23.575855 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.573827    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 20:51:23.575855 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.573830    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 20:51:23.575855 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.573832    2560 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 20:51:23.575855 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.573835    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 20:51:23.575855 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.573837    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 20:51:23.575855 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.573840    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 20:51:23.575855 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.573843    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 20:51:23.575855 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.573845    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 20:51:23.575855 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.573848    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 20:51:23.575855 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.573851    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 20:51:23.575855 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.573854    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 20:51:23.575855 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.573856    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 20:51:23.575855 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.573859    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 20:51:23.575855 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.573862    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 20:51:23.575855 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.573864    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 20:51:23.575855 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.573867    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 20:51:23.575855 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.573869    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 20:51:23.575855 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.573873    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 20:51:23.576361 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.573875    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 20:51:23.576361 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.573878    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 20:51:23.576361 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.573881    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 20:51:23.576361 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.573883    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 20:51:23.576361 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.573886    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 20:51:23.576361 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.573888    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 20:51:23.576361 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.573891    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 20:51:23.576361 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.573894    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 20:51:23.576361 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.573896    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 20:51:23.576361 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.573899    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 20:51:23.576361 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.573901    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 20:51:23.576361 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.573904    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 20:51:23.576361 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.573908    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 20:51:23.576361 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.573910    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 20:51:23.576361 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.573913    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 20:51:23.576361 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.573916    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 20:51:23.576361 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.573918    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 20:51:23.576361 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.573921    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 20:51:23.576361 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.573924    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 20:51:23.576361 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.573926    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 20:51:23.576847 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.573929    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 20:51:23.576847 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.573931    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 20:51:23.576847 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.573933    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 20:51:23.576847 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.573936    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 20:51:23.576847 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.573939    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 20:51:23.576847 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.573942    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 20:51:23.576847 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.573944    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 20:51:23.576847 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.573947    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 20:51:23.576847 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.573952    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 20:51:23.576847 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.573957    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 20:51:23.576847 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.573960    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 20:51:23.576847 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.573964    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 20:51:23.576847 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.573968    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 20:51:23.576847 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.573972    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 20:51:23.576847 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.573974    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 20:51:23.576847 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.573978    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 20:51:23.576847 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.573980    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 20:51:23.576847 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.573983    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 20:51:23.577320 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.573986    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 20:51:23.577320 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.573988    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 20:51:23.577320 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.573991    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 20:51:23.577320 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.573994    2560 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 20:51:23.577320 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.573996    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 20:51:23.577320 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.573999    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 20:51:23.577320 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.574002    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 20:51:23.577320 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.574005    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 20:51:23.577320 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.574007    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 20:51:23.577320 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.574010    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 20:51:23.577320 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.574012    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 20:51:23.577320 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.574015    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 20:51:23.577320 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.574017    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 20:51:23.577320 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.574020    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 20:51:23.577320 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.574022    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 20:51:23.577320 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.574025    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 20:51:23.577320 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.574028    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 20:51:23.577320 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.574030    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 20:51:23.577320 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.574033    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 20:51:23.577320 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.574035    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 20:51:23.577818 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.574037    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 20:51:23.577818 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.574454    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 20:51:23.577818 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.574460    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 20:51:23.577818 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.574463    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 20:51:23.577818 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.574465    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 20:51:23.577818 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.574468    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 20:51:23.577818 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.574471    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 20:51:23.577818 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.574474    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 20:51:23.577818 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.574477    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 20:51:23.577818 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.574480    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 20:51:23.577818 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.574483    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 20:51:23.577818 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.574486    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 20:51:23.577818 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.574488    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 20:51:23.577818 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.574491    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 20:51:23.577818 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.574494    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 20:51:23.577818 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.574497    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 20:51:23.577818 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.574500    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 20:51:23.577818 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.574503    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 20:51:23.577818 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.574508    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 20:51:23.578294 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.574511    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 20:51:23.578294 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.574514    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 20:51:23.578294 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.574517    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 20:51:23.578294 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.574519    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 20:51:23.578294 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.574522    2560 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 20:51:23.578294 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.574525    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 20:51:23.578294 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.574527    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 20:51:23.578294 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.574530    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 20:51:23.578294 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.574533    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 20:51:23.578294 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.574535    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 20:51:23.578294 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.574538    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 20:51:23.578294 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.574540    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 20:51:23.578294 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.574543    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 20:51:23.578294 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.574546    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 20:51:23.578294 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.574549    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 20:51:23.578294 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.574553    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 20:51:23.578294 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.574555    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 20:51:23.578294 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.574558    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 20:51:23.578294 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.574561    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 20:51:23.578294 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.574563    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 20:51:23.578851 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.574566    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 20:51:23.578851 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.574570    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 20:51:23.578851 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.574573    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 20:51:23.578851 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.574576    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 20:51:23.578851 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.574579    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 20:51:23.578851 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.574581    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 20:51:23.578851 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.574584    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 20:51:23.578851 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.574587    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 20:51:23.578851 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.574589    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 20:51:23.578851 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.574592    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 20:51:23.578851 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.574594    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 20:51:23.578851 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.574597    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 20:51:23.578851 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.574599    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 20:51:23.578851 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.574602    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 20:51:23.578851 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.574604    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 20:51:23.578851 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.574607    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 20:51:23.578851 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.574609    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 20:51:23.578851 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.574612    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 20:51:23.578851 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.574614    2560 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 20:51:23.578851 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.574616    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 20:51:23.579362 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.574619    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 20:51:23.579362 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.574621    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 20:51:23.579362 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.574624    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 20:51:23.579362 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.574626    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 20:51:23.579362 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.574628    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 20:51:23.579362 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.574631    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 20:51:23.579362 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.574635    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 20:51:23.579362 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.574637    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 20:51:23.579362 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.574641    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 20:51:23.579362 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.574644    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 20:51:23.579362 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.574647    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 20:51:23.579362 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.574650    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 20:51:23.579362 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.574653    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 20:51:23.579362 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.574657    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 20:51:23.579362 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.574660    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 20:51:23.579362 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.574663    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 20:51:23.579362 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.574666    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 20:51:23.579362 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.574669    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 20:51:23.579362 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.574672    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 20:51:23.579822 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.574674    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 20:51:23.579822 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.574677    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 20:51:23.579822 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.574679    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 20:51:23.579822 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.574681    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 20:51:23.579822 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.574684    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 20:51:23.579822 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.574686    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 20:51:23.579822 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.574689    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 20:51:23.579822 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.574691    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 20:51:23.579822 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.574694    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 20:51:23.579822 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.574696    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 20:51:23.579822 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.574700    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 20:51:23.579822 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.574702    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 20:51:23.579822 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.574704    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 20:51:23.579822 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.574707    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 20:51:23.579822 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.574710    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 20:51:23.579822 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.574712    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 20:51:23.579822 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.574715    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 20:51:23.579822 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.574717    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 20:51:23.579822 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.574719    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 20:51:23.579822 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.574722    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 20:51:23.580352 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.574724    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 20:51:23.580352 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.574727    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 20:51:23.580352 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.574807    2560 flags.go:64] FLAG: --address="0.0.0.0"
Jul 27 20:51:23.580352 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.574821    2560 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 27 20:51:23.580352 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.574827    2560 flags.go:64] FLAG: --anonymous-auth="true"
Jul 27 20:51:23.580352 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.574832    2560 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 27 20:51:23.580352 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.574837    2560 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 27 20:51:23.580352 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.574841    2560 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 27 20:51:23.580352 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.574845    2560 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 27 20:51:23.580352 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.574850    2560 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 27 20:51:23.580352 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.574854    2560 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 27 20:51:23.580352 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.574857    2560 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 27 20:51:23.580352 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.574861    2560 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 27 20:51:23.580352 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.574864    2560 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 27 20:51:23.580352 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.574867    2560 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 27 20:51:23.580352 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.574870    2560 flags.go:64] FLAG: --cgroup-root=""
Jul 27 20:51:23.580352 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.574873    2560 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 27 20:51:23.580352 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.574876    2560 flags.go:64] FLAG: --client-ca-file=""
Jul 27 20:51:23.580352 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.574878    2560 flags.go:64] FLAG: --cloud-provider="external"
Jul 27 20:51:23.580352 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.574881    2560 flags.go:64] FLAG: --cluster-dns="[]"
Jul 27 20:51:23.580352 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.574886    2560 flags.go:64] FLAG: --cluster-domain=""
Jul 27 20:51:23.580352 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.574888    2560 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 27 20:51:23.580352 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.574891    2560 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 27 20:51:23.580352 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.574894    2560 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 27 20:51:23.580926 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.574898    2560 flags.go:64] FLAG: --container-log-max-files="5"
Jul 27 20:51:23.580926 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.574902    2560 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 27 20:51:23.580926 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.574904    2560 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 27 20:51:23.580926 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.574907    2560 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 27 20:51:23.580926 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.574911    2560 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 27 20:51:23.580926 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.574914    2560 flags.go:64] FLAG: --contention-profiling="false"
Jul 27 20:51:23.580926 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.574917    2560 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 27 20:51:23.580926 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.574920    2560 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 27 20:51:23.580926 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.574923    2560 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 27 20:51:23.580926 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.574925    2560 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 27 20:51:23.580926 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.574929    2560 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 27 20:51:23.580926 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.574932    2560 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 27 20:51:23.580926 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.574935    2560 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 27 20:51:23.580926 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.574938    2560 flags.go:64] FLAG: --enable-load-reader="false"
Jul 27 20:51:23.580926 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.574942    2560 flags.go:64] FLAG: --enable-server="true"
Jul 27 20:51:23.580926 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.574947    2560 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 27 20:51:23.580926 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.574951    2560 flags.go:64] FLAG: --event-burst="100"
Jul 27 20:51:23.580926 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.574954    2560 flags.go:64] FLAG: --event-qps="50"
Jul 27 20:51:23.580926 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.574957    2560 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 27 20:51:23.580926 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.574960    2560 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 27 20:51:23.580926 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.574963    2560 flags.go:64] FLAG: --eviction-hard=""
Jul 27 20:51:23.580926 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.574967    2560 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 27 20:51:23.580926 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.574970    2560 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 27 20:51:23.580926 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.574973    2560 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 27 20:51:23.580926 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.574976    2560 flags.go:64] FLAG: --eviction-soft=""
Jul 27 20:51:23.581576 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.574979    2560 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 27 20:51:23.581576 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.574982    2560 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 27 20:51:23.581576 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.574985    2560 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 27 20:51:23.581576 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.574988    2560 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 27 20:51:23.581576 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.574990    2560 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 27 20:51:23.581576 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.574993    2560 flags.go:64] FLAG: --fail-swap-on="true"
Jul 27 20:51:23.581576 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.574996    2560 flags.go:64] FLAG: --feature-gates=""
Jul 27 20:51:23.581576 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.575000    2560 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 27 20:51:23.581576 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.575003    2560 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 27 20:51:23.581576 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.575006    2560 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 27 20:51:23.581576 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.575009    2560 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 27 20:51:23.581576 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.575012    2560 flags.go:64] FLAG: --healthz-port="10248"
Jul 27 20:51:23.581576 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.575014    2560 flags.go:64] FLAG: --help="false"
Jul 27 20:51:23.581576 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.575017    2560 flags.go:64] FLAG: --hostname-override="ip-10-0-141-64.ec2.internal"
Jul 27 20:51:23.581576 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.575020    2560 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 27 20:51:23.581576 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.575023    2560 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 27 20:51:23.581576 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.575026    2560 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 27 20:51:23.581576 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.575029    2560 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 27 20:51:23.581576 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.575032    2560 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 27 20:51:23.581576 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.575035    2560 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 27 20:51:23.581576 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.575038    2560 flags.go:64] FLAG: --image-service-endpoint=""
Jul 27 20:51:23.581576 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.575042    2560 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 27 20:51:23.581576 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.575046    2560 flags.go:64] FLAG: --kube-api-burst="100"
Jul 27 20:51:23.582216 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.575049    2560 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 27 20:51:23.582216 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.575052    2560 flags.go:64] FLAG: --kube-api-qps="50"
Jul 27 20:51:23.582216 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.575054    2560 flags.go:64] FLAG: --kube-reserved=""
Jul 27 20:51:23.582216 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.575057    2560 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 27 20:51:23.582216 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.575060    2560 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 27 20:51:23.582216 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.575063    2560 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 27 20:51:23.582216 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.575066    2560 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 27 20:51:23.582216 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.575069    2560 flags.go:64] FLAG: --lock-file=""
Jul 27 20:51:23.582216 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.575071    2560 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 27 20:51:23.582216 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.575074    2560 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 27 20:51:23.582216 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.575077    2560 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 27 20:51:23.582216 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.575097    2560 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 27 20:51:23.582216 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.575101    2560 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 27 20:51:23.582216 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.575103    2560 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 27 20:51:23.582216 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.575106    2560 flags.go:64] FLAG: --logging-format="text"
Jul 27 20:51:23.582216 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.575109    2560 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 27 20:51:23.582216 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.575112    2560 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 27 20:51:23.582216 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.575115    2560 flags.go:64] FLAG: --manifest-url=""
Jul 27 20:51:23.582216 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.575118    2560 flags.go:64] FLAG: --manifest-url-header=""
Jul 27 20:51:23.582216 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.575122    2560 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 27 20:51:23.582216 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.575125    2560 flags.go:64] FLAG: --max-open-files="1000000"
Jul 27 20:51:23.582216 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.575130    2560 flags.go:64] FLAG: --max-pods="110"
Jul 27 20:51:23.582216 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.575132    2560 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 27 20:51:23.582216 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.575136    2560 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 27 20:51:23.582216 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.575138    2560 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 27 20:51:23.582813 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.575141    2560 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 27 20:51:23.582813 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.575144    2560 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 27 20:51:23.582813 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.575147    2560 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 27 20:51:23.582813 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.575150    2560 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 27 20:51:23.582813 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.575158    2560 flags.go:64] FLAG: --node-status-max-images="50"
Jul 27 20:51:23.582813 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.575163    2560 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 27 20:51:23.582813 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.575167    2560 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 27 20:51:23.582813 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.575170    2560 flags.go:64] FLAG: --pod-cidr=""
Jul 27 20:51:23.582813 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.575173    2560 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 27 20:51:23.582813 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.575178    2560 flags.go:64] FLAG: --pod-manifest-path=""
Jul 27 20:51:23.582813 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.575181    2560 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 27 20:51:23.582813 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.575185    2560 flags.go:64] FLAG: --pods-per-core="0"
Jul 27 20:51:23.582813 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.575190    2560 flags.go:64] FLAG: --port="10250"
Jul 27 20:51:23.582813 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.575193    2560 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 27 20:51:23.582813 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.575195    2560 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-04b98b332b2d83159"
Jul 27 20:51:23.582813 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.575199    2560 flags.go:64] FLAG: --qos-reserved=""
Jul 27 20:51:23.582813 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.575202    2560 flags.go:64] FLAG: --read-only-port="10255"
Jul 27 20:51:23.582813 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.575205    2560 flags.go:64] FLAG: --register-node="true"
Jul 27 20:51:23.582813 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.575208    2560 flags.go:64] FLAG: --register-with-taints=""
Jul 27 20:51:23.582813 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.575212    2560 flags.go:64] FLAG: --registry-burst="10"
Jul 27 20:51:23.582813 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.575215    2560 flags.go:64] FLAG: --registry-qps="5"
Jul 27 20:51:23.582813 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.575218    2560 flags.go:64] FLAG: --reserved-cpus=""
Jul 27 20:51:23.582813 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.575221    2560 flags.go:64] FLAG: --reserved-memory=""
Jul 27 20:51:23.582813 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.575224    2560 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 27 20:51:23.582813 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.575227    2560 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 27 20:51:23.583438 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.575230    2560 flags.go:64] FLAG: --rotate-certificates="false"
Jul 27 20:51:23.583438 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.575233    2560 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 27 20:51:23.583438 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.575236    2560 flags.go:64] FLAG: --runonce="false"
Jul 27 20:51:23.583438 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.575239    2560 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 27 20:51:23.583438 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.575242    2560 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 27 20:51:23.583438 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.575245    2560 flags.go:64] FLAG: --seccomp-default="false"
Jul 27 20:51:23.583438 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.575247    2560 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 27 20:51:23.583438 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.575250    2560 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 27 20:51:23.583438 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.575253    2560 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 27 20:51:23.583438 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.575257    2560 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 27 20:51:23.583438 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.575259    2560 flags.go:64] FLAG: --storage-driver-password="root"
Jul 27 20:51:23.583438 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.575262    2560 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 27 20:51:23.583438 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.575265    2560 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 27 20:51:23.583438 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.575270    2560 flags.go:64] FLAG: --storage-driver-user="root"
Jul 27 20:51:23.583438 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.575273    2560 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 27 20:51:23.583438 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.575276    2560 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 27 20:51:23.583438 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.575279    2560 flags.go:64] FLAG: --system-cgroups=""
Jul 27 20:51:23.583438 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.575281    2560 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 27 20:51:23.583438 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.575287    2560 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 27 20:51:23.583438 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.575290    2560 flags.go:64] FLAG: --tls-cert-file=""
Jul 27 20:51:23.583438 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.575293    2560 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 27 20:51:23.583438 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.575297    2560 flags.go:64] FLAG: --tls-min-version=""
Jul 27 20:51:23.583438 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.575300    2560 flags.go:64] FLAG: --tls-private-key-file=""
Jul 27 20:51:23.583438 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.575303    2560 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 27 20:51:23.583438 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.575306    2560 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 27 20:51:23.584099 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.575309    2560 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 27 20:51:23.584099 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.575312    2560 flags.go:64] FLAG: --v="2"
Jul 27 20:51:23.584099 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.575316    2560 flags.go:64] FLAG: --version="false"
Jul 27 20:51:23.584099 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.575321    2560 flags.go:64] FLAG: --vmodule=""
Jul 27 20:51:23.584099 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.575325    2560 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 27 20:51:23.584099 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.575328    2560 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 27 20:51:23.584099 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.575448    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 20:51:23.584099 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.575453    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 20:51:23.584099 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.575456    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 20:51:23.584099 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.575459    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 20:51:23.584099 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.575463    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 20:51:23.584099 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.575466    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 20:51:23.584099 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.575469    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 20:51:23.584099 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.575472    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 20:51:23.584099 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.575476    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 20:51:23.584099 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.575478    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 20:51:23.584099 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.575481    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 20:51:23.584099 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.575484    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 20:51:23.584099 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.575486    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 20:51:23.584099 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.575489    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 20:51:23.584593 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.575495    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 20:51:23.584593 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.575498    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 20:51:23.584593 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.575500    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 20:51:23.584593 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.575503    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 20:51:23.584593 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.575506    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 20:51:23.584593 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.575509    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 20:51:23.584593 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.575511    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 20:51:23.584593 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.575514    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 20:51:23.584593 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.575523    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 20:51:23.584593 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.575526    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 20:51:23.584593 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.575528    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 20:51:23.584593 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.575531    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 20:51:23.584593 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.575533    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 20:51:23.584593 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.575536    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 20:51:23.584593 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.575538    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 20:51:23.584593 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.575540    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 20:51:23.584593 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.575543    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 20:51:23.584593 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.575545    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 20:51:23.584593 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.575548    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 20:51:23.584593 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.575550    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 20:51:23.585070 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.575552    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 20:51:23.585070 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.575555    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 20:51:23.585070 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.575557    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 20:51:23.585070 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.575560    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 20:51:23.585070 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.575563    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 20:51:23.585070 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.575566    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 20:51:23.585070 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.575568    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 20:51:23.585070 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.575571    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 20:51:23.585070 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.575573    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 20:51:23.585070 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.575576    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 20:51:23.585070 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.575578    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 20:51:23.585070 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.575581    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 20:51:23.585070 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.575586    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 20:51:23.585070 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.575588    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 20:51:23.585070 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.575591    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 20:51:23.585070 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.575594    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 20:51:23.585070 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.575596    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 20:51:23.585070 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.575599    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 20:51:23.585070 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.575602    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 20:51:23.585070 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.575604    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 20:51:23.585565 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.575607    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 20:51:23.585565 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.575610    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 20:51:23.585565 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.575612    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 20:51:23.585565 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.575615    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 20:51:23.585565 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.575617    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 20:51:23.585565 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.575620    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 20:51:23.585565 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.575622    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 20:51:23.585565 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.575625    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 20:51:23.585565 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.575627    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 20:51:23.585565 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.575630    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 20:51:23.585565 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.575632    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 20:51:23.585565 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.575634    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 20:51:23.585565 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.575637    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 20:51:23.585565 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.575639    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 20:51:23.585565 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.575642    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 20:51:23.585565 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.575644    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 20:51:23.585565 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.575646    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 20:51:23.585565 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.575649    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 20:51:23.585565 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.575651    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 20:51:23.585565 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.575654    2560 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 20:51:23.585565 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.575656    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 20:51:23.586098 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.575658    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 20:51:23.586098 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.575661    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 20:51:23.586098 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.575664    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 20:51:23.586098 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.575670    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 20:51:23.586098 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.575673    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 20:51:23.586098 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.575676    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 20:51:23.586098 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.575678    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 20:51:23.586098 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.575681    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 20:51:23.586098 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.575684    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 20:51:23.586098 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.575686    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 20:51:23.586098 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.575688    2560 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 20:51:23.586098 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.575691    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 20:51:23.586098 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.575694    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 20:51:23.586098 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.575697    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 20:51:23.586098 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.575700    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 20:51:23.586098 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.575702    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 20:51:23.586098 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.575704    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 20:51:23.586098 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.575707    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 20:51:23.586098 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.575709    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 20:51:23.586556 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.575713    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 20:51:23.586556 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.575717    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 20:51:23.586556 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.575719    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 20:51:23.586556 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.575722    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 20:51:23.586556 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.575724    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 20:51:23.586556 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.576325    2560 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 27 20:51:23.586556 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.583451    2560 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 27 20:51:23.586556 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.583468    2560 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 27 20:51:23.586556 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.584464    2560 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 27 20:51:23.586556 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.584472    2560 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 27 20:51:23.586556 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.584531    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 20:51:23.586556 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.584536    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 20:51:23.586556 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.584540    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 20:51:23.586556 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.584543    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 20:51:23.586902 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.584547    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 20:51:23.586902 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.584549    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 20:51:23.586902 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.584552    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 20:51:23.586902 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.584555    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 20:51:23.586902 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.584558    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 20:51:23.586902 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.584561    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 20:51:23.586902 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.584563    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 20:51:23.586902 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.584566    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 20:51:23.586902 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.584568    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 20:51:23.586902 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.584571    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 20:51:23.586902 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.584573    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 20:51:23.586902 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.584576    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 20:51:23.586902 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.584579    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 20:51:23.586902 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.584582    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 20:51:23.586902 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.584584    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 20:51:23.586902 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.584587    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 20:51:23.586902 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.584590    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 20:51:23.586902 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.584593    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 20:51:23.586902 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.584595    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 20:51:23.586902 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.584598    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 20:51:23.587407 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.584601    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 20:51:23.587407 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.584603    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 20:51:23.587407 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.584606    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 20:51:23.587407 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.584609    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 20:51:23.587407 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.584612    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 20:51:23.587407 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.584616    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 20:51:23.587407 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.584619    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 20:51:23.587407 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.584621    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 20:51:23.587407 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.584624    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 20:51:23.587407 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.584627    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 20:51:23.587407 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.584629    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 20:51:23.587407 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.584632    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 20:51:23.587407 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.584635    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 20:51:23.587407 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.584637    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 20:51:23.587407 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.584640    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 20:51:23.587407 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.584642    2560 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 20:51:23.587407 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.584644    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 20:51:23.587407 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.584647    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 20:51:23.587407 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.584650    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 20:51:23.587407 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.584653    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 20:51:23.587880 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.584655    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 20:51:23.587880 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.584658    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 20:51:23.587880 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.584660    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 20:51:23.587880 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.584663    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 20:51:23.587880 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.584666    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 20:51:23.587880 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.584668    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 20:51:23.587880 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.584670    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 20:51:23.587880 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.584673    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 20:51:23.587880 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.584676    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 20:51:23.587880 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.584678    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 20:51:23.587880 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.584681    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 20:51:23.587880 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.584683    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 20:51:23.587880 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.584686    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 20:51:23.587880 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.584688    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 20:51:23.587880 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.584691    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 20:51:23.587880 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.584693    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 20:51:23.587880 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.584696    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 20:51:23.587880 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.584699    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 20:51:23.587880 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.584703    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 20:51:23.587880 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.584706    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 20:51:23.588387 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.584708    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 20:51:23.588387 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.584710    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 20:51:23.588387 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.584713    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 20:51:23.588387 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.584715    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 20:51:23.588387 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.584718    2560 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 20:51:23.588387 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.584720    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 20:51:23.588387 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.584723    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 20:51:23.588387 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.584725    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 20:51:23.588387 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.584727    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 20:51:23.588387 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.584730    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 20:51:23.588387 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.584734    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 20:51:23.588387 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.584738    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 20:51:23.588387 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.584741    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 20:51:23.588387 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.584743    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 20:51:23.588387 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.584746    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 20:51:23.588387 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.584749    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 20:51:23.588387 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.584753    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 20:51:23.588387 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.584757    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 20:51:23.588873 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.584760    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 20:51:23.588873 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.584763    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 20:51:23.588873 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.584766    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 20:51:23.588873 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.584769    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 20:51:23.588873 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.584771    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 20:51:23.588873 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.584774    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 20:51:23.588873 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.584776    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 20:51:23.588873 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.584779    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 20:51:23.588873 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.584781    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 20:51:23.588873 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.584784    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 20:51:23.588873 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.584786    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 20:51:23.588873 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.584789    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 20:51:23.588873 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.584792    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 20:51:23.588873 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.584795    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 20:51:23.588873 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.584798    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 20:51:23.588873 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.584800    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 20:51:23.588873 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.584803    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 20:51:23.589710 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.584808    2560 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 27 20:51:23.589710 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.584926    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 20:51:23.589710 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.584932    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 20:51:23.589710 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.584935    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 20:51:23.589710 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.584937    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 20:51:23.589710 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.584940    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 20:51:23.589710 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.584943    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 20:51:23.589710 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.584946    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 20:51:23.589710 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.584949    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 20:51:23.589710 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.584952    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 20:51:23.589710 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.584955    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 20:51:23.589710 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.584957    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 20:51:23.589710 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.584960    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 20:51:23.589710 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.584962    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 20:51:23.589710 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.584965    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 20:51:23.590182 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.584967    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 20:51:23.590182 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.584969    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 20:51:23.590182 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.584972    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 20:51:23.590182 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.584974    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 20:51:23.590182 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.584977    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 20:51:23.590182 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.584979    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 20:51:23.590182 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.584982    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 20:51:23.590182 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.584984    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 20:51:23.590182 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.584986    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 20:51:23.590182 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.584989    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 20:51:23.590182 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.584991    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 20:51:23.590182 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.584994    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 20:51:23.590182 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.584997    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 20:51:23.590182 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.585000    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 20:51:23.590182 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.585003    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 20:51:23.590182 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.585006    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 20:51:23.590182 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.585009    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 20:51:23.590182 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.585011    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 20:51:23.590182 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.585014    2560 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 20:51:23.590182 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.585017    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 20:51:23.590693 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.585019    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 20:51:23.590693 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.585022    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 20:51:23.590693 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.585024    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 20:51:23.590693 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.585026    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 20:51:23.590693 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.585029    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 20:51:23.590693 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.585031    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 20:51:23.590693 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.585034    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 20:51:23.590693 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.585036    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 20:51:23.590693 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.585039    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 20:51:23.590693 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.585041    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 20:51:23.590693 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.585043    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 20:51:23.590693 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.585046    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 20:51:23.590693 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.585049    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 20:51:23.590693 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.585052    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 20:51:23.590693 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.585054    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 20:51:23.590693 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.585058    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 20:51:23.590693 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.585062    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 20:51:23.590693 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.585065    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 20:51:23.590693 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.585068    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 20:51:23.591178 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.585070    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 20:51:23.591178 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.585074    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 20:51:23.591178 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.585076    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 20:51:23.591178 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.585079    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 20:51:23.591178 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.585082    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 20:51:23.591178 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.585100    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 20:51:23.591178 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.585104    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 20:51:23.591178 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.585109    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 20:51:23.591178 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.585113    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 20:51:23.591178 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.585116    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 20:51:23.591178 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.585119    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 20:51:23.591178 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.585121    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 20:51:23.591178 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.585124    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 20:51:23.591178 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.585127    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 20:51:23.591178 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.585129    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 20:51:23.591178 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.585132    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 20:51:23.591178 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.585134    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 20:51:23.591178 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.585137    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 20:51:23.591178 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.585139    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 20:51:23.591178 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.585142    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 20:51:23.591658 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.585144    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 20:51:23.591658 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.585147    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 20:51:23.591658 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.585150    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 20:51:23.591658 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.585152    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 20:51:23.591658 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.585154    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 20:51:23.591658 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.585157    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 20:51:23.591658 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.585159    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 20:51:23.591658 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.585162    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 20:51:23.591658 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.585165    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 20:51:23.591658 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.585168    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 20:51:23.591658 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.585171    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 20:51:23.591658 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.585174    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 20:51:23.591658 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.585177    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 20:51:23.591658 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.585180    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 20:51:23.591658 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.585182    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 20:51:23.591658 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.585185    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 20:51:23.591658 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.585188    2560 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 20:51:23.591658 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.585191    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 20:51:23.591658 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.585194    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 20:51:23.592131 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.585196    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 20:51:23.592131 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.585199    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 20:51:23.592131 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.585202    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 20:51:23.592131 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.585205    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 20:51:23.592131 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.585208    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 20:51:23.592131 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.585211    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 20:51:23.592131 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:23.585213    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 20:51:23.592131 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.585218    2560 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 27 20:51:23.592131 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.585351    2560 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 27 20:51:23.592131 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.587459    2560 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 27 20:51:23.592131 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.590511    2560 server.go:1019] "Starting client certificate rotation"
Jul 27 20:51:23.592131 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.590774    2560 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 27 20:51:23.592131 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.590809    2560 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 27 20:51:23.593755 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.593737    2560 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 27 20:51:23.595017 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.594988    2560 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 27 20:51:23.612524 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.612500    2560 log.go:25] "Validated CRI v1 runtime API"
Jul 27 20:51:23.614918 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.614900    2560 log.go:25] "Validated CRI v1 image API"
Jul 27 20:51:23.615851 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.615831    2560 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 27 20:51:23.619290 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.619259    2560 fs.go:135] Filesystem UUIDs: map[3fe6dee2-6e07-4653-9ec2-9cadf545da40:/dev/nvme0n1p3 63d477aa-95d1-45d9-a4bf-64163d2106cf:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Jul 27 20:51:23.619290 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.619285    2560 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 27 20:51:23.622480 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.622459    2560 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 27 20:51:23.624448 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.624328    2560 manager.go:216] Machine: {Timestamp:2026-07-27 20:51:23.622899942 +0000 UTC m=+0.374761739 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3134697 MemoryCapacity:32812175360 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec23d713878736d212336785737b96da SystemUUID:ec23d713-8787-36d2-1233-6785737b96da BootID:13f46450-aa4e-4a60-92b8-269232d500dd Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:46:ce:d4:2f:97 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:46:ce:d4:2f:97 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:4a:b1:9e:98:b7:1e Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812175360 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 27 20:51:23.624448 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.624436    2560 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 27 20:51:23.624584 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.624531    2560 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 27 20:51:23.627005 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.626978    2560 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 27 20:51:23.627167 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.627008    2560 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-64.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 27 20:51:23.627878 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.627867    2560 topology_manager.go:138] "Creating topology manager with none policy"
Jul 27 20:51:23.627913 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.627881    2560 container_manager_linux.go:309] "Creating device plugin manager"
Jul 27 20:51:23.627913 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.627889    2560 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 27 20:51:23.628617 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.628607    2560 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 27 20:51:23.629660 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.629650    2560 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 27 20:51:23.631318 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.631306    2560 state_mem.go:36] "Initialized new in-memory state store"
Jul 27 20:51:23.631479 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.631469    2560 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 27 20:51:23.631518 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.631501    2560 kubelet.go:486] "Attempting to sync node with API server"
Jul 27 20:51:23.631518 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.631511    2560 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 27 20:51:23.631630 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.631522    2560 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 27 20:51:23.631630 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.631532    2560 kubelet.go:398] "Adding apiserver pod source"
Jul 27 20:51:23.631630 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.631544    2560 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 27 20:51:23.631947 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.631933    2560 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 27 20:51:23.631977 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.631968    2560 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 27 20:51:23.635485 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.635385    2560 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 27 20:51:23.637294 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.637276    2560 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 27 20:51:23.637294 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.637309    2560 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 27 20:51:23.639349 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.639335    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 27 20:51:23.639349 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.639352    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 27 20:51:23.639448 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.639359    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 27 20:51:23.639448 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.639365    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 27 20:51:23.639448 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.639374    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 27 20:51:23.639448 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.639380    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 27 20:51:23.639448 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.639387    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 27 20:51:23.639448 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.639392    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 27 20:51:23.639448 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.639399    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 27 20:51:23.639448 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.639405    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 27 20:51:23.639448 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.639414    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 27 20:51:23.639448 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.639419    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 27 20:51:23.639448 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.639440    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 27 20:51:23.639448 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.639446    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 27 20:51:23.642775 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.642757    2560 server.go:1268] "Started kubelet"
Jul 27 20:51:23.642891 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.642834    2560 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 27 20:51:23.643103 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.642933    2560 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 27 20:51:23.643190 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.643130    2560 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 27 20:51:23.643719 ip-10-0-141-64 systemd[1]: Started Kubernetes Kubelet.
Jul 27 20:51:23.644336 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.644193    2560 server.go:310] "Adding debug handlers to kubelet server"
Jul 27 20:51:23.644686 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.644667    2560 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-64.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 27 20:51:23.645055 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.645040    2560 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 27 20:51:23.645501 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:23.645483    2560 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-64.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 27 20:51:23.645569 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:23.645487    2560 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 27 20:51:23.648734 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:23.648711    2560 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 27 20:51:23.651544 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.651528    2560 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 27 20:51:23.651631 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.651608    2560 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 27 20:51:23.651778 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:23.651696    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-64.ec2.internal\" not found"
Jul 27 20:51:23.652400 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.652380    2560 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 27 20:51:23.652400 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.652385    2560 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 27 20:51:23.652584 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.652414    2560 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 27 20:51:23.652584 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.652491    2560 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 27 20:51:23.652584 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.652513    2560 factory.go:55] Registering systemd factory
Jul 27 20:51:23.652584 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.652519    2560 reconstruct.go:97] "Volume reconstruction finished"
Jul 27 20:51:23.652584 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.652531    2560 reconciler.go:29] "Reconciler: start to sync state"
Jul 27 20:51:23.652584 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.652523    2560 factory.go:223] Registration of the systemd container factory successfully
Jul 27 20:51:23.652923 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.652814    2560 factory.go:153] Registering CRI-O factory
Jul 27 20:51:23.652923 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.652832    2560 factory.go:223] Registration of the crio container factory successfully
Jul 27 20:51:23.652923 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.652858    2560 factory.go:103] Registering Raw factory
Jul 27 20:51:23.652923 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.652873    2560 manager.go:1196] Started watching for new ooms in manager
Jul 27 20:51:23.653413 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.653397    2560 manager.go:349] Starting recovery of all containers
Jul 27 20:51:23.654219 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.654188    2560 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-z6tps"
Jul 27 20:51:23.654443 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:23.653244    2560 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-64.ec2.internal.18c640bd32b57713  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-64.ec2.internal,UID:ip-10-0-141-64.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-64.ec2.internal,},FirstTimestamp:2026-07-27 20:51:23.642730259 +0000 UTC m=+0.394592046,LastTimestamp:2026-07-27 20:51:23.642730259 +0000 UTC m=+0.394592046,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-64.ec2.internal,}"
Jul 27 20:51:23.658310 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:23.658282    2560 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-141-64.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 27 20:51:23.658427 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:23.658403    2560 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 27 20:51:23.664053 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.664021    2560 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-z6tps"
Jul 27 20:51:23.664510 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.664491    2560 manager.go:354] Recovery completed
Jul 27 20:51:23.666686 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:23.666654    2560 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Jul 27 20:51:23.669793 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.669779    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 20:51:23.671579 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.671561    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-64.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 20:51:23.671661 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.671588    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-64.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 20:51:23.671661 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.671598    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-64.ec2.internal" event="NodeHasSufficientPID"
Jul 27 20:51:23.672164 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.672150    2560 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 27 20:51:23.672164 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.672158    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 20:51:23.672273 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.672162    2560 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 27 20:51:23.672273 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.672196    2560 state_mem.go:36] "Initialized new in-memory state store"
Jul 27 20:51:23.672848 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.672835    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-64.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 20:51:23.672899 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.672862    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-64.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 20:51:23.672899 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.672872    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-64.ec2.internal" event="NodeHasSufficientPID"
Jul 27 20:51:23.673454 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:23.673371    2560 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-64.ec2.internal.18c640bd346d9ffd  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-64.ec2.internal,UID:ip-10-0-141-64.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-141-64.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-141-64.ec2.internal,},FirstTimestamp:2026-07-27 20:51:23.671576573 +0000 UTC m=+0.423438360,LastTimestamp:2026-07-27 20:51:23.671576573 +0000 UTC m=+0.423438360,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-64.ec2.internal,}"
Jul 27 20:51:23.673717 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.673707    2560 policy_none.go:49] "None policy: Start"
Jul 27 20:51:23.673754 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.673722    2560 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 27 20:51:23.673754 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.673732    2560 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 27 20:51:23.674649 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.674638    2560 policy_none.go:47] "Start"
Jul 27 20:51:23.710908 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.710701    2560 manager.go:337] "Starting Device Plugin manager"
Jul 27 20:51:23.711053 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:23.710934    2560 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 27 20:51:23.711053 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.710947    2560 server.go:87] "Starting device plugin registration server"
Jul 27 20:51:23.711220 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.711208    2560 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 27 20:51:23.711295 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.711230    2560 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 27 20:51:23.711425 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.711404    2560 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 27 20:51:23.711532 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.711510    2560 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 27 20:51:23.711532 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.711530    2560 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 27 20:51:23.712122 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:23.712104    2560 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 27 20:51:23.712220 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:23.712161    2560 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-64.ec2.internal\" not found"
Jul 27 20:51:23.812367 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.812319    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 20:51:23.813405 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.813329    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-64.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 20:51:23.813405 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.813364    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-64.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 20:51:23.813405 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.813374    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-64.ec2.internal" event="NodeHasSufficientPID"
Jul 27 20:51:23.813405 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.813400    2560 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-141-64.ec2.internal"
Jul 27 20:51:23.820502 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.820443    2560 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-141-64.ec2.internal"
Jul 27 20:51:23.820502 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:23.820472    2560 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-64.ec2.internal\": node \"ip-10-0-141-64.ec2.internal\" not found"
Jul 27 20:51:23.831346 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:23.831323    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-64.ec2.internal\" not found"
Jul 27 20:51:23.839828 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.839792    2560 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 27 20:51:23.841196 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.841179    2560 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 27 20:51:23.841260 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.841219    2560 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 27 20:51:23.841260 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.841244    2560 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 27 20:51:23.841349 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:23.841278    2560 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 27 20:51:23.843947 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.843924    2560 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 20:51:23.932118 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:23.932046    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-64.ec2.internal\" not found"
Jul 27 20:51:23.942209 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.942183    2560 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-64.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-64.ec2.internal"]
Jul 27 20:51:23.942275 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.942262    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 20:51:23.943309 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.943292    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-64.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 20:51:23.943386 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.943324    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-64.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 20:51:23.943386 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.943334    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-64.ec2.internal" event="NodeHasSufficientPID"
Jul 27 20:51:23.944581 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.944568    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 20:51:23.944702 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.944684    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-64.ec2.internal"
Jul 27 20:51:23.944737 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.944725    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 20:51:23.945376 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.945358    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-64.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 20:51:23.945456 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.945389    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-64.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 20:51:23.945456 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.945401    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-64.ec2.internal" event="NodeHasSufficientPID"
Jul 27 20:51:23.945456 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.945415    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-64.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 20:51:23.945456 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.945436    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-64.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 20:51:23.945456 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.945449    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-64.ec2.internal" event="NodeHasSufficientPID"
Jul 27 20:51:23.946626 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.946602    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-64.ec2.internal"
Jul 27 20:51:23.946742 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.946633    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 20:51:23.947353 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.947337    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-64.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 20:51:23.947456 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.947361    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-64.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 20:51:23.947456 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.947374    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-64.ec2.internal" event="NodeHasSufficientPID"
Jul 27 20:51:23.953871 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.953853    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e132fc0b72959a63cea0555c65890a47-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-64.ec2.internal\" (UID: \"e132fc0b72959a63cea0555c65890a47\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-64.ec2.internal"
Jul 27 20:51:23.953946 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.953878    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e132fc0b72959a63cea0555c65890a47-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-64.ec2.internal\" (UID: \"e132fc0b72959a63cea0555c65890a47\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-64.ec2.internal"
Jul 27 20:51:23.953946 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:23.953896    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5fa776c99382db40a7e9c25fe5c95038-config\") pod \"kube-apiserver-proxy-ip-10-0-141-64.ec2.internal\" (UID: \"5fa776c99382db40a7e9c25fe5c95038\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-64.ec2.internal"
Jul 27 20:51:23.967878 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:23.967848    2560 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-64.ec2.internal\" not found" node="ip-10-0-141-64.ec2.internal"
Jul 27 20:51:23.972315 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:23.972299    2560 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-64.ec2.internal\" not found" node="ip-10-0-141-64.ec2.internal"
Jul 27 20:51:24.033004 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:24.032979    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-64.ec2.internal\" not found"
Jul 27 20:51:24.054066 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:24.054037    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e132fc0b72959a63cea0555c65890a47-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-64.ec2.internal\" (UID: \"e132fc0b72959a63cea0555c65890a47\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-64.ec2.internal"
Jul 27 20:51:24.054147 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:24.054071    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5fa776c99382db40a7e9c25fe5c95038-config\") pod \"kube-apiserver-proxy-ip-10-0-141-64.ec2.internal\" (UID: \"5fa776c99382db40a7e9c25fe5c95038\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-64.ec2.internal"
Jul 27 20:51:24.054147 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:24.054119    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e132fc0b72959a63cea0555c65890a47-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-64.ec2.internal\" (UID: \"e132fc0b72959a63cea0555c65890a47\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-64.ec2.internal"
Jul 27 20:51:24.054147 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:24.054139    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e132fc0b72959a63cea0555c65890a47-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-64.ec2.internal\" (UID: \"e132fc0b72959a63cea0555c65890a47\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-64.ec2.internal"
Jul 27 20:51:24.054259 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:24.054184    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e132fc0b72959a63cea0555c65890a47-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-64.ec2.internal\" (UID: \"e132fc0b72959a63cea0555c65890a47\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-64.ec2.internal"
Jul 27 20:51:24.054259 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:24.054213    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5fa776c99382db40a7e9c25fe5c95038-config\") pod \"kube-apiserver-proxy-ip-10-0-141-64.ec2.internal\" (UID: \"5fa776c99382db40a7e9c25fe5c95038\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-64.ec2.internal"
Jul 27 20:51:24.133745 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:24.133673    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-64.ec2.internal\" not found"
Jul 27 20:51:24.234191 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:24.234146    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-64.ec2.internal\" not found"
Jul 27 20:51:24.270760 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:24.270731    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-64.ec2.internal"
Jul 27 20:51:24.276889 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:24.276866    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-64.ec2.internal"
Jul 27 20:51:24.334840 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:24.334793    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-64.ec2.internal\" not found"
Jul 27 20:51:24.435412 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:24.435314    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-64.ec2.internal\" not found"
Jul 27 20:51:24.538374 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:24.538327    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-64.ec2.internal\" not found"
Jul 27 20:51:24.589685 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:24.589656    2560 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 27 20:51:24.590183 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:24.589814    2560 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 27 20:51:24.638838 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:24.638781    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-64.ec2.internal\" not found"
Jul 27 20:51:24.669777 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:24.669738    2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-26 20:46:24 +0000 UTC" deadline="2028-03-07 10:07:03.963571396 +0000 UTC"
Jul 27 20:51:24.669777 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:24.669767    2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14125h15m39.293808323s"
Jul 27 20:51:24.739400 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:24.739316    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-64.ec2.internal\" not found"
Jul 27 20:51:24.749460 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:24.749436    2560 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 20:51:24.784253 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:24.784217    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fa776c99382db40a7e9c25fe5c95038.slice/crio-e87e5589e6df9aeaec16c34639dd59985408fd8ac95ef2182d5a5dd86536c008 WatchSource:0}: Error finding container e87e5589e6df9aeaec16c34639dd59985408fd8ac95ef2182d5a5dd86536c008: Status 404 returned error can't find the container with id e87e5589e6df9aeaec16c34639dd59985408fd8ac95ef2182d5a5dd86536c008
Jul 27 20:51:24.784852 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:24.784828    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode132fc0b72959a63cea0555c65890a47.slice/crio-4f20057457d2d595f4881e0feae06a0f0ddce6cb52deccbb9ab26a10e82ff21e WatchSource:0}: Error finding container 4f20057457d2d595f4881e0feae06a0f0ddce6cb52deccbb9ab26a10e82ff21e: Status 404 returned error can't find the container with id 4f20057457d2d595f4881e0feae06a0f0ddce6cb52deccbb9ab26a10e82ff21e
Jul 27 20:51:24.790179 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:24.790159    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 20:51:24.806537 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:24.806514    2560 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 20:51:24.846143 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:24.846058    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-64.ec2.internal" event={"ID":"e132fc0b72959a63cea0555c65890a47","Type":"ContainerStarted","Data":"4f20057457d2d595f4881e0feae06a0f0ddce6cb52deccbb9ab26a10e82ff21e"}
Jul 27 20:51:24.847367 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:24.847341    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-64.ec2.internal" event={"ID":"5fa776c99382db40a7e9c25fe5c95038","Type":"ContainerStarted","Data":"e87e5589e6df9aeaec16c34639dd59985408fd8ac95ef2182d5a5dd86536c008"}
Jul 27 20:51:24.852727 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:24.852706    2560 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-64.ec2.internal"
Jul 27 20:51:24.866170 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:24.866146    2560 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 27 20:51:24.868560 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:24.868544    2560 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-64.ec2.internal"
Jul 27 20:51:24.881144 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:24.881120    2560 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 27 20:51:25.121687 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.121594    2560 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 20:51:25.632288 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.632257    2560 apiserver.go:52] "Watching apiserver"
Jul 27 20:51:25.646600 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.646573    2560 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 27 20:51:25.647351 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.647323    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-8cdfg","openshift-ovn-kubernetes/ovnkube-node-95hg7","kube-system/global-pull-secret-syncer-dtrjg","kube-system/konnectivity-agent-sg57x","kube-system/kube-apiserver-proxy-ip-10-0-141-64.ec2.internal","openshift-cluster-node-tuning-operator/tuned-gn9bl","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-64.ec2.internal","openshift-multus/multus-additional-cni-plugins-djcnj","openshift-multus/network-metrics-daemon-r6g8d","openshift-network-diagnostics/network-check-target-dbzwm","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tjtf6","openshift-dns/node-resolver-g5tm9","openshift-image-registry/node-ca-jm574","openshift-multus/multus-td78h"]
Jul 27 20:51:25.649133 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.649089    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dbzwm"
Jul 27 20:51:25.649268 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:25.649217    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dbzwm" podUID="1f1259cb-be97-4841-bd22-5bba585870b6"
Jul 27 20:51:25.650705 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.650680    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tjtf6"
Jul 27 20:51:25.651940 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.651826    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dtrjg"
Jul 27 20:51:25.651940 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:25.651896    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dtrjg" podUID="4b924e42-83bc-459f-b1a2-f115f9bce63f"
Jul 27 20:51:25.652921 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.652897    2560 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 27 20:51:25.653758 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.653741    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 27 20:51:25.654152 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.654088    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 27 20:51:25.654152 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.654143    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-scm48\""
Jul 27 20:51:25.654326 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.654179    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 27 20:51:25.654462 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.654353    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-sg57x"
Jul 27 20:51:25.655798 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.655757    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-td78h"
Jul 27 20:51:25.656526 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.656503    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 27 20:51:25.657160 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.656909    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 27 20:51:25.657160 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.656968    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-q74xz\""
Jul 27 20:51:25.657317 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.657177    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-gn9bl"
Jul 27 20:51:25.660187 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.658055    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 27 20:51:25.660187 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.658521    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-s2wrx\""
Jul 27 20:51:25.660187 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.658723    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 27 20:51:25.660187 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.658902    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 27 20:51:25.660187 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.659075    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 27 20:51:25.660187 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.659517    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 27 20:51:25.660187 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.659689    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 27 20:51:25.660187 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.659869    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-v66dj\""
Jul 27 20:51:25.660187 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.659992    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r6g8d"
Jul 27 20:51:25.660187 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:25.660054    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r6g8d" podUID="fac6ee6e-225a-4402-8834-69ff744b07f9"
Jul 27 20:51:25.660187 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.660186    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jm574"
Jul 27 20:51:25.662031 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.661970    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a49f6f21-5110-4f7c-8f79-1d9e14d69bc6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tjtf6\" (UID: \"a49f6f21-5110-4f7c-8f79-1d9e14d69bc6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tjtf6"
Jul 27 20:51:25.662031 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.662008    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a49f6f21-5110-4f7c-8f79-1d9e14d69bc6-registration-dir\") pod \"aws-ebs-csi-driver-node-tjtf6\" (UID: \"a49f6f21-5110-4f7c-8f79-1d9e14d69bc6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tjtf6"
Jul 27 20:51:25.662219 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.662038    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a49f6f21-5110-4f7c-8f79-1d9e14d69bc6-sys-fs\") pod \"aws-ebs-csi-driver-node-tjtf6\" (UID: \"a49f6f21-5110-4f7c-8f79-1d9e14d69bc6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tjtf6"
Jul 27 20:51:25.662219 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.662065    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8ns2\" (UniqueName: \"kubernetes.io/projected/1f1259cb-be97-4841-bd22-5bba585870b6-kube-api-access-n8ns2\") pod \"network-check-target-dbzwm\" (UID: \"1f1259cb-be97-4841-bd22-5bba585870b6\") " pod="openshift-network-diagnostics/network-check-target-dbzwm"
Jul 27 20:51:25.662219 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.662090    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a49f6f21-5110-4f7c-8f79-1d9e14d69bc6-socket-dir\") pod \"aws-ebs-csi-driver-node-tjtf6\" (UID: \"a49f6f21-5110-4f7c-8f79-1d9e14d69bc6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tjtf6"
Jul 27 20:51:25.662219 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.662132    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a49f6f21-5110-4f7c-8f79-1d9e14d69bc6-device-dir\") pod \"aws-ebs-csi-driver-node-tjtf6\" (UID: \"a49f6f21-5110-4f7c-8f79-1d9e14d69bc6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tjtf6"
Jul 27 20:51:25.662219 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.662156    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4b924e42-83bc-459f-b1a2-f115f9bce63f-kubelet-config\") pod \"global-pull-secret-syncer-dtrjg\" (UID: \"4b924e42-83bc-459f-b1a2-f115f9bce63f\") " pod="kube-system/global-pull-secret-syncer-dtrjg"
Jul 27 20:51:25.662219 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.662174    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-95hg7"
Jul 27 20:51:25.662219 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.662181    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4b924e42-83bc-459f-b1a2-f115f9bce63f-dbus\") pod \"global-pull-secret-syncer-dtrjg\" (UID: \"4b924e42-83bc-459f-b1a2-f115f9bce63f\") " pod="kube-system/global-pull-secret-syncer-dtrjg"
Jul 27 20:51:25.662219 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.662207    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a49f6f21-5110-4f7c-8f79-1d9e14d69bc6-etc-selinux\") pod \"aws-ebs-csi-driver-node-tjtf6\" (UID: \"a49f6f21-5110-4f7c-8f79-1d9e14d69bc6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tjtf6"
Jul 27 20:51:25.662730 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.662232    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49fsn\" (UniqueName: \"kubernetes.io/projected/a49f6f21-5110-4f7c-8f79-1d9e14d69bc6-kube-api-access-49fsn\") pod \"aws-ebs-csi-driver-node-tjtf6\" (UID: \"a49f6f21-5110-4f7c-8f79-1d9e14d69bc6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tjtf6"
Jul 27 20:51:25.662730 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.662259    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4b924e42-83bc-459f-b1a2-f115f9bce63f-original-pull-secret\") pod \"global-pull-secret-syncer-dtrjg\" (UID: \"4b924e42-83bc-459f-b1a2-f115f9bce63f\") " pod="kube-system/global-pull-secret-syncer-dtrjg"
Jul 27 20:51:25.662730 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.662283    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9a766ad8-ce17-462e-b45f-867a6c6c44e4-agent-certs\") pod \"konnectivity-agent-sg57x\" (UID: \"9a766ad8-ce17-462e-b45f-867a6c6c44e4\") " pod="kube-system/konnectivity-agent-sg57x"
Jul 27 20:51:25.662730 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.662305    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9a766ad8-ce17-462e-b45f-867a6c6c44e4-konnectivity-ca\") pod \"konnectivity-agent-sg57x\" (UID: \"9a766ad8-ce17-462e-b45f-867a6c6c44e4\") " pod="kube-system/konnectivity-agent-sg57x"
Jul 27 20:51:25.662859 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.662834    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-nsqjl\""
Jul 27 20:51:25.662942 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.662917    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 27 20:51:25.663007 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.662917    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 27 20:51:25.663007 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.662991    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 27 20:51:25.663831 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.663812    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-8cdfg"
Jul 27 20:51:25.665453 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.665430    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 27 20:51:25.665805 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.665782    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 27 20:51:25.666618 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.666589    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 27 20:51:25.666854 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.666803    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 27 20:51:25.667059 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.667037    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-lw4z9\""
Jul 27 20:51:25.667317 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.667296    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 27 20:51:25.667605 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.667581    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 27 20:51:25.667895 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.667876    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 27 20:51:25.668403 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.668383    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 27 20:51:25.668611 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.668594    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 27 20:51:25.668809 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.668792    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-jvrwr\""
Jul 27 20:51:25.672046 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.672026    2560 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 27 20:51:25.674325 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.674309    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-djcnj"
Jul 27 20:51:25.675714 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.675695    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-g5tm9"
Jul 27 20:51:25.677098 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.677030    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 27 20:51:25.677503 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.677467    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-67m2m\""
Jul 27 20:51:25.677654 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.677638    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 27 20:51:25.678070 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.678045    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 27 20:51:25.678428 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.678408    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-m6xzk\""
Jul 27 20:51:25.678788 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.678631    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 27 20:51:25.694313 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.694293    2560 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-b6tjl"
Jul 27 20:51:25.700875 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.700849    2560 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-b6tjl"
Jul 27 20:51:25.753964 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.753940    2560 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 27 20:51:25.762729 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.762700    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dbae1ee6-3616-432d-8ef9-255ad4cd444e-cni-binary-copy\") pod \"multus-td78h\" (UID: \"dbae1ee6-3616-432d-8ef9-255ad4cd444e\") " pod="openshift-multus/multus-td78h"
Jul 27 20:51:25.762862 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.762732    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/600fc153-38e0-4874-92e2-5895e14fa506-run\") pod \"tuned-gn9bl\" (UID: \"600fc153-38e0-4874-92e2-5895e14fa506\") " pod="openshift-cluster-node-tuning-operator/tuned-gn9bl"
Jul 27 20:51:25.762862 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.762749    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/600fc153-38e0-4874-92e2-5895e14fa506-tmp\") pod \"tuned-gn9bl\" (UID: \"600fc153-38e0-4874-92e2-5895e14fa506\") " pod="openshift-cluster-node-tuning-operator/tuned-gn9bl"
Jul 27 20:51:25.762862 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.762766    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/433c431a-b5bf-4734-a03f-6f0a16895ded-run-openvswitch\") pod \"ovnkube-node-95hg7\" (UID: \"433c431a-b5bf-4734-a03f-6f0a16895ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-95hg7"
Jul 27 20:51:25.762862 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.762796    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/433c431a-b5bf-4734-a03f-6f0a16895ded-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-95hg7\" (UID: \"433c431a-b5bf-4734-a03f-6f0a16895ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-95hg7"
Jul 27 20:51:25.762862 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.762824    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fac6ee6e-225a-4402-8834-69ff744b07f9-metrics-certs\") pod \"network-metrics-daemon-r6g8d\" (UID: \"fac6ee6e-225a-4402-8834-69ff744b07f9\") " pod="openshift-multus/network-metrics-daemon-r6g8d"
Jul 27 20:51:25.762862 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.762841    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/eae6d6c0-9844-45a8-a5ef-9c6f1ecc2810-cnibin\") pod \"multus-additional-cni-plugins-djcnj\" (UID: \"eae6d6c0-9844-45a8-a5ef-9c6f1ecc2810\") " pod="openshift-multus/multus-additional-cni-plugins-djcnj"
Jul 27 20:51:25.763193 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.762896    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4b924e42-83bc-459f-b1a2-f115f9bce63f-kubelet-config\") pod \"global-pull-secret-syncer-dtrjg\" (UID: \"4b924e42-83bc-459f-b1a2-f115f9bce63f\") " pod="kube-system/global-pull-secret-syncer-dtrjg"
Jul 27 20:51:25.763193 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.762931    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dbae1ee6-3616-432d-8ef9-255ad4cd444e-multus-conf-dir\") pod \"multus-td78h\" (UID: \"dbae1ee6-3616-432d-8ef9-255ad4cd444e\") " pod="openshift-multus/multus-td78h"
Jul 27 20:51:25.763193 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.762951    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/dbae1ee6-3616-432d-8ef9-255ad4cd444e-host-run-multus-certs\") pod \"multus-td78h\" (UID: \"dbae1ee6-3616-432d-8ef9-255ad4cd444e\") " pod="openshift-multus/multus-td78h"
Jul 27 20:51:25.763193 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.762978    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3566e5fb-c112-4f14-9e2f-fce1d833e15a-iptables-alerter-script\") pod \"iptables-alerter-8cdfg\" (UID: \"3566e5fb-c112-4f14-9e2f-fce1d833e15a\") " pod="openshift-network-operator/iptables-alerter-8cdfg"
Jul 27 20:51:25.763193 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.763004    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kkwm\" (UniqueName: \"kubernetes.io/projected/412ea27a-182d-40cc-917e-4686a4ff65cb-kube-api-access-7kkwm\") pod \"node-ca-jm574\" (UID: \"412ea27a-182d-40cc-917e-4686a4ff65cb\") " pod="openshift-image-registry/node-ca-jm574"
Jul 27 20:51:25.763193 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.763004    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4b924e42-83bc-459f-b1a2-f115f9bce63f-kubelet-config\") pod \"global-pull-secret-syncer-dtrjg\" (UID: \"4b924e42-83bc-459f-b1a2-f115f9bce63f\") " pod="kube-system/global-pull-secret-syncer-dtrjg"
Jul 27 20:51:25.763193 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.763023    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzh9m\" (UniqueName: \"kubernetes.io/projected/eae6d6c0-9844-45a8-a5ef-9c6f1ecc2810-kube-api-access-bzh9m\") pod \"multus-additional-cni-plugins-djcnj\" (UID: \"eae6d6c0-9844-45a8-a5ef-9c6f1ecc2810\") " pod="openshift-multus/multus-additional-cni-plugins-djcnj"
Jul 27 20:51:25.763193 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.763052    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a49f6f21-5110-4f7c-8f79-1d9e14d69bc6-etc-selinux\") pod \"aws-ebs-csi-driver-node-tjtf6\" (UID: \"a49f6f21-5110-4f7c-8f79-1d9e14d69bc6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tjtf6"
Jul 27 20:51:25.763193 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.763081    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9a766ad8-ce17-462e-b45f-867a6c6c44e4-konnectivity-ca\") pod \"konnectivity-agent-sg57x\" (UID: \"9a766ad8-ce17-462e-b45f-867a6c6c44e4\") " pod="kube-system/konnectivity-agent-sg57x"
Jul 27 20:51:25.763193 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.763123    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/dbae1ee6-3616-432d-8ef9-255ad4cd444e-hostroot\") pod \"multus-td78h\" (UID: \"dbae1ee6-3616-432d-8ef9-255ad4cd444e\") " pod="openshift-multus/multus-td78h"
Jul 27 20:51:25.763193 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.763144    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/600fc153-38e0-4874-92e2-5895e14fa506-etc-tuned\") pod \"tuned-gn9bl\" (UID: \"600fc153-38e0-4874-92e2-5895e14fa506\") " pod="openshift-cluster-node-tuning-operator/tuned-gn9bl"
Jul 27 20:51:25.763193 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.763164    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/433c431a-b5bf-4734-a03f-6f0a16895ded-ovnkube-script-lib\") pod \"ovnkube-node-95hg7\" (UID: \"433c431a-b5bf-4734-a03f-6f0a16895ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-95hg7"
Jul 27 20:51:25.763193 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.763181    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a49f6f21-5110-4f7c-8f79-1d9e14d69bc6-etc-selinux\") pod \"aws-ebs-csi-driver-node-tjtf6\" (UID: \"a49f6f21-5110-4f7c-8f79-1d9e14d69bc6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tjtf6"
Jul 27 20:51:25.763193 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.763191    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/eae6d6c0-9844-45a8-a5ef-9c6f1ecc2810-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-djcnj\" (UID: \"eae6d6c0-9844-45a8-a5ef-9c6f1ecc2810\") " pod="openshift-multus/multus-additional-cni-plugins-djcnj"
Jul 27 20:51:25.763910 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.763222    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a49f6f21-5110-4f7c-8f79-1d9e14d69bc6-registration-dir\") pod \"aws-ebs-csi-driver-node-tjtf6\" (UID: \"a49f6f21-5110-4f7c-8f79-1d9e14d69bc6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tjtf6"
Jul 27 20:51:25.763910 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.763246    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/600fc153-38e0-4874-92e2-5895e14fa506-etc-modprobe-d\") pod \"tuned-gn9bl\" (UID: \"600fc153-38e0-4874-92e2-5895e14fa506\") " pod="openshift-cluster-node-tuning-operator/tuned-gn9bl"
Jul 27 20:51:25.763910 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.763270    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/600fc153-38e0-4874-92e2-5895e14fa506-host\") pod \"tuned-gn9bl\" (UID: \"600fc153-38e0-4874-92e2-5895e14fa506\") " pod="openshift-cluster-node-tuning-operator/tuned-gn9bl"
Jul 27 20:51:25.763910 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.763288    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/433c431a-b5bf-4734-a03f-6f0a16895ded-run-systemd\") pod \"ovnkube-node-95hg7\" (UID: \"433c431a-b5bf-4734-a03f-6f0a16895ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-95hg7"
Jul 27 20:51:25.763910 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.763295    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a49f6f21-5110-4f7c-8f79-1d9e14d69bc6-registration-dir\") pod \"aws-ebs-csi-driver-node-tjtf6\" (UID: \"a49f6f21-5110-4f7c-8f79-1d9e14d69bc6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tjtf6"
Jul 27 20:51:25.763910 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.763304    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/433c431a-b5bf-4734-a03f-6f0a16895ded-node-log\") pod \"ovnkube-node-95hg7\" (UID: \"433c431a-b5bf-4734-a03f-6f0a16895ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-95hg7"
Jul 27 20:51:25.763910 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.763319    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dn9l\" (UniqueName: \"kubernetes.io/projected/3566e5fb-c112-4f14-9e2f-fce1d833e15a-kube-api-access-9dn9l\") pod \"iptables-alerter-8cdfg\" (UID: \"3566e5fb-c112-4f14-9e2f-fce1d833e15a\") " pod="openshift-network-operator/iptables-alerter-8cdfg"
Jul 27 20:51:25.763910 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.763344    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/eae6d6c0-9844-45a8-a5ef-9c6f1ecc2810-cni-binary-copy\") pod \"multus-additional-cni-plugins-djcnj\" (UID: \"eae6d6c0-9844-45a8-a5ef-9c6f1ecc2810\") " pod="openshift-multus/multus-additional-cni-plugins-djcnj"
Jul 27 20:51:25.763910 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.763385    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a49f6f21-5110-4f7c-8f79-1d9e14d69bc6-sys-fs\") pod \"aws-ebs-csi-driver-node-tjtf6\" (UID: \"a49f6f21-5110-4f7c-8f79-1d9e14d69bc6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tjtf6"
Jul 27 20:51:25.763910 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.763400    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/dbae1ee6-3616-432d-8ef9-255ad4cd444e-host-run-k8s-cni-cncf-io\") pod \"multus-td78h\" (UID: \"dbae1ee6-3616-432d-8ef9-255ad4cd444e\") " pod="openshift-multus/multus-td78h"
Jul 27 20:51:25.763910 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.763418    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/dbae1ee6-3616-432d-8ef9-255ad4cd444e-host-var-lib-cni-multus\") pod \"multus-td78h\" (UID: \"dbae1ee6-3616-432d-8ef9-255ad4cd444e\") " pod="openshift-multus/multus-td78h"
Jul 27 20:51:25.763910 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.763439    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dbae1ee6-3616-432d-8ef9-255ad4cd444e-etc-kubernetes\") pod \"multus-td78h\" (UID: \"dbae1ee6-3616-432d-8ef9-255ad4cd444e\") " pod="openshift-multus/multus-td78h"
Jul 27 20:51:25.763910 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.763457    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9827d\" (UniqueName: \"kubernetes.io/projected/dbae1ee6-3616-432d-8ef9-255ad4cd444e-kube-api-access-9827d\") pod \"multus-td78h\" (UID: \"dbae1ee6-3616-432d-8ef9-255ad4cd444e\") " pod="openshift-multus/multus-td78h"
Jul 27 20:51:25.763910 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.763473    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/600fc153-38e0-4874-92e2-5895e14fa506-sys\") pod \"tuned-gn9bl\" (UID: \"600fc153-38e0-4874-92e2-5895e14fa506\") " pod="openshift-cluster-node-tuning-operator/tuned-gn9bl"
Jul 27 20:51:25.763910 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.763479    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a49f6f21-5110-4f7c-8f79-1d9e14d69bc6-sys-fs\") pod \"aws-ebs-csi-driver-node-tjtf6\" (UID: \"a49f6f21-5110-4f7c-8f79-1d9e14d69bc6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tjtf6"
Jul 27 20:51:25.763910 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.763544    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/600fc153-38e0-4874-92e2-5895e14fa506-lib-modules\") pod \"tuned-gn9bl\" (UID: \"600fc153-38e0-4874-92e2-5895e14fa506\") " pod="openshift-cluster-node-tuning-operator/tuned-gn9bl"
Jul 27 20:51:25.763910 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.763612    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/412ea27a-182d-40cc-917e-4686a4ff65cb-serviceca\") pod \"node-ca-jm574\" (UID: \"412ea27a-182d-40cc-917e-4686a4ff65cb\") " pod="openshift-image-registry/node-ca-jm574"
Jul 27 20:51:25.764592 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.763654    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49fsn\" (UniqueName: \"kubernetes.io/projected/a49f6f21-5110-4f7c-8f79-1d9e14d69bc6-kube-api-access-49fsn\") pod \"aws-ebs-csi-driver-node-tjtf6\" (UID: \"a49f6f21-5110-4f7c-8f79-1d9e14d69bc6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tjtf6"
Jul 27 20:51:25.764592 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.763681    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/dbae1ee6-3616-432d-8ef9-255ad4cd444e-multus-socket-dir-parent\") pod \"multus-td78h\" (UID: \"dbae1ee6-3616-432d-8ef9-255ad4cd444e\") " pod="openshift-multus/multus-td78h"
Jul 27 20:51:25.764592 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.763695    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9a766ad8-ce17-462e-b45f-867a6c6c44e4-konnectivity-ca\") pod \"konnectivity-agent-sg57x\" (UID: \"9a766ad8-ce17-462e-b45f-867a6c6c44e4\") " pod="kube-system/konnectivity-agent-sg57x"
Jul 27 20:51:25.764592 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.763707    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dbae1ee6-3616-432d-8ef9-255ad4cd444e-host-run-netns\") pod \"multus-td78h\" (UID: \"dbae1ee6-3616-432d-8ef9-255ad4cd444e\") " pod="openshift-multus/multus-td78h"
Jul 27 20:51:25.764592 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.763731    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dbae1ee6-3616-432d-8ef9-255ad4cd444e-host-var-lib-cni-bin\") pod \"multus-td78h\" (UID: \"dbae1ee6-3616-432d-8ef9-255ad4cd444e\") " pod="openshift-multus/multus-td78h"
Jul 27 20:51:25.764592 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.763757    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/600fc153-38e0-4874-92e2-5895e14fa506-etc-sysconfig\") pod \"tuned-gn9bl\" (UID: \"600fc153-38e0-4874-92e2-5895e14fa506\") " pod="openshift-cluster-node-tuning-operator/tuned-gn9bl"
Jul 27 20:51:25.764592 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.763803    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/600fc153-38e0-4874-92e2-5895e14fa506-etc-sysctl-d\") pod \"tuned-gn9bl\" (UID: \"600fc153-38e0-4874-92e2-5895e14fa506\") " pod="openshift-cluster-node-tuning-operator/tuned-gn9bl"
Jul 27 20:51:25.764592 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.763831    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/600fc153-38e0-4874-92e2-5895e14fa506-etc-systemd\") pod \"tuned-gn9bl\" (UID: \"600fc153-38e0-4874-92e2-5895e14fa506\") " pod="openshift-cluster-node-tuning-operator/tuned-gn9bl"
Jul 27 20:51:25.764592 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.763855    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/600fc153-38e0-4874-92e2-5895e14fa506-var-lib-kubelet\") pod \"tuned-gn9bl\" (UID: \"600fc153-38e0-4874-92e2-5895e14fa506\") " pod="openshift-cluster-node-tuning-operator/tuned-gn9bl"
Jul 27 20:51:25.764592 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.763892    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dbae1ee6-3616-432d-8ef9-255ad4cd444e-multus-cni-dir\") pod \"multus-td78h\" (UID: \"dbae1ee6-3616-432d-8ef9-255ad4cd444e\") " pod="openshift-multus/multus-td78h"
Jul 27 20:51:25.764592 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.763919    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qslq4\" (UniqueName: \"kubernetes.io/projected/600fc153-38e0-4874-92e2-5895e14fa506-kube-api-access-qslq4\") pod \"tuned-gn9bl\" (UID: \"600fc153-38e0-4874-92e2-5895e14fa506\") " pod="openshift-cluster-node-tuning-operator/tuned-gn9bl"
Jul 27 20:51:25.764592 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.763961    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/433c431a-b5bf-4734-a03f-6f0a16895ded-host-run-ovn-kubernetes\") pod \"ovnkube-node-95hg7\" (UID: \"433c431a-b5bf-4734-a03f-6f0a16895ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-95hg7"
Jul 27 20:51:25.764592 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.763988    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/433c431a-b5bf-4734-a03f-6f0a16895ded-env-overrides\") pod \"ovnkube-node-95hg7\" (UID: \"433c431a-b5bf-4734-a03f-6f0a16895ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-95hg7"
Jul 27 20:51:25.764592 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.764006    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/eae6d6c0-9844-45a8-a5ef-9c6f1ecc2810-tuning-conf-dir\") pod \"multus-additional-cni-plugins-djcnj\" (UID: \"eae6d6c0-9844-45a8-a5ef-9c6f1ecc2810\") " pod="openshift-multus/multus-additional-cni-plugins-djcnj"
Jul 27 20:51:25.764592 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.764062    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/dbae1ee6-3616-432d-8ef9-255ad4cd444e-os-release\") pod \"multus-td78h\" (UID: \"dbae1ee6-3616-432d-8ef9-255ad4cd444e\") " pod="openshift-multus/multus-td78h"
Jul 27 20:51:25.764592 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.764148    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/433c431a-b5bf-4734-a03f-6f0a16895ded-systemd-units\") pod \"ovnkube-node-95hg7\" (UID: \"433c431a-b5bf-4734-a03f-6f0a16895ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-95hg7"
Jul 27 20:51:25.765315 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.764182    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/433c431a-b5bf-4734-a03f-6f0a16895ded-ovnkube-config\") pod \"ovnkube-node-95hg7\" (UID: \"433c431a-b5bf-4734-a03f-6f0a16895ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-95hg7"
Jul 27 20:51:25.765315 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.764217    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/412ea27a-182d-40cc-917e-4686a4ff65cb-host\") pod \"node-ca-jm574\" (UID: \"412ea27a-182d-40cc-917e-4686a4ff65cb\") " pod="openshift-image-registry/node-ca-jm574"
Jul 27 20:51:25.765315 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.764257    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a49f6f21-5110-4f7c-8f79-1d9e14d69bc6-device-dir\") pod \"aws-ebs-csi-driver-node-tjtf6\" (UID: \"a49f6f21-5110-4f7c-8f79-1d9e14d69bc6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tjtf6"
Jul 27 20:51:25.765315 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.764294    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4b924e42-83bc-459f-b1a2-f115f9bce63f-dbus\") pod \"global-pull-secret-syncer-dtrjg\" (UID: \"4b924e42-83bc-459f-b1a2-f115f9bce63f\") " pod="kube-system/global-pull-secret-syncer-dtrjg"
Jul 27 20:51:25.765315 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.764318    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/600fc153-38e0-4874-92e2-5895e14fa506-etc-kubernetes\") pod \"tuned-gn9bl\" (UID: \"600fc153-38e0-4874-92e2-5895e14fa506\") " pod="openshift-cluster-node-tuning-operator/tuned-gn9bl"
Jul 27 20:51:25.765315 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.764340    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/433c431a-b5bf-4734-a03f-6f0a16895ded-host-kubelet\") pod \"ovnkube-node-95hg7\" (UID: \"433c431a-b5bf-4734-a03f-6f0a16895ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-95hg7"
Jul 27 20:51:25.765315 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.764367    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a49f6f21-5110-4f7c-8f79-1d9e14d69bc6-device-dir\") pod \"aws-ebs-csi-driver-node-tjtf6\" (UID: \"a49f6f21-5110-4f7c-8f79-1d9e14d69bc6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tjtf6"
Jul 27 20:51:25.765315 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.764404    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/433c431a-b5bf-4734-a03f-6f0a16895ded-log-socket\") pod \"ovnkube-node-95hg7\" (UID: \"433c431a-b5bf-4734-a03f-6f0a16895ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-95hg7"
Jul 27 20:51:25.765315 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.764439    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3566e5fb-c112-4f14-9e2f-fce1d833e15a-host-slash\") pod \"iptables-alerter-8cdfg\" (UID: \"3566e5fb-c112-4f14-9e2f-fce1d833e15a\") " pod="openshift-network-operator/iptables-alerter-8cdfg"
Jul 27 20:51:25.765315 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.764462    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6mts\" (UniqueName: \"kubernetes.io/projected/fac6ee6e-225a-4402-8834-69ff744b07f9-kube-api-access-x6mts\") pod \"network-metrics-daemon-r6g8d\" (UID: \"fac6ee6e-225a-4402-8834-69ff744b07f9\") " pod="openshift-multus/network-metrics-daemon-r6g8d"
Jul 27 20:51:25.765315 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.764490    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eae6d6c0-9844-45a8-a5ef-9c6f1ecc2810-system-cni-dir\") pod \"multus-additional-cni-plugins-djcnj\" (UID: \"eae6d6c0-9844-45a8-a5ef-9c6f1ecc2810\") " pod="openshift-multus/multus-additional-cni-plugins-djcnj"
Jul 27 20:51:25.765315 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.764521    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/433c431a-b5bf-4734-a03f-6f0a16895ded-etc-openvswitch\") pod \"ovnkube-node-95hg7\" (UID: \"433c431a-b5bf-4734-a03f-6f0a16895ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-95hg7"
Jul 27 20:51:25.765315 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.764538    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4b924e42-83bc-459f-b1a2-f115f9bce63f-dbus\") pod \"global-pull-secret-syncer-dtrjg\" (UID: \"4b924e42-83bc-459f-b1a2-f115f9bce63f\") " pod="kube-system/global-pull-secret-syncer-dtrjg"
Jul 27 20:51:25.765315 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.764545    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/433c431a-b5bf-4734-a03f-6f0a16895ded-host-cni-bin\") pod \"ovnkube-node-95hg7\" (UID: \"433c431a-b5bf-4734-a03f-6f0a16895ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-95hg7"
Jul 27 20:51:25.765315 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.764570    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtp67\" (UniqueName: \"kubernetes.io/projected/433c431a-b5bf-4734-a03f-6f0a16895ded-kube-api-access-jtp67\") pod \"ovnkube-node-95hg7\" (UID: \"433c431a-b5bf-4734-a03f-6f0a16895ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-95hg7"
Jul 27 20:51:25.765315 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.764606    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95f7e24a-fe2f-451e-ae08-fa9342e04cef-tmp-dir\") pod \"node-resolver-g5tm9\" (UID: \"95f7e24a-fe2f-451e-ae08-fa9342e04cef\") " pod="openshift-dns/node-resolver-g5tm9"
Jul 27 20:51:25.765315 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.764632    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qj65h\" (UniqueName: \"kubernetes.io/projected/95f7e24a-fe2f-451e-ae08-fa9342e04cef-kube-api-access-qj65h\") pod \"node-resolver-g5tm9\" (UID: \"95f7e24a-fe2f-451e-ae08-fa9342e04cef\") " pod="openshift-dns/node-resolver-g5tm9"
Jul 27 20:51:25.766004 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.764657    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dbae1ee6-3616-432d-8ef9-255ad4cd444e-system-cni-dir\") pod \"multus-td78h\" (UID: \"dbae1ee6-3616-432d-8ef9-255ad4cd444e\") " pod="openshift-multus/multus-td78h"
Jul 27 20:51:25.766004 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.764680    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/dbae1ee6-3616-432d-8ef9-255ad4cd444e-multus-daemon-config\") pod \"multus-td78h\" (UID: \"dbae1ee6-3616-432d-8ef9-255ad4cd444e\") " pod="openshift-multus/multus-td78h"
Jul 27 20:51:25.766004 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.764695    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/600fc153-38e0-4874-92e2-5895e14fa506-etc-sysctl-conf\") pod \"tuned-gn9bl\" (UID: \"600fc153-38e0-4874-92e2-5895e14fa506\") " pod="openshift-cluster-node-tuning-operator/tuned-gn9bl"
Jul 27 20:51:25.766004 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.764718    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/eae6d6c0-9844-45a8-a5ef-9c6f1ecc2810-os-release\") pod \"multus-additional-cni-plugins-djcnj\" (UID: \"eae6d6c0-9844-45a8-a5ef-9c6f1ecc2810\") " pod="openshift-multus/multus-additional-cni-plugins-djcnj"
Jul 27 20:51:25.766004 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.764739    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/eae6d6c0-9844-45a8-a5ef-9c6f1ecc2810-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-djcnj\" (UID: \"eae6d6c0-9844-45a8-a5ef-9c6f1ecc2810\") " pod="openshift-multus/multus-additional-cni-plugins-djcnj"
Jul 27 20:51:25.766004 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.764762    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8ns2\" (UniqueName: \"kubernetes.io/projected/1f1259cb-be97-4841-bd22-5bba585870b6-kube-api-access-n8ns2\") pod \"network-check-target-dbzwm\" (UID: \"1f1259cb-be97-4841-bd22-5bba585870b6\") " pod="openshift-network-diagnostics/network-check-target-dbzwm"
Jul 27 20:51:25.766004 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.764790    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a49f6f21-5110-4f7c-8f79-1d9e14d69bc6-socket-dir\") pod \"aws-ebs-csi-driver-node-tjtf6\" (UID: \"a49f6f21-5110-4f7c-8f79-1d9e14d69bc6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tjtf6"
Jul 27 20:51:25.766004 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.764814    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dbae1ee6-3616-432d-8ef9-255ad4cd444e-cnibin\") pod \"multus-td78h\" (UID: \"dbae1ee6-3616-432d-8ef9-255ad4cd444e\") " pod="openshift-multus/multus-td78h"
Jul 27 20:51:25.766004 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.764854    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dbae1ee6-3616-432d-8ef9-255ad4cd444e-host-var-lib-kubelet\") pod \"multus-td78h\" (UID: \"dbae1ee6-3616-432d-8ef9-255ad4cd444e\") " pod="openshift-multus/multus-td78h"
Jul 27 20:51:25.766004 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.764892    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/433c431a-b5bf-4734-a03f-6f0a16895ded-var-lib-openvswitch\") pod \"ovnkube-node-95hg7\" (UID: \"433c431a-b5bf-4734-a03f-6f0a16895ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-95hg7"
Jul 27 20:51:25.766004 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.764907    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a49f6f21-5110-4f7c-8f79-1d9e14d69bc6-socket-dir\") pod \"aws-ebs-csi-driver-node-tjtf6\" (UID: \"a49f6f21-5110-4f7c-8f79-1d9e14d69bc6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tjtf6"
Jul 27 20:51:25.766004 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.764923    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/433c431a-b5bf-4734-a03f-6f0a16895ded-ovn-node-metrics-cert\") pod \"ovnkube-node-95hg7\" (UID: \"433c431a-b5bf-4734-a03f-6f0a16895ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-95hg7"
Jul 27 20:51:25.766004 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.764952    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4b924e42-83bc-459f-b1a2-f115f9bce63f-original-pull-secret\") pod \"global-pull-secret-syncer-dtrjg\" (UID: \"4b924e42-83bc-459f-b1a2-f115f9bce63f\") " pod="kube-system/global-pull-secret-syncer-dtrjg"
Jul 27 20:51:25.766004 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.764978    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9a766ad8-ce17-462e-b45f-867a6c6c44e4-agent-certs\") pod \"konnectivity-agent-sg57x\" (UID: \"9a766ad8-ce17-462e-b45f-867a6c6c44e4\") " pod="kube-system/konnectivity-agent-sg57x"
Jul 27 20:51:25.766004 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.765000    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/433c431a-b5bf-4734-a03f-6f0a16895ded-host-run-netns\") pod \"ovnkube-node-95hg7\" (UID: \"433c431a-b5bf-4734-a03f-6f0a16895ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-95hg7"
Jul 27 20:51:25.766004 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:25.765083    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 20:51:25.766004 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.765119    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/433c431a-b5bf-4734-a03f-6f0a16895ded-run-ovn\") pod \"ovnkube-node-95hg7\" (UID: \"433c431a-b5bf-4734-a03f-6f0a16895ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-95hg7"
Jul 27 20:51:25.766682 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:25.765148    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b924e42-83bc-459f-b1a2-f115f9bce63f-original-pull-secret podName:4b924e42-83bc-459f-b1a2-f115f9bce63f nodeName:}" failed. No retries permitted until 2026-07-27 20:51:26.26513164 +0000 UTC m=+3.016993415 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4b924e42-83bc-459f-b1a2-f115f9bce63f-original-pull-secret") pod "global-pull-secret-syncer-dtrjg" (UID: "4b924e42-83bc-459f-b1a2-f115f9bce63f") : object "kube-system"/"original-pull-secret" not registered
Jul 27 20:51:25.766682 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.765170    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/95f7e24a-fe2f-451e-ae08-fa9342e04cef-hosts-file\") pod \"node-resolver-g5tm9\" (UID: \"95f7e24a-fe2f-451e-ae08-fa9342e04cef\") " pod="openshift-dns/node-resolver-g5tm9"
Jul 27 20:51:25.766682 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.765194    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a49f6f21-5110-4f7c-8f79-1d9e14d69bc6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tjtf6\" (UID: \"a49f6f21-5110-4f7c-8f79-1d9e14d69bc6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tjtf6"
Jul 27 20:51:25.766682 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.765216    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/433c431a-b5bf-4734-a03f-6f0a16895ded-host-slash\") pod \"ovnkube-node-95hg7\" (UID: \"433c431a-b5bf-4734-a03f-6f0a16895ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-95hg7"
Jul 27 20:51:25.766682 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.765232    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/433c431a-b5bf-4734-a03f-6f0a16895ded-host-cni-netd\") pod \"ovnkube-node-95hg7\" (UID: \"433c431a-b5bf-4734-a03f-6f0a16895ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-95hg7"
Jul 27 20:51:25.766682 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.765247    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a49f6f21-5110-4f7c-8f79-1d9e14d69bc6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tjtf6\" (UID: \"a49f6f21-5110-4f7c-8f79-1d9e14d69bc6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tjtf6"
Jul 27 20:51:25.766682 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.766122    2560 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 27 20:51:25.769921 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.769853    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9a766ad8-ce17-462e-b45f-867a6c6c44e4-agent-certs\") pod \"konnectivity-agent-sg57x\" (UID: \"9a766ad8-ce17-462e-b45f-867a6c6c44e4\") " pod="kube-system/konnectivity-agent-sg57x"
Jul 27 20:51:25.774041 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:25.773998    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 20:51:25.774041 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:25.774041    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 20:51:25.774224 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:25.774058    2560 projected.go:196] Error preparing data for projected volume kube-api-access-n8ns2 for pod openshift-network-diagnostics/network-check-target-dbzwm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 20:51:25.774224 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:25.774156    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1f1259cb-be97-4841-bd22-5bba585870b6-kube-api-access-n8ns2 podName:1f1259cb-be97-4841-bd22-5bba585870b6 nodeName:}" failed. No retries permitted until 2026-07-27 20:51:26.274135225 +0000 UTC m=+3.025997024 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-n8ns2" (UniqueName: "kubernetes.io/projected/1f1259cb-be97-4841-bd22-5bba585870b6-kube-api-access-n8ns2") pod "network-check-target-dbzwm" (UID: "1f1259cb-be97-4841-bd22-5bba585870b6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 20:51:25.774860 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.774816    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-49fsn\" (UniqueName: \"kubernetes.io/projected/a49f6f21-5110-4f7c-8f79-1d9e14d69bc6-kube-api-access-49fsn\") pod \"aws-ebs-csi-driver-node-tjtf6\" (UID: \"a49f6f21-5110-4f7c-8f79-1d9e14d69bc6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tjtf6"
Jul 27 20:51:25.865753 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.865720    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dn9l\" (UniqueName: \"kubernetes.io/projected/3566e5fb-c112-4f14-9e2f-fce1d833e15a-kube-api-access-9dn9l\") pod \"iptables-alerter-8cdfg\" (UID: \"3566e5fb-c112-4f14-9e2f-fce1d833e15a\") " pod="openshift-network-operator/iptables-alerter-8cdfg"
Jul 27 20:51:25.865753 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.865762    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/eae6d6c0-9844-45a8-a5ef-9c6f1ecc2810-cni-binary-copy\") pod \"multus-additional-cni-plugins-djcnj\" (UID: \"eae6d6c0-9844-45a8-a5ef-9c6f1ecc2810\") " pod="openshift-multus/multus-additional-cni-plugins-djcnj"
Jul 27 20:51:25.865996 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.865790    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/dbae1ee6-3616-432d-8ef9-255ad4cd444e-host-run-k8s-cni-cncf-io\") pod \"multus-td78h\" (UID: \"dbae1ee6-3616-432d-8ef9-255ad4cd444e\") " pod="openshift-multus/multus-td78h"
Jul 27 20:51:25.865996 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.865809    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/dbae1ee6-3616-432d-8ef9-255ad4cd444e-host-var-lib-cni-multus\") pod \"multus-td78h\" (UID: \"dbae1ee6-3616-432d-8ef9-255ad4cd444e\") " pod="openshift-multus/multus-td78h"
Jul 27 20:51:25.865996 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.865826    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dbae1ee6-3616-432d-8ef9-255ad4cd444e-etc-kubernetes\") pod \"multus-td78h\" (UID: \"dbae1ee6-3616-432d-8ef9-255ad4cd444e\") " pod="openshift-multus/multus-td78h"
Jul 27 20:51:25.865996 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.865865    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9827d\" (UniqueName: \"kubernetes.io/projected/dbae1ee6-3616-432d-8ef9-255ad4cd444e-kube-api-access-9827d\") pod \"multus-td78h\" (UID: \"dbae1ee6-3616-432d-8ef9-255ad4cd444e\") " pod="openshift-multus/multus-td78h"
Jul 27 20:51:25.865996 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.865887    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/600fc153-38e0-4874-92e2-5895e14fa506-sys\") pod \"tuned-gn9bl\" (UID: \"600fc153-38e0-4874-92e2-5895e14fa506\") " pod="openshift-cluster-node-tuning-operator/tuned-gn9bl"
Jul 27 20:51:25.865996 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.865911    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/600fc153-38e0-4874-92e2-5895e14fa506-lib-modules\") pod \"tuned-gn9bl\" (UID: \"600fc153-38e0-4874-92e2-5895e14fa506\") " pod="openshift-cluster-node-tuning-operator/tuned-gn9bl"
Jul 27 20:51:25.865996 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.865955    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/412ea27a-182d-40cc-917e-4686a4ff65cb-serviceca\") pod \"node-ca-jm574\" (UID: \"412ea27a-182d-40cc-917e-4686a4ff65cb\") " pod="openshift-image-registry/node-ca-jm574"
Jul 27 20:51:25.865996 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.865978    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/dbae1ee6-3616-432d-8ef9-255ad4cd444e-multus-socket-dir-parent\") pod \"multus-td78h\" (UID: \"dbae1ee6-3616-432d-8ef9-255ad4cd444e\") " pod="openshift-multus/multus-td78h"
Jul 27 20:51:25.866330 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.866006    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dbae1ee6-3616-432d-8ef9-255ad4cd444e-host-run-netns\") pod \"multus-td78h\" (UID: \"dbae1ee6-3616-432d-8ef9-255ad4cd444e\") " pod="openshift-multus/multus-td78h"
Jul 27 20:51:25.866330 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.866009    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/dbae1ee6-3616-432d-8ef9-255ad4cd444e-host-var-lib-cni-multus\") pod \"multus-td78h\" (UID: \"dbae1ee6-3616-432d-8ef9-255ad4cd444e\") " pod="openshift-multus/multus-td78h"
Jul 27 20:51:25.866330 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.866025    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dbae1ee6-3616-432d-8ef9-255ad4cd444e-host-var-lib-cni-bin\") pod \"multus-td78h\" (UID: \"dbae1ee6-3616-432d-8ef9-255ad4cd444e\") " pod="openshift-multus/multus-td78h"
Jul 27 20:51:25.866330 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.866074    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/600fc153-38e0-4874-92e2-5895e14fa506-etc-sysconfig\") pod \"tuned-gn9bl\" (UID: \"600fc153-38e0-4874-92e2-5895e14fa506\") " pod="openshift-cluster-node-tuning-operator/tuned-gn9bl"
Jul 27 20:51:25.866330 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.866103    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/600fc153-38e0-4874-92e2-5895e14fa506-etc-sysctl-d\") pod \"tuned-gn9bl\" (UID: \"600fc153-38e0-4874-92e2-5895e14fa506\") " pod="openshift-cluster-node-tuning-operator/tuned-gn9bl"
Jul 27 20:51:25.866330 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.866149    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/600fc153-38e0-4874-92e2-5895e14fa506-etc-systemd\") pod \"tuned-gn9bl\" (UID: \"600fc153-38e0-4874-92e2-5895e14fa506\") " pod="openshift-cluster-node-tuning-operator/tuned-gn9bl"
Jul 27 20:51:25.866330 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.866174    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/dbae1ee6-3616-432d-8ef9-255ad4cd444e-host-run-k8s-cni-cncf-io\") pod \"multus-td78h\" (UID: \"dbae1ee6-3616-432d-8ef9-255ad4cd444e\") " pod="openshift-multus/multus-td78h"
Jul 27 20:51:25.866330 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.866238    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dbae1ee6-3616-432d-8ef9-255ad4cd444e-etc-kubernetes\") pod \"multus-td78h\" (UID: \"dbae1ee6-3616-432d-8ef9-255ad4cd444e\") " pod="openshift-multus/multus-td78h"
Jul 27 20:51:25.866330 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.866251    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/600fc153-38e0-4874-92e2-5895e14fa506-var-lib-kubelet\") pod \"tuned-gn9bl\" (UID: \"600fc153-38e0-4874-92e2-5895e14fa506\") " pod="openshift-cluster-node-tuning-operator/tuned-gn9bl"
Jul 27 20:51:25.866330 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.866186    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/600fc153-38e0-4874-92e2-5895e14fa506-var-lib-kubelet\") pod \"tuned-gn9bl\" (UID: \"600fc153-38e0-4874-92e2-5895e14fa506\") " pod="openshift-cluster-node-tuning-operator/tuned-gn9bl"
Jul 27 20:51:25.866330 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.866303    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dbae1ee6-3616-432d-8ef9-255ad4cd444e-multus-cni-dir\") pod \"multus-td78h\" (UID: \"dbae1ee6-3616-432d-8ef9-255ad4cd444e\") " pod="openshift-multus/multus-td78h"
Jul 27 20:51:25.866691 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.866358    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qslq4\" (UniqueName: \"kubernetes.io/projected/600fc153-38e0-4874-92e2-5895e14fa506-kube-api-access-qslq4\") pod \"tuned-gn9bl\" (UID: \"600fc153-38e0-4874-92e2-5895e14fa506\") " pod="openshift-cluster-node-tuning-operator/tuned-gn9bl"
Jul 27 20:51:25.866691 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.866386    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/433c431a-b5bf-4734-a03f-6f0a16895ded-host-run-ovn-kubernetes\") pod \"ovnkube-node-95hg7\" (UID: \"433c431a-b5bf-4734-a03f-6f0a16895ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-95hg7"
Jul 27 20:51:25.866691 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.866461    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/433c431a-b5bf-4734-a03f-6f0a16895ded-env-overrides\") pod \"ovnkube-node-95hg7\" (UID: \"433c431a-b5bf-4734-a03f-6f0a16895ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-95hg7"
Jul 27 20:51:25.866691 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.866493    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/eae6d6c0-9844-45a8-a5ef-9c6f1ecc2810-tuning-conf-dir\") pod \"multus-additional-cni-plugins-djcnj\" (UID: \"eae6d6c0-9844-45a8-a5ef-9c6f1ecc2810\") " pod="openshift-multus/multus-additional-cni-plugins-djcnj"
Jul 27 20:51:25.866691 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.866547    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/600fc153-38e0-4874-92e2-5895e14fa506-sys\") pod \"tuned-gn9bl\" (UID: \"600fc153-38e0-4874-92e2-5895e14fa506\") " pod="openshift-cluster-node-tuning-operator/tuned-gn9bl"
Jul 27 20:51:25.866691 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.866553    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/dbae1ee6-3616-432d-8ef9-255ad4cd444e-os-release\") pod \"multus-td78h\" (UID: \"dbae1ee6-3616-432d-8ef9-255ad4cd444e\") " pod="openshift-multus/multus-td78h"
Jul 27 20:51:25.866691 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.866604    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/433c431a-b5bf-4734-a03f-6f0a16895ded-systemd-units\") pod \"ovnkube-node-95hg7\" (UID: \"433c431a-b5bf-4734-a03f-6f0a16895ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-95hg7"
Jul 27 20:51:25.866691 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.866623    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/dbae1ee6-3616-432d-8ef9-255ad4cd444e-os-release\") pod \"multus-td78h\" (UID: \"dbae1ee6-3616-432d-8ef9-255ad4cd444e\") " pod="openshift-multus/multus-td78h"
Jul 27 20:51:25.866691 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.866679    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/433c431a-b5bf-4734-a03f-6f0a16895ded-ovnkube-config\") pod \"ovnkube-node-95hg7\" (UID: \"433c431a-b5bf-4734-a03f-6f0a16895ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-95hg7"
Jul 27 20:51:25.866691 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.866684    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dbae1ee6-3616-432d-8ef9-255ad4cd444e-multus-cni-dir\") pod \"multus-td78h\" (UID: \"dbae1ee6-3616-432d-8ef9-255ad4cd444e\") " pod="openshift-multus/multus-td78h"
Jul 27 20:51:25.867074 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.866713    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/412ea27a-182d-40cc-917e-4686a4ff65cb-host\") pod \"node-ca-jm574\" (UID: \"412ea27a-182d-40cc-917e-4686a4ff65cb\") " pod="openshift-image-registry/node-ca-jm574"
Jul 27 20:51:25.867074 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.866745    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/600fc153-38e0-4874-92e2-5895e14fa506-etc-kubernetes\") pod \"tuned-gn9bl\" (UID: \"600fc153-38e0-4874-92e2-5895e14fa506\") " pod="openshift-cluster-node-tuning-operator/tuned-gn9bl"
Jul 27 20:51:25.867074 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.866805    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/eae6d6c0-9844-45a8-a5ef-9c6f1ecc2810-cni-binary-copy\") pod \"multus-additional-cni-plugins-djcnj\" (UID: \"eae6d6c0-9844-45a8-a5ef-9c6f1ecc2810\") " pod="openshift-multus/multus-additional-cni-plugins-djcnj"
Jul 27 20:51:25.867074 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.866813    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/433c431a-b5bf-4734-a03f-6f0a16895ded-host-kubelet\") pod \"ovnkube-node-95hg7\" (UID: \"433c431a-b5bf-4734-a03f-6f0a16895ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-95hg7"
Jul 27 20:51:25.867074 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.866840    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/433c431a-b5bf-4734-a03f-6f0a16895ded-log-socket\") pod \"ovnkube-node-95hg7\" (UID: \"433c431a-b5bf-4734-a03f-6f0a16895ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-95hg7"
Jul 27 20:51:25.867074 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.866923    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dbae1ee6-3616-432d-8ef9-255ad4cd444e-host-var-lib-cni-bin\") pod \"multus-td78h\" (UID: \"dbae1ee6-3616-432d-8ef9-255ad4cd444e\") " pod="openshift-multus/multus-td78h"
Jul 27 20:51:25.867074 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.866923    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/600fc153-38e0-4874-92e2-5895e14fa506-etc-sysconfig\") pod \"tuned-gn9bl\" (UID: \"600fc153-38e0-4874-92e2-5895e14fa506\") " pod="openshift-cluster-node-tuning-operator/tuned-gn9bl"
Jul 27 20:51:25.867074 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.866944    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3566e5fb-c112-4f14-9e2f-fce1d833e15a-host-slash\") pod \"iptables-alerter-8cdfg\" (UID: \"3566e5fb-c112-4f14-9e2f-fce1d833e15a\") " pod="openshift-network-operator/iptables-alerter-8cdfg"
Jul 27 20:51:25.867074 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.866981    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/600fc153-38e0-4874-92e2-5895e14fa506-etc-systemd\") pod \"tuned-gn9bl\" (UID: \"600fc153-38e0-4874-92e2-5895e14fa506\") " pod="openshift-cluster-node-tuning-operator/tuned-gn9bl"
Jul 27 20:51:25.867074 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.866998    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6mts\" (UniqueName: \"kubernetes.io/projected/fac6ee6e-225a-4402-8834-69ff744b07f9-kube-api-access-x6mts\") pod \"network-metrics-daemon-r6g8d\" (UID: \"fac6ee6e-225a-4402-8834-69ff744b07f9\") " pod="openshift-multus/network-metrics-daemon-r6g8d"
Jul 27 20:51:25.867074 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.867036    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eae6d6c0-9844-45a8-a5ef-9c6f1ecc2810-system-cni-dir\") pod \"multus-additional-cni-plugins-djcnj\" (UID: \"eae6d6c0-9844-45a8-a5ef-9c6f1ecc2810\") " pod="openshift-multus/multus-additional-cni-plugins-djcnj"
Jul 27 20:51:25.867074 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.867062    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/600fc153-38e0-4874-92e2-5895e14fa506-etc-sysctl-d\") pod \"tuned-gn9bl\" (UID: \"600fc153-38e0-4874-92e2-5895e14fa506\") " pod="openshift-cluster-node-tuning-operator/tuned-gn9bl"
Jul 27 20:51:25.867074 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.867064    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/433c431a-b5bf-4734-a03f-6f0a16895ded-etc-openvswitch\") pod \"ovnkube-node-95hg7\" (UID: \"433c431a-b5bf-4734-a03f-6f0a16895ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-95hg7"
Jul 27 20:51:25.867598 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.867087    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/433c431a-b5bf-4734-a03f-6f0a16895ded-etc-openvswitch\") pod \"ovnkube-node-95hg7\" (UID: \"433c431a-b5bf-4734-a03f-6f0a16895ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-95hg7"
Jul 27 20:51:25.867598 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.867095    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/433c431a-b5bf-4734-a03f-6f0a16895ded-host-cni-bin\") pod \"ovnkube-node-95hg7\" (UID: \"433c431a-b5bf-4734-a03f-6f0a16895ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-95hg7"
Jul 27 20:51:25.867598 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.867133    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/433c431a-b5bf-4734-a03f-6f0a16895ded-log-socket\") pod \"ovnkube-node-95hg7\" (UID: \"433c431a-b5bf-4734-a03f-6f0a16895ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-95hg7"
Jul 27 20:51:25.867598 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.867196    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/433c431a-b5bf-4734-a03f-6f0a16895ded-host-run-ovn-kubernetes\") pod \"ovnkube-node-95hg7\" (UID: \"433c431a-b5bf-4734-a03f-6f0a16895ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-95hg7"
Jul 27 20:51:25.867598 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.867238    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/433c431a-b5bf-4734-a03f-6f0a16895ded-host-kubelet\") pod \"ovnkube-node-95hg7\" (UID: \"433c431a-b5bf-4734-a03f-6f0a16895ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-95hg7"
Jul 27 20:51:25.867598 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.867245    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eae6d6c0-9844-45a8-a5ef-9c6f1ecc2810-system-cni-dir\") pod \"multus-additional-cni-plugins-djcnj\" (UID: \"eae6d6c0-9844-45a8-a5ef-9c6f1ecc2810\") " pod="openshift-multus/multus-additional-cni-plugins-djcnj"
Jul 27 20:51:25.867598 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.867269    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtp67\" (UniqueName: \"kubernetes.io/projected/433c431a-b5bf-4734-a03f-6f0a16895ded-kube-api-access-jtp67\") pod \"ovnkube-node-95hg7\" (UID: \"433c431a-b5bf-4734-a03f-6f0a16895ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-95hg7"
Jul 27 20:51:25.867598 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.867289    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95f7e24a-fe2f-451e-ae08-fa9342e04cef-tmp-dir\") pod \"node-resolver-g5tm9\" (UID: \"95f7e24a-fe2f-451e-ae08-fa9342e04cef\") " pod="openshift-dns/node-resolver-g5tm9"
Jul 27 20:51:25.867598 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.867311    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qj65h\" (UniqueName: \"kubernetes.io/projected/95f7e24a-fe2f-451e-ae08-fa9342e04cef-kube-api-access-qj65h\") pod \"node-resolver-g5tm9\" (UID: \"95f7e24a-fe2f-451e-ae08-fa9342e04cef\") " pod="openshift-dns/node-resolver-g5tm9"
Jul 27 20:51:25.867598 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.867363    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dbae1ee6-3616-432d-8ef9-255ad4cd444e-system-cni-dir\") pod \"multus-td78h\" (UID: \"dbae1ee6-3616-432d-8ef9-255ad4cd444e\") " pod="openshift-multus/multus-td78h"
Jul 27 20:51:25.867598 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.867408    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/dbae1ee6-3616-432d-8ef9-255ad4cd444e-multus-daemon-config\") pod \"multus-td78h\" (UID: \"dbae1ee6-3616-432d-8ef9-255ad4cd444e\") " pod="openshift-multus/multus-td78h"
Jul 27 20:51:25.867598 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.867516    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/433c431a-b5bf-4734-a03f-6f0a16895ded-env-overrides\") pod \"ovnkube-node-95hg7\" (UID: \"433c431a-b5bf-4734-a03f-6f0a16895ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-95hg7"
Jul 27 20:51:25.867598 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.867527    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dbae1ee6-3616-432d-8ef9-255ad4cd444e-host-run-netns\") pod \"multus-td78h\" (UID: \"dbae1ee6-3616-432d-8ef9-255ad4cd444e\") " pod="openshift-multus/multus-td78h"
Jul 27 20:51:25.867598 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.867529    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/600fc153-38e0-4874-92e2-5895e14fa506-etc-sysctl-conf\") pod \"tuned-gn9bl\" (UID: \"600fc153-38e0-4874-92e2-5895e14fa506\") " pod="openshift-cluster-node-tuning-operator/tuned-gn9bl"
Jul 27 20:51:25.867598 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.867578    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/eae6d6c0-9844-45a8-a5ef-9c6f1ecc2810-os-release\") pod \"multus-additional-cni-plugins-djcnj\" (UID: \"eae6d6c0-9844-45a8-a5ef-9c6f1ecc2810\") " pod="openshift-multus/multus-additional-cni-plugins-djcnj"
Jul 27 20:51:25.867598 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.867607    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/eae6d6c0-9844-45a8-a5ef-9c6f1ecc2810-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-djcnj\" (UID: \"eae6d6c0-9844-45a8-a5ef-9c6f1ecc2810\") " pod="openshift-multus/multus-additional-cni-plugins-djcnj"
Jul 27 20:51:25.868155 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.867644    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/600fc153-38e0-4874-92e2-5895e14fa506-lib-modules\") pod \"tuned-gn9bl\" (UID: \"600fc153-38e0-4874-92e2-5895e14fa506\") " pod="openshift-cluster-node-tuning-operator/tuned-gn9bl"
Jul 27 20:51:25.868155 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.867663    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dbae1ee6-3616-432d-8ef9-255ad4cd444e-cnibin\") pod \"multus-td78h\" (UID: \"dbae1ee6-3616-432d-8ef9-255ad4cd444e\") " pod="openshift-multus/multus-td78h"
Jul 27 20:51:25.868155 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.867710    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dbae1ee6-3616-432d-8ef9-255ad4cd444e-host-var-lib-kubelet\") pod \"multus-td78h\" (UID: \"dbae1ee6-3616-432d-8ef9-255ad4cd444e\") " pod="openshift-multus/multus-td78h"
Jul 27 20:51:25.868155 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.867721    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/eae6d6c0-9844-45a8-a5ef-9c6f1ecc2810-tuning-conf-dir\") pod \"multus-additional-cni-plugins-djcnj\" (UID: \"eae6d6c0-9844-45a8-a5ef-9c6f1ecc2810\") " pod="openshift-multus/multus-additional-cni-plugins-djcnj"
Jul 27 20:51:25.868155 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.867734    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/433c431a-b5bf-4734-a03f-6f0a16895ded-host-cni-bin\") pod \"ovnkube-node-95hg7\" (UID: \"433c431a-b5bf-4734-a03f-6f0a16895ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-95hg7"
Jul 27 20:51:25.868155 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.867739    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/433c431a-b5bf-4734-a03f-6f0a16895ded-var-lib-openvswitch\") pod \"ovnkube-node-95hg7\" (UID: \"433c431a-b5bf-4734-a03f-6f0a16895ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-95hg7"
Jul 27 20:51:25.868155 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.867793    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/433c431a-b5bf-4734-a03f-6f0a16895ded-ovn-node-metrics-cert\") pod \"ovnkube-node-95hg7\" (UID: \"433c431a-b5bf-4734-a03f-6f0a16895ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-95hg7"
Jul 27 20:51:25.868155 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.867817    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/433c431a-b5bf-4734-a03f-6f0a16895ded-var-lib-openvswitch\") pod \"ovnkube-node-95hg7\" (UID: \"433c431a-b5bf-4734-a03f-6f0a16895ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-95hg7"
Jul 27 20:51:25.868155 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.867871    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/600fc153-38e0-4874-92e2-5895e14fa506-etc-kubernetes\") pod \"tuned-gn9bl\" (UID: \"600fc153-38e0-4874-92e2-5895e14fa506\") " pod="openshift-cluster-node-tuning-operator/tuned-gn9bl"
Jul 27 20:51:25.868155 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.867856    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/433c431a-b5bf-4734-a03f-6f0a16895ded-host-run-netns\") pod \"ovnkube-node-95hg7\" (UID: \"433c431a-b5bf-4734-a03f-6f0a16895ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-95hg7"
Jul 27 20:51:25.868155 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.867915    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/433c431a-b5bf-4734-a03f-6f0a16895ded-host-run-netns\") pod \"ovnkube-node-95hg7\" (UID: \"433c431a-b5bf-4734-a03f-6f0a16895ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-95hg7"
Jul 27 20:51:25.868155 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.867915    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/433c431a-b5bf-4734-a03f-6f0a16895ded-run-ovn\") pod \"ovnkube-node-95hg7\" (UID: \"433c431a-b5bf-4734-a03f-6f0a16895ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-95hg7"
Jul 27 20:51:25.868155 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.867938    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/95f7e24a-fe2f-451e-ae08-fa9342e04cef-hosts-file\") pod \"node-resolver-g5tm9\" (UID: \"95f7e24a-fe2f-451e-ae08-fa9342e04cef\") " pod="openshift-dns/node-resolver-g5tm9"
Jul 27 20:51:25.868155 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.867950    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/433c431a-b5bf-4734-a03f-6f0a16895ded-run-ovn\") pod \"ovnkube-node-95hg7\" (UID: \"433c431a-b5bf-4734-a03f-6f0a16895ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-95hg7"
Jul 27 20:51:25.868155 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.867977    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/433c431a-b5bf-4734-a03f-6f0a16895ded-host-slash\") pod \"ovnkube-node-95hg7\" (UID: \"433c431a-b5bf-4734-a03f-6f0a16895ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-95hg7"
Jul 27 20:51:25.868155 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.868100    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/dbae1ee6-3616-432d-8ef9-255ad4cd444e-multus-socket-dir-parent\") pod \"multus-td78h\" (UID: \"dbae1ee6-3616-432d-8ef9-255ad4cd444e\") " pod="openshift-multus/multus-td78h"
Jul 27 20:51:25.868762 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.868249    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/412ea27a-182d-40cc-917e-4686a4ff65cb-serviceca\") pod \"node-ca-jm574\" (UID: \"412ea27a-182d-40cc-917e-4686a4ff65cb\") " pod="openshift-image-registry/node-ca-jm574"
Jul 27 20:51:25.868762 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.868592    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95f7e24a-fe2f-451e-ae08-fa9342e04cef-tmp-dir\") pod \"node-resolver-g5tm9\" (UID: \"95f7e24a-fe2f-451e-ae08-fa9342e04cef\") " pod="openshift-dns/node-resolver-g5tm9"
Jul 27 20:51:25.868762 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.868636    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/dbae1ee6-3616-432d-8ef9-255ad4cd444e-multus-daemon-config\") pod \"multus-td78h\" (UID: \"dbae1ee6-3616-432d-8ef9-255ad4cd444e\") " pod="openshift-multus/multus-td78h"
Jul 27 20:51:25.868762 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.868722    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/eae6d6c0-9844-45a8-a5ef-9c6f1ecc2810-os-release\") pod \"multus-additional-cni-plugins-djcnj\" (UID: \"eae6d6c0-9844-45a8-a5ef-9c6f1ecc2810\") " pod="openshift-multus/multus-additional-cni-plugins-djcnj"
Jul 27 20:51:25.868762 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.867578    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3566e5fb-c112-4f14-9e2f-fce1d833e15a-host-slash\") pod \"iptables-alerter-8cdfg\" (UID: \"3566e5fb-c112-4f14-9e2f-fce1d833e15a\") " pod="openshift-network-operator/iptables-alerter-8cdfg"
Jul 27 20:51:25.868941 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.868776    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/433c431a-b5bf-4734-a03f-6f0a16895ded-ovnkube-config\") pod \"ovnkube-node-95hg7\" (UID: \"433c431a-b5bf-4734-a03f-6f0a16895ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-95hg7"
Jul 27 20:51:25.868941 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.868823    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/433c431a-b5bf-4734-a03f-6f0a16895ded-systemd-units\") pod \"ovnkube-node-95hg7\" (UID: \"433c431a-b5bf-4734-a03f-6f0a16895ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-95hg7"
Jul 27 20:51:25.868941 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.868861    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/412ea27a-182d-40cc-917e-4686a4ff65cb-host\") pod \"node-ca-jm574\" (UID: \"412ea27a-182d-40cc-917e-4686a4ff65cb\") " pod="openshift-image-registry/node-ca-jm574"
Jul 27 20:51:25.868941 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.868862    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dbae1ee6-3616-432d-8ef9-255ad4cd444e-system-cni-dir\") pod \"multus-td78h\" (UID: \"dbae1ee6-3616-432d-8ef9-255ad4cd444e\") " pod="openshift-multus/multus-td78h"
Jul 27 20:51:25.868941 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.868916    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dbae1ee6-3616-432d-8ef9-255ad4cd444e-cnibin\") pod \"multus-td78h\" (UID: \"dbae1ee6-3616-432d-8ef9-255ad4cd444e\") " pod="openshift-multus/multus-td78h"
Jul 27 20:51:25.869154 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.868944    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/433c431a-b5bf-4734-a03f-6f0a16895ded-host-cni-netd\") pod \"ovnkube-node-95hg7\" (UID: \"433c431a-b5bf-4734-a03f-6f0a16895ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-95hg7"
Jul 27 20:51:25.869154 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.868978    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dbae1ee6-3616-432d-8ef9-255ad4cd444e-cni-binary-copy\") pod \"multus-td78h\" (UID: \"dbae1ee6-3616-432d-8ef9-255ad4cd444e\") " pod="openshift-multus/multus-td78h"
Jul 27 20:51:25.869154 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.869020    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/600fc153-38e0-4874-92e2-5895e14fa506-etc-sysctl-conf\") pod \"tuned-gn9bl\" (UID: \"600fc153-38e0-4874-92e2-5895e14fa506\") " pod="openshift-cluster-node-tuning-operator/tuned-gn9bl"
Jul 27 20:51:25.869154 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.869030    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/600fc153-38e0-4874-92e2-5895e14fa506-run\") pod \"tuned-gn9bl\" (UID: \"600fc153-38e0-4874-92e2-5895e14fa506\") " pod="openshift-cluster-node-tuning-operator/tuned-gn9bl"
Jul 27 20:51:25.869154 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.869070    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dbae1ee6-3616-432d-8ef9-255ad4cd444e-host-var-lib-kubelet\") pod \"multus-td78h\" (UID: \"dbae1ee6-3616-432d-8ef9-255ad4cd444e\") " pod="openshift-multus/multus-td78h"
Jul 27 20:51:25.869430 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.869237    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/433c431a-b5bf-4734-a03f-6f0a16895ded-host-slash\") pod \"ovnkube-node-95hg7\" (UID: \"433c431a-b5bf-4734-a03f-6f0a16895ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-95hg7"
Jul 27 20:51:25.869430 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.869243    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/600fc153-38e0-4874-92e2-5895e14fa506-run\") pod \"tuned-gn9bl\" (UID: \"600fc153-38e0-4874-92e2-5895e14fa506\") " pod="openshift-cluster-node-tuning-operator/tuned-gn9bl"
Jul 27 20:51:25.869430 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.869276    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/600fc153-38e0-4874-92e2-5895e14fa506-tmp\") pod \"tuned-gn9bl\" (UID: \"600fc153-38e0-4874-92e2-5895e14fa506\") " pod="openshift-cluster-node-tuning-operator/tuned-gn9bl"
Jul 27 20:51:25.869430 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.869310    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/433c431a-b5bf-4734-a03f-6f0a16895ded-run-openvswitch\") pod \"ovnkube-node-95hg7\" (UID: \"433c431a-b5bf-4734-a03f-6f0a16895ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-95hg7"
Jul 27 20:51:25.869430 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.869327    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/95f7e24a-fe2f-451e-ae08-fa9342e04cef-hosts-file\") pod \"node-resolver-g5tm9\" (UID: \"95f7e24a-fe2f-451e-ae08-fa9342e04cef\") " pod="openshift-dns/node-resolver-g5tm9"
Jul 27 20:51:25.869430 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.869367    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/433c431a-b5bf-4734-a03f-6f0a16895ded-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-95hg7\" (UID: \"433c431a-b5bf-4734-a03f-6f0a16895ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-95hg7"
Jul 27 20:51:25.869430 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.869422    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/433c431a-b5bf-4734-a03f-6f0a16895ded-host-cni-netd\") pod \"ovnkube-node-95hg7\" (UID: \"433c431a-b5bf-4734-a03f-6f0a16895ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-95hg7"
Jul 27 20:51:25.869430 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.869415    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/433c431a-b5bf-4734-a03f-6f0a16895ded-run-openvswitch\") pod \"ovnkube-node-95hg7\" (UID: \"433c431a-b5bf-4734-a03f-6f0a16895ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-95hg7"
Jul 27 20:51:25.869867 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.869461    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fac6ee6e-225a-4402-8834-69ff744b07f9-metrics-certs\") pod \"network-metrics-daemon-r6g8d\" (UID: \"fac6ee6e-225a-4402-8834-69ff744b07f9\") " pod="openshift-multus/network-metrics-daemon-r6g8d"
Jul 27 20:51:25.869867 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.869490    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/eae6d6c0-9844-45a8-a5ef-9c6f1ecc2810-cnibin\") pod \"multus-additional-cni-plugins-djcnj\" (UID: \"eae6d6c0-9844-45a8-a5ef-9c6f1ecc2810\") " pod="openshift-multus/multus-additional-cni-plugins-djcnj"
Jul 27 20:51:25.869867 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.869539    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/433c431a-b5bf-4734-a03f-6f0a16895ded-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-95hg7\" (UID: \"433c431a-b5bf-4734-a03f-6f0a16895ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-95hg7"
Jul 27 20:51:25.869867 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:25.869618    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 20:51:25.869867 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:25.869671    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fac6ee6e-225a-4402-8834-69ff744b07f9-metrics-certs podName:fac6ee6e-225a-4402-8834-69ff744b07f9 nodeName:}" failed. No retries permitted until 2026-07-27 20:51:26.369653124 +0000 UTC m=+3.121514898 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fac6ee6e-225a-4402-8834-69ff744b07f9-metrics-certs") pod "network-metrics-daemon-r6g8d" (UID: "fac6ee6e-225a-4402-8834-69ff744b07f9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 20:51:25.870161 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.870012    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dbae1ee6-3616-432d-8ef9-255ad4cd444e-cni-binary-copy\") pod \"multus-td78h\" (UID: \"dbae1ee6-3616-432d-8ef9-255ad4cd444e\") " pod="openshift-multus/multus-td78h"
Jul 27 20:51:25.870161 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.870140    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dbae1ee6-3616-432d-8ef9-255ad4cd444e-multus-conf-dir\") pod \"multus-td78h\" (UID: \"dbae1ee6-3616-432d-8ef9-255ad4cd444e\") " pod="openshift-multus/multus-td78h"
Jul 27 20:51:25.870283 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.870193    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/dbae1ee6-3616-432d-8ef9-255ad4cd444e-host-run-multus-certs\") pod \"multus-td78h\" (UID: \"dbae1ee6-3616-432d-8ef9-255ad4cd444e\") " pod="openshift-multus/multus-td78h"
Jul 27 20:51:25.870283 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.870239    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3566e5fb-c112-4f14-9e2f-fce1d833e15a-iptables-alerter-script\") pod \"iptables-alerter-8cdfg\" (UID: \"3566e5fb-c112-4f14-9e2f-fce1d833e15a\") " pod="openshift-network-operator/iptables-alerter-8cdfg"
Jul 27 20:51:25.870283 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.870269    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kkwm\" (UniqueName: \"kubernetes.io/projected/412ea27a-182d-40cc-917e-4686a4ff65cb-kube-api-access-7kkwm\") pod \"node-ca-jm574\" (UID: \"412ea27a-182d-40cc-917e-4686a4ff65cb\") " pod="openshift-image-registry/node-ca-jm574"
Jul 27 20:51:25.870413 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.870297    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzh9m\" (UniqueName: \"kubernetes.io/projected/eae6d6c0-9844-45a8-a5ef-9c6f1ecc2810-kube-api-access-bzh9m\") pod \"multus-additional-cni-plugins-djcnj\" (UID: \"eae6d6c0-9844-45a8-a5ef-9c6f1ecc2810\") " pod="openshift-multus/multus-additional-cni-plugins-djcnj"
Jul 27 20:51:25.870413 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.870349    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/dbae1ee6-3616-432d-8ef9-255ad4cd444e-host-run-multus-certs\") pod \"multus-td78h\" (UID: \"dbae1ee6-3616-432d-8ef9-255ad4cd444e\") " pod="openshift-multus/multus-td78h"
Jul 27 20:51:25.870413 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.870389    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dbae1ee6-3616-432d-8ef9-255ad4cd444e-multus-conf-dir\") pod \"multus-td78h\" (UID: \"dbae1ee6-3616-432d-8ef9-255ad4cd444e\") " pod="openshift-multus/multus-td78h"
Jul 27 20:51:25.870807 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.870779    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/eae6d6c0-9844-45a8-a5ef-9c6f1ecc2810-cnibin\") pod \"multus-additional-cni-plugins-djcnj\" (UID: \"eae6d6c0-9844-45a8-a5ef-9c6f1ecc2810\") " pod="openshift-multus/multus-additional-cni-plugins-djcnj"
Jul 27 20:51:25.870917 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.870889    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3566e5fb-c112-4f14-9e2f-fce1d833e15a-iptables-alerter-script\") pod \"iptables-alerter-8cdfg\" (UID: \"3566e5fb-c112-4f14-9e2f-fce1d833e15a\") " pod="openshift-network-operator/iptables-alerter-8cdfg"
Jul 27 20:51:25.870983 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.870926    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/dbae1ee6-3616-432d-8ef9-255ad4cd444e-hostroot\") pod \"multus-td78h\" (UID: \"dbae1ee6-3616-432d-8ef9-255ad4cd444e\") " pod="openshift-multus/multus-td78h"
Jul 27 20:51:25.870983 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.870964    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/600fc153-38e0-4874-92e2-5895e14fa506-etc-tuned\") pod \"tuned-gn9bl\" (UID: \"600fc153-38e0-4874-92e2-5895e14fa506\") " pod="openshift-cluster-node-tuning-operator/tuned-gn9bl"
Jul 27 20:51:25.871107 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.870992    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/433c431a-b5bf-4734-a03f-6f0a16895ded-ovnkube-script-lib\") pod \"ovnkube-node-95hg7\" (UID: \"433c431a-b5bf-4734-a03f-6f0a16895ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-95hg7"
Jul 27 20:51:25.871107 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.871014    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/dbae1ee6-3616-432d-8ef9-255ad4cd444e-hostroot\") pod \"multus-td78h\" (UID: \"dbae1ee6-3616-432d-8ef9-255ad4cd444e\") " pod="openshift-multus/multus-td78h"
Jul 27 20:51:25.871107 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.871026    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/eae6d6c0-9844-45a8-a5ef-9c6f1ecc2810-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-djcnj\" (UID: \"eae6d6c0-9844-45a8-a5ef-9c6f1ecc2810\") " pod="openshift-multus/multus-additional-cni-plugins-djcnj"
Jul 27 20:51:25.871107 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.871070    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/600fc153-38e0-4874-92e2-5895e14fa506-etc-modprobe-d\") pod \"tuned-gn9bl\" (UID: \"600fc153-38e0-4874-92e2-5895e14fa506\") " pod="openshift-cluster-node-tuning-operator/tuned-gn9bl"
Jul 27 20:51:25.871615 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.871587    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/600fc153-38e0-4874-92e2-5895e14fa506-etc-modprobe-d\") pod \"tuned-gn9bl\" (UID: \"600fc153-38e0-4874-92e2-5895e14fa506\") " pod="openshift-cluster-node-tuning-operator/tuned-gn9bl"
Jul 27 20:51:25.871682 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.871068    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/eae6d6c0-9844-45a8-a5ef-9c6f1ecc2810-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-djcnj\" (UID: \"eae6d6c0-9844-45a8-a5ef-9c6f1ecc2810\") " pod="openshift-multus/multus-additional-cni-plugins-djcnj"
Jul 27 20:51:25.871732 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.871700    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/600fc153-38e0-4874-92e2-5895e14fa506-host\") pod \"tuned-gn9bl\" (UID: \"600fc153-38e0-4874-92e2-5895e14fa506\") " pod="openshift-cluster-node-tuning-operator/tuned-gn9bl"
Jul 27 20:51:25.871806 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.871744    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/433c431a-b5bf-4734-a03f-6f0a16895ded-run-systemd\") pod \"ovnkube-node-95hg7\" (UID: \"433c431a-b5bf-4734-a03f-6f0a16895ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-95hg7"
Jul 27 20:51:25.871877 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.871850    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/433c431a-b5bf-4734-a03f-6f0a16895ded-ovnkube-script-lib\") pod \"ovnkube-node-95hg7\" (UID: \"433c431a-b5bf-4734-a03f-6f0a16895ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-95hg7"
Jul 27 20:51:25.871933 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.871868    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/433c431a-b5bf-4734-a03f-6f0a16895ded-run-systemd\") pod \"ovnkube-node-95hg7\" (UID: \"433c431a-b5bf-4734-a03f-6f0a16895ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-95hg7"
Jul 27 20:51:25.871981 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.871932    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/600fc153-38e0-4874-92e2-5895e14fa506-host\") pod \"tuned-gn9bl\" (UID: \"600fc153-38e0-4874-92e2-5895e14fa506\") " pod="openshift-cluster-node-tuning-operator/tuned-gn9bl"
Jul 27 20:51:25.872041 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.872020    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/433c431a-b5bf-4734-a03f-6f0a16895ded-node-log\") pod \"ovnkube-node-95hg7\" (UID: \"433c431a-b5bf-4734-a03f-6f0a16895ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-95hg7"
Jul 27 20:51:25.872259 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.872233    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/433c431a-b5bf-4734-a03f-6f0a16895ded-node-log\") pod \"ovnkube-node-95hg7\" (UID: \"433c431a-b5bf-4734-a03f-6f0a16895ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-95hg7"
Jul 27 20:51:25.872934 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.872875    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/eae6d6c0-9844-45a8-a5ef-9c6f1ecc2810-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-djcnj\" (UID: \"eae6d6c0-9844-45a8-a5ef-9c6f1ecc2810\") " pod="openshift-multus/multus-additional-cni-plugins-djcnj"
Jul 27 20:51:25.875370 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.873233    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/600fc153-38e0-4874-92e2-5895e14fa506-tmp\") pod \"tuned-gn9bl\" (UID: \"600fc153-38e0-4874-92e2-5895e14fa506\") " pod="openshift-cluster-node-tuning-operator/tuned-gn9bl"
Jul 27 20:51:25.876190 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.876070    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/600fc153-38e0-4874-92e2-5895e14fa506-etc-tuned\") pod \"tuned-gn9bl\" (UID: \"600fc153-38e0-4874-92e2-5895e14fa506\") " pod="openshift-cluster-node-tuning-operator/tuned-gn9bl"
Jul 27 20:51:25.876688 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.876667    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/433c431a-b5bf-4734-a03f-6f0a16895ded-ovn-node-metrics-cert\") pod \"ovnkube-node-95hg7\" (UID: \"433c431a-b5bf-4734-a03f-6f0a16895ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-95hg7"
Jul 27 20:51:25.878759 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.878740    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9827d\" (UniqueName: \"kubernetes.io/projected/dbae1ee6-3616-432d-8ef9-255ad4cd444e-kube-api-access-9827d\") pod \"multus-td78h\" (UID: \"dbae1ee6-3616-432d-8ef9-255ad4cd444e\") " pod="openshift-multus/multus-td78h"
Jul 27 20:51:25.881375 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.881344    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtp67\" (UniqueName: \"kubernetes.io/projected/433c431a-b5bf-4734-a03f-6f0a16895ded-kube-api-access-jtp67\") pod \"ovnkube-node-95hg7\" (UID: \"433c431a-b5bf-4734-a03f-6f0a16895ded\") " pod="openshift-ovn-kubernetes/ovnkube-node-95hg7"
Jul 27 20:51:25.882331 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.882272    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kkwm\" (UniqueName: \"kubernetes.io/projected/412ea27a-182d-40cc-917e-4686a4ff65cb-kube-api-access-7kkwm\") pod \"node-ca-jm574\" (UID: \"412ea27a-182d-40cc-917e-4686a4ff65cb\") " pod="openshift-image-registry/node-ca-jm574"
Jul 27 20:51:25.882460 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.882440    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qj65h\" (UniqueName: \"kubernetes.io/projected/95f7e24a-fe2f-451e-ae08-fa9342e04cef-kube-api-access-qj65h\") pod \"node-resolver-g5tm9\" (UID: \"95f7e24a-fe2f-451e-ae08-fa9342e04cef\") " pod="openshift-dns/node-resolver-g5tm9"
Jul 27 20:51:25.882561 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.882537    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6mts\" (UniqueName: \"kubernetes.io/projected/fac6ee6e-225a-4402-8834-69ff744b07f9-kube-api-access-x6mts\") pod \"network-metrics-daemon-r6g8d\" (UID: \"fac6ee6e-225a-4402-8834-69ff744b07f9\") " pod="openshift-multus/network-metrics-daemon-r6g8d"
Jul 27 20:51:25.882697 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.882670    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qslq4\" (UniqueName: \"kubernetes.io/projected/600fc153-38e0-4874-92e2-5895e14fa506-kube-api-access-qslq4\") pod \"tuned-gn9bl\" (UID: \"600fc153-38e0-4874-92e2-5895e14fa506\") " pod="openshift-cluster-node-tuning-operator/tuned-gn9bl"
Jul 27 20:51:25.883127 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.883087    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dn9l\" (UniqueName: \"kubernetes.io/projected/3566e5fb-c112-4f14-9e2f-fce1d833e15a-kube-api-access-9dn9l\") pod \"iptables-alerter-8cdfg\" (UID: \"3566e5fb-c112-4f14-9e2f-fce1d833e15a\") " pod="openshift-network-operator/iptables-alerter-8cdfg"
Jul 27 20:51:25.884071 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.884052    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzh9m\" (UniqueName: \"kubernetes.io/projected/eae6d6c0-9844-45a8-a5ef-9c6f1ecc2810-kube-api-access-bzh9m\") pod \"multus-additional-cni-plugins-djcnj\" (UID: \"eae6d6c0-9844-45a8-a5ef-9c6f1ecc2810\") " pod="openshift-multus/multus-additional-cni-plugins-djcnj"
Jul 27 20:51:25.916970 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.916939    2560 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 20:51:25.974164 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.974139    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tjtf6"
Jul 27 20:51:25.985286 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.985263    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-sg57x"
Jul 27 20:51:25.996314 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:25.996296    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-g5tm9"
Jul 27 20:51:26.003060 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:26.003042    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-td78h"
Jul 27 20:51:26.013695 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:26.013675    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-gn9bl"
Jul 27 20:51:26.022610 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:26.022580    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jm574"
Jul 27 20:51:26.032476 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:26.032457    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-95hg7"
Jul 27 20:51:26.039426 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:26.039405    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-8cdfg"
Jul 27 20:51:26.044156 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:26.044136    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-djcnj"
Jul 27 20:51:26.275554 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:26.275460    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8ns2\" (UniqueName: \"kubernetes.io/projected/1f1259cb-be97-4841-bd22-5bba585870b6-kube-api-access-n8ns2\") pod \"network-check-target-dbzwm\" (UID: \"1f1259cb-be97-4841-bd22-5bba585870b6\") " pod="openshift-network-diagnostics/network-check-target-dbzwm"
Jul 27 20:51:26.275554 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:26.275499    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4b924e42-83bc-459f-b1a2-f115f9bce63f-original-pull-secret\") pod \"global-pull-secret-syncer-dtrjg\" (UID: \"4b924e42-83bc-459f-b1a2-f115f9bce63f\") " pod="kube-system/global-pull-secret-syncer-dtrjg"
Jul 27 20:51:26.275758 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:26.275651    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 20:51:26.275758 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:26.275684    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 20:51:26.275758 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:26.275652    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 20:51:26.275758 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:26.275700    2560 projected.go:196] Error preparing data for projected volume kube-api-access-n8ns2 for pod openshift-network-diagnostics/network-check-target-dbzwm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 20:51:26.275758 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:26.275756    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b924e42-83bc-459f-b1a2-f115f9bce63f-original-pull-secret podName:4b924e42-83bc-459f-b1a2-f115f9bce63f nodeName:}" failed. No retries permitted until 2026-07-27 20:51:27.275735751 +0000 UTC m=+4.027597530 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4b924e42-83bc-459f-b1a2-f115f9bce63f-original-pull-secret") pod "global-pull-secret-syncer-dtrjg" (UID: "4b924e42-83bc-459f-b1a2-f115f9bce63f") : object "kube-system"/"original-pull-secret" not registered
Jul 27 20:51:26.275960 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:26.275778    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1f1259cb-be97-4841-bd22-5bba585870b6-kube-api-access-n8ns2 podName:1f1259cb-be97-4841-bd22-5bba585870b6 nodeName:}" failed. No retries permitted until 2026-07-27 20:51:27.27576916 +0000 UTC m=+4.027630938 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-n8ns2" (UniqueName: "kubernetes.io/projected/1f1259cb-be97-4841-bd22-5bba585870b6-kube-api-access-n8ns2") pod "network-check-target-dbzwm" (UID: "1f1259cb-be97-4841-bd22-5bba585870b6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 20:51:26.376055 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:26.376021    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fac6ee6e-225a-4402-8834-69ff744b07f9-metrics-certs\") pod \"network-metrics-daemon-r6g8d\" (UID: \"fac6ee6e-225a-4402-8834-69ff744b07f9\") " pod="openshift-multus/network-metrics-daemon-r6g8d"
Jul 27 20:51:26.376269 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:26.376221    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 20:51:26.376335 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:26.376293    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fac6ee6e-225a-4402-8834-69ff744b07f9-metrics-certs podName:fac6ee6e-225a-4402-8834-69ff744b07f9 nodeName:}" failed. No retries permitted until 2026-07-27 20:51:27.376272171 +0000 UTC m=+4.128133946 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fac6ee6e-225a-4402-8834-69ff744b07f9-metrics-certs") pod "network-metrics-daemon-r6g8d" (UID: "fac6ee6e-225a-4402-8834-69ff744b07f9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 20:51:26.703304 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:26.703173    2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-26 20:46:26 +0000 UTC" deadline="2028-01-28 07:46:30.835877855 +0000 UTC"
Jul 27 20:51:26.703304 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:26.703253    2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13186h55m4.132630222s"
Jul 27 20:51:26.858352 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:26.858317    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod412ea27a_182d_40cc_917e_4686a4ff65cb.slice/crio-949b0a7bf09ef728534ee1aaf4a631b10240b3e9cdde977fea956e74ffa93761 WatchSource:0}: Error finding container 949b0a7bf09ef728534ee1aaf4a631b10240b3e9cdde977fea956e74ffa93761: Status 404 returned error can't find the container with id 949b0a7bf09ef728534ee1aaf4a631b10240b3e9cdde977fea956e74ffa93761
Jul 27 20:51:26.859077 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:26.859045    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbae1ee6_3616_432d_8ef9_255ad4cd444e.slice/crio-62cef117f16dfeb5fea5bf09351faef31f690fd05b56b8aa1b8614461f21ae56 WatchSource:0}: Error finding container 62cef117f16dfeb5fea5bf09351faef31f690fd05b56b8aa1b8614461f21ae56: Status 404 returned error can't find the container with id 62cef117f16dfeb5fea5bf09351faef31f690fd05b56b8aa1b8614461f21ae56
Jul 27 20:51:26.860411 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:26.860387    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3566e5fb_c112_4f14_9e2f_fce1d833e15a.slice/crio-52b4ed7d5fb0667e8ae40a09117145cc32eab244454fe477ee06faef32a446bd WatchSource:0}: Error finding container 52b4ed7d5fb0667e8ae40a09117145cc32eab244454fe477ee06faef32a446bd: Status 404 returned error can't find the container with id 52b4ed7d5fb0667e8ae40a09117145cc32eab244454fe477ee06faef32a446bd
Jul 27 20:51:26.863027 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:26.861401    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95f7e24a_fe2f_451e_ae08_fa9342e04cef.slice/crio-d4cf40940f54e0d85b057d28bc4346fc7ac8499d05a1ace2891e400e92c4c773 WatchSource:0}: Error finding container d4cf40940f54e0d85b057d28bc4346fc7ac8499d05a1ace2891e400e92c4c773: Status 404 returned error can't find the container with id d4cf40940f54e0d85b057d28bc4346fc7ac8499d05a1ace2891e400e92c4c773
Jul 27 20:51:27.284007 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:27.283734    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8ns2\" (UniqueName: \"kubernetes.io/projected/1f1259cb-be97-4841-bd22-5bba585870b6-kube-api-access-n8ns2\") pod \"network-check-target-dbzwm\" (UID: \"1f1259cb-be97-4841-bd22-5bba585870b6\") " pod="openshift-network-diagnostics/network-check-target-dbzwm"
Jul 27 20:51:27.284192 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:27.284034    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4b924e42-83bc-459f-b1a2-f115f9bce63f-original-pull-secret\") pod \"global-pull-secret-syncer-dtrjg\" (UID: \"4b924e42-83bc-459f-b1a2-f115f9bce63f\") " pod="kube-system/global-pull-secret-syncer-dtrjg"
Jul 27 20:51:27.284356 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:27.284336    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 20:51:27.284428 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:27.284412    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b924e42-83bc-459f-b1a2-f115f9bce63f-original-pull-secret podName:4b924e42-83bc-459f-b1a2-f115f9bce63f nodeName:}" failed. No retries permitted until 2026-07-27 20:51:29.284391445 +0000 UTC m=+6.036253221 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4b924e42-83bc-459f-b1a2-f115f9bce63f-original-pull-secret") pod "global-pull-secret-syncer-dtrjg" (UID: "4b924e42-83bc-459f-b1a2-f115f9bce63f") : object "kube-system"/"original-pull-secret" not registered
Jul 27 20:51:27.284730 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:27.284695    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 20:51:27.284730 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:27.284727    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 20:51:27.284880 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:27.284741    2560 projected.go:196] Error preparing data for projected volume kube-api-access-n8ns2 for pod openshift-network-diagnostics/network-check-target-dbzwm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 20:51:27.284880 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:27.284797    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1f1259cb-be97-4841-bd22-5bba585870b6-kube-api-access-n8ns2 podName:1f1259cb-be97-4841-bd22-5bba585870b6 nodeName:}" failed. No retries permitted until 2026-07-27 20:51:29.284779155 +0000 UTC m=+6.036640945 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-n8ns2" (UniqueName: "kubernetes.io/projected/1f1259cb-be97-4841-bd22-5bba585870b6-kube-api-access-n8ns2") pod "network-check-target-dbzwm" (UID: "1f1259cb-be97-4841-bd22-5bba585870b6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 20:51:27.386315 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:27.385607    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fac6ee6e-225a-4402-8834-69ff744b07f9-metrics-certs\") pod \"network-metrics-daemon-r6g8d\" (UID: \"fac6ee6e-225a-4402-8834-69ff744b07f9\") " pod="openshift-multus/network-metrics-daemon-r6g8d"
Jul 27 20:51:27.386315 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:27.385797    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 20:51:27.386315 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:27.385865    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fac6ee6e-225a-4402-8834-69ff744b07f9-metrics-certs podName:fac6ee6e-225a-4402-8834-69ff744b07f9 nodeName:}" failed. No retries permitted until 2026-07-27 20:51:29.385845442 +0000 UTC m=+6.137707219 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fac6ee6e-225a-4402-8834-69ff744b07f9-metrics-certs") pod "network-metrics-daemon-r6g8d" (UID: "fac6ee6e-225a-4402-8834-69ff744b07f9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 20:51:27.703772 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:27.703650    2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-26 20:46:26 +0000 UTC" deadline="2028-02-18 07:04:28.495702249 +0000 UTC"
Jul 27 20:51:27.703772 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:27.703687    2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13690h13m0.792020303s"
Jul 27 20:51:27.844316 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:27.844283    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dbzwm"
Jul 27 20:51:27.844564 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:27.844418    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dbzwm" podUID="1f1259cb-be97-4841-bd22-5bba585870b6"
Jul 27 20:51:27.844886 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:27.844863    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dtrjg"
Jul 27 20:51:27.845022 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:27.844965    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dtrjg" podUID="4b924e42-83bc-459f-b1a2-f115f9bce63f"
Jul 27 20:51:27.845783 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:27.845728    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r6g8d"
Jul 27 20:51:27.845884 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:27.845844    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r6g8d" podUID="fac6ee6e-225a-4402-8834-69ff744b07f9"
Jul 27 20:51:27.872243 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:27.872207    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-td78h" event={"ID":"dbae1ee6-3616-432d-8ef9-255ad4cd444e","Type":"ContainerStarted","Data":"62cef117f16dfeb5fea5bf09351faef31f690fd05b56b8aa1b8614461f21ae56"}
Jul 27 20:51:27.882161 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:27.882052    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-8cdfg" event={"ID":"3566e5fb-c112-4f14-9e2f-fce1d833e15a","Type":"ContainerStarted","Data":"52b4ed7d5fb0667e8ae40a09117145cc32eab244454fe477ee06faef32a446bd"}
Jul 27 20:51:27.892323 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:27.892290    2560 generic.go:356] "Generic (PLEG): container finished" podID="e132fc0b72959a63cea0555c65890a47" containerID="75dcafadc8ac673d619440f3a22dd3b0be0b9c5e4ca70bed88f71e62462171d6" exitCode=0
Jul 27 20:51:27.892560 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:27.892387    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-64.ec2.internal" event={"ID":"e132fc0b72959a63cea0555c65890a47","Type":"ContainerDied","Data":"75dcafadc8ac673d619440f3a22dd3b0be0b9c5e4ca70bed88f71e62462171d6"}
Jul 27 20:51:27.902952 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:27.902807    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-gn9bl" event={"ID":"600fc153-38e0-4874-92e2-5895e14fa506","Type":"ContainerStarted","Data":"aa4e34f79ed43d5c88c41172e8a3bdfd0ce57d09f1d9f188c0849387fbe50983"}
Jul 27 20:51:27.912376 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:27.912347    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-sg57x" event={"ID":"9a766ad8-ce17-462e-b45f-867a6c6c44e4","Type":"ContainerStarted","Data":"6951ff1e77628435b6e32791fbf67c81c3b7fdc485ba28d326ec37f5531497c7"}
Jul 27 20:51:27.924348 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:27.924310    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-g5tm9" event={"ID":"95f7e24a-fe2f-451e-ae08-fa9342e04cef","Type":"ContainerStarted","Data":"d4cf40940f54e0d85b057d28bc4346fc7ac8499d05a1ace2891e400e92c4c773"}
Jul 27 20:51:27.928746 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:27.928720    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-95hg7" event={"ID":"433c431a-b5bf-4734-a03f-6f0a16895ded","Type":"ContainerStarted","Data":"7cbc6fea5e6de33c406247fe7ba05411d3b2e8638a90e054f041942934dde6a9"}
Jul 27 20:51:27.943490 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:27.943459    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jm574" event={"ID":"412ea27a-182d-40cc-917e-4686a4ff65cb","Type":"ContainerStarted","Data":"949b0a7bf09ef728534ee1aaf4a631b10240b3e9cdde977fea956e74ffa93761"}
Jul 27 20:51:27.949644 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:27.949606    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-64.ec2.internal" event={"ID":"5fa776c99382db40a7e9c25fe5c95038","Type":"ContainerStarted","Data":"85371d020d360229701ff42863522ce383716519a8abfeb52c9b135598cad108"}
Jul 27 20:51:27.952580 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:27.952548    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tjtf6" event={"ID":"a49f6f21-5110-4f7c-8f79-1d9e14d69bc6","Type":"ContainerStarted","Data":"d846802bad49bb3ba477cd7f2a28d1ae57200579aac6fe983e2476f9429aecd9"}
Jul 27 20:51:27.957855 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:27.957788    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-djcnj" event={"ID":"eae6d6c0-9844-45a8-a5ef-9c6f1ecc2810","Type":"ContainerStarted","Data":"7723a92ee3ecb675300c2387c8a101ff2a600cfb6afd0c6bc4c00d820d39efa2"}
Jul 27 20:51:27.966972 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:27.966921    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-64.ec2.internal" podStartSLOduration=2.966901895 podStartE2EDuration="2.966901895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 20:51:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:51:27.966340494 +0000 UTC m=+4.718202294" watchObservedRunningTime="2026-07-27 20:51:27.966901895 +0000 UTC m=+4.718763694"
Jul 27 20:51:28.991239 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:28.991143    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-64.ec2.internal" event={"ID":"e132fc0b72959a63cea0555c65890a47","Type":"ContainerStarted","Data":"eb283c3396a085713aa3b4a85a1f0cfe8bf33b5faedd72c4dfb78e6c03d7dd29"}
Jul 27 20:51:29.302524 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:29.302421    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8ns2\" (UniqueName: \"kubernetes.io/projected/1f1259cb-be97-4841-bd22-5bba585870b6-kube-api-access-n8ns2\") pod \"network-check-target-dbzwm\" (UID: \"1f1259cb-be97-4841-bd22-5bba585870b6\") " pod="openshift-network-diagnostics/network-check-target-dbzwm"
Jul 27 20:51:29.302524 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:29.302484    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4b924e42-83bc-459f-b1a2-f115f9bce63f-original-pull-secret\") pod \"global-pull-secret-syncer-dtrjg\" (UID: \"4b924e42-83bc-459f-b1a2-f115f9bce63f\") " pod="kube-system/global-pull-secret-syncer-dtrjg"
Jul 27 20:51:29.302777 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:29.302663    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 20:51:29.302777 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:29.302730    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b924e42-83bc-459f-b1a2-f115f9bce63f-original-pull-secret podName:4b924e42-83bc-459f-b1a2-f115f9bce63f nodeName:}" failed. No retries permitted until 2026-07-27 20:51:33.302710881 +0000 UTC m=+10.054572656 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4b924e42-83bc-459f-b1a2-f115f9bce63f-original-pull-secret") pod "global-pull-secret-syncer-dtrjg" (UID: "4b924e42-83bc-459f-b1a2-f115f9bce63f") : object "kube-system"/"original-pull-secret" not registered
Jul 27 20:51:29.303232 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:29.303069    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 20:51:29.303232 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:29.303099    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 20:51:29.303232 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:29.303114    2560 projected.go:196] Error preparing data for projected volume kube-api-access-n8ns2 for pod openshift-network-diagnostics/network-check-target-dbzwm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 20:51:29.303232 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:29.303205    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1f1259cb-be97-4841-bd22-5bba585870b6-kube-api-access-n8ns2 podName:1f1259cb-be97-4841-bd22-5bba585870b6 nodeName:}" failed. No retries permitted until 2026-07-27 20:51:33.303185297 +0000 UTC m=+10.055047078 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-n8ns2" (UniqueName: "kubernetes.io/projected/1f1259cb-be97-4841-bd22-5bba585870b6-kube-api-access-n8ns2") pod "network-check-target-dbzwm" (UID: "1f1259cb-be97-4841-bd22-5bba585870b6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 20:51:29.403174 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:29.402872    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fac6ee6e-225a-4402-8834-69ff744b07f9-metrics-certs\") pod \"network-metrics-daemon-r6g8d\" (UID: \"fac6ee6e-225a-4402-8834-69ff744b07f9\") " pod="openshift-multus/network-metrics-daemon-r6g8d"
Jul 27 20:51:29.403174 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:29.403052    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 20:51:29.403174 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:29.403129    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fac6ee6e-225a-4402-8834-69ff744b07f9-metrics-certs podName:fac6ee6e-225a-4402-8834-69ff744b07f9 nodeName:}" failed. No retries permitted until 2026-07-27 20:51:33.403109035 +0000 UTC m=+10.154970814 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fac6ee6e-225a-4402-8834-69ff744b07f9-metrics-certs") pod "network-metrics-daemon-r6g8d" (UID: "fac6ee6e-225a-4402-8834-69ff744b07f9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 20:51:29.846084 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:29.845256    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dbzwm"
Jul 27 20:51:29.846084 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:29.845397    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dbzwm" podUID="1f1259cb-be97-4841-bd22-5bba585870b6"
Jul 27 20:51:29.846084 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:29.845787    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dtrjg"
Jul 27 20:51:29.846084 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:29.845884    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dtrjg" podUID="4b924e42-83bc-459f-b1a2-f115f9bce63f"
Jul 27 20:51:29.846084 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:29.845935    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r6g8d"
Jul 27 20:51:29.846084 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:29.846021    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r6g8d" podUID="fac6ee6e-225a-4402-8834-69ff744b07f9"
Jul 27 20:51:31.842708 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:31.842579    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dtrjg"
Jul 27 20:51:31.842708 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:31.842623    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r6g8d"
Jul 27 20:51:31.843244 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:31.842718    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dtrjg" podUID="4b924e42-83bc-459f-b1a2-f115f9bce63f"
Jul 27 20:51:31.843244 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:31.842579    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dbzwm"
Jul 27 20:51:31.843532 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:31.843363    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dbzwm" podUID="1f1259cb-be97-4841-bd22-5bba585870b6"
Jul 27 20:51:31.843532 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:31.843497    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r6g8d" podUID="fac6ee6e-225a-4402-8834-69ff744b07f9"
Jul 27 20:51:33.339040 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:33.338996    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8ns2\" (UniqueName: \"kubernetes.io/projected/1f1259cb-be97-4841-bd22-5bba585870b6-kube-api-access-n8ns2\") pod \"network-check-target-dbzwm\" (UID: \"1f1259cb-be97-4841-bd22-5bba585870b6\") " pod="openshift-network-diagnostics/network-check-target-dbzwm"
Jul 27 20:51:33.339581 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:33.339059    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4b924e42-83bc-459f-b1a2-f115f9bce63f-original-pull-secret\") pod \"global-pull-secret-syncer-dtrjg\" (UID: \"4b924e42-83bc-459f-b1a2-f115f9bce63f\") " pod="kube-system/global-pull-secret-syncer-dtrjg"
Jul 27 20:51:33.339581 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:33.339235    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 20:51:33.339581 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:33.339241    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 20:51:33.339581 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:33.339266    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 20:51:33.339581 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:33.339278    2560 projected.go:196] Error preparing data for projected volume kube-api-access-n8ns2 for pod openshift-network-diagnostics/network-check-target-dbzwm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 20:51:33.339581 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:33.339311    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b924e42-83bc-459f-b1a2-f115f9bce63f-original-pull-secret podName:4b924e42-83bc-459f-b1a2-f115f9bce63f nodeName:}" failed. No retries permitted until 2026-07-27 20:51:41.339290194 +0000 UTC m=+18.091151986 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4b924e42-83bc-459f-b1a2-f115f9bce63f-original-pull-secret") pod "global-pull-secret-syncer-dtrjg" (UID: "4b924e42-83bc-459f-b1a2-f115f9bce63f") : object "kube-system"/"original-pull-secret" not registered
Jul 27 20:51:33.339581 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:33.339332    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1f1259cb-be97-4841-bd22-5bba585870b6-kube-api-access-n8ns2 podName:1f1259cb-be97-4841-bd22-5bba585870b6 nodeName:}" failed. No retries permitted until 2026-07-27 20:51:41.339322607 +0000 UTC m=+18.091184395 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-n8ns2" (UniqueName: "kubernetes.io/projected/1f1259cb-be97-4841-bd22-5bba585870b6-kube-api-access-n8ns2") pod "network-check-target-dbzwm" (UID: "1f1259cb-be97-4841-bd22-5bba585870b6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 20:51:33.439830 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:33.439791    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fac6ee6e-225a-4402-8834-69ff744b07f9-metrics-certs\") pod \"network-metrics-daemon-r6g8d\" (UID: \"fac6ee6e-225a-4402-8834-69ff744b07f9\") " pod="openshift-multus/network-metrics-daemon-r6g8d"
Jul 27 20:51:33.439981 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:33.439943    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 20:51:33.440020 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:33.440005    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fac6ee6e-225a-4402-8834-69ff744b07f9-metrics-certs podName:fac6ee6e-225a-4402-8834-69ff744b07f9 nodeName:}" failed. No retries permitted until 2026-07-27 20:51:41.439986559 +0000 UTC m=+18.191848334 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fac6ee6e-225a-4402-8834-69ff744b07f9-metrics-certs") pod "network-metrics-daemon-r6g8d" (UID: "fac6ee6e-225a-4402-8834-69ff744b07f9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 20:51:33.843558 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:33.843517    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dtrjg"
Jul 27 20:51:33.843730 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:33.843647    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dbzwm"
Jul 27 20:51:33.843730 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:33.843666    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r6g8d"
Jul 27 20:51:33.843730 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:33.843669    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dtrjg" podUID="4b924e42-83bc-459f-b1a2-f115f9bce63f"
Jul 27 20:51:33.843953 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:33.843743    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dbzwm" podUID="1f1259cb-be97-4841-bd22-5bba585870b6"
Jul 27 20:51:33.843953 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:33.843831    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r6g8d" podUID="fac6ee6e-225a-4402-8834-69ff744b07f9"
Jul 27 20:51:35.844702 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:35.844655    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dbzwm"
Jul 27 20:51:35.845111 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:35.844798    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dbzwm" podUID="1f1259cb-be97-4841-bd22-5bba585870b6"
Jul 27 20:51:35.845252 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:35.845235    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dtrjg"
Jul 27 20:51:35.845356 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:35.845336    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dtrjg" podUID="4b924e42-83bc-459f-b1a2-f115f9bce63f"
Jul 27 20:51:35.845407 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:35.845399    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r6g8d"
Jul 27 20:51:35.845500 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:35.845483    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r6g8d" podUID="fac6ee6e-225a-4402-8834-69ff744b07f9"
Jul 27 20:51:37.841889 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:37.841857    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dbzwm"
Jul 27 20:51:37.841889 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:37.841891    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r6g8d"
Jul 27 20:51:37.842417 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:37.841858    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dtrjg"
Jul 27 20:51:37.842417 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:37.841976    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dbzwm" podUID="1f1259cb-be97-4841-bd22-5bba585870b6"
Jul 27 20:51:37.842417 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:37.842112    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dtrjg" podUID="4b924e42-83bc-459f-b1a2-f115f9bce63f"
Jul 27 20:51:37.842417 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:37.842222    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r6g8d" podUID="fac6ee6e-225a-4402-8834-69ff744b07f9"
Jul 27 20:51:39.842483 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:39.842449    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dtrjg"
Jul 27 20:51:39.842979 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:39.842592    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r6g8d"
Jul 27 20:51:39.842979 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:39.842448    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dbzwm"
Jul 27 20:51:39.842979 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:39.842707    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r6g8d" podUID="fac6ee6e-225a-4402-8834-69ff744b07f9"
Jul 27 20:51:39.842979 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:39.842587    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dtrjg" podUID="4b924e42-83bc-459f-b1a2-f115f9bce63f"
Jul 27 20:51:39.842979 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:39.842792    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dbzwm" podUID="1f1259cb-be97-4841-bd22-5bba585870b6"
Jul 27 20:51:41.401595 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:41.401565    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8ns2\" (UniqueName: \"kubernetes.io/projected/1f1259cb-be97-4841-bd22-5bba585870b6-kube-api-access-n8ns2\") pod \"network-check-target-dbzwm\" (UID: \"1f1259cb-be97-4841-bd22-5bba585870b6\") " pod="openshift-network-diagnostics/network-check-target-dbzwm"
Jul 27 20:51:41.401595 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:41.401598    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4b924e42-83bc-459f-b1a2-f115f9bce63f-original-pull-secret\") pod \"global-pull-secret-syncer-dtrjg\" (UID: \"4b924e42-83bc-459f-b1a2-f115f9bce63f\") " pod="kube-system/global-pull-secret-syncer-dtrjg"
Jul 27 20:51:41.402046 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:41.401710    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 20:51:41.402046 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:41.401741    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 20:51:41.402046 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:41.401758    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b924e42-83bc-459f-b1a2-f115f9bce63f-original-pull-secret podName:4b924e42-83bc-459f-b1a2-f115f9bce63f nodeName:}" failed. No retries permitted until 2026-07-27 20:51:57.401743595 +0000 UTC m=+34.153605369 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4b924e42-83bc-459f-b1a2-f115f9bce63f-original-pull-secret") pod "global-pull-secret-syncer-dtrjg" (UID: "4b924e42-83bc-459f-b1a2-f115f9bce63f") : object "kube-system"/"original-pull-secret" not registered
Jul 27 20:51:41.402046 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:41.401760    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 20:51:41.402046 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:41.401772    2560 projected.go:196] Error preparing data for projected volume kube-api-access-n8ns2 for pod openshift-network-diagnostics/network-check-target-dbzwm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 20:51:41.402046 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:41.401809    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1f1259cb-be97-4841-bd22-5bba585870b6-kube-api-access-n8ns2 podName:1f1259cb-be97-4841-bd22-5bba585870b6 nodeName:}" failed. No retries permitted until 2026-07-27 20:51:57.401797482 +0000 UTC m=+34.153659255 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-n8ns2" (UniqueName: "kubernetes.io/projected/1f1259cb-be97-4841-bd22-5bba585870b6-kube-api-access-n8ns2") pod "network-check-target-dbzwm" (UID: "1f1259cb-be97-4841-bd22-5bba585870b6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 20:51:41.502747 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:41.502712    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fac6ee6e-225a-4402-8834-69ff744b07f9-metrics-certs\") pod \"network-metrics-daemon-r6g8d\" (UID: \"fac6ee6e-225a-4402-8834-69ff744b07f9\") " pod="openshift-multus/network-metrics-daemon-r6g8d"
Jul 27 20:51:41.502924 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:41.502837    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 20:51:41.502924 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:41.502891    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fac6ee6e-225a-4402-8834-69ff744b07f9-metrics-certs podName:fac6ee6e-225a-4402-8834-69ff744b07f9 nodeName:}" failed. No retries permitted until 2026-07-27 20:51:57.502876605 +0000 UTC m=+34.254738377 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fac6ee6e-225a-4402-8834-69ff744b07f9-metrics-certs") pod "network-metrics-daemon-r6g8d" (UID: "fac6ee6e-225a-4402-8834-69ff744b07f9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 20:51:41.842250 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:41.842221    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dbzwm"
Jul 27 20:51:41.842250 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:41.842239    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dtrjg"
Jul 27 20:51:41.842416 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:41.842329    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dbzwm" podUID="1f1259cb-be97-4841-bd22-5bba585870b6"
Jul 27 20:51:41.842416 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:41.842390    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dtrjg" podUID="4b924e42-83bc-459f-b1a2-f115f9bce63f"
Jul 27 20:51:41.842491 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:41.842419    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r6g8d"
Jul 27 20:51:41.842544 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:41.842519    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r6g8d" podUID="fac6ee6e-225a-4402-8834-69ff744b07f9"
Jul 27 20:51:43.843223 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:43.843186    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dbzwm"
Jul 27 20:51:43.843687 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:43.843274    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dtrjg"
Jul 27 20:51:43.843687 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:43.843292    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dbzwm" podUID="1f1259cb-be97-4841-bd22-5bba585870b6"
Jul 27 20:51:43.843687 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:43.843387    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r6g8d"
Jul 27 20:51:43.843687 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:43.843446    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dtrjg" podUID="4b924e42-83bc-459f-b1a2-f115f9bce63f"
Jul 27 20:51:43.843687 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:43.843509    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r6g8d" podUID="fac6ee6e-225a-4402-8834-69ff744b07f9"
Jul 27 20:51:45.842382 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:45.842359    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r6g8d"
Jul 27 20:51:45.842789 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:45.842394    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dbzwm"
Jul 27 20:51:45.842789 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:45.842359    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dtrjg"
Jul 27 20:51:45.842789 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:45.842492    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r6g8d" podUID="fac6ee6e-225a-4402-8834-69ff744b07f9"
Jul 27 20:51:45.842789 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:45.842591    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dtrjg" podUID="4b924e42-83bc-459f-b1a2-f115f9bce63f"
Jul 27 20:51:45.842789 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:45.842658    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dbzwm" podUID="1f1259cb-be97-4841-bd22-5bba585870b6"
Jul 27 20:51:47.029356 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:47.028975    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-gn9bl" event={"ID":"600fc153-38e0-4874-92e2-5895e14fa506","Type":"ContainerStarted","Data":"e4d021179f28f9591782f4bf5dcc4d520235dcca5ce4f9681e46a1fa70d6bb0a"}
Jul 27 20:51:47.030444 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:47.030415    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-sg57x" event={"ID":"9a766ad8-ce17-462e-b45f-867a6c6c44e4","Type":"ContainerStarted","Data":"acaf5f2cce061005c91d2a20a2bfcd58028c7201e08db0816768b811756679ab"}
Jul 27 20:51:47.031850 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:47.031818    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-g5tm9" event={"ID":"95f7e24a-fe2f-451e-ae08-fa9342e04cef","Type":"ContainerStarted","Data":"e64a188a1f931dfdf0938b1e14205ce3755156d9a22fd1a05695f2880dfa930c"}
Jul 27 20:51:47.034564 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:47.034545    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-95hg7_433c431a-b5bf-4734-a03f-6f0a16895ded/ovn-acl-logging/0.log"
Jul 27 20:51:47.034887 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:47.034866    2560 generic.go:356] "Generic (PLEG): container finished" podID="433c431a-b5bf-4734-a03f-6f0a16895ded" containerID="bde5ed442cafaa6693764d4068392fc6af0183c0737e52aba7f122222f621dc5" exitCode=1
Jul 27 20:51:47.034957 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:47.034934    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-95hg7" event={"ID":"433c431a-b5bf-4734-a03f-6f0a16895ded","Type":"ContainerStarted","Data":"34d7604910e1ec0b778c54cddc5b96aeb1155d32a654c0ed6b0ffd416efeec1e"}
Jul 27 20:51:47.035027 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:47.035011    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-95hg7" event={"ID":"433c431a-b5bf-4734-a03f-6f0a16895ded","Type":"ContainerStarted","Data":"dfa3c917ca2a5e991fbb15589208f580f5aed0d2ffee3ef99d893939bae02dee"}
Jul 27 20:51:47.035080 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:47.035066    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-95hg7" event={"ID":"433c431a-b5bf-4734-a03f-6f0a16895ded","Type":"ContainerStarted","Data":"60edb382af9dcc8d1f8938d18adbae9ea7ead4d41c72a6000d41c9e89af1d50e"}
Jul 27 20:51:47.035149 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:47.035136    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-95hg7" event={"ID":"433c431a-b5bf-4734-a03f-6f0a16895ded","Type":"ContainerStarted","Data":"8cb49e768b6772f1c9344a90115794883a844d96e1820aa21f6ac585b4bd340a"}
Jul 27 20:51:47.035200 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:47.035170    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-95hg7" event={"ID":"433c431a-b5bf-4734-a03f-6f0a16895ded","Type":"ContainerDied","Data":"bde5ed442cafaa6693764d4068392fc6af0183c0737e52aba7f122222f621dc5"}
Jul 27 20:51:47.035248 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:47.035210    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-95hg7" event={"ID":"433c431a-b5bf-4734-a03f-6f0a16895ded","Type":"ContainerStarted","Data":"7f478645091bb40e24ebad94018facf8dbdde8203a3e973f35be19ee32243767"}
Jul 27 20:51:47.036405 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:47.036381    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jm574" event={"ID":"412ea27a-182d-40cc-917e-4686a4ff65cb","Type":"ContainerStarted","Data":"0ea273f97e148116c69cef1c407fc96a1257fb841cbbf98e3cb9aee00d9373c8"}
Jul 27 20:51:47.037777 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:47.037751    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tjtf6" event={"ID":"a49f6f21-5110-4f7c-8f79-1d9e14d69bc6","Type":"ContainerStarted","Data":"3362ff51c492abaf77b8e9ef5d679e8bd7022c5126638374029e85d006f46c82"}
Jul 27 20:51:47.039156 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:47.039132    2560 generic.go:356] "Generic (PLEG): container finished" podID="eae6d6c0-9844-45a8-a5ef-9c6f1ecc2810" containerID="0b07ef3655ed8a3bf88668a02d2fbcad7d08e708c8b3adad827805bf20c671f9" exitCode=0
Jul 27 20:51:47.039247 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:47.039214    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-djcnj" event={"ID":"eae6d6c0-9844-45a8-a5ef-9c6f1ecc2810","Type":"ContainerDied","Data":"0b07ef3655ed8a3bf88668a02d2fbcad7d08e708c8b3adad827805bf20c671f9"}
Jul 27 20:51:47.040750 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:47.040698    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-td78h" event={"ID":"dbae1ee6-3616-432d-8ef9-255ad4cd444e","Type":"ContainerStarted","Data":"6ca8cae5a80c8e93c19602e30826f5506b1c5dddd99357ccc08c0b2dcab4f720"}
Jul 27 20:51:47.048058 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:47.048020    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-gn9bl" podStartSLOduration=4.065893943 podStartE2EDuration="23.048007737s" totalImagesPullingTime="18.982113794s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 20:51:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:51:47.047695268 +0000 UTC m=+23.799557062" watchObservedRunningTime="2026-07-27 20:51:47.048007737 +0000 UTC m=+23.799869575"
Jul 27 20:51:47.048193 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:47.048134    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-64.ec2.internal" podStartSLOduration=22.048129437 podStartE2EDuration="22.048129437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 20:51:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:51:29.009855505 +0000 UTC m=+5.761717304" watchObservedRunningTime="2026-07-27 20:51:47.048129437 +0000 UTC m=+23.799991233"
Jul 27 20:51:47.068737 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:47.068693    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-td78h" podStartSLOduration=3.989377204 podStartE2EDuration="23.068678118s" totalImagesPullingTime="19.079300914s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 20:51:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:51:47.068503904 +0000 UTC m=+23.820365699" watchObservedRunningTime="2026-07-27 20:51:47.068678118 +0000 UTC m=+23.820539914"
Jul 27 20:51:47.108255 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:47.108209    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-g5tm9" podStartSLOduration=4.111364625 podStartE2EDuration="23.108190559s" totalImagesPullingTime="18.996825934s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 20:51:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:51:47.107471609 +0000 UTC m=+23.859333406" watchObservedRunningTime="2026-07-27 20:51:47.108190559 +0000 UTC m=+23.860052356"
Jul 27 20:51:47.123105 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:47.123050    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-sg57x" podStartSLOduration=4.153893308 podStartE2EDuration="23.12303436s" totalImagesPullingTime="18.969141052s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:51:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:51:47.122995931 +0000 UTC m=+23.874857725" watchObservedRunningTime="2026-07-27 20:51:47.12303436 +0000 UTC m=+23.874896154"
Jul 27 20:51:47.272439 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:47.272414    2560 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 27 20:51:47.724603 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:47.724488    2560 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-27T20:51:47.272433751Z","UUID":"4009addd-86b0-4c94-b5ee-9ec41b7569fd","Handler":null,"Name":"","Endpoint":""}
Jul 27 20:51:47.726459 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:47.726426    2560 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 27 20:51:47.726602 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:47.726470    2560 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 27 20:51:47.842280 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:47.842242    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dbzwm"
Jul 27 20:51:47.842459 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:47.842377    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dbzwm" podUID="1f1259cb-be97-4841-bd22-5bba585870b6"
Jul 27 20:51:47.842817 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:47.842794    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dtrjg"
Jul 27 20:51:47.842927 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:47.842902    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dtrjg" podUID="4b924e42-83bc-459f-b1a2-f115f9bce63f"
Jul 27 20:51:47.842998 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:47.842988    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r6g8d"
Jul 27 20:51:47.843082 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:47.843066    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r6g8d" podUID="fac6ee6e-225a-4402-8834-69ff744b07f9"
Jul 27 20:51:48.045910 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:48.045870    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tjtf6" event={"ID":"a49f6f21-5110-4f7c-8f79-1d9e14d69bc6","Type":"ContainerStarted","Data":"c7537be06be6cc4f53a58a1fd8f69c1d7cd4c8c1b1dc088b0f6566491e2d27e6"}
Jul 27 20:51:48.048313 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:48.047711    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-8cdfg" event={"ID":"3566e5fb-c112-4f14-9e2f-fce1d833e15a","Type":"ContainerStarted","Data":"55f39c357927c7a72c5fb71e29cf44f5f400adad50f9ef0e3f0d29a38b3ba98d"}
Jul 27 20:51:48.067340 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:48.066857    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-jm574" podStartSLOduration=5.071125893 podStartE2EDuration="24.066835871s" totalImagesPullingTime="18.995709978s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 20:51:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:51:47.146208514 +0000 UTC m=+23.898070309" watchObservedRunningTime="2026-07-27 20:51:48.066835871 +0000 UTC m=+24.818697667"
Jul 27 20:51:48.067340 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:48.067032    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-8cdfg" podStartSLOduration=5.072323491 podStartE2EDuration="24.067026989s" totalImagesPullingTime="18.994703498s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 20:51:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:51:48.06633349 +0000 UTC m=+24.818195287" watchObservedRunningTime="2026-07-27 20:51:48.067026989 +0000 UTC m=+24.818888782"
Jul 27 20:51:49.053980 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:49.053953    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-95hg7_433c431a-b5bf-4734-a03f-6f0a16895ded/ovn-acl-logging/0.log"
Jul 27 20:51:49.054457 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:49.054359    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-95hg7" event={"ID":"433c431a-b5bf-4734-a03f-6f0a16895ded","Type":"ContainerStarted","Data":"0743b72d61d06a7f24ba3c655efa43329335f675e327178441128ef4aa8d7818"}
Jul 27 20:51:49.056882 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:49.056853    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tjtf6" event={"ID":"a49f6f21-5110-4f7c-8f79-1d9e14d69bc6","Type":"ContainerStarted","Data":"c2b59a4a1dd0fe67fc43f385899587ed1cc99fd509d56fb9d0b6ed212323d1b3"}
Jul 27 20:51:49.075501 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:49.075459    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tjtf6" podStartSLOduration=4.16295637 podStartE2EDuration="25.07544348s" totalImagesPullingTime="20.91248711s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 20:51:24 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:51:49.075380605 +0000 UTC m=+25.827242414" watchObservedRunningTime="2026-07-27 20:51:49.07544348 +0000 UTC m=+25.827305275"
Jul 27 20:51:49.841772 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:49.841590    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r6g8d"
Jul 27 20:51:49.841964 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:49.841587    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dbzwm"
Jul 27 20:51:49.841964 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:49.841862    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r6g8d" podUID="fac6ee6e-225a-4402-8834-69ff744b07f9"
Jul 27 20:51:49.841964 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:49.841914    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dbzwm" podUID="1f1259cb-be97-4841-bd22-5bba585870b6"
Jul 27 20:51:49.841964 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:49.841590    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dtrjg"
Jul 27 20:51:49.842195 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:49.842044    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dtrjg" podUID="4b924e42-83bc-459f-b1a2-f115f9bce63f"
Jul 27 20:51:50.657816 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:50.657780    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-sg57x"
Jul 27 20:51:50.658460 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:50.658439    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-sg57x"
Jul 27 20:51:51.060876 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:51.060853    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-sg57x"
Jul 27 20:51:51.061417 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:51.061399    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-sg57x"
Jul 27 20:51:51.842027 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:51.841824    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dbzwm"
Jul 27 20:51:51.842694 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:51.841824    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dtrjg"
Jul 27 20:51:51.842694 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:51.842134    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dbzwm" podUID="1f1259cb-be97-4841-bd22-5bba585870b6"
Jul 27 20:51:51.842694 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:51.841890    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r6g8d"
Jul 27 20:51:51.842694 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:51.842212    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dtrjg" podUID="4b924e42-83bc-459f-b1a2-f115f9bce63f"
Jul 27 20:51:51.842694 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:51.842258    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r6g8d" podUID="fac6ee6e-225a-4402-8834-69ff744b07f9"
Jul 27 20:51:52.065068 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:52.065038    2560 generic.go:356] "Generic (PLEG): container finished" podID="eae6d6c0-9844-45a8-a5ef-9c6f1ecc2810" containerID="326c78806189e311f0207188b43f282af90b5f0a5e139d433276dd965697fd96" exitCode=0
Jul 27 20:51:52.065253 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:52.065133    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-djcnj" event={"ID":"eae6d6c0-9844-45a8-a5ef-9c6f1ecc2810","Type":"ContainerDied","Data":"326c78806189e311f0207188b43f282af90b5f0a5e139d433276dd965697fd96"}
Jul 27 20:51:52.068737 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:52.068719    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-95hg7_433c431a-b5bf-4734-a03f-6f0a16895ded/ovn-acl-logging/0.log"
Jul 27 20:51:52.069129 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:52.069106    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-95hg7" event={"ID":"433c431a-b5bf-4734-a03f-6f0a16895ded","Type":"ContainerStarted","Data":"a445aa2282a04ea0b8c890023f245ef6fbf478bb7641db20138c4ec65d1b79ba"}
Jul 27 20:51:52.069527 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:52.069510    2560 scope.go:117] "RemoveContainer" containerID="bde5ed442cafaa6693764d4068392fc6af0183c0737e52aba7f122222f621dc5"
Jul 27 20:51:53.075267 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:53.075196    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-95hg7_433c431a-b5bf-4734-a03f-6f0a16895ded/ovn-acl-logging/0.log"
Jul 27 20:51:53.075670 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:53.075622    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-95hg7" event={"ID":"433c431a-b5bf-4734-a03f-6f0a16895ded","Type":"ContainerStarted","Data":"cbc26b7927a83c23252b3ce616df240ff38284669925b86e6c59635bab308903"}
Jul 27 20:51:53.075878 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:53.075861    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-95hg7"
Jul 27 20:51:53.075947 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:53.075917    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-95hg7"
Jul 27 20:51:53.075991 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:53.075960    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-95hg7"
Jul 27 20:51:53.078175 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:53.078147    2560 generic.go:356] "Generic (PLEG): container finished" podID="eae6d6c0-9844-45a8-a5ef-9c6f1ecc2810" containerID="6658830c25b1da667b44d13c6a7c215a3690c39ddbb8d7ae8cedaa708e4b4e87" exitCode=0
Jul 27 20:51:53.078286 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:53.078179    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-djcnj" event={"ID":"eae6d6c0-9844-45a8-a5ef-9c6f1ecc2810","Type":"ContainerDied","Data":"6658830c25b1da667b44d13c6a7c215a3690c39ddbb8d7ae8cedaa708e4b4e87"}
Jul 27 20:51:53.092038 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:53.092020    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-95hg7"
Jul 27 20:51:53.092213 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:53.092199    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-95hg7"
Jul 27 20:51:53.108806 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:53.108756    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-95hg7" podStartSLOduration=9.967879496 podStartE2EDuration="29.108744571s" totalImagesPullingTime="19.140865075s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 20:51:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:51:53.107400573 +0000 UTC m=+29.859262388" watchObservedRunningTime="2026-07-27 20:51:53.108744571 +0000 UTC m=+29.860606366"
Jul 27 20:51:53.111218 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:53.111194    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-dtrjg"]
Jul 27 20:51:53.111345 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:53.111330    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dtrjg"
Jul 27 20:51:53.111475 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:53.111451    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dtrjg" podUID="4b924e42-83bc-459f-b1a2-f115f9bce63f"
Jul 27 20:51:53.111805 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:53.111785    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-dbzwm"]
Jul 27 20:51:53.111893 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:53.111880    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dbzwm"
Jul 27 20:51:53.111993 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:53.111971    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dbzwm" podUID="1f1259cb-be97-4841-bd22-5bba585870b6"
Jul 27 20:51:53.113368 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:53.113346    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-r6g8d"]
Jul 27 20:51:53.113455 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:53.113447    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r6g8d"
Jul 27 20:51:53.113560 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:53.113543    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r6g8d" podUID="fac6ee6e-225a-4402-8834-69ff744b07f9"
Jul 27 20:51:54.083221 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:54.082999    2560 generic.go:356] "Generic (PLEG): container finished" podID="eae6d6c0-9844-45a8-a5ef-9c6f1ecc2810" containerID="02bfc2163187f473557d696eb753a24080f17d64f51e7791294c5ecebeafcabe" exitCode=0
Jul 27 20:51:54.083587 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:54.083104    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-djcnj" event={"ID":"eae6d6c0-9844-45a8-a5ef-9c6f1ecc2810","Type":"ContainerDied","Data":"02bfc2163187f473557d696eb753a24080f17d64f51e7791294c5ecebeafcabe"}
Jul 27 20:51:54.841685 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:54.841655    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r6g8d"
Jul 27 20:51:54.841856 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:54.841655    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dbzwm"
Jul 27 20:51:54.841856 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:54.841808    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r6g8d" podUID="fac6ee6e-225a-4402-8834-69ff744b07f9"
Jul 27 20:51:54.841856 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:54.841656    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dtrjg"
Jul 27 20:51:54.842002 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:54.841881    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dbzwm" podUID="1f1259cb-be97-4841-bd22-5bba585870b6"
Jul 27 20:51:54.842002 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:54.841933    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dtrjg" podUID="4b924e42-83bc-459f-b1a2-f115f9bce63f"
Jul 27 20:51:56.841646 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:56.841610    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r6g8d"
Jul 27 20:51:56.842397 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:56.841739    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dbzwm"
Jul 27 20:51:56.842397 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:56.841743    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r6g8d" podUID="fac6ee6e-225a-4402-8834-69ff744b07f9"
Jul 27 20:51:56.842397 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:56.841776    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dtrjg"
Jul 27 20:51:56.842397 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:56.841832    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dbzwm" podUID="1f1259cb-be97-4841-bd22-5bba585870b6"
Jul 27 20:51:56.842397 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:56.841903    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dtrjg" podUID="4b924e42-83bc-459f-b1a2-f115f9bce63f"
Jul 27 20:51:57.420752 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:57.420707    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8ns2\" (UniqueName: \"kubernetes.io/projected/1f1259cb-be97-4841-bd22-5bba585870b6-kube-api-access-n8ns2\") pod \"network-check-target-dbzwm\" (UID: \"1f1259cb-be97-4841-bd22-5bba585870b6\") " pod="openshift-network-diagnostics/network-check-target-dbzwm"
Jul 27 20:51:57.420978 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:57.420776    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4b924e42-83bc-459f-b1a2-f115f9bce63f-original-pull-secret\") pod \"global-pull-secret-syncer-dtrjg\" (UID: \"4b924e42-83bc-459f-b1a2-f115f9bce63f\") " pod="kube-system/global-pull-secret-syncer-dtrjg"
Jul 27 20:51:57.420978 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:57.420898    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 20:51:57.420978 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:57.420929    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 20:51:57.420978 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:57.420903    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 20:51:57.420978 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:57.420943    2560 projected.go:196] Error preparing data for projected volume kube-api-access-n8ns2 for pod openshift-network-diagnostics/network-check-target-dbzwm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 20:51:57.421221 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:57.421001    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b924e42-83bc-459f-b1a2-f115f9bce63f-original-pull-secret podName:4b924e42-83bc-459f-b1a2-f115f9bce63f nodeName:}" failed. No retries permitted until 2026-07-27 20:52:29.420980892 +0000 UTC m=+66.172842669 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4b924e42-83bc-459f-b1a2-f115f9bce63f-original-pull-secret") pod "global-pull-secret-syncer-dtrjg" (UID: "4b924e42-83bc-459f-b1a2-f115f9bce63f") : object "kube-system"/"original-pull-secret" not registered
Jul 27 20:51:57.421221 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:57.421031    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1f1259cb-be97-4841-bd22-5bba585870b6-kube-api-access-n8ns2 podName:1f1259cb-be97-4841-bd22-5bba585870b6 nodeName:}" failed. No retries permitted until 2026-07-27 20:52:29.421014576 +0000 UTC m=+66.172876363 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-n8ns2" (UniqueName: "kubernetes.io/projected/1f1259cb-be97-4841-bd22-5bba585870b6-kube-api-access-n8ns2") pod "network-check-target-dbzwm" (UID: "1f1259cb-be97-4841-bd22-5bba585870b6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 20:51:57.521291 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:57.521256    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fac6ee6e-225a-4402-8834-69ff744b07f9-metrics-certs\") pod \"network-metrics-daemon-r6g8d\" (UID: \"fac6ee6e-225a-4402-8834-69ff744b07f9\") " pod="openshift-multus/network-metrics-daemon-r6g8d"
Jul 27 20:51:57.521473 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:57.521364    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 20:51:57.521473 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:57.521420    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fac6ee6e-225a-4402-8834-69ff744b07f9-metrics-certs podName:fac6ee6e-225a-4402-8834-69ff744b07f9 nodeName:}" failed. No retries permitted until 2026-07-27 20:52:29.521407179 +0000 UTC m=+66.273268953 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fac6ee6e-225a-4402-8834-69ff744b07f9-metrics-certs") pod "network-metrics-daemon-r6g8d" (UID: "fac6ee6e-225a-4402-8834-69ff744b07f9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 20:51:58.841986 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:58.841913    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r6g8d"
Jul 27 20:51:58.842463 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:58.842037    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dtrjg"
Jul 27 20:51:58.842463 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:58.842049    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r6g8d" podUID="fac6ee6e-225a-4402-8834-69ff744b07f9"
Jul 27 20:51:58.842463 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:58.842067    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dbzwm"
Jul 27 20:51:58.842463 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:58.842157    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dtrjg" podUID="4b924e42-83bc-459f-b1a2-f115f9bce63f"
Jul 27 20:51:58.842463 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:58.842232    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dbzwm" podUID="1f1259cb-be97-4841-bd22-5bba585870b6"
Jul 27 20:51:59.071406 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:59.071382    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-64.ec2.internal" event="NodeReady"
Jul 27 20:51:59.071581 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:59.071493    2560 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 27 20:51:59.100128 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:59.100040    2560 generic.go:356] "Generic (PLEG): container finished" podID="eae6d6c0-9844-45a8-a5ef-9c6f1ecc2810" containerID="d55620e9354a4d726d0200698de531f09b636f5a4c91261487233e69ceeb3e23" exitCode=0
Jul 27 20:51:59.100247 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:59.100121    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-djcnj" event={"ID":"eae6d6c0-9844-45a8-a5ef-9c6f1ecc2810","Type":"ContainerDied","Data":"d55620e9354a4d726d0200698de531f09b636f5a4c91261487233e69ceeb3e23"}
Jul 27 20:51:59.126674 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:59.126647    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-blb74"]
Jul 27 20:51:59.168018 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:59.167992    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-ds6vw"]
Jul 27 20:51:59.168171 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:59.168136    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-blb74"
Jul 27 20:51:59.170742 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:59.170711    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 27 20:51:59.170844 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:59.170742    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 27 20:51:59.170974 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:59.170959    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-lslmx\""
Jul 27 20:51:59.171130 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:59.171107    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 27 20:51:59.193521 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:59.193500    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-ds6vw"]
Jul 27 20:51:59.193613 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:59.193533    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-blb74"]
Jul 27 20:51:59.193613 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:59.193599    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ds6vw"
Jul 27 20:51:59.196263 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:59.196247    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 27 20:51:59.196263 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:59.196258    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-x9tww\""
Jul 27 20:51:59.196426 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:59.196392    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 27 20:51:59.337145 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:59.337116    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a9187e8b-eb38-4984-b284-39419b5de51b-metrics-tls\") pod \"dns-default-ds6vw\" (UID: \"a9187e8b-eb38-4984-b284-39419b5de51b\") " pod="openshift-dns/dns-default-ds6vw"
Jul 27 20:51:59.337248 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:59.337166    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9187e8b-eb38-4984-b284-39419b5de51b-tmp-dir\") pod \"dns-default-ds6vw\" (UID: \"a9187e8b-eb38-4984-b284-39419b5de51b\") " pod="openshift-dns/dns-default-ds6vw"
Jul 27 20:51:59.337248 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:59.337201    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xk7rs\" (UniqueName: \"kubernetes.io/projected/a9187e8b-eb38-4984-b284-39419b5de51b-kube-api-access-xk7rs\") pod \"dns-default-ds6vw\" (UID: \"a9187e8b-eb38-4984-b284-39419b5de51b\") " pod="openshift-dns/dns-default-ds6vw"
Jul 27 20:51:59.337248 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:59.337229    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a9187e8b-eb38-4984-b284-39419b5de51b-config-volume\") pod \"dns-default-ds6vw\" (UID: \"a9187e8b-eb38-4984-b284-39419b5de51b\") " pod="openshift-dns/dns-default-ds6vw"
Jul 27 20:51:59.337346 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:59.337282    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqq2n\" (UniqueName: \"kubernetes.io/projected/ef72a491-3d46-4039-9c8b-a6a9eb853223-kube-api-access-xqq2n\") pod \"ingress-canary-blb74\" (UID: \"ef72a491-3d46-4039-9c8b-a6a9eb853223\") " pod="openshift-ingress-canary/ingress-canary-blb74"
Jul 27 20:51:59.337346 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:59.337316    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ef72a491-3d46-4039-9c8b-a6a9eb853223-cert\") pod \"ingress-canary-blb74\" (UID: \"ef72a491-3d46-4039-9c8b-a6a9eb853223\") " pod="openshift-ingress-canary/ingress-canary-blb74"
Jul 27 20:51:59.438473 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:59.438390    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xk7rs\" (UniqueName: \"kubernetes.io/projected/a9187e8b-eb38-4984-b284-39419b5de51b-kube-api-access-xk7rs\") pod \"dns-default-ds6vw\" (UID: \"a9187e8b-eb38-4984-b284-39419b5de51b\") " pod="openshift-dns/dns-default-ds6vw"
Jul 27 20:51:59.438473 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:59.438436    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a9187e8b-eb38-4984-b284-39419b5de51b-config-volume\") pod \"dns-default-ds6vw\" (UID: \"a9187e8b-eb38-4984-b284-39419b5de51b\") " pod="openshift-dns/dns-default-ds6vw"
Jul 27 20:51:59.438473 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:59.438474    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqq2n\" (UniqueName: \"kubernetes.io/projected/ef72a491-3d46-4039-9c8b-a6a9eb853223-kube-api-access-xqq2n\") pod \"ingress-canary-blb74\" (UID: \"ef72a491-3d46-4039-9c8b-a6a9eb853223\") " pod="openshift-ingress-canary/ingress-canary-blb74"
Jul 27 20:51:59.438723 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:59.438507    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ef72a491-3d46-4039-9c8b-a6a9eb853223-cert\") pod \"ingress-canary-blb74\" (UID: \"ef72a491-3d46-4039-9c8b-a6a9eb853223\") " pod="openshift-ingress-canary/ingress-canary-blb74"
Jul 27 20:51:59.438723 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:59.438529    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a9187e8b-eb38-4984-b284-39419b5de51b-metrics-tls\") pod \"dns-default-ds6vw\" (UID: \"a9187e8b-eb38-4984-b284-39419b5de51b\") " pod="openshift-dns/dns-default-ds6vw"
Jul 27 20:51:59.438723 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:59.438563    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9187e8b-eb38-4984-b284-39419b5de51b-tmp-dir\") pod \"dns-default-ds6vw\" (UID: \"a9187e8b-eb38-4984-b284-39419b5de51b\") " pod="openshift-dns/dns-default-ds6vw"
Jul 27 20:51:59.438723 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:59.438694    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 20:51:59.438912 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:59.438778    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9187e8b-eb38-4984-b284-39419b5de51b-metrics-tls podName:a9187e8b-eb38-4984-b284-39419b5de51b nodeName:}" failed. No retries permitted until 2026-07-27 20:51:59.938755895 +0000 UTC m=+36.690617684 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a9187e8b-eb38-4984-b284-39419b5de51b-metrics-tls") pod "dns-default-ds6vw" (UID: "a9187e8b-eb38-4984-b284-39419b5de51b") : secret "dns-default-metrics-tls" not found
Jul 27 20:51:59.438912 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:59.438869    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9187e8b-eb38-4984-b284-39419b5de51b-tmp-dir\") pod \"dns-default-ds6vw\" (UID: \"a9187e8b-eb38-4984-b284-39419b5de51b\") " pod="openshift-dns/dns-default-ds6vw"
Jul 27 20:51:59.439051 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:59.439032    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a9187e8b-eb38-4984-b284-39419b5de51b-config-volume\") pod \"dns-default-ds6vw\" (UID: \"a9187e8b-eb38-4984-b284-39419b5de51b\") " pod="openshift-dns/dns-default-ds6vw"
Jul 27 20:51:59.442882 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:59.442762    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ef72a491-3d46-4039-9c8b-a6a9eb853223-cert\") pod \"ingress-canary-blb74\" (UID: \"ef72a491-3d46-4039-9c8b-a6a9eb853223\") " pod="openshift-ingress-canary/ingress-canary-blb74"
Jul 27 20:51:59.449810 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:59.449789    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqq2n\" (UniqueName: \"kubernetes.io/projected/ef72a491-3d46-4039-9c8b-a6a9eb853223-kube-api-access-xqq2n\") pod \"ingress-canary-blb74\" (UID: \"ef72a491-3d46-4039-9c8b-a6a9eb853223\") " pod="openshift-ingress-canary/ingress-canary-blb74"
Jul 27 20:51:59.450054 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:59.450037    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xk7rs\" (UniqueName: \"kubernetes.io/projected/a9187e8b-eb38-4984-b284-39419b5de51b-kube-api-access-xk7rs\") pod \"dns-default-ds6vw\" (UID: \"a9187e8b-eb38-4984-b284-39419b5de51b\") " pod="openshift-dns/dns-default-ds6vw"
Jul 27 20:51:59.489074 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:59.489050    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-blb74"
Jul 27 20:51:59.693177 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:59.693076    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-blb74"]
Jul 27 20:51:59.696835 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:51:59.696812    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef72a491_3d46_4039_9c8b_a6a9eb853223.slice/crio-b73d681c4c8a0032cae4d49051e620b079f9d852893279c172ffab4ad812e9f5 WatchSource:0}: Error finding container b73d681c4c8a0032cae4d49051e620b079f9d852893279c172ffab4ad812e9f5: Status 404 returned error can't find the container with id b73d681c4c8a0032cae4d49051e620b079f9d852893279c172ffab4ad812e9f5
Jul 27 20:51:59.942911 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:51:59.942873    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a9187e8b-eb38-4984-b284-39419b5de51b-metrics-tls\") pod \"dns-default-ds6vw\" (UID: \"a9187e8b-eb38-4984-b284-39419b5de51b\") " pod="openshift-dns/dns-default-ds6vw"
Jul 27 20:51:59.943341 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:59.942984    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 20:51:59.943341 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:51:59.943034    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9187e8b-eb38-4984-b284-39419b5de51b-metrics-tls podName:a9187e8b-eb38-4984-b284-39419b5de51b nodeName:}" failed. No retries permitted until 2026-07-27 20:52:00.943021459 +0000 UTC m=+37.694883231 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a9187e8b-eb38-4984-b284-39419b5de51b-metrics-tls") pod "dns-default-ds6vw" (UID: "a9187e8b-eb38-4984-b284-39419b5de51b") : secret "dns-default-metrics-tls" not found
Jul 27 20:52:00.105673 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:00.105625    2560 generic.go:356] "Generic (PLEG): container finished" podID="eae6d6c0-9844-45a8-a5ef-9c6f1ecc2810" containerID="9fec736c12278732b8fc9538a8020cc45166ddd95b21c9099452384067efa339" exitCode=0
Jul 27 20:52:00.105848 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:00.105715    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-djcnj" event={"ID":"eae6d6c0-9844-45a8-a5ef-9c6f1ecc2810","Type":"ContainerDied","Data":"9fec736c12278732b8fc9538a8020cc45166ddd95b21c9099452384067efa339"}
Jul 27 20:52:00.106815 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:00.106789    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-blb74" event={"ID":"ef72a491-3d46-4039-9c8b-a6a9eb853223","Type":"ContainerStarted","Data":"b73d681c4c8a0032cae4d49051e620b079f9d852893279c172ffab4ad812e9f5"}
Jul 27 20:52:00.842382 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:00.842353    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dtrjg"
Jul 27 20:52:00.842567 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:00.842353    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r6g8d"
Jul 27 20:52:00.842567 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:00.842353    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dbzwm"
Jul 27 20:52:00.845486 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:00.845460    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 27 20:52:00.846637 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:00.846618    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 27 20:52:00.846637 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:00.846631    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-bvgk8\""
Jul 27 20:52:00.846792 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:00.846631    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 27 20:52:00.846864 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:00.846850    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-qtl47\""
Jul 27 20:52:00.846949 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:00.846933    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 27 20:52:00.951517 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:00.951485    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a9187e8b-eb38-4984-b284-39419b5de51b-metrics-tls\") pod \"dns-default-ds6vw\" (UID: \"a9187e8b-eb38-4984-b284-39419b5de51b\") " pod="openshift-dns/dns-default-ds6vw"
Jul 27 20:52:00.951897 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:52:00.951629    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 20:52:00.951897 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:52:00.951710    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9187e8b-eb38-4984-b284-39419b5de51b-metrics-tls podName:a9187e8b-eb38-4984-b284-39419b5de51b nodeName:}" failed. No retries permitted until 2026-07-27 20:52:02.951690514 +0000 UTC m=+39.703552295 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a9187e8b-eb38-4984-b284-39419b5de51b-metrics-tls") pod "dns-default-ds6vw" (UID: "a9187e8b-eb38-4984-b284-39419b5de51b") : secret "dns-default-metrics-tls" not found
Jul 27 20:52:01.112721 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:01.112493    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-djcnj" event={"ID":"eae6d6c0-9844-45a8-a5ef-9c6f1ecc2810","Type":"ContainerStarted","Data":"ae8a005bc959a892626d00a6ced6edc5b30749edb20d5d3d087afe99742ee806"}
Jul 27 20:52:02.116999 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:02.116953    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-blb74" event={"ID":"ef72a491-3d46-4039-9c8b-a6a9eb853223","Type":"ContainerStarted","Data":"933cc407a9a3ed66bb7af08e59e4ac9ba153d34b1296bfde491020c2bd63846c"}
Jul 27 20:52:02.134155 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:02.133976    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-blb74" podStartSLOduration=0.865325304 podStartE2EDuration="3.133957692s" totalImagesPullingTime="2.268632388s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:51:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:52:02.133657296 +0000 UTC m=+38.885519092" watchObservedRunningTime="2026-07-27 20:52:02.133957692 +0000 UTC m=+38.885819488"
Jul 27 20:52:02.134297 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:02.134226    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-djcnj" podStartSLOduration=9.347337789000001 podStartE2EDuration="38.134218566s" totalImagesPullingTime="28.786880777s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 20:51:24 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:52:01.137626437 +0000 UTC m=+37.889488246" watchObservedRunningTime="2026-07-27 20:52:02.134218566 +0000 UTC m=+38.886080363"
Jul 27 20:52:02.968263 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:02.968180    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a9187e8b-eb38-4984-b284-39419b5de51b-metrics-tls\") pod \"dns-default-ds6vw\" (UID: \"a9187e8b-eb38-4984-b284-39419b5de51b\") " pod="openshift-dns/dns-default-ds6vw"
Jul 27 20:52:02.968428 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:52:02.968357    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 20:52:02.968491 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:52:02.968438    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9187e8b-eb38-4984-b284-39419b5de51b-metrics-tls podName:a9187e8b-eb38-4984-b284-39419b5de51b nodeName:}" failed. No retries permitted until 2026-07-27 20:52:06.968415835 +0000 UTC m=+43.720277616 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a9187e8b-eb38-4984-b284-39419b5de51b-metrics-tls") pod "dns-default-ds6vw" (UID: "a9187e8b-eb38-4984-b284-39419b5de51b") : secret "dns-default-metrics-tls" not found
Jul 27 20:52:06.996199 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:06.996157    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a9187e8b-eb38-4984-b284-39419b5de51b-metrics-tls\") pod \"dns-default-ds6vw\" (UID: \"a9187e8b-eb38-4984-b284-39419b5de51b\") " pod="openshift-dns/dns-default-ds6vw"
Jul 27 20:52:06.996651 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:52:06.996317    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 20:52:06.996651 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:52:06.996380    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9187e8b-eb38-4984-b284-39419b5de51b-metrics-tls podName:a9187e8b-eb38-4984-b284-39419b5de51b nodeName:}" failed. No retries permitted until 2026-07-27 20:52:14.996363561 +0000 UTC m=+51.748225354 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a9187e8b-eb38-4984-b284-39419b5de51b-metrics-tls") pod "dns-default-ds6vw" (UID: "a9187e8b-eb38-4984-b284-39419b5de51b") : secret "dns-default-metrics-tls" not found
Jul 27 20:52:14.163243 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:14.163209    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-7vsbq"]
Jul 27 20:52:14.173337 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:14.173308    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-7vsbq"
Jul 27 20:52:14.173771 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:14.173744    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-7vsbq"]
Jul 27 20:52:14.175761 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:14.175723    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Jul 27 20:52:14.175884 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:14.175799    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Jul 27 20:52:14.176117 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:14.176100    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Jul 27 20:52:14.176200 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:14.176100    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Jul 27 20:52:14.176802 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:14.176782    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-26hkr\""
Jul 27 20:52:14.280731 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:14.280697    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-65494b8dd8-cvl2s"]
Jul 27 20:52:14.296153 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:14.296133    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-65494b8dd8-cvl2s"
Jul 27 20:52:14.299450 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:14.299426    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 27 20:52:14.299587 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:14.299522    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-jv4bm\""
Jul 27 20:52:14.299889 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:14.299876    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 27 20:52:14.300254 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:14.300241    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 27 20:52:14.306078 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:14.306060    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 27 20:52:14.310591 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:14.310569    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-65494b8dd8-cvl2s"]
Jul 27 20:52:14.350351 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:14.350328    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1da96bd6-098f-41d7-9be9-904d53068b43-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-7vsbq\" (UID: \"1da96bd6-098f-41d7-9be9-904d53068b43\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-7vsbq"
Jul 27 20:52:14.350451 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:14.350369    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kk7cb\" (UniqueName: \"kubernetes.io/projected/1da96bd6-098f-41d7-9be9-904d53068b43-kube-api-access-kk7cb\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-7vsbq\" (UID: \"1da96bd6-098f-41d7-9be9-904d53068b43\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-7vsbq"
Jul 27 20:52:14.350451 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:14.350410    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1da96bd6-098f-41d7-9be9-904d53068b43-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-7vsbq\" (UID: \"1da96bd6-098f-41d7-9be9-904d53068b43\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-7vsbq"
Jul 27 20:52:14.390288 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:14.390264    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-z6zxv"]
Jul 27 20:52:14.401691 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:14.401671    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-z6zxv"
Jul 27 20:52:14.406059 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:14.406039    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Jul 27 20:52:14.406607 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:14.406587    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Jul 27 20:52:14.406886 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:14.406865    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Jul 27 20:52:14.407108 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:14.406906    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Jul 27 20:52:14.407213 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:14.406957    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-ph4ct\""
Jul 27 20:52:14.408080 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:14.408009    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-z6zxv"]
Jul 27 20:52:14.412228 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:14.412211    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Jul 27 20:52:14.450868 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:14.450809    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kk7cb\" (UniqueName: \"kubernetes.io/projected/1da96bd6-098f-41d7-9be9-904d53068b43-kube-api-access-kk7cb\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-7vsbq\" (UID: \"1da96bd6-098f-41d7-9be9-904d53068b43\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-7vsbq"
Jul 27 20:52:14.450868 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:14.450860    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0bf98545-754e-4371-a262-d78b44f0361d-ca-trust-extracted\") pod \"image-registry-65494b8dd8-cvl2s\" (UID: \"0bf98545-754e-4371-a262-d78b44f0361d\") " pod="openshift-image-registry/image-registry-65494b8dd8-cvl2s"
Jul 27 20:52:14.451006 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:14.450989    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1da96bd6-098f-41d7-9be9-904d53068b43-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-7vsbq\" (UID: \"1da96bd6-098f-41d7-9be9-904d53068b43\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-7vsbq"
Jul 27 20:52:14.451059 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:14.451046    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0bf98545-754e-4371-a262-d78b44f0361d-registry-tls\") pod \"image-registry-65494b8dd8-cvl2s\" (UID: \"0bf98545-754e-4371-a262-d78b44f0361d\") " pod="openshift-image-registry/image-registry-65494b8dd8-cvl2s"
Jul 27 20:52:14.451119 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:14.451069    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68nt5\" (UniqueName: \"kubernetes.io/projected/0bf98545-754e-4371-a262-d78b44f0361d-kube-api-access-68nt5\") pod \"image-registry-65494b8dd8-cvl2s\" (UID: \"0bf98545-754e-4371-a262-d78b44f0361d\") " pod="openshift-image-registry/image-registry-65494b8dd8-cvl2s"
Jul 27 20:52:14.451167 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:14.451153    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1da96bd6-098f-41d7-9be9-904d53068b43-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-7vsbq\" (UID: \"1da96bd6-098f-41d7-9be9-904d53068b43\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-7vsbq"
Jul 27 20:52:14.451210 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:14.451185    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0bf98545-754e-4371-a262-d78b44f0361d-trusted-ca\") pod \"image-registry-65494b8dd8-cvl2s\" (UID: \"0bf98545-754e-4371-a262-d78b44f0361d\") " pod="openshift-image-registry/image-registry-65494b8dd8-cvl2s"
Jul 27 20:52:14.451244 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:14.451218    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0bf98545-754e-4371-a262-d78b44f0361d-bound-sa-token\") pod \"image-registry-65494b8dd8-cvl2s\" (UID: \"0bf98545-754e-4371-a262-d78b44f0361d\") " pod="openshift-image-registry/image-registry-65494b8dd8-cvl2s"
Jul 27 20:52:14.451319 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:14.451302    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0bf98545-754e-4371-a262-d78b44f0361d-registry-certificates\") pod \"image-registry-65494b8dd8-cvl2s\" (UID: \"0bf98545-754e-4371-a262-d78b44f0361d\") " pod="openshift-image-registry/image-registry-65494b8dd8-cvl2s"
Jul 27 20:52:14.451356 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:14.451342    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0bf98545-754e-4371-a262-d78b44f0361d-image-registry-private-configuration\") pod \"image-registry-65494b8dd8-cvl2s\" (UID: \"0bf98545-754e-4371-a262-d78b44f0361d\") " pod="openshift-image-registry/image-registry-65494b8dd8-cvl2s"
Jul 27 20:52:14.451402 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:14.451382    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0bf98545-754e-4371-a262-d78b44f0361d-installation-pull-secrets\") pod \"image-registry-65494b8dd8-cvl2s\" (UID: \"0bf98545-754e-4371-a262-d78b44f0361d\") " pod="openshift-image-registry/image-registry-65494b8dd8-cvl2s"
Jul 27 20:52:14.451600 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:14.451585    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1da96bd6-098f-41d7-9be9-904d53068b43-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-7vsbq\" (UID: \"1da96bd6-098f-41d7-9be9-904d53068b43\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-7vsbq"
Jul 27 20:52:14.454900 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:14.454878    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1da96bd6-098f-41d7-9be9-904d53068b43-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-7vsbq\" (UID: \"1da96bd6-098f-41d7-9be9-904d53068b43\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-7vsbq"
Jul 27 20:52:14.459208 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:14.459184    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kk7cb\" (UniqueName: \"kubernetes.io/projected/1da96bd6-098f-41d7-9be9-904d53068b43-kube-api-access-kk7cb\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-7vsbq\" (UID: \"1da96bd6-098f-41d7-9be9-904d53068b43\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-7vsbq"
Jul 27 20:52:14.483986 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:14.483953    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-7vsbq"
Jul 27 20:52:14.552698 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:14.552667    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0bf98545-754e-4371-a262-d78b44f0361d-trusted-ca\") pod \"image-registry-65494b8dd8-cvl2s\" (UID: \"0bf98545-754e-4371-a262-d78b44f0361d\") " pod="openshift-image-registry/image-registry-65494b8dd8-cvl2s"
Jul 27 20:52:14.552832 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:14.552709    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0bf98545-754e-4371-a262-d78b44f0361d-bound-sa-token\") pod \"image-registry-65494b8dd8-cvl2s\" (UID: \"0bf98545-754e-4371-a262-d78b44f0361d\") " pod="openshift-image-registry/image-registry-65494b8dd8-cvl2s"
Jul 27 20:52:14.552832 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:14.552735    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8a6b7ae6-6c21-453d-99b9-303066d19915-trusted-ca\") pod \"console-operator-584dcb6b54-z6zxv\" (UID: \"8a6b7ae6-6c21-453d-99b9-303066d19915\") " pod="openshift-console-operator/console-operator-584dcb6b54-z6zxv"
Jul 27 20:52:14.552832 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:14.552776    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0bf98545-754e-4371-a262-d78b44f0361d-registry-certificates\") pod \"image-registry-65494b8dd8-cvl2s\" (UID: \"0bf98545-754e-4371-a262-d78b44f0361d\") " pod="openshift-image-registry/image-registry-65494b8dd8-cvl2s"
Jul 27 20:52:14.552832 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:14.552799    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8a6b7ae6-6c21-453d-99b9-303066d19915-serving-cert\") pod \"console-operator-584dcb6b54-z6zxv\" (UID: \"8a6b7ae6-6c21-453d-99b9-303066d19915\") " pod="openshift-console-operator/console-operator-584dcb6b54-z6zxv"
Jul 27 20:52:14.552832 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:14.552826    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0bf98545-754e-4371-a262-d78b44f0361d-image-registry-private-configuration\") pod \"image-registry-65494b8dd8-cvl2s\" (UID: \"0bf98545-754e-4371-a262-d78b44f0361d\") " pod="openshift-image-registry/image-registry-65494b8dd8-cvl2s"
Jul 27 20:52:14.553111 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:14.552853    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0bf98545-754e-4371-a262-d78b44f0361d-installation-pull-secrets\") pod \"image-registry-65494b8dd8-cvl2s\" (UID: \"0bf98545-754e-4371-a262-d78b44f0361d\") " pod="openshift-image-registry/image-registry-65494b8dd8-cvl2s"
Jul 27 20:52:14.553111 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:14.552874    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gf24w\" (UniqueName: \"kubernetes.io/projected/8a6b7ae6-6c21-453d-99b9-303066d19915-kube-api-access-gf24w\") pod \"console-operator-584dcb6b54-z6zxv\" (UID: \"8a6b7ae6-6c21-453d-99b9-303066d19915\") " pod="openshift-console-operator/console-operator-584dcb6b54-z6zxv"
Jul 27 20:52:14.553111 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:14.552922    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0bf98545-754e-4371-a262-d78b44f0361d-ca-trust-extracted\") pod \"image-registry-65494b8dd8-cvl2s\" (UID: \"0bf98545-754e-4371-a262-d78b44f0361d\") " pod="openshift-image-registry/image-registry-65494b8dd8-cvl2s"
Jul 27 20:52:14.553111 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:14.552976    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8a6b7ae6-6c21-453d-99b9-303066d19915-config\") pod \"console-operator-584dcb6b54-z6zxv\" (UID: \"8a6b7ae6-6c21-453d-99b9-303066d19915\") " pod="openshift-console-operator/console-operator-584dcb6b54-z6zxv"
Jul 27 20:52:14.553111 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:14.553000    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0bf98545-754e-4371-a262-d78b44f0361d-registry-tls\") pod \"image-registry-65494b8dd8-cvl2s\" (UID: \"0bf98545-754e-4371-a262-d78b44f0361d\") " pod="openshift-image-registry/image-registry-65494b8dd8-cvl2s"
Jul 27 20:52:14.553111 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:14.553020    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68nt5\" (UniqueName: \"kubernetes.io/projected/0bf98545-754e-4371-a262-d78b44f0361d-kube-api-access-68nt5\") pod \"image-registry-65494b8dd8-cvl2s\" (UID: \"0bf98545-754e-4371-a262-d78b44f0361d\") " pod="openshift-image-registry/image-registry-65494b8dd8-cvl2s"
Jul 27 20:52:14.553447 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:52:14.553424    2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 27 20:52:14.553505 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:52:14.553450    2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-65494b8dd8-cvl2s: secret "image-registry-tls" not found
Jul 27 20:52:14.553505 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:14.553488    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0bf98545-754e-4371-a262-d78b44f0361d-ca-trust-extracted\") pod \"image-registry-65494b8dd8-cvl2s\" (UID: \"0bf98545-754e-4371-a262-d78b44f0361d\") " pod="openshift-image-registry/image-registry-65494b8dd8-cvl2s"
Jul 27 20:52:14.553603 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:52:14.553519    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0bf98545-754e-4371-a262-d78b44f0361d-registry-tls podName:0bf98545-754e-4371-a262-d78b44f0361d nodeName:}" failed. No retries permitted until 2026-07-27 20:52:15.053498137 +0000 UTC m=+51.805359910 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0bf98545-754e-4371-a262-d78b44f0361d-registry-tls") pod "image-registry-65494b8dd8-cvl2s" (UID: "0bf98545-754e-4371-a262-d78b44f0361d") : secret "image-registry-tls" not found
Jul 27 20:52:14.553940 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:14.553917    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0bf98545-754e-4371-a262-d78b44f0361d-registry-certificates\") pod \"image-registry-65494b8dd8-cvl2s\" (UID: \"0bf98545-754e-4371-a262-d78b44f0361d\") " pod="openshift-image-registry/image-registry-65494b8dd8-cvl2s"
Jul 27 20:52:14.554957 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:14.554932    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0bf98545-754e-4371-a262-d78b44f0361d-trusted-ca\") pod \"image-registry-65494b8dd8-cvl2s\" (UID: \"0bf98545-754e-4371-a262-d78b44f0361d\") " pod="openshift-image-registry/image-registry-65494b8dd8-cvl2s"
Jul 27 20:52:14.556306 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:14.556282    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0bf98545-754e-4371-a262-d78b44f0361d-installation-pull-secrets\") pod \"image-registry-65494b8dd8-cvl2s\" (UID: \"0bf98545-754e-4371-a262-d78b44f0361d\") " pod="openshift-image-registry/image-registry-65494b8dd8-cvl2s"
Jul 27 20:52:14.556800 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:14.556774    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0bf98545-754e-4371-a262-d78b44f0361d-image-registry-private-configuration\") pod \"image-registry-65494b8dd8-cvl2s\" (UID: \"0bf98545-754e-4371-a262-d78b44f0361d\") " pod="openshift-image-registry/image-registry-65494b8dd8-cvl2s"
Jul 27 20:52:14.564219 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:14.564195    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0bf98545-754e-4371-a262-d78b44f0361d-bound-sa-token\") pod \"image-registry-65494b8dd8-cvl2s\" (UID: \"0bf98545-754e-4371-a262-d78b44f0361d\") " pod="openshift-image-registry/image-registry-65494b8dd8-cvl2s"
Jul 27 20:52:14.564504 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:14.564486    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-68nt5\" (UniqueName: \"kubernetes.io/projected/0bf98545-754e-4371-a262-d78b44f0361d-kube-api-access-68nt5\") pod \"image-registry-65494b8dd8-cvl2s\" (UID: \"0bf98545-754e-4371-a262-d78b44f0361d\") " pod="openshift-image-registry/image-registry-65494b8dd8-cvl2s"
Jul 27 20:52:14.621119 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:14.621069    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-7vsbq"]
Jul 27 20:52:14.623653 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:52:14.623626    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1da96bd6_098f_41d7_9be9_904d53068b43.slice/crio-9cea619e283dd332b1f00380c68195c45fda55cbf2134acbae13528442e50621 WatchSource:0}: Error finding container 9cea619e283dd332b1f00380c68195c45fda55cbf2134acbae13528442e50621: Status 404 returned error can't find the container with id 9cea619e283dd332b1f00380c68195c45fda55cbf2134acbae13528442e50621
Jul 27 20:52:14.654072 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:14.654051    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8a6b7ae6-6c21-453d-99b9-303066d19915-config\") pod \"console-operator-584dcb6b54-z6zxv\" (UID: \"8a6b7ae6-6c21-453d-99b9-303066d19915\") " pod="openshift-console-operator/console-operator-584dcb6b54-z6zxv"
Jul 27 20:52:14.654182 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:14.654109    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8a6b7ae6-6c21-453d-99b9-303066d19915-trusted-ca\") pod \"console-operator-584dcb6b54-z6zxv\" (UID: \"8a6b7ae6-6c21-453d-99b9-303066d19915\") " pod="openshift-console-operator/console-operator-584dcb6b54-z6zxv"
Jul 27 20:52:14.654182 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:14.654139    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8a6b7ae6-6c21-453d-99b9-303066d19915-serving-cert\") pod \"console-operator-584dcb6b54-z6zxv\" (UID: \"8a6b7ae6-6c21-453d-99b9-303066d19915\") " pod="openshift-console-operator/console-operator-584dcb6b54-z6zxv"
Jul 27 20:52:14.654317 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:14.654292    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gf24w\" (UniqueName: \"kubernetes.io/projected/8a6b7ae6-6c21-453d-99b9-303066d19915-kube-api-access-gf24w\") pod \"console-operator-584dcb6b54-z6zxv\" (UID: \"8a6b7ae6-6c21-453d-99b9-303066d19915\") " pod="openshift-console-operator/console-operator-584dcb6b54-z6zxv"
Jul 27 20:52:14.657966 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:14.655257    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8a6b7ae6-6c21-453d-99b9-303066d19915-config\") pod \"console-operator-584dcb6b54-z6zxv\" (UID: \"8a6b7ae6-6c21-453d-99b9-303066d19915\") " pod="openshift-console-operator/console-operator-584dcb6b54-z6zxv"
Jul 27 20:52:14.657966 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:14.655579    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8a6b7ae6-6c21-453d-99b9-303066d19915-trusted-ca\") pod \"console-operator-584dcb6b54-z6zxv\" (UID: \"8a6b7ae6-6c21-453d-99b9-303066d19915\") " pod="openshift-console-operator/console-operator-584dcb6b54-z6zxv"
Jul 27 20:52:14.657966 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:14.657337    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8a6b7ae6-6c21-453d-99b9-303066d19915-serving-cert\") pod \"console-operator-584dcb6b54-z6zxv\" (UID: \"8a6b7ae6-6c21-453d-99b9-303066d19915\") " pod="openshift-console-operator/console-operator-584dcb6b54-z6zxv"
Jul 27 20:52:14.663264 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:14.663245    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gf24w\" (UniqueName: \"kubernetes.io/projected/8a6b7ae6-6c21-453d-99b9-303066d19915-kube-api-access-gf24w\") pod \"console-operator-584dcb6b54-z6zxv\" (UID: \"8a6b7ae6-6c21-453d-99b9-303066d19915\") " pod="openshift-console-operator/console-operator-584dcb6b54-z6zxv"
Jul 27 20:52:14.711250 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:14.711196    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-z6zxv"
Jul 27 20:52:14.876376 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:14.876346    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-z6zxv"]
Jul 27 20:52:14.879761 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:52:14.879731    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a6b7ae6_6c21_453d_99b9_303066d19915.slice/crio-2d824ceaa82d91570b601678c6d2a84ef8bac6f75a4c9f98c64e2923ab147ffd WatchSource:0}: Error finding container 2d824ceaa82d91570b601678c6d2a84ef8bac6f75a4c9f98c64e2923ab147ffd: Status 404 returned error can't find the container with id 2d824ceaa82d91570b601678c6d2a84ef8bac6f75a4c9f98c64e2923ab147ffd
Jul 27 20:52:15.057651 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:15.057613    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0bf98545-754e-4371-a262-d78b44f0361d-registry-tls\") pod \"image-registry-65494b8dd8-cvl2s\" (UID: \"0bf98545-754e-4371-a262-d78b44f0361d\") " pod="openshift-image-registry/image-registry-65494b8dd8-cvl2s"
Jul 27 20:52:15.057811 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:15.057657    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a9187e8b-eb38-4984-b284-39419b5de51b-metrics-tls\") pod \"dns-default-ds6vw\" (UID: \"a9187e8b-eb38-4984-b284-39419b5de51b\") " pod="openshift-dns/dns-default-ds6vw"
Jul 27 20:52:15.057811 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:52:15.057785    2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 27 20:52:15.057811 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:52:15.057797    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 20:52:15.057811 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:52:15.057803    2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-65494b8dd8-cvl2s: secret "image-registry-tls" not found
Jul 27 20:52:15.057994 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:52:15.057872    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9187e8b-eb38-4984-b284-39419b5de51b-metrics-tls podName:a9187e8b-eb38-4984-b284-39419b5de51b nodeName:}" failed. No retries permitted until 2026-07-27 20:52:31.057858005 +0000 UTC m=+67.809719799 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a9187e8b-eb38-4984-b284-39419b5de51b-metrics-tls") pod "dns-default-ds6vw" (UID: "a9187e8b-eb38-4984-b284-39419b5de51b") : secret "dns-default-metrics-tls" not found
Jul 27 20:52:15.057994 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:52:15.057887    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0bf98545-754e-4371-a262-d78b44f0361d-registry-tls podName:0bf98545-754e-4371-a262-d78b44f0361d nodeName:}" failed. No retries permitted until 2026-07-27 20:52:16.057881423 +0000 UTC m=+52.809743196 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0bf98545-754e-4371-a262-d78b44f0361d-registry-tls") pod "image-registry-65494b8dd8-cvl2s" (UID: "0bf98545-754e-4371-a262-d78b44f0361d") : secret "image-registry-tls" not found
Jul 27 20:52:15.149838 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:15.149807    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-z6zxv" event={"ID":"8a6b7ae6-6c21-453d-99b9-303066d19915","Type":"ContainerStarted","Data":"2d824ceaa82d91570b601678c6d2a84ef8bac6f75a4c9f98c64e2923ab147ffd"}
Jul 27 20:52:15.150996 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:15.150966    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-7vsbq" event={"ID":"1da96bd6-098f-41d7-9be9-904d53068b43","Type":"ContainerStarted","Data":"9cea619e283dd332b1f00380c68195c45fda55cbf2134acbae13528442e50621"}
Jul 27 20:52:16.064916 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:16.064881    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0bf98545-754e-4371-a262-d78b44f0361d-registry-tls\") pod \"image-registry-65494b8dd8-cvl2s\" (UID: \"0bf98545-754e-4371-a262-d78b44f0361d\") " pod="openshift-image-registry/image-registry-65494b8dd8-cvl2s"
Jul 27 20:52:16.065502 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:52:16.065019    2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 27 20:52:16.065502 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:52:16.065032    2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-65494b8dd8-cvl2s: secret "image-registry-tls" not found
Jul 27 20:52:16.065502 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:52:16.065109    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0bf98545-754e-4371-a262-d78b44f0361d-registry-tls podName:0bf98545-754e-4371-a262-d78b44f0361d nodeName:}" failed. No retries permitted until 2026-07-27 20:52:18.065071385 +0000 UTC m=+54.816933181 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0bf98545-754e-4371-a262-d78b44f0361d-registry-tls") pod "image-registry-65494b8dd8-cvl2s" (UID: "0bf98545-754e-4371-a262-d78b44f0361d") : secret "image-registry-tls" not found
Jul 27 20:52:17.158784 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:17.158754    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-7vsbq" event={"ID":"1da96bd6-098f-41d7-9be9-904d53068b43","Type":"ContainerStarted","Data":"2de532f9dfa34a5d0602c559fac709f5c6e1af638581e07a203522b3bb151804"}
Jul 27 20:52:17.182519 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:17.182449    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-7vsbq" podStartSLOduration=0.708916112 podStartE2EDuration="3.182432072s" totalImagesPullingTime="2.47351596s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:52:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:52:17.182137929 +0000 UTC m=+53.933999734" watchObservedRunningTime="2026-07-27 20:52:17.182432072 +0000 UTC m=+53.934293866"
Jul 27 20:52:18.080558 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:18.080526    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0bf98545-754e-4371-a262-d78b44f0361d-registry-tls\") pod \"image-registry-65494b8dd8-cvl2s\" (UID: \"0bf98545-754e-4371-a262-d78b44f0361d\") " pod="openshift-image-registry/image-registry-65494b8dd8-cvl2s"
Jul 27 20:52:18.080702 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:52:18.080636    2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 27 20:52:18.080702 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:52:18.080650    2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-65494b8dd8-cvl2s: secret "image-registry-tls" not found
Jul 27 20:52:18.080702 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:52:18.080697    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0bf98545-754e-4371-a262-d78b44f0361d-registry-tls podName:0bf98545-754e-4371-a262-d78b44f0361d nodeName:}" failed. No retries permitted until 2026-07-27 20:52:22.080683954 +0000 UTC m=+58.832545726 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0bf98545-754e-4371-a262-d78b44f0361d-registry-tls") pod "image-registry-65494b8dd8-cvl2s" (UID: "0bf98545-754e-4371-a262-d78b44f0361d") : secret "image-registry-tls" not found
Jul 27 20:52:18.162911 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:18.162885    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-z6zxv_8a6b7ae6-6c21-453d-99b9-303066d19915/console-operator/0.log"
Jul 27 20:52:18.163279 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:18.162925    2560 generic.go:356] "Generic (PLEG): container finished" podID="8a6b7ae6-6c21-453d-99b9-303066d19915" containerID="284456002c366fb6c2006ddf913e7263b63d208248c619ff8104c7b65a610fea" exitCode=255
Jul 27 20:52:18.163279 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:18.163017    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-z6zxv" event={"ID":"8a6b7ae6-6c21-453d-99b9-303066d19915","Type":"ContainerDied","Data":"284456002c366fb6c2006ddf913e7263b63d208248c619ff8104c7b65a610fea"}
Jul 27 20:52:18.163279 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:18.163239    2560 scope.go:117] "RemoveContainer" containerID="284456002c366fb6c2006ddf913e7263b63d208248c619ff8104c7b65a610fea"
Jul 27 20:52:19.135575 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:19.135535    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-mb7xd"]
Jul 27 20:52:19.137386 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:19.137371    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-mb7xd"
Jul 27 20:52:19.143328 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:19.140201    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 27 20:52:19.143328 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:19.140389    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-5rxbs\""
Jul 27 20:52:19.143764 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:19.143739    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 27 20:52:19.143880 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:19.143745    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 27 20:52:19.143880 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:19.143746    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 27 20:52:19.151166 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:19.151146    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-mb7xd"]
Jul 27 20:52:19.166509 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:19.166493    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-z6zxv_8a6b7ae6-6c21-453d-99b9-303066d19915/console-operator/1.log"
Jul 27 20:52:19.166862 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:19.166848    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-z6zxv_8a6b7ae6-6c21-453d-99b9-303066d19915/console-operator/0.log"
Jul 27 20:52:19.166903 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:19.166883    2560 generic.go:356] "Generic (PLEG): container finished" podID="8a6b7ae6-6c21-453d-99b9-303066d19915" containerID="18921b555aef90835decc2504c918a148d5fdd0d841d7f42ae7fe8a8b418c1c3" exitCode=255
Jul 27 20:52:19.166950 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:19.166934    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-z6zxv" event={"ID":"8a6b7ae6-6c21-453d-99b9-303066d19915","Type":"ContainerDied","Data":"18921b555aef90835decc2504c918a148d5fdd0d841d7f42ae7fe8a8b418c1c3"}
Jul 27 20:52:19.166997 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:19.166985    2560 scope.go:117] "RemoveContainer" containerID="284456002c366fb6c2006ddf913e7263b63d208248c619ff8104c7b65a610fea"
Jul 27 20:52:19.167173 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:19.167161    2560 scope.go:117] "RemoveContainer" containerID="18921b555aef90835decc2504c918a148d5fdd0d841d7f42ae7fe8a8b418c1c3"
Jul 27 20:52:19.167330 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:52:19.167313    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-z6zxv_openshift-console-operator(8a6b7ae6-6c21-453d-99b9-303066d19915)\"" pod="openshift-console-operator/console-operator-584dcb6b54-z6zxv" podUID="8a6b7ae6-6c21-453d-99b9-303066d19915"
Jul 27 20:52:19.188333 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:19.188306    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1c5f5fd5-7bf8-4d82-9c39-ca9e923ec6b4-data-volume\") pod \"insights-runtime-extractor-mb7xd\" (UID: \"1c5f5fd5-7bf8-4d82-9c39-ca9e923ec6b4\") " pod="openshift-insights/insights-runtime-extractor-mb7xd"
Jul 27 20:52:19.188430 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:19.188374    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1c5f5fd5-7bf8-4d82-9c39-ca9e923ec6b4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mb7xd\" (UID: \"1c5f5fd5-7bf8-4d82-9c39-ca9e923ec6b4\") " pod="openshift-insights/insights-runtime-extractor-mb7xd"
Jul 27 20:52:19.188485 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:19.188450    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1c5f5fd5-7bf8-4d82-9c39-ca9e923ec6b4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mb7xd\" (UID: \"1c5f5fd5-7bf8-4d82-9c39-ca9e923ec6b4\") " pod="openshift-insights/insights-runtime-extractor-mb7xd"
Jul 27 20:52:19.188581 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:19.188563    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1c5f5fd5-7bf8-4d82-9c39-ca9e923ec6b4-crio-socket\") pod \"insights-runtime-extractor-mb7xd\" (UID: \"1c5f5fd5-7bf8-4d82-9c39-ca9e923ec6b4\") " pod="openshift-insights/insights-runtime-extractor-mb7xd"
Jul 27 20:52:19.188635 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:19.188596    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrt7j\" (UniqueName: \"kubernetes.io/projected/1c5f5fd5-7bf8-4d82-9c39-ca9e923ec6b4-kube-api-access-wrt7j\") pod \"insights-runtime-extractor-mb7xd\" (UID: \"1c5f5fd5-7bf8-4d82-9c39-ca9e923ec6b4\") " pod="openshift-insights/insights-runtime-extractor-mb7xd"
Jul 27 20:52:19.289060 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:19.289022    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1c5f5fd5-7bf8-4d82-9c39-ca9e923ec6b4-crio-socket\") pod \"insights-runtime-extractor-mb7xd\" (UID: \"1c5f5fd5-7bf8-4d82-9c39-ca9e923ec6b4\") " pod="openshift-insights/insights-runtime-extractor-mb7xd"
Jul 27 20:52:19.289242 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:19.289067    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrt7j\" (UniqueName: \"kubernetes.io/projected/1c5f5fd5-7bf8-4d82-9c39-ca9e923ec6b4-kube-api-access-wrt7j\") pod \"insights-runtime-extractor-mb7xd\" (UID: \"1c5f5fd5-7bf8-4d82-9c39-ca9e923ec6b4\") " pod="openshift-insights/insights-runtime-extractor-mb7xd"
Jul 27 20:52:19.289242 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:19.289123    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1c5f5fd5-7bf8-4d82-9c39-ca9e923ec6b4-data-volume\") pod \"insights-runtime-extractor-mb7xd\" (UID: \"1c5f5fd5-7bf8-4d82-9c39-ca9e923ec6b4\") " pod="openshift-insights/insights-runtime-extractor-mb7xd"
Jul 27 20:52:19.289242 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:19.289160    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1c5f5fd5-7bf8-4d82-9c39-ca9e923ec6b4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mb7xd\" (UID: \"1c5f5fd5-7bf8-4d82-9c39-ca9e923ec6b4\") " pod="openshift-insights/insights-runtime-extractor-mb7xd"
Jul 27 20:52:19.289242 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:19.289204    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1c5f5fd5-7bf8-4d82-9c39-ca9e923ec6b4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mb7xd\" (UID: \"1c5f5fd5-7bf8-4d82-9c39-ca9e923ec6b4\") " pod="openshift-insights/insights-runtime-extractor-mb7xd"
Jul 27 20:52:19.289440 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:19.289250    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1c5f5fd5-7bf8-4d82-9c39-ca9e923ec6b4-crio-socket\") pod \"insights-runtime-extractor-mb7xd\" (UID: \"1c5f5fd5-7bf8-4d82-9c39-ca9e923ec6b4\") " pod="openshift-insights/insights-runtime-extractor-mb7xd"
Jul 27 20:52:19.289440 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:52:19.289309    2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 27 20:52:19.289440 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:52:19.289330    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 20:52:19.289440 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:52:19.289370    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1c5f5fd5-7bf8-4d82-9c39-ca9e923ec6b4-kube-rbac-proxy-cm podName:1c5f5fd5-7bf8-4d82-9c39-ca9e923ec6b4 nodeName:}" failed. No retries permitted until 2026-07-27 20:52:19.789355064 +0000 UTC m=+56.541216837 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1c5f5fd5-7bf8-4d82-9c39-ca9e923ec6b4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mb7xd" (UID: "1c5f5fd5-7bf8-4d82-9c39-ca9e923ec6b4") : configmap "kube-rbac-proxy" not found
Jul 27 20:52:19.289440 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:52:19.289391    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c5f5fd5-7bf8-4d82-9c39-ca9e923ec6b4-insights-runtime-extractor-tls podName:1c5f5fd5-7bf8-4d82-9c39-ca9e923ec6b4 nodeName:}" failed. No retries permitted until 2026-07-27 20:52:19.789375129 +0000 UTC m=+56.541236922 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1c5f5fd5-7bf8-4d82-9c39-ca9e923ec6b4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mb7xd" (UID: "1c5f5fd5-7bf8-4d82-9c39-ca9e923ec6b4") : secret "insights-runtime-extractor-tls" not found
Jul 27 20:52:19.289619 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:19.289491    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1c5f5fd5-7bf8-4d82-9c39-ca9e923ec6b4-data-volume\") pod \"insights-runtime-extractor-mb7xd\" (UID: \"1c5f5fd5-7bf8-4d82-9c39-ca9e923ec6b4\") " pod="openshift-insights/insights-runtime-extractor-mb7xd"
Jul 27 20:52:19.300698 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:19.300677    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrt7j\" (UniqueName: \"kubernetes.io/projected/1c5f5fd5-7bf8-4d82-9c39-ca9e923ec6b4-kube-api-access-wrt7j\") pod \"insights-runtime-extractor-mb7xd\" (UID: \"1c5f5fd5-7bf8-4d82-9c39-ca9e923ec6b4\") " pod="openshift-insights/insights-runtime-extractor-mb7xd"
Jul 27 20:52:19.793548 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:19.793518    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1c5f5fd5-7bf8-4d82-9c39-ca9e923ec6b4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mb7xd\" (UID: \"1c5f5fd5-7bf8-4d82-9c39-ca9e923ec6b4\") " pod="openshift-insights/insights-runtime-extractor-mb7xd"
Jul 27 20:52:19.793726 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:19.793560    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1c5f5fd5-7bf8-4d82-9c39-ca9e923ec6b4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mb7xd\" (UID: \"1c5f5fd5-7bf8-4d82-9c39-ca9e923ec6b4\") " pod="openshift-insights/insights-runtime-extractor-mb7xd"
Jul 27 20:52:19.793726 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:52:19.793646    2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 27 20:52:19.793726 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:52:19.793665    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 20:52:19.793726 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:52:19.793721    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c5f5fd5-7bf8-4d82-9c39-ca9e923ec6b4-insights-runtime-extractor-tls podName:1c5f5fd5-7bf8-4d82-9c39-ca9e923ec6b4 nodeName:}" failed. No retries permitted until 2026-07-27 20:52:20.793707594 +0000 UTC m=+57.545569368 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1c5f5fd5-7bf8-4d82-9c39-ca9e923ec6b4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mb7xd" (UID: "1c5f5fd5-7bf8-4d82-9c39-ca9e923ec6b4") : secret "insights-runtime-extractor-tls" not found
Jul 27 20:52:19.793880 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:52:19.793733    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1c5f5fd5-7bf8-4d82-9c39-ca9e923ec6b4-kube-rbac-proxy-cm podName:1c5f5fd5-7bf8-4d82-9c39-ca9e923ec6b4 nodeName:}" failed. No retries permitted until 2026-07-27 20:52:20.793726533 +0000 UTC m=+57.545588306 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1c5f5fd5-7bf8-4d82-9c39-ca9e923ec6b4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mb7xd" (UID: "1c5f5fd5-7bf8-4d82-9c39-ca9e923ec6b4") : configmap "kube-rbac-proxy" not found
Jul 27 20:52:20.170736 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:20.170660    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-z6zxv_8a6b7ae6-6c21-453d-99b9-303066d19915/console-operator/1.log"
Jul 27 20:52:20.171080 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:20.170994    2560 scope.go:117] "RemoveContainer" containerID="18921b555aef90835decc2504c918a148d5fdd0d841d7f42ae7fe8a8b418c1c3"
Jul 27 20:52:20.171176 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:52:20.171156    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-z6zxv_openshift-console-operator(8a6b7ae6-6c21-453d-99b9-303066d19915)\"" pod="openshift-console-operator/console-operator-584dcb6b54-z6zxv" podUID="8a6b7ae6-6c21-453d-99b9-303066d19915"
Jul 27 20:52:20.801377 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:20.801337    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1c5f5fd5-7bf8-4d82-9c39-ca9e923ec6b4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mb7xd\" (UID: \"1c5f5fd5-7bf8-4d82-9c39-ca9e923ec6b4\") " pod="openshift-insights/insights-runtime-extractor-mb7xd"
Jul 27 20:52:20.801377 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:20.801390    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1c5f5fd5-7bf8-4d82-9c39-ca9e923ec6b4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mb7xd\" (UID: \"1c5f5fd5-7bf8-4d82-9c39-ca9e923ec6b4\") " pod="openshift-insights/insights-runtime-extractor-mb7xd"
Jul 27 20:52:20.801594 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:52:20.801476    2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 27 20:52:20.801594 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:52:20.801531    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 20:52:20.801594 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:52:20.801538    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1c5f5fd5-7bf8-4d82-9c39-ca9e923ec6b4-kube-rbac-proxy-cm podName:1c5f5fd5-7bf8-4d82-9c39-ca9e923ec6b4 nodeName:}" failed. No retries permitted until 2026-07-27 20:52:22.801523745 +0000 UTC m=+59.553385521 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1c5f5fd5-7bf8-4d82-9c39-ca9e923ec6b4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mb7xd" (UID: "1c5f5fd5-7bf8-4d82-9c39-ca9e923ec6b4") : configmap "kube-rbac-proxy" not found
Jul 27 20:52:20.801689 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:52:20.801621    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c5f5fd5-7bf8-4d82-9c39-ca9e923ec6b4-insights-runtime-extractor-tls podName:1c5f5fd5-7bf8-4d82-9c39-ca9e923ec6b4 nodeName:}" failed. No retries permitted until 2026-07-27 20:52:22.801605527 +0000 UTC m=+59.553467300 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1c5f5fd5-7bf8-4d82-9c39-ca9e923ec6b4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mb7xd" (UID: "1c5f5fd5-7bf8-4d82-9c39-ca9e923ec6b4") : secret "insights-runtime-extractor-tls" not found
Jul 27 20:52:22.110983 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:22.110947    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0bf98545-754e-4371-a262-d78b44f0361d-registry-tls\") pod \"image-registry-65494b8dd8-cvl2s\" (UID: \"0bf98545-754e-4371-a262-d78b44f0361d\") " pod="openshift-image-registry/image-registry-65494b8dd8-cvl2s"
Jul 27 20:52:22.111486 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:52:22.111153    2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 27 20:52:22.111486 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:52:22.111175    2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-65494b8dd8-cvl2s: secret "image-registry-tls" not found
Jul 27 20:52:22.111486 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:52:22.111233    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0bf98545-754e-4371-a262-d78b44f0361d-registry-tls podName:0bf98545-754e-4371-a262-d78b44f0361d nodeName:}" failed. No retries permitted until 2026-07-27 20:52:30.111215192 +0000 UTC m=+66.863077027 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0bf98545-754e-4371-a262-d78b44f0361d-registry-tls") pod "image-registry-65494b8dd8-cvl2s" (UID: "0bf98545-754e-4371-a262-d78b44f0361d") : secret "image-registry-tls" not found
Jul 27 20:52:22.111486 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:22.111356    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-g5tm9_95f7e24a-fe2f-451e-ae08-fa9342e04cef/dns-node-resolver/0.log"
Jul 27 20:52:22.710472 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:22.710445    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-jm574_412ea27a-182d-40cc-917e-4686a4ff65cb/node-ca/0.log"
Jul 27 20:52:22.817295 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:22.817250    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1c5f5fd5-7bf8-4d82-9c39-ca9e923ec6b4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mb7xd\" (UID: \"1c5f5fd5-7bf8-4d82-9c39-ca9e923ec6b4\") " pod="openshift-insights/insights-runtime-extractor-mb7xd"
Jul 27 20:52:22.817454 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:22.817312    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1c5f5fd5-7bf8-4d82-9c39-ca9e923ec6b4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mb7xd\" (UID: \"1c5f5fd5-7bf8-4d82-9c39-ca9e923ec6b4\") " pod="openshift-insights/insights-runtime-extractor-mb7xd"
Jul 27 20:52:22.817454 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:52:22.817384    2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 27 20:52:22.817454 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:52:22.817407    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 20:52:22.817566 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:52:22.817472    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1c5f5fd5-7bf8-4d82-9c39-ca9e923ec6b4-kube-rbac-proxy-cm podName:1c5f5fd5-7bf8-4d82-9c39-ca9e923ec6b4 nodeName:}" failed. No retries permitted until 2026-07-27 20:52:26.817443522 +0000 UTC m=+63.569305295 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1c5f5fd5-7bf8-4d82-9c39-ca9e923ec6b4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mb7xd" (UID: "1c5f5fd5-7bf8-4d82-9c39-ca9e923ec6b4") : configmap "kube-rbac-proxy" not found
Jul 27 20:52:22.817566 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:52:22.817492    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c5f5fd5-7bf8-4d82-9c39-ca9e923ec6b4-insights-runtime-extractor-tls podName:1c5f5fd5-7bf8-4d82-9c39-ca9e923ec6b4 nodeName:}" failed. No retries permitted until 2026-07-27 20:52:26.817484063 +0000 UTC m=+63.569345875 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1c5f5fd5-7bf8-4d82-9c39-ca9e923ec6b4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mb7xd" (UID: "1c5f5fd5-7bf8-4d82-9c39-ca9e923ec6b4") : secret "insights-runtime-extractor-tls" not found
Jul 27 20:52:23.511650 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:23.511620    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-blb74_ef72a491-3d46-4039-9c8b-a6a9eb853223/serve-healthcheck-canary/0.log"
Jul 27 20:52:24.512105 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:24.512051    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-7vsbq_1da96bd6-098f-41d7-9be9-904d53068b43/kube-storage-version-migrator-operator/0.log"
Jul 27 20:52:24.712081 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:24.712049    2560 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-584dcb6b54-z6zxv"
Jul 27 20:52:24.712223 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:24.712117    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-z6zxv"
Jul 27 20:52:24.712438 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:24.712426    2560 scope.go:117] "RemoveContainer" containerID="18921b555aef90835decc2504c918a148d5fdd0d841d7f42ae7fe8a8b418c1c3"
Jul 27 20:52:24.712575 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:52:24.712556    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-z6zxv_openshift-console-operator(8a6b7ae6-6c21-453d-99b9-303066d19915)\"" pod="openshift-console-operator/console-operator-584dcb6b54-z6zxv" podUID="8a6b7ae6-6c21-453d-99b9-303066d19915"
Jul 27 20:52:25.097416 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:25.097383    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-95hg7"
Jul 27 20:52:26.847722 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:26.847685    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1c5f5fd5-7bf8-4d82-9c39-ca9e923ec6b4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mb7xd\" (UID: \"1c5f5fd5-7bf8-4d82-9c39-ca9e923ec6b4\") " pod="openshift-insights/insights-runtime-extractor-mb7xd"
Jul 27 20:52:26.848082 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:26.847811    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1c5f5fd5-7bf8-4d82-9c39-ca9e923ec6b4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mb7xd\" (UID: \"1c5f5fd5-7bf8-4d82-9c39-ca9e923ec6b4\") " pod="openshift-insights/insights-runtime-extractor-mb7xd"
Jul 27 20:52:26.848082 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:52:26.847831    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 20:52:26.848082 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:52:26.847894    2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 27 20:52:26.848082 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:52:26.847897    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c5f5fd5-7bf8-4d82-9c39-ca9e923ec6b4-insights-runtime-extractor-tls podName:1c5f5fd5-7bf8-4d82-9c39-ca9e923ec6b4 nodeName:}" failed. No retries permitted until 2026-07-27 20:52:34.84787938 +0000 UTC m=+71.599741170 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1c5f5fd5-7bf8-4d82-9c39-ca9e923ec6b4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mb7xd" (UID: "1c5f5fd5-7bf8-4d82-9c39-ca9e923ec6b4") : secret "insights-runtime-extractor-tls" not found
Jul 27 20:52:26.848082 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:52:26.847948    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1c5f5fd5-7bf8-4d82-9c39-ca9e923ec6b4-kube-rbac-proxy-cm podName:1c5f5fd5-7bf8-4d82-9c39-ca9e923ec6b4 nodeName:}" failed. No retries permitted until 2026-07-27 20:52:34.847937363 +0000 UTC m=+71.599799142 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1c5f5fd5-7bf8-4d82-9c39-ca9e923ec6b4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mb7xd" (UID: "1c5f5fd5-7bf8-4d82-9c39-ca9e923ec6b4") : configmap "kube-rbac-proxy" not found
Jul 27 20:52:29.469032 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:29.468998    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8ns2\" (UniqueName: \"kubernetes.io/projected/1f1259cb-be97-4841-bd22-5bba585870b6-kube-api-access-n8ns2\") pod \"network-check-target-dbzwm\" (UID: \"1f1259cb-be97-4841-bd22-5bba585870b6\") " pod="openshift-network-diagnostics/network-check-target-dbzwm"
Jul 27 20:52:29.469032 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:29.469040    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4b924e42-83bc-459f-b1a2-f115f9bce63f-original-pull-secret\") pod \"global-pull-secret-syncer-dtrjg\" (UID: \"4b924e42-83bc-459f-b1a2-f115f9bce63f\") " pod="kube-system/global-pull-secret-syncer-dtrjg"
Jul 27 20:52:29.471689 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:29.471673    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 27 20:52:29.471756 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:29.471673    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 27 20:52:29.481664 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:29.481638    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 27 20:52:29.481885 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:29.481868    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4b924e42-83bc-459f-b1a2-f115f9bce63f-original-pull-secret\") pod \"global-pull-secret-syncer-dtrjg\" (UID: \"4b924e42-83bc-459f-b1a2-f115f9bce63f\") " pod="kube-system/global-pull-secret-syncer-dtrjg"
Jul 27 20:52:29.491792 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:29.491772    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8ns2\" (UniqueName: \"kubernetes.io/projected/1f1259cb-be97-4841-bd22-5bba585870b6-kube-api-access-n8ns2\") pod \"network-check-target-dbzwm\" (UID: \"1f1259cb-be97-4841-bd22-5bba585870b6\") " pod="openshift-network-diagnostics/network-check-target-dbzwm"
Jul 27 20:52:29.570072 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:29.570044    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fac6ee6e-225a-4402-8834-69ff744b07f9-metrics-certs\") pod \"network-metrics-daemon-r6g8d\" (UID: \"fac6ee6e-225a-4402-8834-69ff744b07f9\") " pod="openshift-multus/network-metrics-daemon-r6g8d"
Jul 27 20:52:29.572493 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:29.572476    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 27 20:52:29.580449 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:52:29.580431    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 27 20:52:29.580501 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:52:29.580492    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fac6ee6e-225a-4402-8834-69ff744b07f9-metrics-certs podName:fac6ee6e-225a-4402-8834-69ff744b07f9 nodeName:}" failed. No retries permitted until 2026-07-27 20:53:33.580477346 +0000 UTC m=+130.332339120 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fac6ee6e-225a-4402-8834-69ff744b07f9-metrics-certs") pod "network-metrics-daemon-r6g8d" (UID: "fac6ee6e-225a-4402-8834-69ff744b07f9") : secret "metrics-daemon-secret" not found
Jul 27 20:52:29.655379 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:29.655359    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dtrjg"
Jul 27 20:52:29.669822 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:29.669802    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-qtl47\""
Jul 27 20:52:29.677825 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:29.677807    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dbzwm"
Jul 27 20:52:29.801394 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:29.801362    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-dtrjg"]
Jul 27 20:52:29.804563 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:52:29.804529    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b924e42_83bc_459f_b1a2_f115f9bce63f.slice/crio-217900242db22c199cf7365e24e719704d748f785ea8b717b42b8af5d02a3050 WatchSource:0}: Error finding container 217900242db22c199cf7365e24e719704d748f785ea8b717b42b8af5d02a3050: Status 404 returned error can't find the container with id 217900242db22c199cf7365e24e719704d748f785ea8b717b42b8af5d02a3050
Jul 27 20:52:29.816381 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:29.816287    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-dbzwm"]
Jul 27 20:52:29.821220 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:52:29.821198    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f1259cb_be97_4841_bd22_5bba585870b6.slice/crio-dec0abe3eb88fb9f5d49657acded14518380fc5bc070a471e9a7cf1d3a9edb28 WatchSource:0}: Error finding container dec0abe3eb88fb9f5d49657acded14518380fc5bc070a471e9a7cf1d3a9edb28: Status 404 returned error can't find the container with id dec0abe3eb88fb9f5d49657acded14518380fc5bc070a471e9a7cf1d3a9edb28
Jul 27 20:52:30.174244 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:30.174204    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0bf98545-754e-4371-a262-d78b44f0361d-registry-tls\") pod \"image-registry-65494b8dd8-cvl2s\" (UID: \"0bf98545-754e-4371-a262-d78b44f0361d\") " pod="openshift-image-registry/image-registry-65494b8dd8-cvl2s"
Jul 27 20:52:30.176765 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:30.176743    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0bf98545-754e-4371-a262-d78b44f0361d-registry-tls\") pod \"image-registry-65494b8dd8-cvl2s\" (UID: \"0bf98545-754e-4371-a262-d78b44f0361d\") " pod="openshift-image-registry/image-registry-65494b8dd8-cvl2s"
Jul 27 20:52:30.195965 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:30.195928    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-dtrjg" event={"ID":"4b924e42-83bc-459f-b1a2-f115f9bce63f","Type":"ContainerStarted","Data":"217900242db22c199cf7365e24e719704d748f785ea8b717b42b8af5d02a3050"}
Jul 27 20:52:30.196893 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:30.196871    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-dbzwm" event={"ID":"1f1259cb-be97-4841-bd22-5bba585870b6","Type":"ContainerStarted","Data":"dec0abe3eb88fb9f5d49657acded14518380fc5bc070a471e9a7cf1d3a9edb28"}
Jul 27 20:52:30.206775 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:30.206752    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-jv4bm\""
Jul 27 20:52:30.214981 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:30.214957    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-65494b8dd8-cvl2s"
Jul 27 20:52:30.386222 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:30.386189    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-65494b8dd8-cvl2s"]
Jul 27 20:52:30.389587 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:52:30.389560    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bf98545_754e_4371_a262_d78b44f0361d.slice/crio-832c866fee0cd8e35ac64df78c7c247551badb458abf540455e499a0fc2fd138 WatchSource:0}: Error finding container 832c866fee0cd8e35ac64df78c7c247551badb458abf540455e499a0fc2fd138: Status 404 returned error can't find the container with id 832c866fee0cd8e35ac64df78c7c247551badb458abf540455e499a0fc2fd138
Jul 27 20:52:31.083774 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:31.083356    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a9187e8b-eb38-4984-b284-39419b5de51b-metrics-tls\") pod \"dns-default-ds6vw\" (UID: \"a9187e8b-eb38-4984-b284-39419b5de51b\") " pod="openshift-dns/dns-default-ds6vw"
Jul 27 20:52:31.086890 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:31.086112    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a9187e8b-eb38-4984-b284-39419b5de51b-metrics-tls\") pod \"dns-default-ds6vw\" (UID: \"a9187e8b-eb38-4984-b284-39419b5de51b\") " pod="openshift-dns/dns-default-ds6vw"
Jul 27 20:52:31.201899 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:31.201862    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-65494b8dd8-cvl2s" event={"ID":"0bf98545-754e-4371-a262-d78b44f0361d","Type":"ContainerStarted","Data":"9be5d53352bdb73378eba1ae629c87ecf293c32bcd582892600538af684d0005"}
Jul 27 20:52:31.202064 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:31.201915    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-65494b8dd8-cvl2s" event={"ID":"0bf98545-754e-4371-a262-d78b44f0361d","Type":"ContainerStarted","Data":"832c866fee0cd8e35ac64df78c7c247551badb458abf540455e499a0fc2fd138"}
Jul 27 20:52:31.202596 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:31.202570    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-65494b8dd8-cvl2s"
Jul 27 20:52:31.225698 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:31.225640    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-65494b8dd8-cvl2s" podStartSLOduration=17.225608693 podStartE2EDuration="17.225608693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:52:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:52:31.224438548 +0000 UTC m=+67.976300366" watchObservedRunningTime="2026-07-27 20:52:31.225608693 +0000 UTC m=+67.977470531"
Jul 27 20:52:31.313040 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:31.313005    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-x9tww\""
Jul 27 20:52:31.320070 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:31.320035    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ds6vw"
Jul 27 20:52:31.479740 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:31.478509    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-ds6vw"]
Jul 27 20:52:31.483696 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:52:31.483665    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9187e8b_eb38_4984_b284_39419b5de51b.slice/crio-c041e3386cb6b2dae721193be79e042332184127eb68316fbc368b7cc0fae934 WatchSource:0}: Error finding container c041e3386cb6b2dae721193be79e042332184127eb68316fbc368b7cc0fae934: Status 404 returned error can't find the container with id c041e3386cb6b2dae721193be79e042332184127eb68316fbc368b7cc0fae934
Jul 27 20:52:32.206770 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:32.206725    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ds6vw" event={"ID":"a9187e8b-eb38-4984-b284-39419b5de51b","Type":"ContainerStarted","Data":"c041e3386cb6b2dae721193be79e042332184127eb68316fbc368b7cc0fae934"}
Jul 27 20:52:34.914664 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:34.914620    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1c5f5fd5-7bf8-4d82-9c39-ca9e923ec6b4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mb7xd\" (UID: \"1c5f5fd5-7bf8-4d82-9c39-ca9e923ec6b4\") " pod="openshift-insights/insights-runtime-extractor-mb7xd"
Jul 27 20:52:34.915007 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:34.914673    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1c5f5fd5-7bf8-4d82-9c39-ca9e923ec6b4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mb7xd\" (UID: \"1c5f5fd5-7bf8-4d82-9c39-ca9e923ec6b4\") " pod="openshift-insights/insights-runtime-extractor-mb7xd"
Jul 27 20:52:34.915007 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:52:34.914925    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 20:52:34.915197 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:52:34.915016    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c5f5fd5-7bf8-4d82-9c39-ca9e923ec6b4-insights-runtime-extractor-tls podName:1c5f5fd5-7bf8-4d82-9c39-ca9e923ec6b4 nodeName:}" failed. No retries permitted until 2026-07-27 20:52:50.914993606 +0000 UTC m=+87.666855393 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1c5f5fd5-7bf8-4d82-9c39-ca9e923ec6b4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mb7xd" (UID: "1c5f5fd5-7bf8-4d82-9c39-ca9e923ec6b4") : secret "insights-runtime-extractor-tls" not found
Jul 27 20:52:34.915265 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:34.915248    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1c5f5fd5-7bf8-4d82-9c39-ca9e923ec6b4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mb7xd\" (UID: \"1c5f5fd5-7bf8-4d82-9c39-ca9e923ec6b4\") " pod="openshift-insights/insights-runtime-extractor-mb7xd"
Jul 27 20:52:35.217013 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:35.216917    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-dtrjg" event={"ID":"4b924e42-83bc-459f-b1a2-f115f9bce63f","Type":"ContainerStarted","Data":"f2d8bf004debd8f2d1c51467f312bdc2c3478154c286ae288e877e8daa3eb84a"}
Jul 27 20:52:35.218709 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:35.218682    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ds6vw" event={"ID":"a9187e8b-eb38-4984-b284-39419b5de51b","Type":"ContainerStarted","Data":"a03efb9412ce30a36eebcb35429356664551952fd06cacb7ae7490c52fae9394"}
Jul 27 20:52:35.218837 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:35.218734    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ds6vw" event={"ID":"a9187e8b-eb38-4984-b284-39419b5de51b","Type":"ContainerStarted","Data":"f1bea6eadb4eabf0038262b04d5f0ac9619d6dccacb01d93e6d80a70b305936d"}
Jul 27 20:52:35.218837 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:35.218765    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-ds6vw"
Jul 27 20:52:35.220316 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:35.220289    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-dbzwm" event={"ID":"1f1259cb-be97-4841-bd22-5bba585870b6","Type":"ContainerStarted","Data":"b349598e067b9b4c1c23889a05d80cbc6df0d4b65df21a4cc9218a4a5b9d6e35"}
Jul 27 20:52:35.220447 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:35.220433    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-dbzwm"
Jul 27 20:52:35.259151 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:35.259108    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-ds6vw" podStartSLOduration=32.884829454 podStartE2EDuration="36.259072129s" totalImagesPullingTime="3.374242675s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:51:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:52:35.258696724 +0000 UTC m=+72.010558519" watchObservedRunningTime="2026-07-27 20:52:35.259072129 +0000 UTC m=+72.010933924"
Jul 27 20:52:35.259294 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:35.259275    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-dtrjg" podStartSLOduration=66.211608715 podStartE2EDuration="1m11.259270436s" totalImagesPullingTime="5.047661721s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 20:51:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:52:35.242448528 +0000 UTC m=+71.994310334" watchObservedRunningTime="2026-07-27 20:52:35.259270436 +0000 UTC m=+72.011132230"
Jul 27 20:52:35.278377 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:35.278341    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-dbzwm" podStartSLOduration=66.270140906 podStartE2EDuration="1m11.278329373s" totalImagesPullingTime="5.008188467s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:51:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:52:35.278255641 +0000 UTC m=+72.030117428" watchObservedRunningTime="2026-07-27 20:52:35.278329373 +0000 UTC m=+72.030191145"
Jul 27 20:52:35.841880 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:35.841848    2560 scope.go:117] "RemoveContainer" containerID="18921b555aef90835decc2504c918a148d5fdd0d841d7f42ae7fe8a8b418c1c3"
Jul 27 20:52:36.225927 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:36.225863    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-z6zxv_8a6b7ae6-6c21-453d-99b9-303066d19915/console-operator/1.log"
Jul 27 20:52:36.226321 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:36.226024    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-z6zxv" event={"ID":"8a6b7ae6-6c21-453d-99b9-303066d19915","Type":"ContainerStarted","Data":"8a703cba73f21c3e528e784ea6fc1ea8cb4dd8a356714b33ca252dfac430f42e"}
Jul 27 20:52:36.226670 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:36.226652    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-z6zxv"
Jul 27 20:52:36.245629 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:36.245577    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-584dcb6b54-z6zxv" podStartSLOduration=20.027163039 podStartE2EDuration="22.245557805s" totalImagesPullingTime="2.218394766s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:52:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:52:36.243810842 +0000 UTC m=+72.995672638" watchObservedRunningTime="2026-07-27 20:52:36.245557805 +0000 UTC m=+72.997419602"
Jul 27 20:52:36.795213 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:36.795181    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-584dcb6b54-z6zxv"
Jul 27 20:52:45.228548 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:45.228437    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-ds6vw"
Jul 27 20:52:47.778889 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:47.778855    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fcb6f7df-zlsd6"]
Jul 27 20:52:47.782008 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:47.781991    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fcb6f7df-zlsd6"
Jul 27 20:52:47.789920 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:47.789885    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 27 20:52:47.790041 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:47.789930    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 27 20:52:47.795880 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:47.795857    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 27 20:52:47.796576 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:47.796544    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 27 20:52:47.801038 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:47.801020    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 27 20:52:47.801331 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:47.801315    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 27 20:52:47.810043 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:47.810019    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 27 20:52:47.902975 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:47.902950    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/bbd1e7a0-40ab-41a0-9702-b816ee71b502-ca\") pod \"cluster-proxy-proxy-agent-6fcb6f7df-zlsd6\" (UID: \"bbd1e7a0-40ab-41a0-9702-b816ee71b502\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fcb6f7df-zlsd6"
Jul 27 20:52:47.903166 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:47.902983    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bbd1e7a0-40ab-41a0-9702-b816ee71b502-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6fcb6f7df-zlsd6\" (UID: \"bbd1e7a0-40ab-41a0-9702-b816ee71b502\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fcb6f7df-zlsd6"
Jul 27 20:52:47.903166 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:47.903013    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-648bb\" (UniqueName: \"kubernetes.io/projected/bbd1e7a0-40ab-41a0-9702-b816ee71b502-kube-api-access-648bb\") pod \"cluster-proxy-proxy-agent-6fcb6f7df-zlsd6\" (UID: \"bbd1e7a0-40ab-41a0-9702-b816ee71b502\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fcb6f7df-zlsd6"
Jul 27 20:52:47.903166 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:47.903124    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/bbd1e7a0-40ab-41a0-9702-b816ee71b502-hub\") pod \"cluster-proxy-proxy-agent-6fcb6f7df-zlsd6\" (UID: \"bbd1e7a0-40ab-41a0-9702-b816ee71b502\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fcb6f7df-zlsd6"
Jul 27 20:52:47.903292 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:47.903173    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbd1e7a0-40ab-41a0-9702-b816ee71b502-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6fcb6f7df-zlsd6\" (UID: \"bbd1e7a0-40ab-41a0-9702-b816ee71b502\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fcb6f7df-zlsd6"
Jul 27 20:52:47.903292 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:47.903224    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/bbd1e7a0-40ab-41a0-9702-b816ee71b502-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6fcb6f7df-zlsd6\" (UID: \"bbd1e7a0-40ab-41a0-9702-b816ee71b502\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fcb6f7df-zlsd6"
Jul 27 20:52:47.909963 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:47.909935    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fcb6f7df-zlsd6"]
Jul 27 20:52:47.932652 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:47.932628    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-65494b8dd8-cvl2s"]
Jul 27 20:52:47.937380 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:47.937357    2560 patch_prober.go:28] interesting pod/image-registry-65494b8dd8-cvl2s container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 27 20:52:47.937480 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:47.937400    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-65494b8dd8-cvl2s" podUID="0bf98545-754e-4371-a262-d78b44f0361d" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 20:52:48.004257 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:48.004222    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbd1e7a0-40ab-41a0-9702-b816ee71b502-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6fcb6f7df-zlsd6\" (UID: \"bbd1e7a0-40ab-41a0-9702-b816ee71b502\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fcb6f7df-zlsd6"
Jul 27 20:52:48.004428 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:48.004273    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/bbd1e7a0-40ab-41a0-9702-b816ee71b502-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6fcb6f7df-zlsd6\" (UID: \"bbd1e7a0-40ab-41a0-9702-b816ee71b502\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fcb6f7df-zlsd6"
Jul 27 20:52:48.004428 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:48.004309    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/bbd1e7a0-40ab-41a0-9702-b816ee71b502-ca\") pod \"cluster-proxy-proxy-agent-6fcb6f7df-zlsd6\" (UID: \"bbd1e7a0-40ab-41a0-9702-b816ee71b502\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fcb6f7df-zlsd6"
Jul 27 20:52:48.004428 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:48.004331    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bbd1e7a0-40ab-41a0-9702-b816ee71b502-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6fcb6f7df-zlsd6\" (UID: \"bbd1e7a0-40ab-41a0-9702-b816ee71b502\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fcb6f7df-zlsd6"
Jul 27 20:52:48.004428 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:48.004358    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-648bb\" (UniqueName: \"kubernetes.io/projected/bbd1e7a0-40ab-41a0-9702-b816ee71b502-kube-api-access-648bb\") pod \"cluster-proxy-proxy-agent-6fcb6f7df-zlsd6\" (UID: \"bbd1e7a0-40ab-41a0-9702-b816ee71b502\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fcb6f7df-zlsd6"
Jul 27 20:52:48.004628 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:48.004512    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/bbd1e7a0-40ab-41a0-9702-b816ee71b502-hub\") pod \"cluster-proxy-proxy-agent-6fcb6f7df-zlsd6\" (UID: \"bbd1e7a0-40ab-41a0-9702-b816ee71b502\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fcb6f7df-zlsd6"
Jul 27 20:52:48.004965 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:48.004936    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbd1e7a0-40ab-41a0-9702-b816ee71b502-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6fcb6f7df-zlsd6\" (UID: \"bbd1e7a0-40ab-41a0-9702-b816ee71b502\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fcb6f7df-zlsd6"
Jul 27 20:52:48.006915 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:48.006889    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/bbd1e7a0-40ab-41a0-9702-b816ee71b502-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6fcb6f7df-zlsd6\" (UID: \"bbd1e7a0-40ab-41a0-9702-b816ee71b502\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fcb6f7df-zlsd6"
Jul 27 20:52:48.007010 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:48.006959    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/bbd1e7a0-40ab-41a0-9702-b816ee71b502-ca\") pod \"cluster-proxy-proxy-agent-6fcb6f7df-zlsd6\" (UID: \"bbd1e7a0-40ab-41a0-9702-b816ee71b502\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fcb6f7df-zlsd6"
Jul 27 20:52:48.007053 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:48.007030    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/bbd1e7a0-40ab-41a0-9702-b816ee71b502-hub\") pod \"cluster-proxy-proxy-agent-6fcb6f7df-zlsd6\" (UID: \"bbd1e7a0-40ab-41a0-9702-b816ee71b502\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fcb6f7df-zlsd6"
Jul 27 20:52:48.007216 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:48.007200    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bbd1e7a0-40ab-41a0-9702-b816ee71b502-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6fcb6f7df-zlsd6\" (UID: \"bbd1e7a0-40ab-41a0-9702-b816ee71b502\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fcb6f7df-zlsd6"
Jul 27 20:52:48.028960 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:48.028898    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-648bb\" (UniqueName: \"kubernetes.io/projected/bbd1e7a0-40ab-41a0-9702-b816ee71b502-kube-api-access-648bb\") pod \"cluster-proxy-proxy-agent-6fcb6f7df-zlsd6\" (UID: \"bbd1e7a0-40ab-41a0-9702-b816ee71b502\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fcb6f7df-zlsd6"
Jul 27 20:52:48.098710 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:48.098677    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fcb6f7df-zlsd6"
Jul 27 20:52:48.242495 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:48.242463    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fcb6f7df-zlsd6"]
Jul 27 20:52:48.245746 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:52:48.245712    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbbd1e7a0_40ab_41a0_9702_b816ee71b502.slice/crio-10df3e3be118e272dc6a478760a34263b31c4a597875d8eb2d354be122a45e5b WatchSource:0}: Error finding container 10df3e3be118e272dc6a478760a34263b31c4a597875d8eb2d354be122a45e5b: Status 404 returned error can't find the container with id 10df3e3be118e272dc6a478760a34263b31c4a597875d8eb2d354be122a45e5b
Jul 27 20:52:48.265496 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:48.265472    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fcb6f7df-zlsd6" event={"ID":"bbd1e7a0-40ab-41a0-9702-b816ee71b502","Type":"ContainerStarted","Data":"10df3e3be118e272dc6a478760a34263b31c4a597875d8eb2d354be122a45e5b"}
Jul 27 20:52:50.926649 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:50.926602    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1c5f5fd5-7bf8-4d82-9c39-ca9e923ec6b4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mb7xd\" (UID: \"1c5f5fd5-7bf8-4d82-9c39-ca9e923ec6b4\") " pod="openshift-insights/insights-runtime-extractor-mb7xd"
Jul 27 20:52:50.929538 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:50.929509    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1c5f5fd5-7bf8-4d82-9c39-ca9e923ec6b4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mb7xd\" (UID: \"1c5f5fd5-7bf8-4d82-9c39-ca9e923ec6b4\") " pod="openshift-insights/insights-runtime-extractor-mb7xd"
Jul 27 20:52:50.951718 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:50.951695    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-5rxbs\""
Jul 27 20:52:50.959686 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:50.959665    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-mb7xd"
Jul 27 20:52:51.980834 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:51.980807    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-mb7xd"]
Jul 27 20:52:51.983459 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:52:51.983434    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c5f5fd5_7bf8_4d82_9c39_ca9e923ec6b4.slice/crio-c030987555133684c4db736b0b506fa925fc8a19796651dc992b546226d66df8 WatchSource:0}: Error finding container c030987555133684c4db736b0b506fa925fc8a19796651dc992b546226d66df8: Status 404 returned error can't find the container with id c030987555133684c4db736b0b506fa925fc8a19796651dc992b546226d66df8
Jul 27 20:52:52.280822 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:52.280769    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mb7xd" event={"ID":"1c5f5fd5-7bf8-4d82-9c39-ca9e923ec6b4","Type":"ContainerStarted","Data":"c030987555133684c4db736b0b506fa925fc8a19796651dc992b546226d66df8"}
Jul 27 20:52:52.282789 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:52.282763    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fcb6f7df-zlsd6" event={"ID":"bbd1e7a0-40ab-41a0-9702-b816ee71b502","Type":"ContainerStarted","Data":"3abc6f2c53707ab205959721f42dc5ff923208de7401c040cfeaf15c092eb16a"}
Jul 27 20:52:52.282911 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:52.282798    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fcb6f7df-zlsd6" event={"ID":"bbd1e7a0-40ab-41a0-9702-b816ee71b502","Type":"ContainerStarted","Data":"92b218c03185b413f510bad502fb7efd995968ca174dbbbc22c5ece044f82c7e"}
Jul 27 20:52:52.282911 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:52.282810    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fcb6f7df-zlsd6" event={"ID":"bbd1e7a0-40ab-41a0-9702-b816ee71b502","Type":"ContainerStarted","Data":"fd020668c5bf3da8532716527d4335a0d31f6ca0fe7ec4a36f2f35103d353002"}
Jul 27 20:52:52.305195 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:52.305141    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6fcb6f7df-zlsd6" podStartSLOduration=1.6639949170000001 podStartE2EDuration="5.30512744s" totalImagesPullingTime="3.641132523s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:52:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:52:52.302687983 +0000 UTC m=+89.054549778" watchObservedRunningTime="2026-07-27 20:52:52.30512744 +0000 UTC m=+89.056989214"
Jul 27 20:52:54.123008 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:54.122973    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7b54c855d7-gwj7h"]
Jul 27 20:52:54.125215 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:54.125196    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b54c855d7-gwj7h"
Jul 27 20:52:54.127798 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:54.127769    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 27 20:52:54.127959 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:54.127933    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-nr66h\""
Jul 27 20:52:54.128895 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:54.128873    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 27 20:52:54.129030 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:54.128879    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 27 20:52:54.129030 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:54.128984    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 27 20:52:54.129264 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:54.129245    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 27 20:52:54.129357 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:54.129302    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 27 20:52:54.129357 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:54.129308    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 27 20:52:54.138268 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:54.138243    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7b54c855d7-gwj7h"]
Jul 27 20:52:54.254270 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:54.254215    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9a73773c-200d-4997-b068-7bcbe585030b-console-serving-cert\") pod \"console-7b54c855d7-gwj7h\" (UID: \"9a73773c-200d-4997-b068-7bcbe585030b\") " pod="openshift-console/console-7b54c855d7-gwj7h"
Jul 27 20:52:54.254438 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:54.254292    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9a73773c-200d-4997-b068-7bcbe585030b-service-ca\") pod \"console-7b54c855d7-gwj7h\" (UID: \"9a73773c-200d-4997-b068-7bcbe585030b\") " pod="openshift-console/console-7b54c855d7-gwj7h"
Jul 27 20:52:54.254438 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:54.254341    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9a73773c-200d-4997-b068-7bcbe585030b-oauth-serving-cert\") pod \"console-7b54c855d7-gwj7h\" (UID: \"9a73773c-200d-4997-b068-7bcbe585030b\") " pod="openshift-console/console-7b54c855d7-gwj7h"
Jul 27 20:52:54.254556 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:54.254456    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9a73773c-200d-4997-b068-7bcbe585030b-console-config\") pod \"console-7b54c855d7-gwj7h\" (UID: \"9a73773c-200d-4997-b068-7bcbe585030b\") " pod="openshift-console/console-7b54c855d7-gwj7h"
Jul 27 20:52:54.254556 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:54.254497    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-246fn\" (UniqueName: \"kubernetes.io/projected/9a73773c-200d-4997-b068-7bcbe585030b-kube-api-access-246fn\") pod \"console-7b54c855d7-gwj7h\" (UID: \"9a73773c-200d-4997-b068-7bcbe585030b\") " pod="openshift-console/console-7b54c855d7-gwj7h"
Jul 27 20:52:54.254556 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:54.254546    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9a73773c-200d-4997-b068-7bcbe585030b-console-oauth-config\") pod \"console-7b54c855d7-gwj7h\" (UID: \"9a73773c-200d-4997-b068-7bcbe585030b\") " pod="openshift-console/console-7b54c855d7-gwj7h"
Jul 27 20:52:54.355729 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:54.355694    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9a73773c-200d-4997-b068-7bcbe585030b-console-serving-cert\") pod \"console-7b54c855d7-gwj7h\" (UID: \"9a73773c-200d-4997-b068-7bcbe585030b\") " pod="openshift-console/console-7b54c855d7-gwj7h"
Jul 27 20:52:54.355902 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:54.355742    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9a73773c-200d-4997-b068-7bcbe585030b-service-ca\") pod \"console-7b54c855d7-gwj7h\" (UID: \"9a73773c-200d-4997-b068-7bcbe585030b\") " pod="openshift-console/console-7b54c855d7-gwj7h"
Jul 27 20:52:54.355902 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:54.355784    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9a73773c-200d-4997-b068-7bcbe585030b-oauth-serving-cert\") pod \"console-7b54c855d7-gwj7h\" (UID: \"9a73773c-200d-4997-b068-7bcbe585030b\") " pod="openshift-console/console-7b54c855d7-gwj7h"
Jul 27 20:52:54.355902 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:54.355841    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9a73773c-200d-4997-b068-7bcbe585030b-console-config\") pod \"console-7b54c855d7-gwj7h\" (UID: \"9a73773c-200d-4997-b068-7bcbe585030b\") " pod="openshift-console/console-7b54c855d7-gwj7h"
Jul 27 20:52:54.355902 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:54.355870    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-246fn\" (UniqueName: \"kubernetes.io/projected/9a73773c-200d-4997-b068-7bcbe585030b-kube-api-access-246fn\") pod \"console-7b54c855d7-gwj7h\" (UID: \"9a73773c-200d-4997-b068-7bcbe585030b\") " pod="openshift-console/console-7b54c855d7-gwj7h"
Jul 27 20:52:54.355902 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:54.355900    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9a73773c-200d-4997-b068-7bcbe585030b-console-oauth-config\") pod \"console-7b54c855d7-gwj7h\" (UID: \"9a73773c-200d-4997-b068-7bcbe585030b\") " pod="openshift-console/console-7b54c855d7-gwj7h"
Jul 27 20:52:54.356571 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:54.356546    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9a73773c-200d-4997-b068-7bcbe585030b-service-ca\") pod \"console-7b54c855d7-gwj7h\" (UID: \"9a73773c-200d-4997-b068-7bcbe585030b\") " pod="openshift-console/console-7b54c855d7-gwj7h"
Jul 27 20:52:54.356848 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:54.356825    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9a73773c-200d-4997-b068-7bcbe585030b-oauth-serving-cert\") pod \"console-7b54c855d7-gwj7h\" (UID: \"9a73773c-200d-4997-b068-7bcbe585030b\") " pod="openshift-console/console-7b54c855d7-gwj7h"
Jul 27 20:52:54.356992 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:54.356972    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9a73773c-200d-4997-b068-7bcbe585030b-console-config\") pod \"console-7b54c855d7-gwj7h\" (UID: \"9a73773c-200d-4997-b068-7bcbe585030b\") " pod="openshift-console/console-7b54c855d7-gwj7h"
Jul 27 20:52:54.358619 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:54.358597    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9a73773c-200d-4997-b068-7bcbe585030b-console-serving-cert\") pod \"console-7b54c855d7-gwj7h\" (UID: \"9a73773c-200d-4997-b068-7bcbe585030b\") " pod="openshift-console/console-7b54c855d7-gwj7h"
Jul 27 20:52:54.358718 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:54.358668    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9a73773c-200d-4997-b068-7bcbe585030b-console-oauth-config\") pod \"console-7b54c855d7-gwj7h\" (UID: \"9a73773c-200d-4997-b068-7bcbe585030b\") " pod="openshift-console/console-7b54c855d7-gwj7h"
Jul 27 20:52:54.372436 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:54.372399    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-246fn\" (UniqueName: \"kubernetes.io/projected/9a73773c-200d-4997-b068-7bcbe585030b-kube-api-access-246fn\") pod \"console-7b54c855d7-gwj7h\" (UID: \"9a73773c-200d-4997-b068-7bcbe585030b\") " pod="openshift-console/console-7b54c855d7-gwj7h"
Jul 27 20:52:54.435341 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:54.435265    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b54c855d7-gwj7h"
Jul 27 20:52:56.864009 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:56.863977    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7b54c855d7-gwj7h"]
Jul 27 20:52:56.868515 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:52:56.868492    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a73773c_200d_4997_b068_7bcbe585030b.slice/crio-bd2418a7f24340d98b36a40f5d55b75d6298432805baad1fb8a5027ddb7eb1c5 WatchSource:0}: Error finding container bd2418a7f24340d98b36a40f5d55b75d6298432805baad1fb8a5027ddb7eb1c5: Status 404 returned error can't find the container with id bd2418a7f24340d98b36a40f5d55b75d6298432805baad1fb8a5027ddb7eb1c5
Jul 27 20:52:57.304177 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:57.304061    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mb7xd" event={"ID":"1c5f5fd5-7bf8-4d82-9c39-ca9e923ec6b4","Type":"ContainerStarted","Data":"2849f5a7ab0ee532eeea642fa32ac0365001597007d34036e2c3738f959e947e"}
Jul 27 20:52:57.305676 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:57.305620    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b54c855d7-gwj7h" event={"ID":"9a73773c-200d-4997-b068-7bcbe585030b","Type":"ContainerStarted","Data":"bd2418a7f24340d98b36a40f5d55b75d6298432805baad1fb8a5027ddb7eb1c5"}
Jul 27 20:52:57.939318 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:52:57.939289    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-65494b8dd8-cvl2s"
Jul 27 20:53:02.326730 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:02.326691    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b54c855d7-gwj7h" event={"ID":"9a73773c-200d-4997-b068-7bcbe585030b","Type":"ContainerStarted","Data":"836f92d4ff2fa654dbfd0c3137c54b11ae4247e1f16716c73d43606bb2a856a7"}
Jul 27 20:53:02.328968 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:02.328939    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mb7xd" event={"ID":"1c5f5fd5-7bf8-4d82-9c39-ca9e923ec6b4","Type":"ContainerStarted","Data":"fc433c76171b6ecbc1523250fb0958718630a191b751e8f947082174dfc3211e"}
Jul 27 20:53:02.349603 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:02.349538    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7b54c855d7-gwj7h" podStartSLOduration=3.726310774 podStartE2EDuration="8.349517042s" totalImagesPullingTime="4.623206268s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:52:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:53:02.348689503 +0000 UTC m=+99.100551321" watchObservedRunningTime="2026-07-27 20:53:02.349517042 +0000 UTC m=+99.101378835"
Jul 27 20:53:02.492135 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:02.492077    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-mj6h5"]
Jul 27 20:53:02.494701 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:02.494672    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mj6h5"
Jul 27 20:53:02.497540 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:02.497508    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 27 20:53:02.497686 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:02.497562    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-nwms7\""
Jul 27 20:53:02.497850 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:02.497816    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 27 20:53:02.497958 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:02.497823    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 27 20:53:02.497958 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:02.497882    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 27 20:53:02.498082 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:02.498066    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 27 20:53:02.498169 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:02.498122    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 27 20:53:02.627515 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:02.627420    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3e1e5615-3312-4080-bbfa-2a92c24bb888-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mj6h5\" (UID: \"3e1e5615-3312-4080-bbfa-2a92c24bb888\") " pod="openshift-monitoring/node-exporter-mj6h5"
Jul 27 20:53:02.627515 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:02.627474    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3e1e5615-3312-4080-bbfa-2a92c24bb888-root\") pod \"node-exporter-mj6h5\" (UID: \"3e1e5615-3312-4080-bbfa-2a92c24bb888\") " pod="openshift-monitoring/node-exporter-mj6h5"
Jul 27 20:53:02.627515 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:02.627508    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9tpv\" (UniqueName: \"kubernetes.io/projected/3e1e5615-3312-4080-bbfa-2a92c24bb888-kube-api-access-r9tpv\") pod \"node-exporter-mj6h5\" (UID: \"3e1e5615-3312-4080-bbfa-2a92c24bb888\") " pod="openshift-monitoring/node-exporter-mj6h5"
Jul 27 20:53:02.627804 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:02.627574    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3e1e5615-3312-4080-bbfa-2a92c24bb888-node-exporter-textfile\") pod \"node-exporter-mj6h5\" (UID: \"3e1e5615-3312-4080-bbfa-2a92c24bb888\") " pod="openshift-monitoring/node-exporter-mj6h5"
Jul 27 20:53:02.627804 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:02.627603    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3e1e5615-3312-4080-bbfa-2a92c24bb888-node-exporter-wtmp\") pod \"node-exporter-mj6h5\" (UID: \"3e1e5615-3312-4080-bbfa-2a92c24bb888\") " pod="openshift-monitoring/node-exporter-mj6h5"
Jul 27 20:53:02.627804 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:02.627649    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3e1e5615-3312-4080-bbfa-2a92c24bb888-sys\") pod \"node-exporter-mj6h5\" (UID: \"3e1e5615-3312-4080-bbfa-2a92c24bb888\") " pod="openshift-monitoring/node-exporter-mj6h5"
Jul 27 20:53:02.627804 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:02.627694    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3e1e5615-3312-4080-bbfa-2a92c24bb888-node-exporter-tls\") pod \"node-exporter-mj6h5\" (UID: \"3e1e5615-3312-4080-bbfa-2a92c24bb888\") " pod="openshift-monitoring/node-exporter-mj6h5"
Jul 27 20:53:02.627804 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:02.627721    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3e1e5615-3312-4080-bbfa-2a92c24bb888-metrics-client-ca\") pod \"node-exporter-mj6h5\" (UID: \"3e1e5615-3312-4080-bbfa-2a92c24bb888\") " pod="openshift-monitoring/node-exporter-mj6h5"
Jul 27 20:53:02.628194 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:02.627807    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3e1e5615-3312-4080-bbfa-2a92c24bb888-node-exporter-accelerators-collector-config\") pod \"node-exporter-mj6h5\" (UID: \"3e1e5615-3312-4080-bbfa-2a92c24bb888\") " pod="openshift-monitoring/node-exporter-mj6h5"
Jul 27 20:53:02.728914 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:02.728873    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3e1e5615-3312-4080-bbfa-2a92c24bb888-sys\") pod \"node-exporter-mj6h5\" (UID: \"3e1e5615-3312-4080-bbfa-2a92c24bb888\") " pod="openshift-monitoring/node-exporter-mj6h5"
Jul 27 20:53:02.729142 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:02.728935    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3e1e5615-3312-4080-bbfa-2a92c24bb888-node-exporter-tls\") pod \"node-exporter-mj6h5\" (UID: \"3e1e5615-3312-4080-bbfa-2a92c24bb888\") " pod="openshift-monitoring/node-exporter-mj6h5"
Jul 27 20:53:02.729142 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:02.728963    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3e1e5615-3312-4080-bbfa-2a92c24bb888-metrics-client-ca\") pod \"node-exporter-mj6h5\" (UID: \"3e1e5615-3312-4080-bbfa-2a92c24bb888\") " pod="openshift-monitoring/node-exporter-mj6h5"
Jul 27 20:53:02.729142 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:02.729003    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3e1e5615-3312-4080-bbfa-2a92c24bb888-sys\") pod \"node-exporter-mj6h5\" (UID: \"3e1e5615-3312-4080-bbfa-2a92c24bb888\") " pod="openshift-monitoring/node-exporter-mj6h5"
Jul 27 20:53:02.729142 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:02.729005    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3e1e5615-3312-4080-bbfa-2a92c24bb888-node-exporter-accelerators-collector-config\") pod \"node-exporter-mj6h5\" (UID: \"3e1e5615-3312-4080-bbfa-2a92c24bb888\") " pod="openshift-monitoring/node-exporter-mj6h5"
Jul 27 20:53:02.729142 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:02.729107    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3e1e5615-3312-4080-bbfa-2a92c24bb888-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mj6h5\" (UID: \"3e1e5615-3312-4080-bbfa-2a92c24bb888\") " pod="openshift-monitoring/node-exporter-mj6h5"
Jul 27 20:53:02.729142 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:02.729140    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3e1e5615-3312-4080-bbfa-2a92c24bb888-root\") pod \"node-exporter-mj6h5\" (UID: \"3e1e5615-3312-4080-bbfa-2a92c24bb888\") " pod="openshift-monitoring/node-exporter-mj6h5"
Jul 27 20:53:02.729464 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:02.729165    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9tpv\" (UniqueName: \"kubernetes.io/projected/3e1e5615-3312-4080-bbfa-2a92c24bb888-kube-api-access-r9tpv\") pod \"node-exporter-mj6h5\" (UID: \"3e1e5615-3312-4080-bbfa-2a92c24bb888\") " pod="openshift-monitoring/node-exporter-mj6h5"
Jul 27 20:53:02.729464 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:02.729227    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3e1e5615-3312-4080-bbfa-2a92c24bb888-node-exporter-textfile\") pod \"node-exporter-mj6h5\" (UID: \"3e1e5615-3312-4080-bbfa-2a92c24bb888\") " pod="openshift-monitoring/node-exporter-mj6h5"
Jul 27 20:53:02.729464 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:02.729243    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3e1e5615-3312-4080-bbfa-2a92c24bb888-root\") pod \"node-exporter-mj6h5\" (UID: \"3e1e5615-3312-4080-bbfa-2a92c24bb888\") " pod="openshift-monitoring/node-exporter-mj6h5"
Jul 27 20:53:02.729464 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:02.729256    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3e1e5615-3312-4080-bbfa-2a92c24bb888-node-exporter-wtmp\") pod \"node-exporter-mj6h5\" (UID: \"3e1e5615-3312-4080-bbfa-2a92c24bb888\") " pod="openshift-monitoring/node-exporter-mj6h5"
Jul 27 20:53:02.729464 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:02.729379    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3e1e5615-3312-4080-bbfa-2a92c24bb888-node-exporter-wtmp\") pod \"node-exporter-mj6h5\" (UID: \"3e1e5615-3312-4080-bbfa-2a92c24bb888\") " pod="openshift-monitoring/node-exporter-mj6h5"
Jul 27 20:53:02.729739 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:02.729707    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3e1e5615-3312-4080-bbfa-2a92c24bb888-node-exporter-textfile\") pod \"node-exporter-mj6h5\" (UID: \"3e1e5615-3312-4080-bbfa-2a92c24bb888\") " pod="openshift-monitoring/node-exporter-mj6h5"
Jul 27 20:53:02.729914 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:02.729696    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3e1e5615-3312-4080-bbfa-2a92c24bb888-node-exporter-accelerators-collector-config\") pod \"node-exporter-mj6h5\" (UID: \"3e1e5615-3312-4080-bbfa-2a92c24bb888\") " pod="openshift-monitoring/node-exporter-mj6h5"
Jul 27 20:53:02.729914 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:02.729779    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3e1e5615-3312-4080-bbfa-2a92c24bb888-metrics-client-ca\") pod \"node-exporter-mj6h5\" (UID: \"3e1e5615-3312-4080-bbfa-2a92c24bb888\") " pod="openshift-monitoring/node-exporter-mj6h5"
Jul 27 20:53:02.732107 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:02.732063    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3e1e5615-3312-4080-bbfa-2a92c24bb888-node-exporter-tls\") pod \"node-exporter-mj6h5\" (UID: \"3e1e5615-3312-4080-bbfa-2a92c24bb888\") " pod="openshift-monitoring/node-exporter-mj6h5"
Jul 27 20:53:02.732272 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:02.732192    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3e1e5615-3312-4080-bbfa-2a92c24bb888-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mj6h5\" (UID: \"3e1e5615-3312-4080-bbfa-2a92c24bb888\") " pod="openshift-monitoring/node-exporter-mj6h5"
Jul 27 20:53:02.741319 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:02.741289    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9tpv\" (UniqueName: \"kubernetes.io/projected/3e1e5615-3312-4080-bbfa-2a92c24bb888-kube-api-access-r9tpv\") pod \"node-exporter-mj6h5\" (UID: \"3e1e5615-3312-4080-bbfa-2a92c24bb888\") " pod="openshift-monitoring/node-exporter-mj6h5"
Jul 27 20:53:02.806775 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:02.806734    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mj6h5"
Jul 27 20:53:02.819074 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:53:02.819042    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e1e5615_3312_4080_bbfa_2a92c24bb888.slice/crio-e4995138ced9a66fe1fabee43c5add53711c49d5afd2d5b74e3970bf55a71703 WatchSource:0}: Error finding container e4995138ced9a66fe1fabee43c5add53711c49d5afd2d5b74e3970bf55a71703: Status 404 returned error can't find the container with id e4995138ced9a66fe1fabee43c5add53711c49d5afd2d5b74e3970bf55a71703
Jul 27 20:53:03.338525 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:03.338474    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mj6h5" event={"ID":"3e1e5615-3312-4080-bbfa-2a92c24bb888","Type":"ContainerStarted","Data":"e4995138ced9a66fe1fabee43c5add53711c49d5afd2d5b74e3970bf55a71703"}
Jul 27 20:53:04.435501 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:04.435433    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7b54c855d7-gwj7h"
Jul 27 20:53:04.435971 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:04.435518    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7b54c855d7-gwj7h"
Jul 27 20:53:04.441275 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:04.441244    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7b54c855d7-gwj7h"
Jul 27 20:53:05.350388 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:05.350354    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7b54c855d7-gwj7h"
Jul 27 20:53:05.655832 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:05.655752    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-7444f7bc55-qltlp"]
Jul 27 20:53:05.680912 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:05.680859    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7444f7bc55-qltlp"]
Jul 27 20:53:05.682752 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:05.681264    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7444f7bc55-qltlp"
Jul 27 20:53:05.689297 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:05.686677    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-gmwfr\""
Jul 27 20:53:05.689297 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:05.686887    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 27 20:53:05.696022 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:05.689752    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 27 20:53:05.696022 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:05.690007    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 27 20:53:05.696022 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:05.690211    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 27 20:53:05.696022 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:05.690387    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-c8cnjkqp3aid5\""
Jul 27 20:53:05.702417 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:05.702397    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 27 20:53:05.756991 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:05.756966    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/39155db8-f2d3-4a9d-8ef8-ed19f16b62d8-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7444f7bc55-qltlp\" (UID: \"39155db8-f2d3-4a9d-8ef8-ed19f16b62d8\") " pod="openshift-monitoring/thanos-querier-7444f7bc55-qltlp"
Jul 27 20:53:05.757153 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:05.757006    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64grv\" (UniqueName: \"kubernetes.io/projected/39155db8-f2d3-4a9d-8ef8-ed19f16b62d8-kube-api-access-64grv\") pod \"thanos-querier-7444f7bc55-qltlp\" (UID: \"39155db8-f2d3-4a9d-8ef8-ed19f16b62d8\") " pod="openshift-monitoring/thanos-querier-7444f7bc55-qltlp"
Jul 27 20:53:05.757153 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:05.757053    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/39155db8-f2d3-4a9d-8ef8-ed19f16b62d8-secret-thanos-querier-tls\") pod \"thanos-querier-7444f7bc55-qltlp\" (UID: \"39155db8-f2d3-4a9d-8ef8-ed19f16b62d8\") " pod="openshift-monitoring/thanos-querier-7444f7bc55-qltlp"
Jul 27 20:53:05.757153 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:05.757125    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/39155db8-f2d3-4a9d-8ef8-ed19f16b62d8-metrics-client-ca\") pod \"thanos-querier-7444f7bc55-qltlp\" (UID: \"39155db8-f2d3-4a9d-8ef8-ed19f16b62d8\") " pod="openshift-monitoring/thanos-querier-7444f7bc55-qltlp"
Jul 27 20:53:05.757247 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:05.757175    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/39155db8-f2d3-4a9d-8ef8-ed19f16b62d8-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7444f7bc55-qltlp\" (UID: \"39155db8-f2d3-4a9d-8ef8-ed19f16b62d8\") " pod="openshift-monitoring/thanos-querier-7444f7bc55-qltlp"
Jul 27 20:53:05.757247 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:05.757199    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/39155db8-f2d3-4a9d-8ef8-ed19f16b62d8-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7444f7bc55-qltlp\" (UID: \"39155db8-f2d3-4a9d-8ef8-ed19f16b62d8\") " pod="openshift-monitoring/thanos-querier-7444f7bc55-qltlp"
Jul 27 20:53:05.757247 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:05.757219    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/39155db8-f2d3-4a9d-8ef8-ed19f16b62d8-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7444f7bc55-qltlp\" (UID: \"39155db8-f2d3-4a9d-8ef8-ed19f16b62d8\") " pod="openshift-monitoring/thanos-querier-7444f7bc55-qltlp"
Jul 27 20:53:05.757247 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:05.757236    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/39155db8-f2d3-4a9d-8ef8-ed19f16b62d8-secret-grpc-tls\") pod \"thanos-querier-7444f7bc55-qltlp\" (UID: \"39155db8-f2d3-4a9d-8ef8-ed19f16b62d8\") " pod="openshift-monitoring/thanos-querier-7444f7bc55-qltlp"
Jul 27 20:53:05.858184 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:05.858153    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/39155db8-f2d3-4a9d-8ef8-ed19f16b62d8-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7444f7bc55-qltlp\" (UID: \"39155db8-f2d3-4a9d-8ef8-ed19f16b62d8\") " pod="openshift-monitoring/thanos-querier-7444f7bc55-qltlp"
Jul 27 20:53:05.858318 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:05.858206    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64grv\" (UniqueName: \"kubernetes.io/projected/39155db8-f2d3-4a9d-8ef8-ed19f16b62d8-kube-api-access-64grv\") pod \"thanos-querier-7444f7bc55-qltlp\" (UID: \"39155db8-f2d3-4a9d-8ef8-ed19f16b62d8\") " pod="openshift-monitoring/thanos-querier-7444f7bc55-qltlp"
Jul 27 20:53:05.858318 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:05.858255    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/39155db8-f2d3-4a9d-8ef8-ed19f16b62d8-secret-thanos-querier-tls\") pod \"thanos-querier-7444f7bc55-qltlp\" (UID: \"39155db8-f2d3-4a9d-8ef8-ed19f16b62d8\") " pod="openshift-monitoring/thanos-querier-7444f7bc55-qltlp"
Jul 27 20:53:05.858318 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:05.858278    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/39155db8-f2d3-4a9d-8ef8-ed19f16b62d8-metrics-client-ca\") pod \"thanos-querier-7444f7bc55-qltlp\" (UID: \"39155db8-f2d3-4a9d-8ef8-ed19f16b62d8\") " pod="openshift-monitoring/thanos-querier-7444f7bc55-qltlp"
Jul 27 20:53:05.858519 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:05.858321    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/39155db8-f2d3-4a9d-8ef8-ed19f16b62d8-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7444f7bc55-qltlp\" (UID: \"39155db8-f2d3-4a9d-8ef8-ed19f16b62d8\") " pod="openshift-monitoring/thanos-querier-7444f7bc55-qltlp"
Jul 27 20:53:05.858519 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:05.858353    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/39155db8-f2d3-4a9d-8ef8-ed19f16b62d8-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7444f7bc55-qltlp\" (UID: \"39155db8-f2d3-4a9d-8ef8-ed19f16b62d8\") " pod="openshift-monitoring/thanos-querier-7444f7bc55-qltlp"
Jul 27 20:53:05.858519 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:05.858378    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/39155db8-f2d3-4a9d-8ef8-ed19f16b62d8-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7444f7bc55-qltlp\" (UID: \"39155db8-f2d3-4a9d-8ef8-ed19f16b62d8\") " pod="openshift-monitoring/thanos-querier-7444f7bc55-qltlp"
Jul 27 20:53:05.858519 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:05.858401    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/39155db8-f2d3-4a9d-8ef8-ed19f16b62d8-secret-grpc-tls\") pod \"thanos-querier-7444f7bc55-qltlp\" (UID: \"39155db8-f2d3-4a9d-8ef8-ed19f16b62d8\") " pod="openshift-monitoring/thanos-querier-7444f7bc55-qltlp"
Jul 27 20:53:05.859133 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:05.859078    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/39155db8-f2d3-4a9d-8ef8-ed19f16b62d8-metrics-client-ca\") pod \"thanos-querier-7444f7bc55-qltlp\" (UID: \"39155db8-f2d3-4a9d-8ef8-ed19f16b62d8\") " pod="openshift-monitoring/thanos-querier-7444f7bc55-qltlp"
Jul 27 20:53:05.861265 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:05.861239    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/39155db8-f2d3-4a9d-8ef8-ed19f16b62d8-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7444f7bc55-qltlp\" (UID: \"39155db8-f2d3-4a9d-8ef8-ed19f16b62d8\") " pod="openshift-monitoring/thanos-querier-7444f7bc55-qltlp"
Jul 27 20:53:05.861380 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:05.861305    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/39155db8-f2d3-4a9d-8ef8-ed19f16b62d8-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7444f7bc55-qltlp\" (UID: \"39155db8-f2d3-4a9d-8ef8-ed19f16b62d8\") " pod="openshift-monitoring/thanos-querier-7444f7bc55-qltlp"
Jul 27 20:53:05.861490 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:05.861469    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/39155db8-f2d3-4a9d-8ef8-ed19f16b62d8-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7444f7bc55-qltlp\" (UID: \"39155db8-f2d3-4a9d-8ef8-ed19f16b62d8\") " pod="openshift-monitoring/thanos-querier-7444f7bc55-qltlp"
Jul 27 20:53:05.861554 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:05.861506    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/39155db8-f2d3-4a9d-8ef8-ed19f16b62d8-secret-grpc-tls\") pod \"thanos-querier-7444f7bc55-qltlp\" (UID: \"39155db8-f2d3-4a9d-8ef8-ed19f16b62d8\") " pod="openshift-monitoring/thanos-querier-7444f7bc55-qltlp"
Jul 27 20:53:05.861605 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:05.861584    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/39155db8-f2d3-4a9d-8ef8-ed19f16b62d8-secret-thanos-querier-tls\") pod \"thanos-querier-7444f7bc55-qltlp\" (UID: \"39155db8-f2d3-4a9d-8ef8-ed19f16b62d8\") " pod="openshift-monitoring/thanos-querier-7444f7bc55-qltlp"
Jul 27 20:53:05.861698 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:05.861677    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/39155db8-f2d3-4a9d-8ef8-ed19f16b62d8-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7444f7bc55-qltlp\" (UID: \"39155db8-f2d3-4a9d-8ef8-ed19f16b62d8\") " pod="openshift-monitoring/thanos-querier-7444f7bc55-qltlp"
Jul 27 20:53:05.866988 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:05.866971    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-64grv\" (UniqueName: \"kubernetes.io/projected/39155db8-f2d3-4a9d-8ef8-ed19f16b62d8-kube-api-access-64grv\") pod \"thanos-querier-7444f7bc55-qltlp\" (UID: \"39155db8-f2d3-4a9d-8ef8-ed19f16b62d8\") " pod="openshift-monitoring/thanos-querier-7444f7bc55-qltlp"
Jul 27 20:53:06.000767 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:06.000745    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7444f7bc55-qltlp"
Jul 27 20:53:06.171915 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:06.171878    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7444f7bc55-qltlp"]
Jul 27 20:53:06.174959 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:53:06.174930    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39155db8_f2d3_4a9d_8ef8_ed19f16b62d8.slice/crio-4e69d58fe0a8d0803d1d18cafbf9e0bdb6293fd56f830693f5b89866f2891091 WatchSource:0}: Error finding container 4e69d58fe0a8d0803d1d18cafbf9e0bdb6293fd56f830693f5b89866f2891091: Status 404 returned error can't find the container with id 4e69d58fe0a8d0803d1d18cafbf9e0bdb6293fd56f830693f5b89866f2891091
Jul 27 20:53:06.228859 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:06.228828    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-dbzwm"
Jul 27 20:53:06.350920 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:06.350883    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mb7xd" event={"ID":"1c5f5fd5-7bf8-4d82-9c39-ca9e923ec6b4","Type":"ContainerStarted","Data":"b18fb17eacb6b2d029809f07e2df9dd9c32fa13e9eecfc8fa6b87832654feef4"}
Jul 27 20:53:06.352100 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:06.352058    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7444f7bc55-qltlp" event={"ID":"39155db8-f2d3-4a9d-8ef8-ed19f16b62d8","Type":"ContainerStarted","Data":"4e69d58fe0a8d0803d1d18cafbf9e0bdb6293fd56f830693f5b89866f2891091"}
Jul 27 20:53:06.353459 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:06.353432    2560 generic.go:356] "Generic (PLEG): container finished" podID="3e1e5615-3312-4080-bbfa-2a92c24bb888" containerID="0b475c5dc416babed45fa3f0a68c50ec35bb923cb88477a7d2f691b96be70c00" exitCode=0
Jul 27 20:53:06.353561 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:06.353518    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mj6h5" event={"ID":"3e1e5615-3312-4080-bbfa-2a92c24bb888","Type":"ContainerDied","Data":"0b475c5dc416babed45fa3f0a68c50ec35bb923cb88477a7d2f691b96be70c00"}
Jul 27 20:53:06.383164 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:06.383121    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-mb7xd" podStartSLOduration=33.520158777 podStartE2EDuration="47.383103852s" totalImagesPullingTime="13.862945075s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 20:52:19 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:53:06.382474089 +0000 UTC m=+103.134335960" watchObservedRunningTime="2026-07-27 20:53:06.383103852 +0000 UTC m=+103.134965642"
Jul 27 20:53:07.360255 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:07.360219    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mj6h5" event={"ID":"3e1e5615-3312-4080-bbfa-2a92c24bb888","Type":"ContainerStarted","Data":"bf392d12d582473f9ac55faf611289c3dd913f2bac82f33d022eb1f31f0d66e7"}
Jul 27 20:53:07.360638 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:07.360286    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mj6h5" event={"ID":"3e1e5615-3312-4080-bbfa-2a92c24bb888","Type":"ContainerStarted","Data":"a505b07d05aae573332691ca787ef2bfd25bfc7a88eebd44d7d6e6c2c70c594a"}
Jul 27 20:53:07.382469 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:07.382424    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-mj6h5" podStartSLOduration=2.133694942 podStartE2EDuration="5.382409623s" totalImagesPullingTime="3.140635246s" totalInitContainerRuntime="108.079435ms" isStatefulPod=true podCreationTimestamp="2026-07-27 20:53:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:53:07.381842046 +0000 UTC m=+104.133703841" watchObservedRunningTime="2026-07-27 20:53:07.382409623 +0000 UTC m=+104.134271419"
Jul 27 20:53:08.365303 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:08.365266    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7444f7bc55-qltlp" event={"ID":"39155db8-f2d3-4a9d-8ef8-ed19f16b62d8","Type":"ContainerStarted","Data":"8642eb987317686f45db97f211b6fafc3fc179491337d4efa5c4056a89ff5fa7"}
Jul 27 20:53:08.365700 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:08.365341    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7444f7bc55-qltlp" event={"ID":"39155db8-f2d3-4a9d-8ef8-ed19f16b62d8","Type":"ContainerStarted","Data":"a04df98356a0a1b5b52350c5bae5117724482f53d43133999fbf0a6216849d62"}
Jul 27 20:53:08.365700 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:08.365378    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7444f7bc55-qltlp" event={"ID":"39155db8-f2d3-4a9d-8ef8-ed19f16b62d8","Type":"ContainerStarted","Data":"338c810854f31d57588a67c0baa660ab831669714a5ed731d45fc1a9993fe841"}
Jul 27 20:53:09.371841 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:09.371797    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7444f7bc55-qltlp" event={"ID":"39155db8-f2d3-4a9d-8ef8-ed19f16b62d8","Type":"ContainerStarted","Data":"2934f680988cadb8104b2a3bb219707ff9adf9b3efeaebf086d10530a4e112a5"}
Jul 27 20:53:09.372339 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:09.371912    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7444f7bc55-qltlp" event={"ID":"39155db8-f2d3-4a9d-8ef8-ed19f16b62d8","Type":"ContainerStarted","Data":"0a37782b228ed5f3b60188386e733d1ae913ce61ad4c4fbe2f66c3a4941779cf"}
Jul 27 20:53:09.372339 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:09.371958    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7444f7bc55-qltlp" event={"ID":"39155db8-f2d3-4a9d-8ef8-ed19f16b62d8","Type":"ContainerStarted","Data":"14aa28eee3c5affa096c4157bbfde35c1ded60f76e817c8b4b74c5cefa7e3b09"}
Jul 27 20:53:09.372339 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:09.371998    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-7444f7bc55-qltlp"
Jul 27 20:53:09.399624 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:09.399565    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-7444f7bc55-qltlp" podStartSLOduration=1.946593937 podStartE2EDuration="4.399551425s" totalImagesPullingTime="2.452957488s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:53:05 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:53:09.398271659 +0000 UTC m=+106.150133478" watchObservedRunningTime="2026-07-27 20:53:09.399551425 +0000 UTC m=+106.151413220"
Jul 27 20:53:12.952026 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:12.951983    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-65494b8dd8-cvl2s" podUID="0bf98545-754e-4371-a262-d78b44f0361d" containerName="registry" containerID="cri-o://9be5d53352bdb73378eba1ae629c87ecf293c32bcd582892600538af684d0005" gracePeriod=30
Jul 27 20:53:13.188510 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:13.188487    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-65494b8dd8-cvl2s"
Jul 27 20:53:13.315393 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:13.315348    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0bf98545-754e-4371-a262-d78b44f0361d-image-registry-private-configuration\") pod \"0bf98545-754e-4371-a262-d78b44f0361d\" (UID: \"0bf98545-754e-4371-a262-d78b44f0361d\") "
Jul 27 20:53:13.315393 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:13.315402    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-68nt5\" (UniqueName: \"kubernetes.io/projected/0bf98545-754e-4371-a262-d78b44f0361d-kube-api-access-68nt5\") pod \"0bf98545-754e-4371-a262-d78b44f0361d\" (UID: \"0bf98545-754e-4371-a262-d78b44f0361d\") "
Jul 27 20:53:13.315670 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:13.315426    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0bf98545-754e-4371-a262-d78b44f0361d-bound-sa-token\") pod \"0bf98545-754e-4371-a262-d78b44f0361d\" (UID: \"0bf98545-754e-4371-a262-d78b44f0361d\") "
Jul 27 20:53:13.315670 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:13.315463    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0bf98545-754e-4371-a262-d78b44f0361d-ca-trust-extracted\") pod \"0bf98545-754e-4371-a262-d78b44f0361d\" (UID: \"0bf98545-754e-4371-a262-d78b44f0361d\") "
Jul 27 20:53:13.315670 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:13.315482    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0bf98545-754e-4371-a262-d78b44f0361d-installation-pull-secrets\") pod \"0bf98545-754e-4371-a262-d78b44f0361d\" (UID: \"0bf98545-754e-4371-a262-d78b44f0361d\") "
Jul 27 20:53:13.315670 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:13.315503    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0bf98545-754e-4371-a262-d78b44f0361d-registry-certificates\") pod \"0bf98545-754e-4371-a262-d78b44f0361d\" (UID: \"0bf98545-754e-4371-a262-d78b44f0361d\") "
Jul 27 20:53:13.315670 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:13.315570    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0bf98545-754e-4371-a262-d78b44f0361d-trusted-ca\") pod \"0bf98545-754e-4371-a262-d78b44f0361d\" (UID: \"0bf98545-754e-4371-a262-d78b44f0361d\") "
Jul 27 20:53:13.315670 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:13.315607    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0bf98545-754e-4371-a262-d78b44f0361d-registry-tls\") pod \"0bf98545-754e-4371-a262-d78b44f0361d\" (UID: \"0bf98545-754e-4371-a262-d78b44f0361d\") "
Jul 27 20:53:13.316156 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:13.316071    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bf98545-754e-4371-a262-d78b44f0361d-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "0bf98545-754e-4371-a262-d78b44f0361d" (UID: "0bf98545-754e-4371-a262-d78b44f0361d"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:53:13.316419 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:13.316391    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bf98545-754e-4371-a262-d78b44f0361d-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "0bf98545-754e-4371-a262-d78b44f0361d" (UID: "0bf98545-754e-4371-a262-d78b44f0361d"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:53:13.318305 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:13.318245    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0bf98545-754e-4371-a262-d78b44f0361d-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "0bf98545-754e-4371-a262-d78b44f0361d" (UID: "0bf98545-754e-4371-a262-d78b44f0361d"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:53:13.318499 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:13.318442    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bf98545-754e-4371-a262-d78b44f0361d-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "0bf98545-754e-4371-a262-d78b44f0361d" (UID: "0bf98545-754e-4371-a262-d78b44f0361d"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:53:13.318620 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:13.318498    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bf98545-754e-4371-a262-d78b44f0361d-kube-api-access-68nt5" (OuterVolumeSpecName: "kube-api-access-68nt5") pod "0bf98545-754e-4371-a262-d78b44f0361d" (UID: "0bf98545-754e-4371-a262-d78b44f0361d"). InnerVolumeSpecName "kube-api-access-68nt5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:53:13.318687 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:13.318633    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bf98545-754e-4371-a262-d78b44f0361d-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "0bf98545-754e-4371-a262-d78b44f0361d" (UID: "0bf98545-754e-4371-a262-d78b44f0361d"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:53:13.318751 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:13.318728    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0bf98545-754e-4371-a262-d78b44f0361d-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "0bf98545-754e-4371-a262-d78b44f0361d" (UID: "0bf98545-754e-4371-a262-d78b44f0361d"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:53:13.324420 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:13.324394    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bf98545-754e-4371-a262-d78b44f0361d-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "0bf98545-754e-4371-a262-d78b44f0361d" (UID: "0bf98545-754e-4371-a262-d78b44f0361d"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 20:53:13.385499 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:13.385461    2560 generic.go:356] "Generic (PLEG): container finished" podID="0bf98545-754e-4371-a262-d78b44f0361d" containerID="9be5d53352bdb73378eba1ae629c87ecf293c32bcd582892600538af684d0005" exitCode=0
Jul 27 20:53:13.385624 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:13.385509    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-65494b8dd8-cvl2s"
Jul 27 20:53:13.386150 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:13.385871    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-65494b8dd8-cvl2s" event={"ID":"0bf98545-754e-4371-a262-d78b44f0361d","Type":"ContainerDied","Data":"9be5d53352bdb73378eba1ae629c87ecf293c32bcd582892600538af684d0005"}
Jul 27 20:53:13.386150 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:13.385939    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-65494b8dd8-cvl2s" event={"ID":"0bf98545-754e-4371-a262-d78b44f0361d","Type":"ContainerDied","Data":"832c866fee0cd8e35ac64df78c7c247551badb458abf540455e499a0fc2fd138"}
Jul 27 20:53:13.386150 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:13.385967    2560 scope.go:117] "RemoveContainer" containerID="9be5d53352bdb73378eba1ae629c87ecf293c32bcd582892600538af684d0005"
Jul 27 20:53:13.395556 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:13.395534    2560 scope.go:117] "RemoveContainer" containerID="9be5d53352bdb73378eba1ae629c87ecf293c32bcd582892600538af684d0005"
Jul 27 20:53:13.395883 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:53:13.395861    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9be5d53352bdb73378eba1ae629c87ecf293c32bcd582892600538af684d0005\": container with ID starting with 9be5d53352bdb73378eba1ae629c87ecf293c32bcd582892600538af684d0005 not found: ID does not exist" containerID="9be5d53352bdb73378eba1ae629c87ecf293c32bcd582892600538af684d0005"
Jul 27 20:53:13.395954 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:13.395890    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9be5d53352bdb73378eba1ae629c87ecf293c32bcd582892600538af684d0005"} err="failed to get container status \"9be5d53352bdb73378eba1ae629c87ecf293c32bcd582892600538af684d0005\": rpc error: code = NotFound desc = could not find container \"9be5d53352bdb73378eba1ae629c87ecf293c32bcd582892600538af684d0005\": container with ID starting with 9be5d53352bdb73378eba1ae629c87ecf293c32bcd582892600538af684d0005 not found: ID does not exist"
Jul 27 20:53:13.408446 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:13.408426    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-65494b8dd8-cvl2s"]
Jul 27 20:53:13.414292 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:13.414273    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-65494b8dd8-cvl2s"]
Jul 27 20:53:13.416438 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:13.416422    2560 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0bf98545-754e-4371-a262-d78b44f0361d-image-registry-private-configuration\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 20:53:13.416497 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:13.416442    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68nt5\" (UniqueName: \"kubernetes.io/projected/0bf98545-754e-4371-a262-d78b44f0361d-kube-api-access-68nt5\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 20:53:13.416497 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:13.416453    2560 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0bf98545-754e-4371-a262-d78b44f0361d-bound-sa-token\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 20:53:13.416497 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:13.416463    2560 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0bf98545-754e-4371-a262-d78b44f0361d-ca-trust-extracted\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 20:53:13.416497 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:13.416472    2560 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0bf98545-754e-4371-a262-d78b44f0361d-installation-pull-secrets\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 20:53:13.416497 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:13.416480    2560 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0bf98545-754e-4371-a262-d78b44f0361d-registry-certificates\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 20:53:13.416497 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:13.416489    2560 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0bf98545-754e-4371-a262-d78b44f0361d-trusted-ca\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 20:53:13.416497 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:13.416497    2560 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0bf98545-754e-4371-a262-d78b44f0361d-registry-tls\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 20:53:13.846685 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:13.846653    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0bf98545-754e-4371-a262-d78b44f0361d" path="/var/lib/kubelet/pods/0bf98545-754e-4371-a262-d78b44f0361d/volumes"
Jul 27 20:53:15.380692 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:15.380664    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-7444f7bc55-qltlp"
Jul 27 20:53:28.678752 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:28.678713    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7b54c855d7-gwj7h"]
Jul 27 20:53:33.670866 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:33.670822    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fac6ee6e-225a-4402-8834-69ff744b07f9-metrics-certs\") pod \"network-metrics-daemon-r6g8d\" (UID: \"fac6ee6e-225a-4402-8834-69ff744b07f9\") " pod="openshift-multus/network-metrics-daemon-r6g8d"
Jul 27 20:53:33.673243 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:33.673214    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fac6ee6e-225a-4402-8834-69ff744b07f9-metrics-certs\") pod \"network-metrics-daemon-r6g8d\" (UID: \"fac6ee6e-225a-4402-8834-69ff744b07f9\") " pod="openshift-multus/network-metrics-daemon-r6g8d"
Jul 27 20:53:33.866772 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:33.866739    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-bvgk8\""
Jul 27 20:53:33.872938 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:33.872915    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r6g8d"
Jul 27 20:53:34.029382 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:34.029264    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-r6g8d"]
Jul 27 20:53:34.031791 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:53:34.031764    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfac6ee6e_225a_4402_8834_69ff744b07f9.slice/crio-4c325a1afac42b7a238e8bf331fb3280dbdf1b24352dd4f72011c8a3ccc7ac94 WatchSource:0}: Error finding container 4c325a1afac42b7a238e8bf331fb3280dbdf1b24352dd4f72011c8a3ccc7ac94: Status 404 returned error can't find the container with id 4c325a1afac42b7a238e8bf331fb3280dbdf1b24352dd4f72011c8a3ccc7ac94
Jul 27 20:53:34.457234 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:34.457198    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-r6g8d" event={"ID":"fac6ee6e-225a-4402-8834-69ff744b07f9","Type":"ContainerStarted","Data":"4c325a1afac42b7a238e8bf331fb3280dbdf1b24352dd4f72011c8a3ccc7ac94"}
Jul 27 20:53:36.466184 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:36.466142    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-r6g8d" event={"ID":"fac6ee6e-225a-4402-8834-69ff744b07f9","Type":"ContainerStarted","Data":"663d508e5bcda9d5e6ab10cad8b014d941a56636a29d2a2cc39a1061a732aca2"}
Jul 27 20:53:36.466764 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:36.466197    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-r6g8d" event={"ID":"fac6ee6e-225a-4402-8834-69ff744b07f9","Type":"ContainerStarted","Data":"656b7527db4296329d08afa44ee147db8fc73b6753fe399cd6ba2bef5dd7bacc"}
Jul 27 20:53:36.484317 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:36.484271    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-r6g8d" podStartSLOduration=130.977257278 podStartE2EDuration="2m12.484254972s" totalImagesPullingTime="1.506997694s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:51:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:53:36.483474474 +0000 UTC m=+133.235336270" watchObservedRunningTime="2026-07-27 20:53:36.484254972 +0000 UTC m=+133.236116766"
Jul 27 20:53:53.529940 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:53.529902    2560 generic.go:356] "Generic (PLEG): container finished" podID="1da96bd6-098f-41d7-9be9-904d53068b43" containerID="2de532f9dfa34a5d0602c559fac709f5c6e1af638581e07a203522b3bb151804" exitCode=0
Jul 27 20:53:53.530526 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:53.529971    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-7vsbq" event={"ID":"1da96bd6-098f-41d7-9be9-904d53068b43","Type":"ContainerDied","Data":"2de532f9dfa34a5d0602c559fac709f5c6e1af638581e07a203522b3bb151804"}
Jul 27 20:53:53.530526 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:53.530313    2560 scope.go:117] "RemoveContainer" containerID="2de532f9dfa34a5d0602c559fac709f5c6e1af638581e07a203522b3bb151804"
Jul 27 20:53:53.699348 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:53.699276    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7b54c855d7-gwj7h" podUID="9a73773c-200d-4997-b068-7bcbe585030b" containerName="console" containerID="cri-o://836f92d4ff2fa654dbfd0c3137c54b11ae4247e1f16716c73d43606bb2a856a7" gracePeriod=15
Jul 27 20:53:53.947244 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:53.947221    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b54c855d7-gwj7h"
Jul 27 20:53:54.023694 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:54.023665    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9a73773c-200d-4997-b068-7bcbe585030b-oauth-serving-cert\") pod \"9a73773c-200d-4997-b068-7bcbe585030b\" (UID: \"9a73773c-200d-4997-b068-7bcbe585030b\") "
Jul 27 20:53:54.023866 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:54.023731    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9a73773c-200d-4997-b068-7bcbe585030b-console-oauth-config\") pod \"9a73773c-200d-4997-b068-7bcbe585030b\" (UID: \"9a73773c-200d-4997-b068-7bcbe585030b\") "
Jul 27 20:53:54.023932 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:54.023873    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9a73773c-200d-4997-b068-7bcbe585030b-console-config\") pod \"9a73773c-200d-4997-b068-7bcbe585030b\" (UID: \"9a73773c-200d-4997-b068-7bcbe585030b\") "
Jul 27 20:53:54.023932 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:54.023914    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9a73773c-200d-4997-b068-7bcbe585030b-service-ca\") pod \"9a73773c-200d-4997-b068-7bcbe585030b\" (UID: \"9a73773c-200d-4997-b068-7bcbe585030b\") "
Jul 27 20:53:54.024040 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:54.023938    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-246fn\" (UniqueName: \"kubernetes.io/projected/9a73773c-200d-4997-b068-7bcbe585030b-kube-api-access-246fn\") pod \"9a73773c-200d-4997-b068-7bcbe585030b\" (UID: \"9a73773c-200d-4997-b068-7bcbe585030b\") "
Jul 27 20:53:54.024040 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:54.023980    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9a73773c-200d-4997-b068-7bcbe585030b-console-serving-cert\") pod \"9a73773c-200d-4997-b068-7bcbe585030b\" (UID: \"9a73773c-200d-4997-b068-7bcbe585030b\") "
Jul 27 20:53:54.024214 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:54.024189    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a73773c-200d-4997-b068-7bcbe585030b-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "9a73773c-200d-4997-b068-7bcbe585030b" (UID: "9a73773c-200d-4997-b068-7bcbe585030b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:53:54.024290 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:54.024263    2560 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9a73773c-200d-4997-b068-7bcbe585030b-oauth-serving-cert\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 20:53:54.024290 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:54.024264    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a73773c-200d-4997-b068-7bcbe585030b-service-ca" (OuterVolumeSpecName: "service-ca") pod "9a73773c-200d-4997-b068-7bcbe585030b" (UID: "9a73773c-200d-4997-b068-7bcbe585030b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:53:54.024366 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:54.024276    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a73773c-200d-4997-b068-7bcbe585030b-console-config" (OuterVolumeSpecName: "console-config") pod "9a73773c-200d-4997-b068-7bcbe585030b" (UID: "9a73773c-200d-4997-b068-7bcbe585030b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 20:53:54.026109 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:54.026068    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a73773c-200d-4997-b068-7bcbe585030b-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "9a73773c-200d-4997-b068-7bcbe585030b" (UID: "9a73773c-200d-4997-b068-7bcbe585030b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:53:54.026292 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:54.026265    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a73773c-200d-4997-b068-7bcbe585030b-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "9a73773c-200d-4997-b068-7bcbe585030b" (UID: "9a73773c-200d-4997-b068-7bcbe585030b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 20:53:54.026390 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:54.026331    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a73773c-200d-4997-b068-7bcbe585030b-kube-api-access-246fn" (OuterVolumeSpecName: "kube-api-access-246fn") pod "9a73773c-200d-4997-b068-7bcbe585030b" (UID: "9a73773c-200d-4997-b068-7bcbe585030b"). InnerVolumeSpecName "kube-api-access-246fn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 20:53:54.125494 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:54.125415    2560 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9a73773c-200d-4997-b068-7bcbe585030b-console-oauth-config\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 20:53:54.125494 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:54.125442    2560 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9a73773c-200d-4997-b068-7bcbe585030b-console-config\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 20:53:54.125494 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:54.125451    2560 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9a73773c-200d-4997-b068-7bcbe585030b-service-ca\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 20:53:54.125494 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:54.125458    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-246fn\" (UniqueName: \"kubernetes.io/projected/9a73773c-200d-4997-b068-7bcbe585030b-kube-api-access-246fn\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 20:53:54.125494 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:54.125467    2560 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9a73773c-200d-4997-b068-7bcbe585030b-console-serving-cert\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 20:53:54.535416 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:54.535382    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-7vsbq" event={"ID":"1da96bd6-098f-41d7-9be9-904d53068b43","Type":"ContainerStarted","Data":"4d16b4ff01bb31b85a7fc22834cb1b6635b576605e89d70434040525accf9984"}
Jul 27 20:53:54.536676 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:54.536651    2560 generic.go:356] "Generic (PLEG): container finished" podID="9a73773c-200d-4997-b068-7bcbe585030b" containerID="836f92d4ff2fa654dbfd0c3137c54b11ae4247e1f16716c73d43606bb2a856a7" exitCode=0
Jul 27 20:53:54.536777 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:54.536693    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b54c855d7-gwj7h"
Jul 27 20:53:54.536777 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:54.536689    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b54c855d7-gwj7h" event={"ID":"9a73773c-200d-4997-b068-7bcbe585030b","Type":"ContainerDied","Data":"836f92d4ff2fa654dbfd0c3137c54b11ae4247e1f16716c73d43606bb2a856a7"}
Jul 27 20:53:54.536849 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:54.536804    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b54c855d7-gwj7h" event={"ID":"9a73773c-200d-4997-b068-7bcbe585030b","Type":"ContainerDied","Data":"bd2418a7f24340d98b36a40f5d55b75d6298432805baad1fb8a5027ddb7eb1c5"}
Jul 27 20:53:54.536849 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:54.536823    2560 scope.go:117] "RemoveContainer" containerID="836f92d4ff2fa654dbfd0c3137c54b11ae4247e1f16716c73d43606bb2a856a7"
Jul 27 20:53:54.545019 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:54.545004    2560 scope.go:117] "RemoveContainer" containerID="836f92d4ff2fa654dbfd0c3137c54b11ae4247e1f16716c73d43606bb2a856a7"
Jul 27 20:53:54.545269 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:53:54.545248    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"836f92d4ff2fa654dbfd0c3137c54b11ae4247e1f16716c73d43606bb2a856a7\": container with ID starting with 836f92d4ff2fa654dbfd0c3137c54b11ae4247e1f16716c73d43606bb2a856a7 not found: ID does not exist" containerID="836f92d4ff2fa654dbfd0c3137c54b11ae4247e1f16716c73d43606bb2a856a7"
Jul 27 20:53:54.545337 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:54.545273    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"836f92d4ff2fa654dbfd0c3137c54b11ae4247e1f16716c73d43606bb2a856a7"} err="failed to get container status \"836f92d4ff2fa654dbfd0c3137c54b11ae4247e1f16716c73d43606bb2a856a7\": rpc error: code = NotFound desc = could not find container \"836f92d4ff2fa654dbfd0c3137c54b11ae4247e1f16716c73d43606bb2a856a7\": container with ID starting with 836f92d4ff2fa654dbfd0c3137c54b11ae4247e1f16716c73d43606bb2a856a7 not found: ID does not exist"
Jul 27 20:53:54.569220 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:54.569198    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7b54c855d7-gwj7h"]
Jul 27 20:53:54.575501 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:54.575479    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7b54c855d7-gwj7h"]
Jul 27 20:53:55.845842 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:53:55.845808    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a73773c-200d-4997-b068-7bcbe585030b" path="/var/lib/kubelet/pods/9a73773c-200d-4997-b068-7bcbe585030b/volumes"
Jul 27 20:55:49.721971 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:55:49.721935    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-768cd866db-7n9mq"]
Jul 27 20:55:49.722455 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:55:49.722189    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a73773c-200d-4997-b068-7bcbe585030b" containerName="console"
Jul 27 20:55:49.722455 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:55:49.722200    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a73773c-200d-4997-b068-7bcbe585030b" containerName="console"
Jul 27 20:55:49.722455 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:55:49.722218    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0bf98545-754e-4371-a262-d78b44f0361d" containerName="registry"
Jul 27 20:55:49.722455 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:55:49.722223    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="0bf98545-754e-4371-a262-d78b44f0361d" containerName="registry"
Jul 27 20:55:49.722455 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:55:49.722263    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a73773c-200d-4997-b068-7bcbe585030b" containerName="console"
Jul 27 20:55:49.722455 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:55:49.722273    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="0bf98545-754e-4371-a262-d78b44f0361d" containerName="registry"
Jul 27 20:55:49.725063 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:55:49.725047    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-768cd866db-7n9mq"
Jul 27 20:55:49.728825 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:55:49.728802    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 27 20:55:49.728922 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:55:49.728802    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-nr66h\""
Jul 27 20:55:49.729733 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:55:49.729710    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 27 20:55:49.729850 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:55:49.729718    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 27 20:55:49.729929 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:55:49.729917    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 27 20:55:49.729978 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:55:49.729919    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 27 20:55:49.729978 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:55:49.729964    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 27 20:55:49.730759 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:55:49.730741    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 27 20:55:49.738261 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:55:49.738238    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 27 20:55:49.739144 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:55:49.739127    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-768cd866db-7n9mq"]
Jul 27 20:55:49.829525 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:55:49.829495    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6b851a82-227d-49cf-a910-ecbc2b59ecac-service-ca\") pod \"console-768cd866db-7n9mq\" (UID: \"6b851a82-227d-49cf-a910-ecbc2b59ecac\") " pod="openshift-console/console-768cd866db-7n9mq"
Jul 27 20:55:49.829525 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:55:49.829529    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6b851a82-227d-49cf-a910-ecbc2b59ecac-console-oauth-config\") pod \"console-768cd866db-7n9mq\" (UID: \"6b851a82-227d-49cf-a910-ecbc2b59ecac\") " pod="openshift-console/console-768cd866db-7n9mq"
Jul 27 20:55:49.829706 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:55:49.829549    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnj8r\" (UniqueName: \"kubernetes.io/projected/6b851a82-227d-49cf-a910-ecbc2b59ecac-kube-api-access-cnj8r\") pod \"console-768cd866db-7n9mq\" (UID: \"6b851a82-227d-49cf-a910-ecbc2b59ecac\") " pod="openshift-console/console-768cd866db-7n9mq"
Jul 27 20:55:49.829706 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:55:49.829630    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6b851a82-227d-49cf-a910-ecbc2b59ecac-console-serving-cert\") pod \"console-768cd866db-7n9mq\" (UID: \"6b851a82-227d-49cf-a910-ecbc2b59ecac\") " pod="openshift-console/console-768cd866db-7n9mq"
Jul 27 20:55:49.829706 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:55:49.829657    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6b851a82-227d-49cf-a910-ecbc2b59ecac-console-config\") pod \"console-768cd866db-7n9mq\" (UID: \"6b851a82-227d-49cf-a910-ecbc2b59ecac\") " pod="openshift-console/console-768cd866db-7n9mq"
Jul 27 20:55:49.829706 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:55:49.829685    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6b851a82-227d-49cf-a910-ecbc2b59ecac-oauth-serving-cert\") pod \"console-768cd866db-7n9mq\" (UID: \"6b851a82-227d-49cf-a910-ecbc2b59ecac\") " pod="openshift-console/console-768cd866db-7n9mq"
Jul 27 20:55:49.829848 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:55:49.829712    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b851a82-227d-49cf-a910-ecbc2b59ecac-trusted-ca-bundle\") pod \"console-768cd866db-7n9mq\" (UID: \"6b851a82-227d-49cf-a910-ecbc2b59ecac\") " pod="openshift-console/console-768cd866db-7n9mq"
Jul 27 20:55:49.930352 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:55:49.930323    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6b851a82-227d-49cf-a910-ecbc2b59ecac-service-ca\") pod \"console-768cd866db-7n9mq\" (UID: \"6b851a82-227d-49cf-a910-ecbc2b59ecac\") " pod="openshift-console/console-768cd866db-7n9mq"
Jul 27 20:55:49.930352 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:55:49.930357    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6b851a82-227d-49cf-a910-ecbc2b59ecac-console-oauth-config\") pod \"console-768cd866db-7n9mq\" (UID: \"6b851a82-227d-49cf-a910-ecbc2b59ecac\") " pod="openshift-console/console-768cd866db-7n9mq"
Jul 27 20:55:49.930581 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:55:49.930376    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnj8r\" (UniqueName: \"kubernetes.io/projected/6b851a82-227d-49cf-a910-ecbc2b59ecac-kube-api-access-cnj8r\") pod \"console-768cd866db-7n9mq\" (UID: \"6b851a82-227d-49cf-a910-ecbc2b59ecac\") " pod="openshift-console/console-768cd866db-7n9mq"
Jul 27 20:55:49.930581 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:55:49.930405    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6b851a82-227d-49cf-a910-ecbc2b59ecac-console-serving-cert\") pod \"console-768cd866db-7n9mq\" (UID: \"6b851a82-227d-49cf-a910-ecbc2b59ecac\") " pod="openshift-console/console-768cd866db-7n9mq"
Jul 27 20:55:49.930581 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:55:49.930423    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6b851a82-227d-49cf-a910-ecbc2b59ecac-console-config\") pod \"console-768cd866db-7n9mq\" (UID: \"6b851a82-227d-49cf-a910-ecbc2b59ecac\") " pod="openshift-console/console-768cd866db-7n9mq"
Jul 27 20:55:49.930581 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:55:49.930452    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6b851a82-227d-49cf-a910-ecbc2b59ecac-oauth-serving-cert\") pod \"console-768cd866db-7n9mq\" (UID: \"6b851a82-227d-49cf-a910-ecbc2b59ecac\") " pod="openshift-console/console-768cd866db-7n9mq"
Jul 27 20:55:49.930581 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:55:49.930470    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b851a82-227d-49cf-a910-ecbc2b59ecac-trusted-ca-bundle\") pod \"console-768cd866db-7n9mq\" (UID: \"6b851a82-227d-49cf-a910-ecbc2b59ecac\") " pod="openshift-console/console-768cd866db-7n9mq"
Jul 27 20:55:49.931138 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:55:49.931117    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6b851a82-227d-49cf-a910-ecbc2b59ecac-service-ca\") pod \"console-768cd866db-7n9mq\" (UID: \"6b851a82-227d-49cf-a910-ecbc2b59ecac\") " pod="openshift-console/console-768cd866db-7n9mq"
Jul 27 20:55:49.931244 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:55:49.931202    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6b851a82-227d-49cf-a910-ecbc2b59ecac-console-config\") pod \"console-768cd866db-7n9mq\" (UID: \"6b851a82-227d-49cf-a910-ecbc2b59ecac\") " pod="openshift-console/console-768cd866db-7n9mq"
Jul 27 20:55:49.931244 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:55:49.931202    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6b851a82-227d-49cf-a910-ecbc2b59ecac-oauth-serving-cert\") pod \"console-768cd866db-7n9mq\" (UID: \"6b851a82-227d-49cf-a910-ecbc2b59ecac\") " pod="openshift-console/console-768cd866db-7n9mq"
Jul 27 20:55:49.931642 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:55:49.931621    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b851a82-227d-49cf-a910-ecbc2b59ecac-trusted-ca-bundle\") pod \"console-768cd866db-7n9mq\" (UID: \"6b851a82-227d-49cf-a910-ecbc2b59ecac\") " pod="openshift-console/console-768cd866db-7n9mq"
Jul 27 20:55:49.933066 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:55:49.933046    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6b851a82-227d-49cf-a910-ecbc2b59ecac-console-oauth-config\") pod \"console-768cd866db-7n9mq\" (UID: \"6b851a82-227d-49cf-a910-ecbc2b59ecac\") " pod="openshift-console/console-768cd866db-7n9mq"
Jul 27 20:55:49.933198 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:55:49.933183    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6b851a82-227d-49cf-a910-ecbc2b59ecac-console-serving-cert\") pod \"console-768cd866db-7n9mq\" (UID: \"6b851a82-227d-49cf-a910-ecbc2b59ecac\") " pod="openshift-console/console-768cd866db-7n9mq"
Jul 27 20:55:49.939350 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:55:49.939331    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnj8r\" (UniqueName: \"kubernetes.io/projected/6b851a82-227d-49cf-a910-ecbc2b59ecac-kube-api-access-cnj8r\") pod \"console-768cd866db-7n9mq\" (UID: \"6b851a82-227d-49cf-a910-ecbc2b59ecac\") " pod="openshift-console/console-768cd866db-7n9mq"
Jul 27 20:55:50.035359 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:55:50.035331    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-768cd866db-7n9mq"
Jul 27 20:55:50.176725 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:55:50.176693    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-768cd866db-7n9mq"]
Jul 27 20:55:50.180793 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:55:50.180765    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b851a82_227d_49cf_a910_ecbc2b59ecac.slice/crio-c0d78e33fb945c8ce5bf390ec2ca5ed835376ce4f6d9953925eda518433c93df WatchSource:0}: Error finding container c0d78e33fb945c8ce5bf390ec2ca5ed835376ce4f6d9953925eda518433c93df: Status 404 returned error can't find the container with id c0d78e33fb945c8ce5bf390ec2ca5ed835376ce4f6d9953925eda518433c93df
Jul 27 20:55:50.923967 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:55:50.923933    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-768cd866db-7n9mq" event={"ID":"6b851a82-227d-49cf-a910-ecbc2b59ecac","Type":"ContainerStarted","Data":"54b6225f9525b93000db1e6527f8a5a6c5e9619e61ce2943b564bd802ed6b4ec"}
Jul 27 20:55:50.924374 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:55:50.923982    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-768cd866db-7n9mq" event={"ID":"6b851a82-227d-49cf-a910-ecbc2b59ecac","Type":"ContainerStarted","Data":"c0d78e33fb945c8ce5bf390ec2ca5ed835376ce4f6d9953925eda518433c93df"}
Jul 27 20:55:50.951965 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:55:50.951911    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-768cd866db-7n9mq" podStartSLOduration=1.951895843 podStartE2EDuration="1.951895843s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:55:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:55:50.950793934 +0000 UTC m=+267.702655729" watchObservedRunningTime="2026-07-27 20:55:50.951895843 +0000 UTC m=+267.703757638"
Jul 27 20:56:00.035688 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:56:00.035651    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-768cd866db-7n9mq"
Jul 27 20:56:00.036078 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:56:00.035715    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-768cd866db-7n9mq"
Jul 27 20:56:00.040397 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:56:00.040374    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-768cd866db-7n9mq"
Jul 27 20:56:00.961804 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:56:00.961773    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-768cd866db-7n9mq"
Jul 27 20:56:23.660486 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:56:23.660456    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-z6zxv_8a6b7ae6-6c21-453d-99b9-303066d19915/console-operator/1.log"
Jul 27 20:56:23.663473 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:56:23.663446    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-z6zxv_8a6b7ae6-6c21-453d-99b9-303066d19915/console-operator/1.log"
Jul 27 20:56:23.666769 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:56:23.666751    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-95hg7_433c431a-b5bf-4734-a03f-6f0a16895ded/ovn-acl-logging/0.log"
Jul 27 20:56:23.669784 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:56:23.669764    2560 kubelet.go:1648] "Image garbage collection succeeded"
Jul 27 20:56:23.670046 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:56:23.670028    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-95hg7_433c431a-b5bf-4734-a03f-6f0a16895ded/ovn-acl-logging/0.log"
Jul 27 20:57:14.410349 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:57:14.410271    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-hf25t"]
Jul 27 20:57:14.413139 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:57:14.413123    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-hf25t"
Jul 27 20:57:14.415452 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:57:14.415430    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 27 20:57:14.416470 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:57:14.416454    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 27 20:57:14.416559 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:57:14.416488    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-5x5v2\""
Jul 27 20:57:14.421559 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:57:14.421526    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-hf25t"]
Jul 27 20:57:14.429801 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:57:14.429779    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krmw4\" (UniqueName: \"kubernetes.io/projected/6b75195b-ed8f-411f-ae54-5ee692445947-kube-api-access-krmw4\") pod \"cert-manager-cainjector-559cdd8c87-hf25t\" (UID: \"6b75195b-ed8f-411f-ae54-5ee692445947\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-hf25t"
Jul 27 20:57:14.429899 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:57:14.429823    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6b75195b-ed8f-411f-ae54-5ee692445947-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-hf25t\" (UID: \"6b75195b-ed8f-411f-ae54-5ee692445947\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-hf25t"
Jul 27 20:57:14.530761 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:57:14.530733    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6b75195b-ed8f-411f-ae54-5ee692445947-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-hf25t\" (UID: \"6b75195b-ed8f-411f-ae54-5ee692445947\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-hf25t"
Jul 27 20:57:14.530905 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:57:14.530789    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krmw4\" (UniqueName: \"kubernetes.io/projected/6b75195b-ed8f-411f-ae54-5ee692445947-kube-api-access-krmw4\") pod \"cert-manager-cainjector-559cdd8c87-hf25t\" (UID: \"6b75195b-ed8f-411f-ae54-5ee692445947\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-hf25t"
Jul 27 20:57:14.539599 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:57:14.539571    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-krmw4\" (UniqueName: \"kubernetes.io/projected/6b75195b-ed8f-411f-ae54-5ee692445947-kube-api-access-krmw4\") pod \"cert-manager-cainjector-559cdd8c87-hf25t\" (UID: \"6b75195b-ed8f-411f-ae54-5ee692445947\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-hf25t"
Jul 27 20:57:14.539693 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:57:14.539573    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6b75195b-ed8f-411f-ae54-5ee692445947-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-hf25t\" (UID: \"6b75195b-ed8f-411f-ae54-5ee692445947\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-hf25t"
Jul 27 20:57:14.721809 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:57:14.721726    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-hf25t"
Jul 27 20:57:14.853726 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:57:14.853699    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-hf25t"]
Jul 27 20:57:14.856914 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:57:14.856887    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6b75195b_ed8f_411f_ae54_5ee692445947.slice/crio-6a682d3a48d96029b19ee0be169ee842a5ca701eef914f1ac81bc35e09e9ac2e WatchSource:0}: Error finding container 6a682d3a48d96029b19ee0be169ee842a5ca701eef914f1ac81bc35e09e9ac2e: Status 404 returned error can't find the container with id 6a682d3a48d96029b19ee0be169ee842a5ca701eef914f1ac81bc35e09e9ac2e
Jul 27 20:57:14.858771 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:57:14.858756    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 20:57:15.212842 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:57:15.212807    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-hf25t" event={"ID":"6b75195b-ed8f-411f-ae54-5ee692445947","Type":"ContainerStarted","Data":"6a682d3a48d96029b19ee0be169ee842a5ca701eef914f1ac81bc35e09e9ac2e"}
Jul 27 20:57:18.058812 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:57:18.058754    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-hqdrq"]
Jul 27 20:57:18.062203 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:57:18.062182    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-hqdrq"
Jul 27 20:57:18.064664 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:57:18.064646    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-kr4tk\""
Jul 27 20:57:18.074286 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:57:18.074249    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-hqdrq"]
Jul 27 20:57:18.157082 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:57:18.157050    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d67c23da-ef59-449a-907d-822b837bba9e-bound-sa-token\") pod \"cert-manager-55b4d954d5-hqdrq\" (UID: \"d67c23da-ef59-449a-907d-822b837bba9e\") " pod="cert-manager/cert-manager-55b4d954d5-hqdrq"
Jul 27 20:57:18.157252 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:57:18.157160    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-565p6\" (UniqueName: \"kubernetes.io/projected/d67c23da-ef59-449a-907d-822b837bba9e-kube-api-access-565p6\") pod \"cert-manager-55b4d954d5-hqdrq\" (UID: \"d67c23da-ef59-449a-907d-822b837bba9e\") " pod="cert-manager/cert-manager-55b4d954d5-hqdrq"
Jul 27 20:57:18.258367 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:57:18.258332    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-565p6\" (UniqueName: \"kubernetes.io/projected/d67c23da-ef59-449a-907d-822b837bba9e-kube-api-access-565p6\") pod \"cert-manager-55b4d954d5-hqdrq\" (UID: \"d67c23da-ef59-449a-907d-822b837bba9e\") " pod="cert-manager/cert-manager-55b4d954d5-hqdrq"
Jul 27 20:57:18.258576 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:57:18.258421    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d67c23da-ef59-449a-907d-822b837bba9e-bound-sa-token\") pod \"cert-manager-55b4d954d5-hqdrq\" (UID: \"d67c23da-ef59-449a-907d-822b837bba9e\") " pod="cert-manager/cert-manager-55b4d954d5-hqdrq"
Jul 27 20:57:18.267235 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:57:18.267203    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d67c23da-ef59-449a-907d-822b837bba9e-bound-sa-token\") pod \"cert-manager-55b4d954d5-hqdrq\" (UID: \"d67c23da-ef59-449a-907d-822b837bba9e\") " pod="cert-manager/cert-manager-55b4d954d5-hqdrq"
Jul 27 20:57:18.267370 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:57:18.267312    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-565p6\" (UniqueName: \"kubernetes.io/projected/d67c23da-ef59-449a-907d-822b837bba9e-kube-api-access-565p6\") pod \"cert-manager-55b4d954d5-hqdrq\" (UID: \"d67c23da-ef59-449a-907d-822b837bba9e\") " pod="cert-manager/cert-manager-55b4d954d5-hqdrq"
Jul 27 20:57:18.372661 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:57:18.372588    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-hqdrq"
Jul 27 20:57:18.517143 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:57:18.517120    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-hqdrq"]
Jul 27 20:57:18.519687 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:57:18.519660    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd67c23da_ef59_449a_907d_822b837bba9e.slice/crio-01beea04f1d2da3c1d167cbb359d685f32b456df6fc6a1d30afeedc824bb3135 WatchSource:0}: Error finding container 01beea04f1d2da3c1d167cbb359d685f32b456df6fc6a1d30afeedc824bb3135: Status 404 returned error can't find the container with id 01beea04f1d2da3c1d167cbb359d685f32b456df6fc6a1d30afeedc824bb3135
Jul 27 20:57:19.230767 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:57:19.230728    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-hf25t" event={"ID":"6b75195b-ed8f-411f-ae54-5ee692445947","Type":"ContainerStarted","Data":"f5d51f209c2ef2998aeb1d3a8882142ec2852c9e8597176f67b320f146d6ebc0"}
Jul 27 20:57:19.232164 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:57:19.232143    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-hqdrq" event={"ID":"d67c23da-ef59-449a-907d-822b837bba9e","Type":"ContainerStarted","Data":"2aabfaeb56c13b676b56dc610d17fa12560d7c83c047c6bbc17ccbf65742219d"}
Jul 27 20:57:19.232280 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:57:19.232170    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-hqdrq" event={"ID":"d67c23da-ef59-449a-907d-822b837bba9e","Type":"ContainerStarted","Data":"01beea04f1d2da3c1d167cbb359d685f32b456df6fc6a1d30afeedc824bb3135"}
Jul 27 20:57:19.252842 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:57:19.252799    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-hf25t" podStartSLOduration=1.662518304 podStartE2EDuration="5.252785342s" totalImagesPullingTime="3.590267038s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:57:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:57:19.251895051 +0000 UTC m=+356.003756847" watchObservedRunningTime="2026-07-27 20:57:19.252785342 +0000 UTC m=+356.004647137"
Jul 27 20:57:19.271962 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:57:19.271925    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-hqdrq" podStartSLOduration=1.271915561 podStartE2EDuration="1.271915561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:57:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:57:19.270616514 +0000 UTC m=+356.022478308" watchObservedRunningTime="2026-07-27 20:57:19.271915561 +0000 UTC m=+356.023777355"
Jul 27 20:57:41.558819 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:57:41.558782    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5c6d5848fb-b8zrz"]
Jul 27 20:57:41.564941 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:57:41.564920    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5c6d5848fb-b8zrz"
Jul 27 20:57:41.568172 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:57:41.568151    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 27 20:57:41.568385 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:57:41.568365    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-sqfzw\""
Jul 27 20:57:41.568385 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:57:41.568380    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 27 20:57:41.568691 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:57:41.568674    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 27 20:57:41.568734 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:57:41.568688    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 27 20:57:41.584503 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:57:41.584483    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5c6d5848fb-b8zrz"]
Jul 27 20:57:41.621334 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:57:41.621306    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thqqf\" (UniqueName: \"kubernetes.io/projected/164f878d-db0a-4fa2-aed1-0ae818bee556-kube-api-access-thqqf\") pod \"opendatahub-operator-controller-manager-5c6d5848fb-b8zrz\" (UID: \"164f878d-db0a-4fa2-aed1-0ae818bee556\") " pod="opendatahub/opendatahub-operator-controller-manager-5c6d5848fb-b8zrz"
Jul 27 20:57:41.621525 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:57:41.621346    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/164f878d-db0a-4fa2-aed1-0ae818bee556-manifests\") pod \"opendatahub-operator-controller-manager-5c6d5848fb-b8zrz\" (UID: \"164f878d-db0a-4fa2-aed1-0ae818bee556\") " pod="opendatahub/opendatahub-operator-controller-manager-5c6d5848fb-b8zrz"
Jul 27 20:57:41.621525 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:57:41.621364    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/164f878d-db0a-4fa2-aed1-0ae818bee556-tmp\") pod \"opendatahub-operator-controller-manager-5c6d5848fb-b8zrz\" (UID: \"164f878d-db0a-4fa2-aed1-0ae818bee556\") " pod="opendatahub/opendatahub-operator-controller-manager-5c6d5848fb-b8zrz"
Jul 27 20:57:41.621525 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:57:41.621406    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/164f878d-db0a-4fa2-aed1-0ae818bee556-webhook-cert\") pod \"opendatahub-operator-controller-manager-5c6d5848fb-b8zrz\" (UID: \"164f878d-db0a-4fa2-aed1-0ae818bee556\") " pod="opendatahub/opendatahub-operator-controller-manager-5c6d5848fb-b8zrz"
Jul 27 20:57:41.621525 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:57:41.621445    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/164f878d-db0a-4fa2-aed1-0ae818bee556-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5c6d5848fb-b8zrz\" (UID: \"164f878d-db0a-4fa2-aed1-0ae818bee556\") " pod="opendatahub/opendatahub-operator-controller-manager-5c6d5848fb-b8zrz"
Jul 27 20:57:41.722799 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:57:41.722758    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thqqf\" (UniqueName: \"kubernetes.io/projected/164f878d-db0a-4fa2-aed1-0ae818bee556-kube-api-access-thqqf\") pod \"opendatahub-operator-controller-manager-5c6d5848fb-b8zrz\" (UID: \"164f878d-db0a-4fa2-aed1-0ae818bee556\") " pod="opendatahub/opendatahub-operator-controller-manager-5c6d5848fb-b8zrz"
Jul 27 20:57:41.722799 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:57:41.722803    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/164f878d-db0a-4fa2-aed1-0ae818bee556-manifests\") pod \"opendatahub-operator-controller-manager-5c6d5848fb-b8zrz\" (UID: \"164f878d-db0a-4fa2-aed1-0ae818bee556\") " pod="opendatahub/opendatahub-operator-controller-manager-5c6d5848fb-b8zrz"
Jul 27 20:57:41.723059 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:57:41.722821    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/164f878d-db0a-4fa2-aed1-0ae818bee556-tmp\") pod \"opendatahub-operator-controller-manager-5c6d5848fb-b8zrz\" (UID: \"164f878d-db0a-4fa2-aed1-0ae818bee556\") " pod="opendatahub/opendatahub-operator-controller-manager-5c6d5848fb-b8zrz"
Jul 27 20:57:41.723059 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:57:41.722938    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/164f878d-db0a-4fa2-aed1-0ae818bee556-webhook-cert\") pod \"opendatahub-operator-controller-manager-5c6d5848fb-b8zrz\" (UID: \"164f878d-db0a-4fa2-aed1-0ae818bee556\") " pod="opendatahub/opendatahub-operator-controller-manager-5c6d5848fb-b8zrz"
Jul 27 20:57:41.723059 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:57:41.722983    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/164f878d-db0a-4fa2-aed1-0ae818bee556-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5c6d5848fb-b8zrz\" (UID: \"164f878d-db0a-4fa2-aed1-0ae818bee556\") " pod="opendatahub/opendatahub-operator-controller-manager-5c6d5848fb-b8zrz"
Jul 27 20:57:41.723277 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:57:41.723255    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/164f878d-db0a-4fa2-aed1-0ae818bee556-manifests\") pod \"opendatahub-operator-controller-manager-5c6d5848fb-b8zrz\" (UID: \"164f878d-db0a-4fa2-aed1-0ae818bee556\") " pod="opendatahub/opendatahub-operator-controller-manager-5c6d5848fb-b8zrz"
Jul 27 20:57:41.725305 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:57:41.725281    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/164f878d-db0a-4fa2-aed1-0ae818bee556-tmp\") pod \"opendatahub-operator-controller-manager-5c6d5848fb-b8zrz\" (UID: \"164f878d-db0a-4fa2-aed1-0ae818bee556\") " pod="opendatahub/opendatahub-operator-controller-manager-5c6d5848fb-b8zrz"
Jul 27 20:57:41.725544 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:57:41.725526    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/164f878d-db0a-4fa2-aed1-0ae818bee556-webhook-cert\") pod \"opendatahub-operator-controller-manager-5c6d5848fb-b8zrz\" (UID: \"164f878d-db0a-4fa2-aed1-0ae818bee556\") " pod="opendatahub/opendatahub-operator-controller-manager-5c6d5848fb-b8zrz"
Jul 27 20:57:41.725655 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:57:41.725636    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/164f878d-db0a-4fa2-aed1-0ae818bee556-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5c6d5848fb-b8zrz\" (UID: \"164f878d-db0a-4fa2-aed1-0ae818bee556\") " pod="opendatahub/opendatahub-operator-controller-manager-5c6d5848fb-b8zrz"
Jul 27 20:57:41.731658 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:57:41.731629    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-thqqf\" (UniqueName: \"kubernetes.io/projected/164f878d-db0a-4fa2-aed1-0ae818bee556-kube-api-access-thqqf\") pod \"opendatahub-operator-controller-manager-5c6d5848fb-b8zrz\" (UID: \"164f878d-db0a-4fa2-aed1-0ae818bee556\") " pod="opendatahub/opendatahub-operator-controller-manager-5c6d5848fb-b8zrz"
Jul 27 20:57:41.874626 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:57:41.874528    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5c6d5848fb-b8zrz"
Jul 27 20:57:42.022161 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:57:42.022134    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5c6d5848fb-b8zrz"]
Jul 27 20:57:42.024647 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:57:42.024615    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod164f878d_db0a_4fa2_aed1_0ae818bee556.slice/crio-2ee99df92a241f7f024575cef828d50f14e3e3993a23db488da17d0222566940 WatchSource:0}: Error finding container 2ee99df92a241f7f024575cef828d50f14e3e3993a23db488da17d0222566940: Status 404 returned error can't find the container with id 2ee99df92a241f7f024575cef828d50f14e3e3993a23db488da17d0222566940
Jul 27 20:57:42.313830 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:57:42.313794    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5c6d5848fb-b8zrz" event={"ID":"164f878d-db0a-4fa2-aed1-0ae818bee556","Type":"ContainerStarted","Data":"2ee99df92a241f7f024575cef828d50f14e3e3993a23db488da17d0222566940"}
Jul 27 20:57:45.328624 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:57:45.328583    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5c6d5848fb-b8zrz" event={"ID":"164f878d-db0a-4fa2-aed1-0ae818bee556","Type":"ContainerStarted","Data":"d2180f39af5aeec76701965c9272b2b9b975cb42f7a815e5bb1bc08ef748d044"}
Jul 27 20:57:46.333589 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:57:46.333514    2560 generic.go:356] "Generic (PLEG): container finished" podID="164f878d-db0a-4fa2-aed1-0ae818bee556" containerID="d2180f39af5aeec76701965c9272b2b9b975cb42f7a815e5bb1bc08ef748d044" exitCode=0
Jul 27 20:57:46.333589 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:57:46.333558    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5c6d5848fb-b8zrz" event={"ID":"164f878d-db0a-4fa2-aed1-0ae818bee556","Type":"ContainerDied","Data":"d2180f39af5aeec76701965c9272b2b9b975cb42f7a815e5bb1bc08ef748d044"}
Jul 27 20:57:47.339400 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:57:47.339360    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5c6d5848fb-b8zrz" event={"ID":"164f878d-db0a-4fa2-aed1-0ae818bee556","Type":"ContainerStarted","Data":"2cfce88fefbeae34c482b48a65d5715d7dee1b4254978eb8bc9a8103eac4ff2f"}
Jul 27 20:57:47.339805 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:57:47.339501    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5c6d5848fb-b8zrz"
Jul 27 20:57:47.367919 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:57:47.367859    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5c6d5848fb-b8zrz" podStartSLOduration=2.000841659 podStartE2EDuration="6.367839338s" totalImagesPullingTime="3.33660634s" totalInitContainerRuntime="1.030391339s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:57:41 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:57:47.365242982 +0000 UTC m=+384.117104804" watchObservedRunningTime="2026-07-27 20:57:47.367839338 +0000 UTC m=+384.119701134"
Jul 27 20:57:58.346484 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:57:58.346451    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5c6d5848fb-b8zrz"
Jul 27 20:58:01.723042 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:58:01.723004    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-8f6xg"]
Jul 27 20:58:01.729078 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:58:01.729061    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-8f6xg"
Jul 27 20:58:01.733387 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:58:01.733363    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Jul 27 20:58:01.733510 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:58:01.733405    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-dqbzk\""
Jul 27 20:58:01.742218 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:58:01.742195    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-8f6xg"]
Jul 27 20:58:01.782703 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:58:01.782671    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a5bf166f-b209-416f-ae7e-98f18db1c2e5-cert\") pod \"odh-model-controller-7998c67dc4-8f6xg\" (UID: \"a5bf166f-b209-416f-ae7e-98f18db1c2e5\") " pod="opendatahub/odh-model-controller-7998c67dc4-8f6xg"
Jul 27 20:58:01.782703 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:58:01.782714    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zn4bt\" (UniqueName: \"kubernetes.io/projected/a5bf166f-b209-416f-ae7e-98f18db1c2e5-kube-api-access-zn4bt\") pod \"odh-model-controller-7998c67dc4-8f6xg\" (UID: \"a5bf166f-b209-416f-ae7e-98f18db1c2e5\") " pod="opendatahub/odh-model-controller-7998c67dc4-8f6xg"
Jul 27 20:58:01.883462 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:58:01.883422    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a5bf166f-b209-416f-ae7e-98f18db1c2e5-cert\") pod \"odh-model-controller-7998c67dc4-8f6xg\" (UID: \"a5bf166f-b209-416f-ae7e-98f18db1c2e5\") " pod="opendatahub/odh-model-controller-7998c67dc4-8f6xg"
Jul 27 20:58:01.883656 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:58:01.883489    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zn4bt\" (UniqueName: \"kubernetes.io/projected/a5bf166f-b209-416f-ae7e-98f18db1c2e5-kube-api-access-zn4bt\") pod \"odh-model-controller-7998c67dc4-8f6xg\" (UID: \"a5bf166f-b209-416f-ae7e-98f18db1c2e5\") " pod="opendatahub/odh-model-controller-7998c67dc4-8f6xg"
Jul 27 20:58:01.883656 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:58:01.883592    2560 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Jul 27 20:58:01.883740 ip-10-0-141-64 kubenswrapper[2560]: E0727 20:58:01.883661    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5bf166f-b209-416f-ae7e-98f18db1c2e5-cert podName:a5bf166f-b209-416f-ae7e-98f18db1c2e5 nodeName:}" failed. No retries permitted until 2026-07-27 20:58:02.383643175 +0000 UTC m=+399.135504949 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/a5bf166f-b209-416f-ae7e-98f18db1c2e5-cert") pod "odh-model-controller-7998c67dc4-8f6xg" (UID: "a5bf166f-b209-416f-ae7e-98f18db1c2e5") : secret "odh-model-controller-webhook-cert" not found
Jul 27 20:58:01.895185 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:58:01.895152    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zn4bt\" (UniqueName: \"kubernetes.io/projected/a5bf166f-b209-416f-ae7e-98f18db1c2e5-kube-api-access-zn4bt\") pod \"odh-model-controller-7998c67dc4-8f6xg\" (UID: \"a5bf166f-b209-416f-ae7e-98f18db1c2e5\") " pod="opendatahub/odh-model-controller-7998c67dc4-8f6xg"
Jul 27 20:58:02.387128 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:58:02.387075    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a5bf166f-b209-416f-ae7e-98f18db1c2e5-cert\") pod \"odh-model-controller-7998c67dc4-8f6xg\" (UID: \"a5bf166f-b209-416f-ae7e-98f18db1c2e5\") " pod="opendatahub/odh-model-controller-7998c67dc4-8f6xg"
Jul 27 20:58:02.389634 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:58:02.389609    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a5bf166f-b209-416f-ae7e-98f18db1c2e5-cert\") pod \"odh-model-controller-7998c67dc4-8f6xg\" (UID: \"a5bf166f-b209-416f-ae7e-98f18db1c2e5\") " pod="opendatahub/odh-model-controller-7998c67dc4-8f6xg"
Jul 27 20:58:02.638595 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:58:02.638489    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-8f6xg"
Jul 27 20:58:02.791543 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:58:02.791506    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-8f6xg"]
Jul 27 20:58:02.795269 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:58:02.795194    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5bf166f_b209_416f_ae7e_98f18db1c2e5.slice/crio-5024ab03ddf637a23eba279402b5025d5f763ed875386eceba1e75772c3ac369 WatchSource:0}: Error finding container 5024ab03ddf637a23eba279402b5025d5f763ed875386eceba1e75772c3ac369: Status 404 returned error can't find the container with id 5024ab03ddf637a23eba279402b5025d5f763ed875386eceba1e75772c3ac369
Jul 27 20:58:03.400941 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:58:03.400903    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-8f6xg" event={"ID":"a5bf166f-b209-416f-ae7e-98f18db1c2e5","Type":"ContainerStarted","Data":"5024ab03ddf637a23eba279402b5025d5f763ed875386eceba1e75772c3ac369"}
Jul 27 20:58:06.416596 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:58:06.416558    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-8f6xg" event={"ID":"a5bf166f-b209-416f-ae7e-98f18db1c2e5","Type":"ContainerStarted","Data":"5303810810ad6e1a42919dfea3cef03a0f09ae537f7c12dbb9e01da61976fbbe"}
Jul 27 20:58:06.417060 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:58:06.416633    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-8f6xg"
Jul 27 20:58:06.437036 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:58:06.436978    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-8f6xg" podStartSLOduration=2.652974614 podStartE2EDuration="5.436961017s" totalImagesPullingTime="2.783986403s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:58:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:58:06.435528791 +0000 UTC m=+403.187390592" watchObservedRunningTime="2026-07-27 20:58:06.436961017 +0000 UTC m=+403.188822813"
Jul 27 20:58:12.084411 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:58:12.084375    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-6598f7b856-nsqwk"]
Jul 27 20:58:12.086541 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:58:12.086517    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6598f7b856-nsqwk"
Jul 27 20:58:12.088162 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:58:12.088139    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-cd8kf"]
Jul 27 20:58:12.090138 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:58:12.090122    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-cd8kf"
Jul 27 20:58:12.098624 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:58:12.098607    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-f4gh6\""
Jul 27 20:58:12.099122 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:58:12.099079    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Jul 27 20:58:12.099122 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:58:12.099112    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 27 20:58:12.104447 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:58:12.104432    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 27 20:58:12.105262 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:58:12.105242    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 27 20:58:12.105353 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:58:12.105300    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 27 20:58:12.114445 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:58:12.114426    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6598f7b856-nsqwk"]
Jul 27 20:58:12.122036 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:58:12.122014    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-bgxv4"]
Jul 27 20:58:12.124538 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:58:12.124518    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-cd8kf"]
Jul 27 20:58:12.124624 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:58:12.124607    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bgxv4"
Jul 27 20:58:12.132215 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:58:12.132193    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Jul 27 20:58:12.132320 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:58:12.132199    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-jmcq2\""
Jul 27 20:58:12.151956 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:58:12.151930    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-bgxv4"]
Jul 27 20:58:12.165967 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:58:12.165943    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40340708-0cf9-4b9c-9331-6155dbee39b3-tls-certs\") pod \"kube-auth-proxy-6598f7b856-nsqwk\" (UID: \"40340708-0cf9-4b9c-9331-6155dbee39b3\") " pod="openshift-ingress/kube-auth-proxy-6598f7b856-nsqwk"
Jul 27 20:58:12.166053 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:58:12.165975    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b88c26a8-d71c-460d-a2db-386767155bba-cert\") pod \"kserve-controller-manager-d6f4b65b8-cd8kf\" (UID: \"b88c26a8-d71c-460d-a2db-386767155bba\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-cd8kf"
Jul 27 20:58:12.166053 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:58:12.165994    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqvrt\" (UniqueName: \"kubernetes.io/projected/40340708-0cf9-4b9c-9331-6155dbee39b3-kube-api-access-gqvrt\") pod \"kube-auth-proxy-6598f7b856-nsqwk\" (UID: \"40340708-0cf9-4b9c-9331-6155dbee39b3\") " pod="openshift-ingress/kube-auth-proxy-6598f7b856-nsqwk"
Jul 27 20:58:12.166053 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:58:12.166020    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hb2qx\" (UniqueName: \"kubernetes.io/projected/b88c26a8-d71c-460d-a2db-386767155bba-kube-api-access-hb2qx\") pod \"kserve-controller-manager-d6f4b65b8-cd8kf\" (UID: \"b88c26a8-d71c-460d-a2db-386767155bba\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-cd8kf"
Jul 27 20:58:12.166053 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:58:12.166039    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/40340708-0cf9-4b9c-9331-6155dbee39b3-tmp\") pod \"kube-auth-proxy-6598f7b856-nsqwk\" (UID: \"40340708-0cf9-4b9c-9331-6155dbee39b3\") " pod="openshift-ingress/kube-auth-proxy-6598f7b856-nsqwk"
Jul 27 20:58:12.266572 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:58:12.266535    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/40340708-0cf9-4b9c-9331-6155dbee39b3-tmp\") pod \"kube-auth-proxy-6598f7b856-nsqwk\" (UID: \"40340708-0cf9-4b9c-9331-6155dbee39b3\") " pod="openshift-ingress/kube-auth-proxy-6598f7b856-nsqwk"
Jul 27 20:58:12.266729 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:58:12.266640    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40340708-0cf9-4b9c-9331-6155dbee39b3-tls-certs\") pod \"kube-auth-proxy-6598f7b856-nsqwk\" (UID: \"40340708-0cf9-4b9c-9331-6155dbee39b3\") " pod="openshift-ingress/kube-auth-proxy-6598f7b856-nsqwk"
Jul 27 20:58:12.266729 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:58:12.266669    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/83957a08-9031-4434-a11d-7dc22b82220c-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-bgxv4\" (UID: \"83957a08-9031-4434-a11d-7dc22b82220c\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bgxv4"
Jul 27 20:58:12.266729 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:58:12.266702    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmpc6\" (UniqueName: \"kubernetes.io/projected/83957a08-9031-4434-a11d-7dc22b82220c-kube-api-access-rmpc6\") pod \"llmisvc-controller-manager-777c6bd7d5-bgxv4\" (UID: \"83957a08-9031-4434-a11d-7dc22b82220c\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bgxv4"
Jul 27 20:58:12.266853 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:58:12.266733    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b88c26a8-d71c-460d-a2db-386767155bba-cert\") pod \"kserve-controller-manager-d6f4b65b8-cd8kf\" (UID: \"b88c26a8-d71c-460d-a2db-386767155bba\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-cd8kf"
Jul 27 20:58:12.266853 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:58:12.266782    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqvrt\" (UniqueName: \"kubernetes.io/projected/40340708-0cf9-4b9c-9331-6155dbee39b3-kube-api-access-gqvrt\") pod \"kube-auth-proxy-6598f7b856-nsqwk\" (UID: \"40340708-0cf9-4b9c-9331-6155dbee39b3\") " pod="openshift-ingress/kube-auth-proxy-6598f7b856-nsqwk"
Jul 27 20:58:12.266853 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:58:12.266834    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hb2qx\" (UniqueName: \"kubernetes.io/projected/b88c26a8-d71c-460d-a2db-386767155bba-kube-api-access-hb2qx\") pod \"kserve-controller-manager-d6f4b65b8-cd8kf\" (UID: \"b88c26a8-d71c-460d-a2db-386767155bba\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-cd8kf"
Jul 27 20:58:12.268950 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:58:12.268929    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/40340708-0cf9-4b9c-9331-6155dbee39b3-tmp\") pod \"kube-auth-proxy-6598f7b856-nsqwk\" (UID: \"40340708-0cf9-4b9c-9331-6155dbee39b3\") " pod="openshift-ingress/kube-auth-proxy-6598f7b856-nsqwk"
Jul 27 20:58:12.269214 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:58:12.269198    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b88c26a8-d71c-460d-a2db-386767155bba-cert\") pod \"kserve-controller-manager-d6f4b65b8-cd8kf\" (UID: \"b88c26a8-d71c-460d-a2db-386767155bba\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-cd8kf"
Jul 27 20:58:12.269299 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:58:12.269279    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40340708-0cf9-4b9c-9331-6155dbee39b3-tls-certs\") pod \"kube-auth-proxy-6598f7b856-nsqwk\" (UID: \"40340708-0cf9-4b9c-9331-6155dbee39b3\") " pod="openshift-ingress/kube-auth-proxy-6598f7b856-nsqwk"
Jul 27 20:58:12.284896 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:58:12.284872    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqvrt\" (UniqueName: \"kubernetes.io/projected/40340708-0cf9-4b9c-9331-6155dbee39b3-kube-api-access-gqvrt\") pod \"kube-auth-proxy-6598f7b856-nsqwk\" (UID: \"40340708-0cf9-4b9c-9331-6155dbee39b3\") " pod="openshift-ingress/kube-auth-proxy-6598f7b856-nsqwk"
Jul 27 20:58:12.288553 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:58:12.288530    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hb2qx\" (UniqueName: \"kubernetes.io/projected/b88c26a8-d71c-460d-a2db-386767155bba-kube-api-access-hb2qx\") pod \"kserve-controller-manager-d6f4b65b8-cd8kf\" (UID: \"b88c26a8-d71c-460d-a2db-386767155bba\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-cd8kf"
Jul 27 20:58:12.368204 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:58:12.368136    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/83957a08-9031-4434-a11d-7dc22b82220c-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-bgxv4\" (UID: \"83957a08-9031-4434-a11d-7dc22b82220c\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bgxv4"
Jul 27 20:58:12.368204 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:58:12.368171    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmpc6\" (UniqueName: \"kubernetes.io/projected/83957a08-9031-4434-a11d-7dc22b82220c-kube-api-access-rmpc6\") pod \"llmisvc-controller-manager-777c6bd7d5-bgxv4\" (UID: \"83957a08-9031-4434-a11d-7dc22b82220c\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bgxv4"
Jul 27 20:58:12.370741 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:58:12.370720    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/83957a08-9031-4434-a11d-7dc22b82220c-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-bgxv4\" (UID: \"83957a08-9031-4434-a11d-7dc22b82220c\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bgxv4"
Jul 27 20:58:12.382493 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:58:12.382472    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmpc6\" (UniqueName: \"kubernetes.io/projected/83957a08-9031-4434-a11d-7dc22b82220c-kube-api-access-rmpc6\") pod \"llmisvc-controller-manager-777c6bd7d5-bgxv4\" (UID: \"83957a08-9031-4434-a11d-7dc22b82220c\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bgxv4"
Jul 27 20:58:12.397304 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:58:12.397275    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6598f7b856-nsqwk"
Jul 27 20:58:12.403800 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:58:12.403781    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-cd8kf"
Jul 27 20:58:12.434568 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:58:12.434536    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bgxv4"
Jul 27 20:58:12.578243 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:58:12.578209    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6598f7b856-nsqwk"]
Jul 27 20:58:12.581869 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:58:12.581843    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40340708_0cf9_4b9c_9331_6155dbee39b3.slice/crio-742dff4f4b39b51f75127fa2da61ad12b886038e81cc11727f3b247cffca8594 WatchSource:0}: Error finding container 742dff4f4b39b51f75127fa2da61ad12b886038e81cc11727f3b247cffca8594: Status 404 returned error can't find the container with id 742dff4f4b39b51f75127fa2da61ad12b886038e81cc11727f3b247cffca8594
Jul 27 20:58:12.615892 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:58:12.615867    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb88c26a8_d71c_460d_a2db_386767155bba.slice/crio-fe05d2879906e41de77718c64dda02194033485f5a2524671462613e34d0aa16 WatchSource:0}: Error finding container fe05d2879906e41de77718c64dda02194033485f5a2524671462613e34d0aa16: Status 404 returned error can't find the container with id fe05d2879906e41de77718c64dda02194033485f5a2524671462613e34d0aa16
Jul 27 20:58:12.622860 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:58:12.622834    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-cd8kf"]
Jul 27 20:58:12.687006 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:58:12.686977    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-bgxv4"]
Jul 27 20:58:12.690377 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:58:12.690353    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod83957a08_9031_4434_a11d_7dc22b82220c.slice/crio-d81c9cb73183f721f1660348ce9a9675a3787841082138853f09ed2c38880c1b WatchSource:0}: Error finding container d81c9cb73183f721f1660348ce9a9675a3787841082138853f09ed2c38880c1b: Status 404 returned error can't find the container with id d81c9cb73183f721f1660348ce9a9675a3787841082138853f09ed2c38880c1b
Jul 27 20:58:13.450836 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:58:13.450740    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bgxv4" event={"ID":"83957a08-9031-4434-a11d-7dc22b82220c","Type":"ContainerStarted","Data":"d81c9cb73183f721f1660348ce9a9675a3787841082138853f09ed2c38880c1b"}
Jul 27 20:58:13.454873 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:58:13.454802    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-cd8kf" event={"ID":"b88c26a8-d71c-460d-a2db-386767155bba","Type":"ContainerStarted","Data":"fe05d2879906e41de77718c64dda02194033485f5a2524671462613e34d0aa16"}
Jul 27 20:58:13.456671 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:58:13.456637    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6598f7b856-nsqwk" event={"ID":"40340708-0cf9-4b9c-9331-6155dbee39b3","Type":"ContainerStarted","Data":"742dff4f4b39b51f75127fa2da61ad12b886038e81cc11727f3b247cffca8594"}
Jul 27 20:58:15.016733 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:58:15.016701    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-bch22"]
Jul 27 20:58:15.019234 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:58:15.019205    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bch22"
Jul 27 20:58:15.025110 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:58:15.025070    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Jul 27 20:58:15.025228 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:58:15.025110    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Jul 27 20:58:15.025228 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:58:15.025166    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-x8tfx\""
Jul 27 20:58:15.025512 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:58:15.025483    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 27 20:58:15.025616 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:58:15.025584    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Jul 27 20:58:15.039510 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:58:15.039486    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-bch22"]
Jul 27 20:58:15.196631 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:58:15.196588    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/235c0fce-11cf-42aa-93c8-74a39b3348b4-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-bch22\" (UID: \"235c0fce-11cf-42aa-93c8-74a39b3348b4\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bch22"
Jul 27 20:58:15.196817 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:58:15.196656    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/235c0fce-11cf-42aa-93c8-74a39b3348b4-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-bch22\" (UID: \"235c0fce-11cf-42aa-93c8-74a39b3348b4\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bch22"
Jul 27 20:58:15.196817 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:58:15.196756    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7f26t\" (UniqueName: \"kubernetes.io/projected/235c0fce-11cf-42aa-93c8-74a39b3348b4-kube-api-access-7f26t\") pod \"istiod-openshift-gateway-5677ccb49b-bch22\" (UID: \"235c0fce-11cf-42aa-93c8-74a39b3348b4\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bch22"
Jul 27 20:58:15.196940 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:58:15.196830    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/235c0fce-11cf-42aa-93c8-74a39b3348b4-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-bch22\" (UID: \"235c0fce-11cf-42aa-93c8-74a39b3348b4\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bch22"
Jul 27 20:58:15.196940 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:58:15.196858    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/235c0fce-11cf-42aa-93c8-74a39b3348b4-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-bch22\" (UID: \"235c0fce-11cf-42aa-93c8-74a39b3348b4\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bch22"
Jul 27 20:58:15.196940 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:58:15.196878    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/235c0fce-11cf-42aa-93c8-74a39b3348b4-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-bch22\" (UID: \"235c0fce-11cf-42aa-93c8-74a39b3348b4\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bch22"
Jul 27 20:58:15.196940 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:58:15.196924    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/235c0fce-11cf-42aa-93c8-74a39b3348b4-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-bch22\" (UID: \"235c0fce-11cf-42aa-93c8-74a39b3348b4\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bch22"
Jul 27 20:58:15.298391 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:58:15.298286    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/235c0fce-11cf-42aa-93c8-74a39b3348b4-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-bch22\" (UID: \"235c0fce-11cf-42aa-93c8-74a39b3348b4\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bch22"
Jul 27 20:58:15.298391 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:58:15.298336    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/235c0fce-11cf-42aa-93c8-74a39b3348b4-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-bch22\" (UID: \"235c0fce-11cf-42aa-93c8-74a39b3348b4\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bch22"
Jul 27 20:58:15.298391 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:58:15.298364    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/235c0fce-11cf-42aa-93c8-74a39b3348b4-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-bch22\" (UID: \"235c0fce-11cf-42aa-93c8-74a39b3348b4\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bch22"
Jul 27 20:58:15.298739 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:58:15.298417    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/235c0fce-11cf-42aa-93c8-74a39b3348b4-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-bch22\" (UID: \"235c0fce-11cf-42aa-93c8-74a39b3348b4\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bch22"
Jul 27 20:58:15.298739 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:58:15.298480    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/235c0fce-11cf-42aa-93c8-74a39b3348b4-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-bch22\" (UID: \"235c0fce-11cf-42aa-93c8-74a39b3348b4\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bch22"
Jul 27 20:58:15.298739 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:58:15.298522    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/235c0fce-11cf-42aa-93c8-74a39b3348b4-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-bch22\" (UID: \"235c0fce-11cf-42aa-93c8-74a39b3348b4\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bch22"
Jul 27 20:58:15.298739 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:58:15.298581    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7f26t\" (UniqueName: \"kubernetes.io/projected/235c0fce-11cf-42aa-93c8-74a39b3348b4-kube-api-access-7f26t\") pod \"istiod-openshift-gateway-5677ccb49b-bch22\" (UID: \"235c0fce-11cf-42aa-93c8-74a39b3348b4\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bch22"
Jul 27 20:58:15.299112 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:58:15.299044    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/235c0fce-11cf-42aa-93c8-74a39b3348b4-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-bch22\" (UID: \"235c0fce-11cf-42aa-93c8-74a39b3348b4\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bch22"
Jul 27 20:58:15.301474 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:58:15.301450    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/235c0fce-11cf-42aa-93c8-74a39b3348b4-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-bch22\" (UID: \"235c0fce-11cf-42aa-93c8-74a39b3348b4\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bch22"
Jul 27 20:58:15.301593 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:58:15.301462    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/235c0fce-11cf-42aa-93c8-74a39b3348b4-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-bch22\" (UID: \"235c0fce-11cf-42aa-93c8-74a39b3348b4\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bch22"
Jul 27 20:58:15.301593 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:58:15.301495    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/235c0fce-11cf-42aa-93c8-74a39b3348b4-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-bch22\" (UID: \"235c0fce-11cf-42aa-93c8-74a39b3348b4\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bch22"
Jul 27 20:58:15.301593 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:58:15.301517    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/235c0fce-11cf-42aa-93c8-74a39b3348b4-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-bch22\" (UID: \"235c0fce-11cf-42aa-93c8-74a39b3348b4\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bch22"
Jul 27 20:58:15.307787 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:58:15.307761    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/235c0fce-11cf-42aa-93c8-74a39b3348b4-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-bch22\" (UID: \"235c0fce-11cf-42aa-93c8-74a39b3348b4\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bch22"
Jul 27 20:58:15.308367 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:58:15.308344    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7f26t\" (UniqueName: \"kubernetes.io/projected/235c0fce-11cf-42aa-93c8-74a39b3348b4-kube-api-access-7f26t\") pod \"istiod-openshift-gateway-5677ccb49b-bch22\" (UID: \"235c0fce-11cf-42aa-93c8-74a39b3348b4\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bch22"
Jul 27 20:58:15.330309 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:58:15.330283    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bch22"
Jul 27 20:58:16.426447 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:58:16.426409    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-bch22"]
Jul 27 20:58:16.430309 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:58:16.430281    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod235c0fce_11cf_42aa_93c8_74a39b3348b4.slice/crio-5040e93477e12f27bf5accace7c618ecbcd54ed4a674aace8a41ae4396adcad0 WatchSource:0}: Error finding container 5040e93477e12f27bf5accace7c618ecbcd54ed4a674aace8a41ae4396adcad0: Status 404 returned error can't find the container with id 5040e93477e12f27bf5accace7c618ecbcd54ed4a674aace8a41ae4396adcad0
Jul 27 20:58:16.472134 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:58:16.472038    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6598f7b856-nsqwk" event={"ID":"40340708-0cf9-4b9c-9331-6155dbee39b3","Type":"ContainerStarted","Data":"b04e687a5d503899833e6e772cdef2f5cdeaa4d712edeeb797885c1739f6ea22"}
Jul 27 20:58:16.473415 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:58:16.473383    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bch22" event={"ID":"235c0fce-11cf-42aa-93c8-74a39b3348b4","Type":"ContainerStarted","Data":"5040e93477e12f27bf5accace7c618ecbcd54ed4a674aace8a41ae4396adcad0"}
Jul 27 20:58:16.474804 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:58:16.474775    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bgxv4" event={"ID":"83957a08-9031-4434-a11d-7dc22b82220c","Type":"ContainerStarted","Data":"b94683457b06af501ed2cc1d6dc23c2fbdbbf2312e9f228aabe1ca869efe5aab"}
Jul 27 20:58:16.474948 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:58:16.474826    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bgxv4"
Jul 27 20:58:16.476320 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:58:16.476294    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-cd8kf" event={"ID":"b88c26a8-d71c-460d-a2db-386767155bba","Type":"ContainerStarted","Data":"4f981ebea8f6814cc8306f999f9d4da50a007a61a9dba55b13d854af17278393"}
Jul 27 20:58:16.476400 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:58:16.476358    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-cd8kf"
Jul 27 20:58:16.499768 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:58:16.499716    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-6598f7b856-nsqwk" podStartSLOduration=0.850163636 podStartE2EDuration="4.499700096s" totalImagesPullingTime="3.64953646s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:58:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:58:16.498760246 +0000 UTC m=+413.250622042" watchObservedRunningTime="2026-07-27 20:58:16.499700096 +0000 UTC m=+413.251561891"
Jul 27 20:58:16.521377 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:58:16.521329    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bgxv4" podStartSLOduration=0.97880445 podStartE2EDuration="4.521313279s" totalImagesPullingTime="3.542508829s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:58:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:58:16.520296337 +0000 UTC m=+413.272158133" watchObservedRunningTime="2026-07-27 20:58:16.521313279 +0000 UTC m=+413.273175073"
Jul 27 20:58:16.553927 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:58:16.553877    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-cd8kf" podStartSLOduration=0.935112227 podStartE2EDuration="4.553858888s" totalImagesPullingTime="3.618746661s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:58:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:58:16.553263849 +0000 UTC m=+413.305125665" watchObservedRunningTime="2026-07-27 20:58:16.553858888 +0000 UTC m=+413.305720685"
Jul 27 20:58:17.423656 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:58:17.423619    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-8f6xg"
Jul 27 20:58:19.306181 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:58:19.306143    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 27 20:58:19.306455 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:58:19.306211    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 27 20:58:19.508307 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:58:19.508262    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bch22" event={"ID":"235c0fce-11cf-42aa-93c8-74a39b3348b4","Type":"ContainerStarted","Data":"3923862f3a45c7c2b5185cb9bd59dc9733fcbe5815c7d20ebe8bda7866741028"}
Jul 27 20:58:19.509036 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:58:19.509007    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bch22"
Jul 27 20:58:19.543067 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:58:19.543009    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bch22" podStartSLOduration=2.669532742 podStartE2EDuration="5.542990373s" totalImagesPullingTime="2.873457631s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:58:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:58:19.542071902 +0000 UTC m=+416.293933712" watchObservedRunningTime="2026-07-27 20:58:19.542990373 +0000 UTC m=+416.294852165"
Jul 27 20:58:20.514896 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:58:20.514859    2560 patch_prober.go:28] interesting pod/istiod-openshift-gateway-5677ccb49b-bch22 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 27 20:58:20.515286 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:58:20.514916    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bch22" podUID="235c0fce-11cf-42aa-93c8-74a39b3348b4" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 20:58:23.514673 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:58:23.514644    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-bch22"
Jul 27 20:58:47.491800 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:58:47.491715    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bgxv4"
Jul 27 20:58:47.495232 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:58:47.495197    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-cd8kf"
Jul 27 20:59:36.482826 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:59:36.482793    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-95d667bb8-sxpj4"]
Jul 27 20:59:36.485302 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:59:36.485284    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-95d667bb8-sxpj4"
Jul 27 20:59:36.496390 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:59:36.496363    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-95d667bb8-sxpj4"]
Jul 27 20:59:36.553248 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:59:36.553202    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d62a4442-c805-40e3-919f-4bf0f1f4dd47-trusted-ca-bundle\") pod \"console-95d667bb8-sxpj4\" (UID: \"d62a4442-c805-40e3-919f-4bf0f1f4dd47\") " pod="openshift-console/console-95d667bb8-sxpj4"
Jul 27 20:59:36.553248 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:59:36.553251    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d62a4442-c805-40e3-919f-4bf0f1f4dd47-oauth-serving-cert\") pod \"console-95d667bb8-sxpj4\" (UID: \"d62a4442-c805-40e3-919f-4bf0f1f4dd47\") " pod="openshift-console/console-95d667bb8-sxpj4"
Jul 27 20:59:36.553456 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:59:36.553273    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d62a4442-c805-40e3-919f-4bf0f1f4dd47-service-ca\") pod \"console-95d667bb8-sxpj4\" (UID: \"d62a4442-c805-40e3-919f-4bf0f1f4dd47\") " pod="openshift-console/console-95d667bb8-sxpj4"
Jul 27 20:59:36.553456 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:59:36.553350    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-679cx\" (UniqueName: \"kubernetes.io/projected/d62a4442-c805-40e3-919f-4bf0f1f4dd47-kube-api-access-679cx\") pod \"console-95d667bb8-sxpj4\" (UID: \"d62a4442-c805-40e3-919f-4bf0f1f4dd47\") " pod="openshift-console/console-95d667bb8-sxpj4"
Jul 27 20:59:36.553456 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:59:36.553390    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d62a4442-c805-40e3-919f-4bf0f1f4dd47-console-serving-cert\") pod \"console-95d667bb8-sxpj4\" (UID: \"d62a4442-c805-40e3-919f-4bf0f1f4dd47\") " pod="openshift-console/console-95d667bb8-sxpj4"
Jul 27 20:59:36.553567 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:59:36.553457    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d62a4442-c805-40e3-919f-4bf0f1f4dd47-console-oauth-config\") pod \"console-95d667bb8-sxpj4\" (UID: \"d62a4442-c805-40e3-919f-4bf0f1f4dd47\") " pod="openshift-console/console-95d667bb8-sxpj4"
Jul 27 20:59:36.553567 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:59:36.553481    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d62a4442-c805-40e3-919f-4bf0f1f4dd47-console-config\") pod \"console-95d667bb8-sxpj4\" (UID: \"d62a4442-c805-40e3-919f-4bf0f1f4dd47\") " pod="openshift-console/console-95d667bb8-sxpj4"
Jul 27 20:59:36.654492 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:59:36.654453    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d62a4442-c805-40e3-919f-4bf0f1f4dd47-console-oauth-config\") pod \"console-95d667bb8-sxpj4\" (UID: \"d62a4442-c805-40e3-919f-4bf0f1f4dd47\") " pod="openshift-console/console-95d667bb8-sxpj4"
Jul 27 20:59:36.654492 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:59:36.654496    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d62a4442-c805-40e3-919f-4bf0f1f4dd47-console-config\") pod \"console-95d667bb8-sxpj4\" (UID: \"d62a4442-c805-40e3-919f-4bf0f1f4dd47\") " pod="openshift-console/console-95d667bb8-sxpj4"
Jul 27 20:59:36.654703 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:59:36.654525    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d62a4442-c805-40e3-919f-4bf0f1f4dd47-trusted-ca-bundle\") pod \"console-95d667bb8-sxpj4\" (UID: \"d62a4442-c805-40e3-919f-4bf0f1f4dd47\") " pod="openshift-console/console-95d667bb8-sxpj4"
Jul 27 20:59:36.654810 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:59:36.654790    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d62a4442-c805-40e3-919f-4bf0f1f4dd47-oauth-serving-cert\") pod \"console-95d667bb8-sxpj4\" (UID: \"d62a4442-c805-40e3-919f-4bf0f1f4dd47\") " pod="openshift-console/console-95d667bb8-sxpj4"
Jul 27 20:59:36.654863 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:59:36.654833    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d62a4442-c805-40e3-919f-4bf0f1f4dd47-service-ca\") pod \"console-95d667bb8-sxpj4\" (UID: \"d62a4442-c805-40e3-919f-4bf0f1f4dd47\") " pod="openshift-console/console-95d667bb8-sxpj4"
Jul 27 20:59:36.654899 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:59:36.654871    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-679cx\" (UniqueName: \"kubernetes.io/projected/d62a4442-c805-40e3-919f-4bf0f1f4dd47-kube-api-access-679cx\") pod \"console-95d667bb8-sxpj4\" (UID: \"d62a4442-c805-40e3-919f-4bf0f1f4dd47\") " pod="openshift-console/console-95d667bb8-sxpj4"
Jul 27 20:59:36.654899 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:59:36.654892    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d62a4442-c805-40e3-919f-4bf0f1f4dd47-console-serving-cert\") pod \"console-95d667bb8-sxpj4\" (UID: \"d62a4442-c805-40e3-919f-4bf0f1f4dd47\") " pod="openshift-console/console-95d667bb8-sxpj4"
Jul 27 20:59:36.655425 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:59:36.655397    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d62a4442-c805-40e3-919f-4bf0f1f4dd47-console-config\") pod \"console-95d667bb8-sxpj4\" (UID: \"d62a4442-c805-40e3-919f-4bf0f1f4dd47\") " pod="openshift-console/console-95d667bb8-sxpj4"
Jul 27 20:59:36.655539 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:59:36.655455    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d62a4442-c805-40e3-919f-4bf0f1f4dd47-trusted-ca-bundle\") pod \"console-95d667bb8-sxpj4\" (UID: \"d62a4442-c805-40e3-919f-4bf0f1f4dd47\") " pod="openshift-console/console-95d667bb8-sxpj4"
Jul 27 20:59:36.655606 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:59:36.655556    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d62a4442-c805-40e3-919f-4bf0f1f4dd47-service-ca\") pod \"console-95d667bb8-sxpj4\" (UID: \"d62a4442-c805-40e3-919f-4bf0f1f4dd47\") " pod="openshift-console/console-95d667bb8-sxpj4"
Jul 27 20:59:36.655756 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:59:36.655638    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d62a4442-c805-40e3-919f-4bf0f1f4dd47-oauth-serving-cert\") pod \"console-95d667bb8-sxpj4\" (UID: \"d62a4442-c805-40e3-919f-4bf0f1f4dd47\") " pod="openshift-console/console-95d667bb8-sxpj4"
Jul 27 20:59:36.657123 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:59:36.657078    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d62a4442-c805-40e3-919f-4bf0f1f4dd47-console-oauth-config\") pod \"console-95d667bb8-sxpj4\" (UID: \"d62a4442-c805-40e3-919f-4bf0f1f4dd47\") " pod="openshift-console/console-95d667bb8-sxpj4"
Jul 27 20:59:36.657419 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:59:36.657404    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d62a4442-c805-40e3-919f-4bf0f1f4dd47-console-serving-cert\") pod \"console-95d667bb8-sxpj4\" (UID: \"d62a4442-c805-40e3-919f-4bf0f1f4dd47\") " pod="openshift-console/console-95d667bb8-sxpj4"
Jul 27 20:59:36.664946 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:59:36.664925    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-679cx\" (UniqueName: \"kubernetes.io/projected/d62a4442-c805-40e3-919f-4bf0f1f4dd47-kube-api-access-679cx\") pod \"console-95d667bb8-sxpj4\" (UID: \"d62a4442-c805-40e3-919f-4bf0f1f4dd47\") " pod="openshift-console/console-95d667bb8-sxpj4"
Jul 27 20:59:36.796040 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:59:36.796008    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-95d667bb8-sxpj4"
Jul 27 20:59:36.930304 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:59:36.930142    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-95d667bb8-sxpj4"]
Jul 27 20:59:36.933210 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:59:36.933182    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd62a4442_c805_40e3_919f_4bf0f1f4dd47.slice/crio-236dd7c40b0989355de57f3d3d9e1197e1c7ac2f41dc5e02fe3af301c3c9f402 WatchSource:0}: Error finding container 236dd7c40b0989355de57f3d3d9e1197e1c7ac2f41dc5e02fe3af301c3c9f402: Status 404 returned error can't find the container with id 236dd7c40b0989355de57f3d3d9e1197e1c7ac2f41dc5e02fe3af301c3c9f402
Jul 27 20:59:37.828498 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:59:37.828464    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-95d667bb8-sxpj4" event={"ID":"d62a4442-c805-40e3-919f-4bf0f1f4dd47","Type":"ContainerStarted","Data":"0d6b22d128aa3822a09045fdee554a05736028ff9b86518039c3ea544aaccf0b"}
Jul 27 20:59:37.828880 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:59:37.828518    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-95d667bb8-sxpj4" event={"ID":"d62a4442-c805-40e3-919f-4bf0f1f4dd47","Type":"ContainerStarted","Data":"236dd7c40b0989355de57f3d3d9e1197e1c7ac2f41dc5e02fe3af301c3c9f402"}
Jul 27 20:59:37.847275 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:59:37.847233    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-95d667bb8-sxpj4" podStartSLOduration=1.847218815 podStartE2EDuration="1.847218815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:59:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:59:37.846663861 +0000 UTC m=+494.598525671" watchObservedRunningTime="2026-07-27 20:59:37.847218815 +0000 UTC m=+494.599080609"
Jul 27 20:59:38.120538 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:59:38.120459    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-nm6pr"]
Jul 27 20:59:38.122727 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:59:38.122710    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-nm6pr"
Jul 27 20:59:38.125904 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:59:38.125885    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Jul 27 20:59:38.126178 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:59:38.126163    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-n67qd\""
Jul 27 20:59:38.126421 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:59:38.126401    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 27 20:59:38.126460 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:59:38.126437    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 27 20:59:38.140193 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:59:38.136461    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-nm6pr"]
Jul 27 20:59:38.167146 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:59:38.167111    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksh7w\" (UniqueName: \"kubernetes.io/projected/7df103ae-114b-4595-b00c-e1d2b44d882d-kube-api-access-ksh7w\") pod \"dns-operator-controller-manager-864dd78c4c-nm6pr\" (UID: \"7df103ae-114b-4595-b00c-e1d2b44d882d\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-nm6pr"
Jul 27 20:59:38.268406 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:59:38.268367    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksh7w\" (UniqueName: \"kubernetes.io/projected/7df103ae-114b-4595-b00c-e1d2b44d882d-kube-api-access-ksh7w\") pod \"dns-operator-controller-manager-864dd78c4c-nm6pr\" (UID: \"7df103ae-114b-4595-b00c-e1d2b44d882d\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-nm6pr"
Jul 27 20:59:38.278712 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:59:38.278691    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksh7w\" (UniqueName: \"kubernetes.io/projected/7df103ae-114b-4595-b00c-e1d2b44d882d-kube-api-access-ksh7w\") pod \"dns-operator-controller-manager-864dd78c4c-nm6pr\" (UID: \"7df103ae-114b-4595-b00c-e1d2b44d882d\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-nm6pr"
Jul 27 20:59:38.433241 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:59:38.433154    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-nm6pr"
Jul 27 20:59:38.567970 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:59:38.567946    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-nm6pr"]
Jul 27 20:59:38.570374 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:59:38.570346    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7df103ae_114b_4595_b00c_e1d2b44d882d.slice/crio-0841b79c34f49df5e7c4e354d0013e2cee1e445232e890312664d18ff994e6d4 WatchSource:0}: Error finding container 0841b79c34f49df5e7c4e354d0013e2cee1e445232e890312664d18ff994e6d4: Status 404 returned error can't find the container with id 0841b79c34f49df5e7c4e354d0013e2cee1e445232e890312664d18ff994e6d4
Jul 27 20:59:38.834385 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:59:38.834351    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-nm6pr" event={"ID":"7df103ae-114b-4595-b00c-e1d2b44d882d","Type":"ContainerStarted","Data":"0841b79c34f49df5e7c4e354d0013e2cee1e445232e890312664d18ff994e6d4"}
Jul 27 20:59:40.828413 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:59:40.828379    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-wqzzm"]
Jul 27 20:59:40.830677 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:59:40.830652    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-wqzzm"
Jul 27 20:59:40.833204 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:59:40.833179    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-bmr5j\""
Jul 27 20:59:40.841700 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:59:40.841676    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-wqzzm"]
Jul 27 20:59:40.846808 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:59:40.846784    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-nm6pr" event={"ID":"7df103ae-114b-4595-b00c-e1d2b44d882d","Type":"ContainerStarted","Data":"cffdcca7061fcf22d25c8377858118b69b0abe72eeb1f38b08821131a42b7679"}
Jul 27 20:59:40.846923 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:59:40.846911    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-nm6pr"
Jul 27 20:59:40.882300 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:59:40.882247    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-nm6pr" podStartSLOduration=1.027835496 podStartE2EDuration="2.88223412s" totalImagesPullingTime="1.854398624s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:59:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:59:40.881319167 +0000 UTC m=+497.633180962" watchObservedRunningTime="2026-07-27 20:59:40.88223412 +0000 UTC m=+497.634095915"
Jul 27 20:59:40.891694 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:59:40.891666    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qngr7\" (UniqueName: \"kubernetes.io/projected/57435120-3648-4a21-8c84-7c0d3d8abc3a-kube-api-access-qngr7\") pod \"limitador-operator-controller-manager-859d7b78bd-wqzzm\" (UID: \"57435120-3648-4a21-8c84-7c0d3d8abc3a\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-wqzzm"
Jul 27 20:59:40.992742 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:59:40.992711    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qngr7\" (UniqueName: \"kubernetes.io/projected/57435120-3648-4a21-8c84-7c0d3d8abc3a-kube-api-access-qngr7\") pod \"limitador-operator-controller-manager-859d7b78bd-wqzzm\" (UID: \"57435120-3648-4a21-8c84-7c0d3d8abc3a\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-wqzzm"
Jul 27 20:59:41.001461 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:59:41.001436    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qngr7\" (UniqueName: \"kubernetes.io/projected/57435120-3648-4a21-8c84-7c0d3d8abc3a-kube-api-access-qngr7\") pod \"limitador-operator-controller-manager-859d7b78bd-wqzzm\" (UID: \"57435120-3648-4a21-8c84-7c0d3d8abc3a\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-wqzzm"
Jul 27 20:59:41.140846 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:59:41.140772    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-wqzzm"
Jul 27 20:59:41.274167 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:59:41.274140    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-wqzzm"]
Jul 27 20:59:41.276305 ip-10-0-141-64 kubenswrapper[2560]: W0727 20:59:41.276275    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57435120_3648_4a21_8c84_7c0d3d8abc3a.slice/crio-4670b92a8c7af416db2b82141241774b9813cc7ced8092d1831ca2ad919d928e WatchSource:0}: Error finding container 4670b92a8c7af416db2b82141241774b9813cc7ced8092d1831ca2ad919d928e: Status 404 returned error can't find the container with id 4670b92a8c7af416db2b82141241774b9813cc7ced8092d1831ca2ad919d928e
Jul 27 20:59:41.857986 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:59:41.857945    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-wqzzm" event={"ID":"57435120-3648-4a21-8c84-7c0d3d8abc3a","Type":"ContainerStarted","Data":"4670b92a8c7af416db2b82141241774b9813cc7ced8092d1831ca2ad919d928e"}
Jul 27 20:59:42.868888 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:59:42.868856    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-wqzzm" event={"ID":"57435120-3648-4a21-8c84-7c0d3d8abc3a","Type":"ContainerStarted","Data":"0f0015da7d73a1a894bef04e57b3c064888b9b1b8d438a313c4d2912b5052a67"}
Jul 27 20:59:42.869214 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:59:42.868993    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-wqzzm"
Jul 27 20:59:42.887812 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:59:42.887770    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-wqzzm" podStartSLOduration=1.392106939 podStartE2EDuration="2.887757367s" totalImagesPullingTime="1.495650428s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:59:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:59:42.886267514 +0000 UTC m=+499.638129308" watchObservedRunningTime="2026-07-27 20:59:42.887757367 +0000 UTC m=+499.639619161"
Jul 27 20:59:46.796697 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:59:46.796659    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-95d667bb8-sxpj4"
Jul 27 20:59:46.797116 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:59:46.796725    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-95d667bb8-sxpj4"
Jul 27 20:59:46.801460 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:59:46.801435    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-95d667bb8-sxpj4"
Jul 27 20:59:46.887730 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:59:46.887699    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-95d667bb8-sxpj4"
Jul 27 20:59:46.937667 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:59:46.937633    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-768cd866db-7n9mq"]
Jul 27 20:59:51.861236 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:59:51.861205    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-nm6pr"
Jul 27 20:59:53.875221 ip-10-0-141-64 kubenswrapper[2560]: I0727 20:59:53.875186    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-wqzzm"
Jul 27 21:00:03.444796 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:03.444756    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-rhlnk"]
Jul 27 21:00:03.448275 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:03.448257    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-rhlnk"
Jul 27 21:00:03.451386 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:03.451365    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-qm5lx\""
Jul 27 21:00:03.462329 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:03.462303    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-rhlnk"]
Jul 27 21:00:03.594399 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:03.594361    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hffmz\" (UniqueName: \"kubernetes.io/projected/4cc93eed-ee04-4b12-bfc2-a9205e94f992-kube-api-access-hffmz\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-rhlnk\" (UID: \"4cc93eed-ee04-4b12-bfc2-a9205e94f992\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-rhlnk"
Jul 27 21:00:03.594580 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:03.594409    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4cc93eed-ee04-4b12-bfc2-a9205e94f992-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-rhlnk\" (UID: \"4cc93eed-ee04-4b12-bfc2-a9205e94f992\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-rhlnk"
Jul 27 21:00:03.695361 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:03.695273    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4cc93eed-ee04-4b12-bfc2-a9205e94f992-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-rhlnk\" (UID: \"4cc93eed-ee04-4b12-bfc2-a9205e94f992\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-rhlnk"
Jul 27 21:00:03.695361 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:03.695351    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hffmz\" (UniqueName: \"kubernetes.io/projected/4cc93eed-ee04-4b12-bfc2-a9205e94f992-kube-api-access-hffmz\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-rhlnk\" (UID: \"4cc93eed-ee04-4b12-bfc2-a9205e94f992\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-rhlnk"
Jul 27 21:00:03.695715 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:03.695690    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4cc93eed-ee04-4b12-bfc2-a9205e94f992-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-rhlnk\" (UID: \"4cc93eed-ee04-4b12-bfc2-a9205e94f992\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-rhlnk"
Jul 27 21:00:03.721673 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:03.721638    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hffmz\" (UniqueName: \"kubernetes.io/projected/4cc93eed-ee04-4b12-bfc2-a9205e94f992-kube-api-access-hffmz\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-rhlnk\" (UID: \"4cc93eed-ee04-4b12-bfc2-a9205e94f992\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-rhlnk"
Jul 27 21:00:03.758131 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:03.758100    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-rhlnk"
Jul 27 21:00:03.908620 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:03.908584    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-rhlnk"]
Jul 27 21:00:03.910762 ip-10-0-141-64 kubenswrapper[2560]: W0727 21:00:03.910735    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cc93eed_ee04_4b12_bfc2_a9205e94f992.slice/crio-26f9a979f9bba048afe89214d7edb993d8736c536def4c3a9657769c26d6a0ec WatchSource:0}: Error finding container 26f9a979f9bba048afe89214d7edb993d8736c536def4c3a9657769c26d6a0ec: Status 404 returned error can't find the container with id 26f9a979f9bba048afe89214d7edb993d8736c536def4c3a9657769c26d6a0ec
Jul 27 21:00:03.964855 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:03.964786    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-rhlnk" event={"ID":"4cc93eed-ee04-4b12-bfc2-a9205e94f992","Type":"ContainerStarted","Data":"26f9a979f9bba048afe89214d7edb993d8736c536def4c3a9657769c26d6a0ec"}
Jul 27 21:00:04.180898 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:04.180866    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-rhlnk"]
Jul 27 21:00:04.185458 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:04.185433    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-rhlnk"]
Jul 27 21:00:04.235599 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:04.235519    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-b9j4q"]
Jul 27 21:00:04.238896 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:04.238875    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-b9j4q"
Jul 27 21:00:04.247003 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:04.246951    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-wqzzm"]
Jul 27 21:00:04.247194 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:04.247161    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-wqzzm" podUID="57435120-3648-4a21-8c84-7c0d3d8abc3a" containerName="manager" containerID="cri-o://0f0015da7d73a1a894bef04e57b3c064888b9b1b8d438a313c4d2912b5052a67" gracePeriod=2
Jul 27 21:00:04.269158 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:04.263611    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-b9j4q"]
Jul 27 21:00:04.273076 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:04.273051    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-wqzzm"]
Jul 27 21:00:04.291239 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:04.291217    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zk927"]
Jul 27 21:00:04.291513 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:04.291502    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57435120-3648-4a21-8c84-7c0d3d8abc3a" containerName="manager"
Jul 27 21:00:04.291554 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:04.291515    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="57435120-3648-4a21-8c84-7c0d3d8abc3a" containerName="manager"
Jul 27 21:00:04.291584 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:04.291569    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="57435120-3648-4a21-8c84-7c0d3d8abc3a" containerName="manager"
Jul 27 21:00:04.293390 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:04.293376    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zk927"
Jul 27 21:00:04.306731 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:04.306704    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zk927"]
Jul 27 21:00:04.325927 ip-10-0-141-64 kubenswrapper[2560]: E0727 21:00:04.325896    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-wqzzm\" is forbidden: User \"system:node:ip-10-0-141-64.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-64.ec2.internal' and this object" podUID="57435120-3648-4a21-8c84-7c0d3d8abc3a" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-wqzzm"
Jul 27 21:00:04.401223 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:04.401191    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8lnw\" (UniqueName: \"kubernetes.io/projected/9e1a3adc-36d5-4b45-a507-8eeacdd33818-kube-api-access-f8lnw\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-b9j4q\" (UID: \"9e1a3adc-36d5-4b45-a507-8eeacdd33818\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-b9j4q"
Jul 27 21:00:04.401384 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:04.401265    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvrpb\" (UniqueName: \"kubernetes.io/projected/8c076ad1-452f-4f41-9c2c-7684990f388f-kube-api-access-nvrpb\") pod \"limitador-operator-controller-manager-859d7b78bd-zk927\" (UID: \"8c076ad1-452f-4f41-9c2c-7684990f388f\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zk927"
Jul 27 21:00:04.401384 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:04.401295    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9e1a3adc-36d5-4b45-a507-8eeacdd33818-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-b9j4q\" (UID: \"9e1a3adc-36d5-4b45-a507-8eeacdd33818\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-b9j4q"
Jul 27 21:00:04.484729 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:04.484703    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-wqzzm"
Jul 27 21:00:04.489429 ip-10-0-141-64 kubenswrapper[2560]: E0727 21:00:04.489370    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-wqzzm\" is forbidden: User \"system:node:ip-10-0-141-64.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-64.ec2.internal' and this object" podUID="57435120-3648-4a21-8c84-7c0d3d8abc3a" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-wqzzm"
Jul 27 21:00:04.501753 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:04.501731    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8lnw\" (UniqueName: \"kubernetes.io/projected/9e1a3adc-36d5-4b45-a507-8eeacdd33818-kube-api-access-f8lnw\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-b9j4q\" (UID: \"9e1a3adc-36d5-4b45-a507-8eeacdd33818\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-b9j4q"
Jul 27 21:00:04.501826 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:04.501786    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvrpb\" (UniqueName: \"kubernetes.io/projected/8c076ad1-452f-4f41-9c2c-7684990f388f-kube-api-access-nvrpb\") pod \"limitador-operator-controller-manager-859d7b78bd-zk927\" (UID: \"8c076ad1-452f-4f41-9c2c-7684990f388f\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zk927"
Jul 27 21:00:04.501826 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:04.501813    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9e1a3adc-36d5-4b45-a507-8eeacdd33818-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-b9j4q\" (UID: \"9e1a3adc-36d5-4b45-a507-8eeacdd33818\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-b9j4q"
Jul 27 21:00:04.502185 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:04.502170    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9e1a3adc-36d5-4b45-a507-8eeacdd33818-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-b9j4q\" (UID: \"9e1a3adc-36d5-4b45-a507-8eeacdd33818\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-b9j4q"
Jul 27 21:00:04.511423 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:04.511400    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvrpb\" (UniqueName: \"kubernetes.io/projected/8c076ad1-452f-4f41-9c2c-7684990f388f-kube-api-access-nvrpb\") pod \"limitador-operator-controller-manager-859d7b78bd-zk927\" (UID: \"8c076ad1-452f-4f41-9c2c-7684990f388f\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zk927"
Jul 27 21:00:04.511717 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:04.511698    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8lnw\" (UniqueName: \"kubernetes.io/projected/9e1a3adc-36d5-4b45-a507-8eeacdd33818-kube-api-access-f8lnw\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-b9j4q\" (UID: \"9e1a3adc-36d5-4b45-a507-8eeacdd33818\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-b9j4q"
Jul 27 21:00:04.550181 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:04.550156    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-b9j4q"
Jul 27 21:00:04.602601 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:04.602570    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qngr7\" (UniqueName: \"kubernetes.io/projected/57435120-3648-4a21-8c84-7c0d3d8abc3a-kube-api-access-qngr7\") pod \"57435120-3648-4a21-8c84-7c0d3d8abc3a\" (UID: \"57435120-3648-4a21-8c84-7c0d3d8abc3a\") "
Jul 27 21:00:04.605327 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:04.605265    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57435120-3648-4a21-8c84-7c0d3d8abc3a-kube-api-access-qngr7" (OuterVolumeSpecName: "kube-api-access-qngr7") pod "57435120-3648-4a21-8c84-7c0d3d8abc3a" (UID: "57435120-3648-4a21-8c84-7c0d3d8abc3a"). InnerVolumeSpecName "kube-api-access-qngr7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:00:04.618418 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:04.618398    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zk927"
Jul 27 21:00:04.694420 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:04.694392    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-b9j4q"]
Jul 27 21:00:04.695461 ip-10-0-141-64 kubenswrapper[2560]: W0727 21:00:04.695435    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e1a3adc_36d5_4b45_a507_8eeacdd33818.slice/crio-d29bbb4e839aa1eb027a0391247c3ade2cfcd15bc4aaba4101e7c44b120da0ff WatchSource:0}: Error finding container d29bbb4e839aa1eb027a0391247c3ade2cfcd15bc4aaba4101e7c44b120da0ff: Status 404 returned error can't find the container with id d29bbb4e839aa1eb027a0391247c3ade2cfcd15bc4aaba4101e7c44b120da0ff
Jul 27 21:00:04.703751 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:04.703719    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qngr7\" (UniqueName: \"kubernetes.io/projected/57435120-3648-4a21-8c84-7c0d3d8abc3a-kube-api-access-qngr7\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:00:04.755769 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:04.755744    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zk927"]
Jul 27 21:00:04.758360 ip-10-0-141-64 kubenswrapper[2560]: W0727 21:00:04.758337    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c076ad1_452f_4f41_9c2c_7684990f388f.slice/crio-27ad4d1943b8db61584699eb50514e4ae06cc1a9f5877a78c0818ddb7e8a79bd WatchSource:0}: Error finding container 27ad4d1943b8db61584699eb50514e4ae06cc1a9f5877a78c0818ddb7e8a79bd: Status 404 returned error can't find the container with id 27ad4d1943b8db61584699eb50514e4ae06cc1a9f5877a78c0818ddb7e8a79bd
Jul 27 21:00:04.971456 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:04.971416    2560 generic.go:356] "Generic (PLEG): container finished" podID="57435120-3648-4a21-8c84-7c0d3d8abc3a" containerID="0f0015da7d73a1a894bef04e57b3c064888b9b1b8d438a313c4d2912b5052a67" exitCode=0
Jul 27 21:00:04.971645 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:04.971472    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-wqzzm"
Jul 27 21:00:04.971645 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:04.971517    2560 scope.go:117] "RemoveContainer" containerID="0f0015da7d73a1a894bef04e57b3c064888b9b1b8d438a313c4d2912b5052a67"
Jul 27 21:00:04.973390 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:04.973353    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-b9j4q" event={"ID":"9e1a3adc-36d5-4b45-a507-8eeacdd33818","Type":"ContainerStarted","Data":"d29bbb4e839aa1eb027a0391247c3ade2cfcd15bc4aaba4101e7c44b120da0ff"}
Jul 27 21:00:04.974186 ip-10-0-141-64 kubenswrapper[2560]: E0727 21:00:04.974152    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-wqzzm\" is forbidden: User \"system:node:ip-10-0-141-64.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-64.ec2.internal' and this object" podUID="57435120-3648-4a21-8c84-7c0d3d8abc3a" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-wqzzm"
Jul 27 21:00:04.975840 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:04.975812    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zk927" event={"ID":"8c076ad1-452f-4f41-9c2c-7684990f388f","Type":"ContainerStarted","Data":"7cfc78d2472de64a34c385e393f559b8c09f9022f5c9892f25d9193401192cb7"}
Jul 27 21:00:04.975966 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:04.975859    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zk927" event={"ID":"8c076ad1-452f-4f41-9c2c-7684990f388f","Type":"ContainerStarted","Data":"27ad4d1943b8db61584699eb50514e4ae06cc1a9f5877a78c0818ddb7e8a79bd"}
Jul 27 21:00:04.975966 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:04.975944    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zk927"
Jul 27 21:00:04.978041 ip-10-0-141-64 kubenswrapper[2560]: E0727 21:00:04.977977    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-wqzzm\" is forbidden: User \"system:node:ip-10-0-141-64.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-64.ec2.internal' and this object" podUID="57435120-3648-4a21-8c84-7c0d3d8abc3a" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-wqzzm"
Jul 27 21:00:04.984054 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:04.984033    2560 scope.go:117] "RemoveContainer" containerID="0f0015da7d73a1a894bef04e57b3c064888b9b1b8d438a313c4d2912b5052a67"
Jul 27 21:00:04.984395 ip-10-0-141-64 kubenswrapper[2560]: E0727 21:00:04.984375    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f0015da7d73a1a894bef04e57b3c064888b9b1b8d438a313c4d2912b5052a67\": container with ID starting with 0f0015da7d73a1a894bef04e57b3c064888b9b1b8d438a313c4d2912b5052a67 not found: ID does not exist" containerID="0f0015da7d73a1a894bef04e57b3c064888b9b1b8d438a313c4d2912b5052a67"
Jul 27 21:00:04.984484 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:04.984402    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f0015da7d73a1a894bef04e57b3c064888b9b1b8d438a313c4d2912b5052a67"} err="failed to get container status \"0f0015da7d73a1a894bef04e57b3c064888b9b1b8d438a313c4d2912b5052a67\": rpc error: code = NotFound desc = could not find container \"0f0015da7d73a1a894bef04e57b3c064888b9b1b8d438a313c4d2912b5052a67\": container with ID starting with 0f0015da7d73a1a894bef04e57b3c064888b9b1b8d438a313c4d2912b5052a67 not found: ID does not exist"
Jul 27 21:00:04.999122 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:04.999054    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zk927" podStartSLOduration=0.999036969 podStartE2EDuration="999.036969ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:00:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:00:04.997616443 +0000 UTC m=+521.749478257" watchObservedRunningTime="2026-07-27 21:00:04.999036969 +0000 UTC m=+521.750898766"
Jul 27 21:00:05.000459 ip-10-0-141-64 kubenswrapper[2560]: E0727 21:00:05.000425    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-wqzzm\" is forbidden: User \"system:node:ip-10-0-141-64.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-64.ec2.internal' and this object" podUID="57435120-3648-4a21-8c84-7c0d3d8abc3a" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-wqzzm"
Jul 27 21:00:05.849578 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:05.849537    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="57435120-3648-4a21-8c84-7c0d3d8abc3a" path="/var/lib/kubelet/pods/57435120-3648-4a21-8c84-7c0d3d8abc3a/volumes"
Jul 27 21:00:09.001382 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:09.001276    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-rhlnk" podUID="4cc93eed-ee04-4b12-bfc2-a9205e94f992" containerName="manager" containerID="cri-o://9ce352ee8acc8c8acc65efa0e80138b34b82fbff1041dd3413c2ba4ec2a7f823" gracePeriod=2
Jul 27 21:00:09.002991 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:09.002961    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-b9j4q" event={"ID":"9e1a3adc-36d5-4b45-a507-8eeacdd33818","Type":"ContainerStarted","Data":"26c5e3d630842a675a4801c4658f9cc50c1363aa2fc317d5c8d1980edcfe2b13"}
Jul 27 21:00:09.003134 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:09.003120    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-b9j4q"
Jul 27 21:00:09.025451 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:09.025406    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-b9j4q" podStartSLOduration=1.01933305 podStartE2EDuration="5.02538672s" totalImagesPullingTime="4.00605367s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:00:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:00:09.023840476 +0000 UTC m=+525.775702291" watchObservedRunningTime="2026-07-27 21:00:09.02538672 +0000 UTC m=+525.777248515"
Jul 27 21:00:09.245259 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:09.245232    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-rhlnk"
Jul 27 21:00:09.247937 ip-10-0-141-64 kubenswrapper[2560]: E0727 21:00:09.247909    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-rhlnk\" is forbidden: User \"system:node:ip-10-0-141-64.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-64.ec2.internal' and this object" podUID="4cc93eed-ee04-4b12-bfc2-a9205e94f992" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-rhlnk"
Jul 27 21:00:09.348024 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:09.347996    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4cc93eed-ee04-4b12-bfc2-a9205e94f992-extensions-socket-volume\") pod \"4cc93eed-ee04-4b12-bfc2-a9205e94f992\" (UID: \"4cc93eed-ee04-4b12-bfc2-a9205e94f992\") "
Jul 27 21:00:09.348211 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:09.348130    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hffmz\" (UniqueName: \"kubernetes.io/projected/4cc93eed-ee04-4b12-bfc2-a9205e94f992-kube-api-access-hffmz\") pod \"4cc93eed-ee04-4b12-bfc2-a9205e94f992\" (UID: \"4cc93eed-ee04-4b12-bfc2-a9205e94f992\") "
Jul 27 21:00:09.348349 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:09.348324    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cc93eed-ee04-4b12-bfc2-a9205e94f992-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "4cc93eed-ee04-4b12-bfc2-a9205e94f992" (UID: "4cc93eed-ee04-4b12-bfc2-a9205e94f992"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:00:09.350273 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:09.350251    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cc93eed-ee04-4b12-bfc2-a9205e94f992-kube-api-access-hffmz" (OuterVolumeSpecName: "kube-api-access-hffmz") pod "4cc93eed-ee04-4b12-bfc2-a9205e94f992" (UID: "4cc93eed-ee04-4b12-bfc2-a9205e94f992"). InnerVolumeSpecName "kube-api-access-hffmz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:00:09.449438 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:09.449406    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hffmz\" (UniqueName: \"kubernetes.io/projected/4cc93eed-ee04-4b12-bfc2-a9205e94f992-kube-api-access-hffmz\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:00:09.449438 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:09.449436    2560 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4cc93eed-ee04-4b12-bfc2-a9205e94f992-extensions-socket-volume\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:00:09.847611 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:09.847578    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4cc93eed-ee04-4b12-bfc2-a9205e94f992" path="/var/lib/kubelet/pods/4cc93eed-ee04-4b12-bfc2-a9205e94f992/volumes"
Jul 27 21:00:10.008279 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:10.008171    2560 generic.go:356] "Generic (PLEG): container finished" podID="4cc93eed-ee04-4b12-bfc2-a9205e94f992" containerID="9ce352ee8acc8c8acc65efa0e80138b34b82fbff1041dd3413c2ba4ec2a7f823" exitCode=2
Jul 27 21:00:10.008279 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:10.008223    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-rhlnk"
Jul 27 21:00:10.008279 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:10.008274    2560 scope.go:117] "RemoveContainer" containerID="9ce352ee8acc8c8acc65efa0e80138b34b82fbff1041dd3413c2ba4ec2a7f823"
Jul 27 21:00:10.012528 ip-10-0-141-64 kubenswrapper[2560]: E0727 21:00:10.012505    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-rhlnk\" is forbidden: User \"system:node:ip-10-0-141-64.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-64.ec2.internal' and this object" podUID="4cc93eed-ee04-4b12-bfc2-a9205e94f992" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-rhlnk"
Jul 27 21:00:10.016702 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:10.016686    2560 scope.go:117] "RemoveContainer" containerID="9ce352ee8acc8c8acc65efa0e80138b34b82fbff1041dd3413c2ba4ec2a7f823"
Jul 27 21:00:10.016946 ip-10-0-141-64 kubenswrapper[2560]: E0727 21:00:10.016928    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ce352ee8acc8c8acc65efa0e80138b34b82fbff1041dd3413c2ba4ec2a7f823\": container with ID starting with 9ce352ee8acc8c8acc65efa0e80138b34b82fbff1041dd3413c2ba4ec2a7f823 not found: ID does not exist" containerID="9ce352ee8acc8c8acc65efa0e80138b34b82fbff1041dd3413c2ba4ec2a7f823"
Jul 27 21:00:10.017008 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:10.016953    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ce352ee8acc8c8acc65efa0e80138b34b82fbff1041dd3413c2ba4ec2a7f823"} err="failed to get container status \"9ce352ee8acc8c8acc65efa0e80138b34b82fbff1041dd3413c2ba4ec2a7f823\": rpc error: code = NotFound desc = could not find container \"9ce352ee8acc8c8acc65efa0e80138b34b82fbff1041dd3413c2ba4ec2a7f823\": container with ID starting with 9ce352ee8acc8c8acc65efa0e80138b34b82fbff1041dd3413c2ba4ec2a7f823 not found: ID does not exist"
Jul 27 21:00:11.961029 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:11.960988    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-768cd866db-7n9mq" podUID="6b851a82-227d-49cf-a910-ecbc2b59ecac" containerName="console" containerID="cri-o://54b6225f9525b93000db1e6527f8a5a6c5e9619e61ce2943b564bd802ed6b4ec" gracePeriod=15
Jul 27 21:00:12.210068 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:12.210040    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-768cd866db-7n9mq"
Jul 27 21:00:12.373059 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:12.373027    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b851a82-227d-49cf-a910-ecbc2b59ecac-trusted-ca-bundle\") pod \"6b851a82-227d-49cf-a910-ecbc2b59ecac\" (UID: \"6b851a82-227d-49cf-a910-ecbc2b59ecac\") "
Jul 27 21:00:12.373262 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:12.373070    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6b851a82-227d-49cf-a910-ecbc2b59ecac-console-oauth-config\") pod \"6b851a82-227d-49cf-a910-ecbc2b59ecac\" (UID: \"6b851a82-227d-49cf-a910-ecbc2b59ecac\") "
Jul 27 21:00:12.373262 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:12.373133    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6b851a82-227d-49cf-a910-ecbc2b59ecac-oauth-serving-cert\") pod \"6b851a82-227d-49cf-a910-ecbc2b59ecac\" (UID: \"6b851a82-227d-49cf-a910-ecbc2b59ecac\") "
Jul 27 21:00:12.373262 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:12.373160    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6b851a82-227d-49cf-a910-ecbc2b59ecac-service-ca\") pod \"6b851a82-227d-49cf-a910-ecbc2b59ecac\" (UID: \"6b851a82-227d-49cf-a910-ecbc2b59ecac\") "
Jul 27 21:00:12.373262 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:12.373180    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cnj8r\" (UniqueName: \"kubernetes.io/projected/6b851a82-227d-49cf-a910-ecbc2b59ecac-kube-api-access-cnj8r\") pod \"6b851a82-227d-49cf-a910-ecbc2b59ecac\" (UID: \"6b851a82-227d-49cf-a910-ecbc2b59ecac\") "
Jul 27 21:00:12.373262 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:12.373213    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6b851a82-227d-49cf-a910-ecbc2b59ecac-console-config\") pod \"6b851a82-227d-49cf-a910-ecbc2b59ecac\" (UID: \"6b851a82-227d-49cf-a910-ecbc2b59ecac\") "
Jul 27 21:00:12.373262 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:12.373250    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6b851a82-227d-49cf-a910-ecbc2b59ecac-console-serving-cert\") pod \"6b851a82-227d-49cf-a910-ecbc2b59ecac\" (UID: \"6b851a82-227d-49cf-a910-ecbc2b59ecac\") "
Jul 27 21:00:12.373617 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:12.373584    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b851a82-227d-49cf-a910-ecbc2b59ecac-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "6b851a82-227d-49cf-a910-ecbc2b59ecac" (UID: "6b851a82-227d-49cf-a910-ecbc2b59ecac"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:00:12.373738 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:12.373701    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b851a82-227d-49cf-a910-ecbc2b59ecac-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "6b851a82-227d-49cf-a910-ecbc2b59ecac" (UID: "6b851a82-227d-49cf-a910-ecbc2b59ecac"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:00:12.373738 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:12.373710    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b851a82-227d-49cf-a910-ecbc2b59ecac-service-ca" (OuterVolumeSpecName: "service-ca") pod "6b851a82-227d-49cf-a910-ecbc2b59ecac" (UID: "6b851a82-227d-49cf-a910-ecbc2b59ecac"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:00:12.373738 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:12.373717    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b851a82-227d-49cf-a910-ecbc2b59ecac-console-config" (OuterVolumeSpecName: "console-config") pod "6b851a82-227d-49cf-a910-ecbc2b59ecac" (UID: "6b851a82-227d-49cf-a910-ecbc2b59ecac"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:00:12.375478 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:12.375455    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b851a82-227d-49cf-a910-ecbc2b59ecac-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "6b851a82-227d-49cf-a910-ecbc2b59ecac" (UID: "6b851a82-227d-49cf-a910-ecbc2b59ecac"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:00:12.375554 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:12.375495    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b851a82-227d-49cf-a910-ecbc2b59ecac-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "6b851a82-227d-49cf-a910-ecbc2b59ecac" (UID: "6b851a82-227d-49cf-a910-ecbc2b59ecac"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:00:12.375554 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:12.375529    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b851a82-227d-49cf-a910-ecbc2b59ecac-kube-api-access-cnj8r" (OuterVolumeSpecName: "kube-api-access-cnj8r") pod "6b851a82-227d-49cf-a910-ecbc2b59ecac" (UID: "6b851a82-227d-49cf-a910-ecbc2b59ecac"). InnerVolumeSpecName "kube-api-access-cnj8r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:00:12.474279 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:12.474239    2560 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6b851a82-227d-49cf-a910-ecbc2b59ecac-console-serving-cert\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:00:12.474279 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:12.474278    2560 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b851a82-227d-49cf-a910-ecbc2b59ecac-trusted-ca-bundle\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:00:12.474454 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:12.474292    2560 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6b851a82-227d-49cf-a910-ecbc2b59ecac-console-oauth-config\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:00:12.474454 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:12.474302    2560 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6b851a82-227d-49cf-a910-ecbc2b59ecac-oauth-serving-cert\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:00:12.474454 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:12.474310    2560 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6b851a82-227d-49cf-a910-ecbc2b59ecac-service-ca\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:00:12.474454 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:12.474318    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnj8r\" (UniqueName: \"kubernetes.io/projected/6b851a82-227d-49cf-a910-ecbc2b59ecac-kube-api-access-cnj8r\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:00:12.474454 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:12.474327    2560 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6b851a82-227d-49cf-a910-ecbc2b59ecac-console-config\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:00:13.023144 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:13.023073    2560 generic.go:356] "Generic (PLEG): container finished" podID="6b851a82-227d-49cf-a910-ecbc2b59ecac" containerID="54b6225f9525b93000db1e6527f8a5a6c5e9619e61ce2943b564bd802ed6b4ec" exitCode=0
Jul 27 21:00:13.023144 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:13.023145    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-768cd866db-7n9mq"
Jul 27 21:00:13.023665 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:13.023169    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-768cd866db-7n9mq" event={"ID":"6b851a82-227d-49cf-a910-ecbc2b59ecac","Type":"ContainerDied","Data":"54b6225f9525b93000db1e6527f8a5a6c5e9619e61ce2943b564bd802ed6b4ec"}
Jul 27 21:00:13.023665 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:13.023235    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-768cd866db-7n9mq" event={"ID":"6b851a82-227d-49cf-a910-ecbc2b59ecac","Type":"ContainerDied","Data":"c0d78e33fb945c8ce5bf390ec2ca5ed835376ce4f6d9953925eda518433c93df"}
Jul 27 21:00:13.023665 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:13.023267    2560 scope.go:117] "RemoveContainer" containerID="54b6225f9525b93000db1e6527f8a5a6c5e9619e61ce2943b564bd802ed6b4ec"
Jul 27 21:00:13.032014 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:13.031997    2560 scope.go:117] "RemoveContainer" containerID="54b6225f9525b93000db1e6527f8a5a6c5e9619e61ce2943b564bd802ed6b4ec"
Jul 27 21:00:13.032300 ip-10-0-141-64 kubenswrapper[2560]: E0727 21:00:13.032281    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54b6225f9525b93000db1e6527f8a5a6c5e9619e61ce2943b564bd802ed6b4ec\": container with ID starting with 54b6225f9525b93000db1e6527f8a5a6c5e9619e61ce2943b564bd802ed6b4ec not found: ID does not exist" containerID="54b6225f9525b93000db1e6527f8a5a6c5e9619e61ce2943b564bd802ed6b4ec"
Jul 27 21:00:13.032361 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:13.032314    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54b6225f9525b93000db1e6527f8a5a6c5e9619e61ce2943b564bd802ed6b4ec"} err="failed to get container status \"54b6225f9525b93000db1e6527f8a5a6c5e9619e61ce2943b564bd802ed6b4ec\": rpc error: code = NotFound desc = could not find container \"54b6225f9525b93000db1e6527f8a5a6c5e9619e61ce2943b564bd802ed6b4ec\": container with ID starting with 54b6225f9525b93000db1e6527f8a5a6c5e9619e61ce2943b564bd802ed6b4ec not found: ID does not exist"
Jul 27 21:00:13.043905 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:13.043882    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-768cd866db-7n9mq"]
Jul 27 21:00:13.047164 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:13.047139    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-768cd866db-7n9mq"]
Jul 27 21:00:13.848354 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:13.848266    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b851a82-227d-49cf-a910-ecbc2b59ecac" path="/var/lib/kubelet/pods/6b851a82-227d-49cf-a910-ecbc2b59ecac/volumes"
Jul 27 21:00:15.984837 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:15.984807    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-zk927"
Jul 27 21:00:20.010865 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:20.010832    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-b9j4q"
Jul 27 21:00:32.684843 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:32.684808    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-b9j4q"]
Jul 27 21:00:32.685221 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:32.684996    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-b9j4q" podUID="9e1a3adc-36d5-4b45-a507-8eeacdd33818" containerName="manager" containerID="cri-o://26c5e3d630842a675a4801c4658f9cc50c1363aa2fc317d5c8d1980edcfe2b13" gracePeriod=10
Jul 27 21:00:32.939398 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:32.939338    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-b9j4q"
Jul 27 21:00:33.033719 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:33.033666    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9e1a3adc-36d5-4b45-a507-8eeacdd33818-extensions-socket-volume\") pod \"9e1a3adc-36d5-4b45-a507-8eeacdd33818\" (UID: \"9e1a3adc-36d5-4b45-a507-8eeacdd33818\") "
Jul 27 21:00:33.033894 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:33.033745    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f8lnw\" (UniqueName: \"kubernetes.io/projected/9e1a3adc-36d5-4b45-a507-8eeacdd33818-kube-api-access-f8lnw\") pod \"9e1a3adc-36d5-4b45-a507-8eeacdd33818\" (UID: \"9e1a3adc-36d5-4b45-a507-8eeacdd33818\") "
Jul 27 21:00:33.034135 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:33.034075    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e1a3adc-36d5-4b45-a507-8eeacdd33818-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "9e1a3adc-36d5-4b45-a507-8eeacdd33818" (UID: "9e1a3adc-36d5-4b45-a507-8eeacdd33818"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:00:33.035989 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:33.035962    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e1a3adc-36d5-4b45-a507-8eeacdd33818-kube-api-access-f8lnw" (OuterVolumeSpecName: "kube-api-access-f8lnw") pod "9e1a3adc-36d5-4b45-a507-8eeacdd33818" (UID: "9e1a3adc-36d5-4b45-a507-8eeacdd33818"). InnerVolumeSpecName "kube-api-access-f8lnw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:00:33.116773 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:33.116742    2560 generic.go:356] "Generic (PLEG): container finished" podID="9e1a3adc-36d5-4b45-a507-8eeacdd33818" containerID="26c5e3d630842a675a4801c4658f9cc50c1363aa2fc317d5c8d1980edcfe2b13" exitCode=0
Jul 27 21:00:33.116939 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:33.116788    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-b9j4q"
Jul 27 21:00:33.116939 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:33.116807    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-b9j4q" event={"ID":"9e1a3adc-36d5-4b45-a507-8eeacdd33818","Type":"ContainerDied","Data":"26c5e3d630842a675a4801c4658f9cc50c1363aa2fc317d5c8d1980edcfe2b13"}
Jul 27 21:00:33.116939 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:33.116848    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-b9j4q" event={"ID":"9e1a3adc-36d5-4b45-a507-8eeacdd33818","Type":"ContainerDied","Data":"d29bbb4e839aa1eb027a0391247c3ade2cfcd15bc4aaba4101e7c44b120da0ff"}
Jul 27 21:00:33.116939 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:33.116868    2560 scope.go:117] "RemoveContainer" containerID="26c5e3d630842a675a4801c4658f9cc50c1363aa2fc317d5c8d1980edcfe2b13"
Jul 27 21:00:33.125621 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:33.125600    2560 scope.go:117] "RemoveContainer" containerID="26c5e3d630842a675a4801c4658f9cc50c1363aa2fc317d5c8d1980edcfe2b13"
Jul 27 21:00:33.125861 ip-10-0-141-64 kubenswrapper[2560]: E0727 21:00:33.125840    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26c5e3d630842a675a4801c4658f9cc50c1363aa2fc317d5c8d1980edcfe2b13\": container with ID starting with 26c5e3d630842a675a4801c4658f9cc50c1363aa2fc317d5c8d1980edcfe2b13 not found: ID does not exist" containerID="26c5e3d630842a675a4801c4658f9cc50c1363aa2fc317d5c8d1980edcfe2b13"
Jul 27 21:00:33.125946 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:33.125870    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26c5e3d630842a675a4801c4658f9cc50c1363aa2fc317d5c8d1980edcfe2b13"} err="failed to get container status \"26c5e3d630842a675a4801c4658f9cc50c1363aa2fc317d5c8d1980edcfe2b13\": rpc error: code = NotFound desc = could not find container \"26c5e3d630842a675a4801c4658f9cc50c1363aa2fc317d5c8d1980edcfe2b13\": container with ID starting with 26c5e3d630842a675a4801c4658f9cc50c1363aa2fc317d5c8d1980edcfe2b13 not found: ID does not exist"
Jul 27 21:00:33.134784 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:33.134756    2560 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9e1a3adc-36d5-4b45-a507-8eeacdd33818-extensions-socket-volume\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:00:33.134784 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:33.134787    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8lnw\" (UniqueName: \"kubernetes.io/projected/9e1a3adc-36d5-4b45-a507-8eeacdd33818-kube-api-access-f8lnw\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:00:33.137471 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:33.137449    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-b9j4q"]
Jul 27 21:00:33.143706 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:33.143679    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-b9j4q"]
Jul 27 21:00:33.847028 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:33.846996    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9e1a3adc-36d5-4b45-a507-8eeacdd33818" path="/var/lib/kubelet/pods/9e1a3adc-36d5-4b45-a507-8eeacdd33818/volumes"
Jul 27 21:00:50.780202 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:50.780165    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:00:50.780676 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:50.780484    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4cc93eed-ee04-4b12-bfc2-a9205e94f992" containerName="manager"
Jul 27 21:00:50.780676 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:50.780495    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="4cc93eed-ee04-4b12-bfc2-a9205e94f992" containerName="manager"
Jul 27 21:00:50.780676 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:50.780507    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b851a82-227d-49cf-a910-ecbc2b59ecac" containerName="console"
Jul 27 21:00:50.780676 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:50.780512    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b851a82-227d-49cf-a910-ecbc2b59ecac" containerName="console"
Jul 27 21:00:50.780676 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:50.780520    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e1a3adc-36d5-4b45-a507-8eeacdd33818" containerName="manager"
Jul 27 21:00:50.780676 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:50.780526    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e1a3adc-36d5-4b45-a507-8eeacdd33818" containerName="manager"
Jul 27 21:00:50.780676 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:50.780579    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b851a82-227d-49cf-a910-ecbc2b59ecac" containerName="console"
Jul 27 21:00:50.780676 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:50.780588    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e1a3adc-36d5-4b45-a507-8eeacdd33818" containerName="manager"
Jul 27 21:00:50.780676 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:50.780596    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="4cc93eed-ee04-4b12-bfc2-a9205e94f992" containerName="manager"
Jul 27 21:00:50.782503 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:50.782488    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-7znrn"
Jul 27 21:00:50.785921 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:50.785902    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-lq9jt\""
Jul 27 21:00:50.786173 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:50.786157    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 27 21:00:50.792398 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:50.792375    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:00:50.829383 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:50.829351    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:00:50.879481 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:50.879447    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/bde6ee24-561a-417d-b2f8-1f24d1c55733-config-file\") pod \"limitador-limitador-54d6d988d5-7znrn\" (UID: \"bde6ee24-561a-417d-b2f8-1f24d1c55733\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-7znrn"
Jul 27 21:00:50.879618 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:50.879492    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbcm7\" (UniqueName: \"kubernetes.io/projected/bde6ee24-561a-417d-b2f8-1f24d1c55733-kube-api-access-xbcm7\") pod \"limitador-limitador-54d6d988d5-7znrn\" (UID: \"bde6ee24-561a-417d-b2f8-1f24d1c55733\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-7znrn"
Jul 27 21:00:50.980446 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:50.980408    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/bde6ee24-561a-417d-b2f8-1f24d1c55733-config-file\") pod \"limitador-limitador-54d6d988d5-7znrn\" (UID: \"bde6ee24-561a-417d-b2f8-1f24d1c55733\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-7znrn"
Jul 27 21:00:50.980619 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:50.980464    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbcm7\" (UniqueName: \"kubernetes.io/projected/bde6ee24-561a-417d-b2f8-1f24d1c55733-kube-api-access-xbcm7\") pod \"limitador-limitador-54d6d988d5-7znrn\" (UID: \"bde6ee24-561a-417d-b2f8-1f24d1c55733\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-7znrn"
Jul 27 21:00:50.981043 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:50.981022    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/bde6ee24-561a-417d-b2f8-1f24d1c55733-config-file\") pod \"limitador-limitador-54d6d988d5-7znrn\" (UID: \"bde6ee24-561a-417d-b2f8-1f24d1c55733\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-7znrn"
Jul 27 21:00:50.989709 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:50.989678    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbcm7\" (UniqueName: \"kubernetes.io/projected/bde6ee24-561a-417d-b2f8-1f24d1c55733-kube-api-access-xbcm7\") pod \"limitador-limitador-54d6d988d5-7znrn\" (UID: \"bde6ee24-561a-417d-b2f8-1f24d1c55733\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-7znrn"
Jul 27 21:00:51.093150 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:51.093066    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-7znrn"
Jul 27 21:00:51.233064 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:51.233029    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:00:51.236649 ip-10-0-141-64 kubenswrapper[2560]: W0727 21:00:51.236611    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbde6ee24_561a_417d_b2f8_1f24d1c55733.slice/crio-a4f4b25d76dfbf3a5567dbf579c7f95aa1a114129b657decc2afea7a28bbbd42 WatchSource:0}: Error finding container a4f4b25d76dfbf3a5567dbf579c7f95aa1a114129b657decc2afea7a28bbbd42: Status 404 returned error can't find the container with id a4f4b25d76dfbf3a5567dbf579c7f95aa1a114129b657decc2afea7a28bbbd42
Jul 27 21:00:52.205413 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:52.205342    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-7znrn" event={"ID":"bde6ee24-561a-417d-b2f8-1f24d1c55733","Type":"ContainerStarted","Data":"a4f4b25d76dfbf3a5567dbf579c7f95aa1a114129b657decc2afea7a28bbbd42"}
Jul 27 21:00:54.221304 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:54.221216    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-7znrn" event={"ID":"bde6ee24-561a-417d-b2f8-1f24d1c55733","Type":"ContainerStarted","Data":"3b90bb84995165df906a8bb87d5e3b0f9975b32d8f7320232fc44315f0a72a5d"}
Jul 27 21:00:54.221665 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:54.221360    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-7znrn"
Jul 27 21:00:54.237136 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:00:54.237068    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-7znrn" podStartSLOduration=1.511584475 podStartE2EDuration="4.237054616s" totalImagesPullingTime="2.725470141s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:00:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:00:54.235651233 +0000 UTC m=+570.987513061" watchObservedRunningTime="2026-07-27 21:00:54.237054616 +0000 UTC m=+570.988916411"
Jul 27 21:01:05.228449 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:01:05.228417    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-7znrn"
Jul 27 21:01:23.689873 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:01:23.689841    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-z6zxv_8a6b7ae6-6c21-453d-99b9-303066d19915/console-operator/1.log"
Jul 27 21:01:23.690961 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:01:23.690938    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-z6zxv_8a6b7ae6-6c21-453d-99b9-303066d19915/console-operator/1.log"
Jul 27 21:01:23.696953 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:01:23.696929    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-95hg7_433c431a-b5bf-4734-a03f-6f0a16895ded/ovn-acl-logging/0.log"
Jul 27 21:01:23.698208 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:01:23.698186    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-95hg7_433c431a-b5bf-4734-a03f-6f0a16895ded/ovn-acl-logging/0.log"
Jul 27 21:02:09.953196 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:02:09.953157    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:02:45.189190 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:02:45.189155    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-m5tt9"]
Jul 27 21:02:45.191930 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:02:45.191905    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-m5tt9"
Jul 27 21:02:45.194452 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:02:45.194426    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Jul 27 21:02:45.195516 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:02:45.195491    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 27 21:02:45.195652 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:02:45.195623    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 27 21:02:45.195788 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:02:45.195770    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-bt4k8\""
Jul 27 21:02:45.202235 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:02:45.202211    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-m5tt9"]
Jul 27 21:02:45.224681 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:02:45.224649    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3552283d-6dcb-4cd2-a850-621c4b698567-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-m5tt9\" (UID: \"3552283d-6dcb-4cd2-a850-621c4b698567\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-m5tt9"
Jul 27 21:02:45.224851 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:02:45.224697    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3552283d-6dcb-4cd2-a850-621c4b698567-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-m5tt9\" (UID: \"3552283d-6dcb-4cd2-a850-621c4b698567\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-m5tt9"
Jul 27 21:02:45.224851 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:02:45.224730    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3552283d-6dcb-4cd2-a850-621c4b698567-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-m5tt9\" (UID: \"3552283d-6dcb-4cd2-a850-621c4b698567\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-m5tt9"
Jul 27 21:02:45.224851 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:02:45.224763    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3552283d-6dcb-4cd2-a850-621c4b698567-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-m5tt9\" (UID: \"3552283d-6dcb-4cd2-a850-621c4b698567\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-m5tt9"
Jul 27 21:02:45.224851 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:02:45.224790    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66x98\" (UniqueName: \"kubernetes.io/projected/3552283d-6dcb-4cd2-a850-621c4b698567-kube-api-access-66x98\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-m5tt9\" (UID: \"3552283d-6dcb-4cd2-a850-621c4b698567\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-m5tt9"
Jul 27 21:02:45.224851 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:02:45.224850    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3552283d-6dcb-4cd2-a850-621c4b698567-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-m5tt9\" (UID: \"3552283d-6dcb-4cd2-a850-621c4b698567\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-m5tt9"
Jul 27 21:02:45.326144 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:02:45.326080    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3552283d-6dcb-4cd2-a850-621c4b698567-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-m5tt9\" (UID: \"3552283d-6dcb-4cd2-a850-621c4b698567\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-m5tt9"
Jul 27 21:02:45.326316 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:02:45.326155    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3552283d-6dcb-4cd2-a850-621c4b698567-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-m5tt9\" (UID: \"3552283d-6dcb-4cd2-a850-621c4b698567\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-m5tt9"
Jul 27 21:02:45.326316 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:02:45.326195    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3552283d-6dcb-4cd2-a850-621c4b698567-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-m5tt9\" (UID: \"3552283d-6dcb-4cd2-a850-621c4b698567\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-m5tt9"
Jul 27 21:02:45.326443 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:02:45.326323    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3552283d-6dcb-4cd2-a850-621c4b698567-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-m5tt9\" (UID: \"3552283d-6dcb-4cd2-a850-621c4b698567\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-m5tt9"
Jul 27 21:02:45.326443 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:02:45.326368    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3552283d-6dcb-4cd2-a850-621c4b698567-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-m5tt9\" (UID: \"3552283d-6dcb-4cd2-a850-621c4b698567\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-m5tt9"
Jul 27 21:02:45.326443 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:02:45.326418    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66x98\" (UniqueName: \"kubernetes.io/projected/3552283d-6dcb-4cd2-a850-621c4b698567-kube-api-access-66x98\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-m5tt9\" (UID: \"3552283d-6dcb-4cd2-a850-621c4b698567\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-m5tt9"
Jul 27 21:02:45.326648 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:02:45.326623    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3552283d-6dcb-4cd2-a850-621c4b698567-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-m5tt9\" (UID: \"3552283d-6dcb-4cd2-a850-621c4b698567\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-m5tt9"
Jul 27 21:02:45.326725 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:02:45.326693    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3552283d-6dcb-4cd2-a850-621c4b698567-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-m5tt9\" (UID: \"3552283d-6dcb-4cd2-a850-621c4b698567\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-m5tt9"
Jul 27 21:02:45.326811 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:02:45.326790    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3552283d-6dcb-4cd2-a850-621c4b698567-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-m5tt9\" (UID: \"3552283d-6dcb-4cd2-a850-621c4b698567\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-m5tt9"
Jul 27 21:02:45.328598 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:02:45.328573    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3552283d-6dcb-4cd2-a850-621c4b698567-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-m5tt9\" (UID: \"3552283d-6dcb-4cd2-a850-621c4b698567\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-m5tt9"
Jul 27 21:02:45.328938 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:02:45.328923    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3552283d-6dcb-4cd2-a850-621c4b698567-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-m5tt9\" (UID: \"3552283d-6dcb-4cd2-a850-621c4b698567\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-m5tt9"
Jul 27 21:02:45.334388 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:02:45.334370    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-66x98\" (UniqueName: \"kubernetes.io/projected/3552283d-6dcb-4cd2-a850-621c4b698567-kube-api-access-66x98\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-m5tt9\" (UID: \"3552283d-6dcb-4cd2-a850-621c4b698567\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-m5tt9"
Jul 27 21:02:45.504857 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:02:45.504771    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-m5tt9"
Jul 27 21:02:45.639864 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:02:45.639827    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-m5tt9"]
Jul 27 21:02:45.642838 ip-10-0-141-64 kubenswrapper[2560]: W0727 21:02:45.642812    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3552283d_6dcb_4cd2_a850_621c4b698567.slice/crio-fc4aef4dc434e21d479b7c8b70b0b8fd5b898265bac1a7abe97ebb73a2aeebb7 WatchSource:0}: Error finding container fc4aef4dc434e21d479b7c8b70b0b8fd5b898265bac1a7abe97ebb73a2aeebb7: Status 404 returned error can't find the container with id fc4aef4dc434e21d479b7c8b70b0b8fd5b898265bac1a7abe97ebb73a2aeebb7
Jul 27 21:02:45.644438 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:02:45.644416    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 21:02:45.694338 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:02:45.694306    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:02:45.713783 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:02:45.713750    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-m5tt9" event={"ID":"3552283d-6dcb-4cd2-a850-621c4b698567","Type":"ContainerStarted","Data":"fc4aef4dc434e21d479b7c8b70b0b8fd5b898265bac1a7abe97ebb73a2aeebb7"}
Jul 27 21:02:46.719886 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:02:46.719858    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-m5tt9" event={"ID":"3552283d-6dcb-4cd2-a850-621c4b698567","Type":"ContainerStarted","Data":"18f98652a99d2913723855af952097b1b18e61a861a6c8858014171ff9459b8f"}
Jul 27 21:02:46.742453 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:02:46.742403    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-m5tt9" podStartSLOduration=0.731246259 podStartE2EDuration="1.742387794s" totalImagesPullingTime="1.011141535s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:02:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:02:46.739715124 +0000 UTC m=+683.491576941" watchObservedRunningTime="2026-07-27 21:02:46.742387794 +0000 UTC m=+683.494249588"
Jul 27 21:02:55.505237 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:02:55.505183    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-m5tt9"
Jul 27 21:02:55.505630 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:02:55.505265    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-m5tt9"
Jul 27 21:02:55.517630 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:02:55.517609    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-m5tt9"
Jul 27 21:02:55.772950 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:02:55.772921    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-m5tt9"
Jul 27 21:02:56.296711 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:02:56.296678    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-jszvk"]
Jul 27 21:02:56.298941 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:02:56.298922    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jszvk"
Jul 27 21:02:56.307237 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:02:56.307213    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Jul 27 21:02:56.325385 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:02:56.325360    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-jszvk"]
Jul 27 21:02:56.427144 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:02:56.427104    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/759ce03b-f36f-48a3-a497-2b2dd4ab5f4e-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jszvk\" (UID: \"759ce03b-f36f-48a3-a497-2b2dd4ab5f4e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jszvk"
Jul 27 21:02:56.427312 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:02:56.427150    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/759ce03b-f36f-48a3-a497-2b2dd4ab5f4e-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jszvk\" (UID: \"759ce03b-f36f-48a3-a497-2b2dd4ab5f4e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jszvk"
Jul 27 21:02:56.427312 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:02:56.427194    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78rlk\" (UniqueName: \"kubernetes.io/projected/759ce03b-f36f-48a3-a497-2b2dd4ab5f4e-kube-api-access-78rlk\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jszvk\" (UID: \"759ce03b-f36f-48a3-a497-2b2dd4ab5f4e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jszvk"
Jul 27 21:02:56.427312 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:02:56.427288    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/759ce03b-f36f-48a3-a497-2b2dd4ab5f4e-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jszvk\" (UID: \"759ce03b-f36f-48a3-a497-2b2dd4ab5f4e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jszvk"
Jul 27 21:02:56.427443 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:02:56.427332    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/759ce03b-f36f-48a3-a497-2b2dd4ab5f4e-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jszvk\" (UID: \"759ce03b-f36f-48a3-a497-2b2dd4ab5f4e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jszvk"
Jul 27 21:02:56.427443 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:02:56.427346    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/759ce03b-f36f-48a3-a497-2b2dd4ab5f4e-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jszvk\" (UID: \"759ce03b-f36f-48a3-a497-2b2dd4ab5f4e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jszvk"
Jul 27 21:02:56.528275 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:02:56.528234    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/759ce03b-f36f-48a3-a497-2b2dd4ab5f4e-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jszvk\" (UID: \"759ce03b-f36f-48a3-a497-2b2dd4ab5f4e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jszvk"
Jul 27 21:02:56.528744 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:02:56.528288    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/759ce03b-f36f-48a3-a497-2b2dd4ab5f4e-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jszvk\" (UID: \"759ce03b-f36f-48a3-a497-2b2dd4ab5f4e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jszvk"
Jul 27 21:02:56.528744 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:02:56.528311    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/759ce03b-f36f-48a3-a497-2b2dd4ab5f4e-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jszvk\" (UID: \"759ce03b-f36f-48a3-a497-2b2dd4ab5f4e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jszvk"
Jul 27 21:02:56.528744 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:02:56.528355    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/759ce03b-f36f-48a3-a497-2b2dd4ab5f4e-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jszvk\" (UID: \"759ce03b-f36f-48a3-a497-2b2dd4ab5f4e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jszvk"
Jul 27 21:02:56.528744 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:02:56.528447    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/759ce03b-f36f-48a3-a497-2b2dd4ab5f4e-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jszvk\" (UID: \"759ce03b-f36f-48a3-a497-2b2dd4ab5f4e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jszvk"
Jul 27 21:02:56.528744 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:02:56.528532    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78rlk\" (UniqueName: \"kubernetes.io/projected/759ce03b-f36f-48a3-a497-2b2dd4ab5f4e-kube-api-access-78rlk\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jszvk\" (UID: \"759ce03b-f36f-48a3-a497-2b2dd4ab5f4e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jszvk"
Jul 27 21:02:56.528744 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:02:56.528667    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/759ce03b-f36f-48a3-a497-2b2dd4ab5f4e-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jszvk\" (UID: \"759ce03b-f36f-48a3-a497-2b2dd4ab5f4e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jszvk"
Jul 27 21:02:56.528744 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:02:56.528727    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/759ce03b-f36f-48a3-a497-2b2dd4ab5f4e-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jszvk\" (UID: \"759ce03b-f36f-48a3-a497-2b2dd4ab5f4e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jszvk"
Jul 27 21:02:56.528995 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:02:56.528749    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/759ce03b-f36f-48a3-a497-2b2dd4ab5f4e-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jszvk\" (UID: \"759ce03b-f36f-48a3-a497-2b2dd4ab5f4e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jszvk"
Jul 27 21:02:56.530696 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:02:56.530675    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/759ce03b-f36f-48a3-a497-2b2dd4ab5f4e-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jszvk\" (UID: \"759ce03b-f36f-48a3-a497-2b2dd4ab5f4e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jszvk"
Jul 27 21:02:56.531074 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:02:56.531055    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/759ce03b-f36f-48a3-a497-2b2dd4ab5f4e-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jszvk\" (UID: \"759ce03b-f36f-48a3-a497-2b2dd4ab5f4e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jszvk"
Jul 27 21:02:56.537779 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:02:56.537742    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-78rlk\" (UniqueName: \"kubernetes.io/projected/759ce03b-f36f-48a3-a497-2b2dd4ab5f4e-kube-api-access-78rlk\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jszvk\" (UID: \"759ce03b-f36f-48a3-a497-2b2dd4ab5f4e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jszvk"
Jul 27 21:02:56.609056 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:02:56.608989    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jszvk"
Jul 27 21:02:56.750225 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:02:56.750198    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-jszvk"]
Jul 27 21:02:56.752769 ip-10-0-141-64 kubenswrapper[2560]: W0727 21:02:56.752742    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod759ce03b_f36f_48a3_a497_2b2dd4ab5f4e.slice/crio-dd7bfa6fa63e7c29ebebeaaae02a69be38a58689eb3ee0e760ab8d21e0998562 WatchSource:0}: Error finding container dd7bfa6fa63e7c29ebebeaaae02a69be38a58689eb3ee0e760ab8d21e0998562: Status 404 returned error can't find the container with id dd7bfa6fa63e7c29ebebeaaae02a69be38a58689eb3ee0e760ab8d21e0998562
Jul 27 21:02:56.769571 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:02:56.769544    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jszvk" event={"ID":"759ce03b-f36f-48a3-a497-2b2dd4ab5f4e","Type":"ContainerStarted","Data":"dd7bfa6fa63e7c29ebebeaaae02a69be38a58689eb3ee0e760ab8d21e0998562"}
Jul 27 21:02:57.121454 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:02:57.121420    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:02:57.775027 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:02:57.774992    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jszvk" event={"ID":"759ce03b-f36f-48a3-a497-2b2dd4ab5f4e","Type":"ContainerStarted","Data":"f2569a6938994e3ea2eeb49c983cea75e0443dcd849ee0f5bf06ca59f1f98033"}
Jul 27 21:02:57.795768 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:02:57.795718    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jszvk" podStartSLOduration=1.604119423 podStartE2EDuration="1.795701973s" totalImagesPullingTime="191.58255ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:02:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:02:57.793733597 +0000 UTC m=+694.545595392" watchObservedRunningTime="2026-07-27 21:02:57.795701973 +0000 UTC m=+694.547563957"
Jul 27 21:03:00.999947 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:03:00.999914    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:03:06.609829 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:03:06.609790    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jszvk"
Jul 27 21:03:06.610231 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:03:06.609858    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jszvk"
Jul 27 21:03:06.622211 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:03:06.622189    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jszvk"
Jul 27 21:03:06.825183 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:03:06.825144    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jszvk"
Jul 27 21:03:09.098883 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:03:09.098844    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:03:21.498763 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:03:21.498674    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:03:33.201609 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:03:33.201565    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:04:39.088441 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:04:39.088407    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:04:48.582130 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:04:48.582075    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:04:59.693861 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:04:59.693830    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:06:23.717515 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:06:23.717426    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-z6zxv_8a6b7ae6-6c21-453d-99b9-303066d19915/console-operator/1.log"
Jul 27 21:06:23.718067 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:06:23.717569    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-z6zxv_8a6b7ae6-6c21-453d-99b9-303066d19915/console-operator/1.log"
Jul 27 21:06:23.724254 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:06:23.724229    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-95hg7_433c431a-b5bf-4734-a03f-6f0a16895ded/ovn-acl-logging/0.log"
Jul 27 21:06:23.724455 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:06:23.724438    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-95hg7_433c431a-b5bf-4734-a03f-6f0a16895ded/ovn-acl-logging/0.log"
Jul 27 21:06:38.594853 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:06:38.594813    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:06:50.697746 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:06:50.697707    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:07:25.993744 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:07:25.993708    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:07:35.495164 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:07:35.495127    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:07:44.292370 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:07:44.292268    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:07:55.890129 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:07:55.890081    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:08:07.201939 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:08:07.201898    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:08:16.504949 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:08:16.504911    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:08:28.096038 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:08:28.096005    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:08:38.093683 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:08:38.093646    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:09:00.594928 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:09:00.594890    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:09:11.289270 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:09:11.289231    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:09:34.599554 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:09:34.599470    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:09:45.491349 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:09:45.491306    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:10:01.091504 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:10:01.091473    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:10:10.890028 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:10:10.889987    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:10:20.699560 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:10:20.699525    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:10:30.289876 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:10:30.289831    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:10:39.891691 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:10:39.891652    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:11:23.742977 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:11:23.742948    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-z6zxv_8a6b7ae6-6c21-453d-99b9-303066d19915/console-operator/1.log"
Jul 27 21:11:23.743815 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:11:23.743793    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-z6zxv_8a6b7ae6-6c21-453d-99b9-303066d19915/console-operator/1.log"
Jul 27 21:11:23.749665 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:11:23.749648    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-95hg7_433c431a-b5bf-4734-a03f-6f0a16895ded/ovn-acl-logging/0.log"
Jul 27 21:11:23.750285 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:11:23.750266    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-95hg7_433c431a-b5bf-4734-a03f-6f0a16895ded/ovn-acl-logging/0.log"
Jul 27 21:12:29.790488 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:12:29.790449    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:12:41.395013 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:12:41.394974    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:12:50.538370 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:12:50.538333    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:13:02.291228 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:13:02.291185    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:13:13.199240 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:13:13.199208    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:13:24.287018 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:13:24.286979    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:13:34.711777 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:13:34.711736    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:13:44.483362 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:13:44.483280    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:13:56.506777 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:13:56.506741    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:14:04.110477 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:14:04.110441    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:14:13.499503 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:14:13.499461    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:14:21.393306 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:14:21.393260    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:14:40.607655 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:14:40.607624    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:14:50.298641 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:14:50.298594    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:15:00.138401 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:15:00.138365    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29753115-t8g6v"]
Jul 27 21:15:00.141753 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:15:00.141737    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753115-t8g6v"
Jul 27 21:15:00.144385 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:15:00.144354    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j5rjm\""
Jul 27 21:15:00.148500 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:15:00.148463    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29753115-t8g6v"]
Jul 27 21:15:00.152474 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:15:00.152454    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753115-t8g6v"
Jul 27 21:15:00.285458 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:15:00.285436    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29753115-t8g6v"]
Jul 27 21:15:00.289448 ip-10-0-141-64 kubenswrapper[2560]: W0727 21:15:00.289411    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8e679a9_15f2_4f34_81f6_399734e7a830.slice/crio-9354115c1bd53fa9381eb0a72eb485d1922f0b85d921fb47850525eeaa308777 WatchSource:0}: Error finding container 9354115c1bd53fa9381eb0a72eb485d1922f0b85d921fb47850525eeaa308777: Status 404 returned error can't find the container with id 9354115c1bd53fa9381eb0a72eb485d1922f0b85d921fb47850525eeaa308777
Jul 27 21:15:00.291762 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:15:00.291741    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 21:15:00.967133 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:15:00.967102    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753115-t8g6v" event={"ID":"c8e679a9-15f2-4f34-81f6-399734e7a830","Type":"ContainerStarted","Data":"9354115c1bd53fa9381eb0a72eb485d1922f0b85d921fb47850525eeaa308777"}
Jul 27 21:15:02.695103 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:15:02.695056    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:15:02.978889 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:15:02.978856    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753115-t8g6v" event={"ID":"c8e679a9-15f2-4f34-81f6-399734e7a830","Type":"ContainerStarted","Data":"44a72bb681750a4c75b96f3a0702508468ad676396db92d8365f30cfc628ea67"}
Jul 27 21:15:02.995216 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:15:02.995170    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753115-t8g6v" podStartSLOduration=0.631016703 podStartE2EDuration="2.995154894s" totalImagesPullingTime="2.364138191s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:15:02.993801814 +0000 UTC m=+1419.745663611" watchObservedRunningTime="2026-07-27 21:15:02.995154894 +0000 UTC m=+1419.747016689"
Jul 27 21:15:03.985079 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:15:03.984992    2560 generic.go:356] "Generic (PLEG): container finished" podID="c8e679a9-15f2-4f34-81f6-399734e7a830" containerID="44a72bb681750a4c75b96f3a0702508468ad676396db92d8365f30cfc628ea67" exitCode=0
Jul 27 21:15:03.985433 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:15:03.985069    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753115-t8g6v" event={"ID":"c8e679a9-15f2-4f34-81f6-399734e7a830","Type":"ContainerDied","Data":"44a72bb681750a4c75b96f3a0702508468ad676396db92d8365f30cfc628ea67"}
Jul 27 21:15:05.123307 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:15:05.123283    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753115-t8g6v"
Jul 27 21:15:05.994709 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:15:05.994680    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753115-t8g6v"
Jul 27 21:15:05.994709 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:15:05.994714    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753115-t8g6v" event={"ID":"c8e679a9-15f2-4f34-81f6-399734e7a830","Type":"ContainerDied","Data":"9354115c1bd53fa9381eb0a72eb485d1922f0b85d921fb47850525eeaa308777"}
Jul 27 21:15:05.994913 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:15:05.994760    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9354115c1bd53fa9381eb0a72eb485d1922f0b85d921fb47850525eeaa308777"
Jul 27 21:15:08.802121 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:15:08.802070    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-fd2c4ccc-kserve-844cc7cc47-7nnp6"]
Jul 27 21:15:08.802519 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:15:08.802423    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8e679a9-15f2-4f34-81f6-399734e7a830" containerName="cleanup"
Jul 27 21:15:08.802519 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:15:08.802435    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8e679a9-15f2-4f34-81f6-399734e7a830" containerName="cleanup"
Jul 27 21:15:08.802519 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:15:08.802493    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8e679a9-15f2-4f34-81f6-399734e7a830" containerName="cleanup"
Jul 27 21:15:08.805318 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:15:08.805301    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-fd2c4ccc-kserve-844cc7cc47-7nnp6"
Jul 27 21:15:08.808232 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:15:08.808208    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-fd2c4ccc-kserve-self-signed-certs\""
Jul 27 21:15:08.814975 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:15:08.814947    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-fd2c4ccc-kserve-844cc7cc47-7nnp6"]
Jul 27 21:15:08.975133 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:15:08.975066    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3329e9db-83a8-474d-bccd-d08bcafe20a7-dshm\") pod \"e2e-same-id-b-fd2c4ccc-kserve-844cc7cc47-7nnp6\" (UID: \"3329e9db-83a8-474d-bccd-d08bcafe20a7\") " pod="llm/e2e-same-id-b-fd2c4ccc-kserve-844cc7cc47-7nnp6"
Jul 27 21:15:08.975133 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:15:08.975135    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mv47g\" (UniqueName: \"kubernetes.io/projected/3329e9db-83a8-474d-bccd-d08bcafe20a7-kube-api-access-mv47g\") pod \"e2e-same-id-b-fd2c4ccc-kserve-844cc7cc47-7nnp6\" (UID: \"3329e9db-83a8-474d-bccd-d08bcafe20a7\") " pod="llm/e2e-same-id-b-fd2c4ccc-kserve-844cc7cc47-7nnp6"
Jul 27 21:15:08.975344 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:15:08.975195    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3329e9db-83a8-474d-bccd-d08bcafe20a7-model-cache\") pod \"e2e-same-id-b-fd2c4ccc-kserve-844cc7cc47-7nnp6\" (UID: \"3329e9db-83a8-474d-bccd-d08bcafe20a7\") " pod="llm/e2e-same-id-b-fd2c4ccc-kserve-844cc7cc47-7nnp6"
Jul 27 21:15:08.975344 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:15:08.975221    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3329e9db-83a8-474d-bccd-d08bcafe20a7-home\") pod \"e2e-same-id-b-fd2c4ccc-kserve-844cc7cc47-7nnp6\" (UID: \"3329e9db-83a8-474d-bccd-d08bcafe20a7\") " pod="llm/e2e-same-id-b-fd2c4ccc-kserve-844cc7cc47-7nnp6"
Jul 27 21:15:08.975344 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:15:08.975262    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3329e9db-83a8-474d-bccd-d08bcafe20a7-tmp-dir\") pod \"e2e-same-id-b-fd2c4ccc-kserve-844cc7cc47-7nnp6\" (UID: \"3329e9db-83a8-474d-bccd-d08bcafe20a7\") " pod="llm/e2e-same-id-b-fd2c4ccc-kserve-844cc7cc47-7nnp6"
Jul 27 21:15:08.975344 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:15:08.975300    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3329e9db-83a8-474d-bccd-d08bcafe20a7-tls-certs\") pod \"e2e-same-id-b-fd2c4ccc-kserve-844cc7cc47-7nnp6\" (UID: \"3329e9db-83a8-474d-bccd-d08bcafe20a7\") " pod="llm/e2e-same-id-b-fd2c4ccc-kserve-844cc7cc47-7nnp6"
Jul 27 21:15:09.076593 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:15:09.076490    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3329e9db-83a8-474d-bccd-d08bcafe20a7-dshm\") pod \"e2e-same-id-b-fd2c4ccc-kserve-844cc7cc47-7nnp6\" (UID: \"3329e9db-83a8-474d-bccd-d08bcafe20a7\") " pod="llm/e2e-same-id-b-fd2c4ccc-kserve-844cc7cc47-7nnp6"
Jul 27 21:15:09.076593 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:15:09.076538    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mv47g\" (UniqueName: \"kubernetes.io/projected/3329e9db-83a8-474d-bccd-d08bcafe20a7-kube-api-access-mv47g\") pod \"e2e-same-id-b-fd2c4ccc-kserve-844cc7cc47-7nnp6\" (UID: \"3329e9db-83a8-474d-bccd-d08bcafe20a7\") " pod="llm/e2e-same-id-b-fd2c4ccc-kserve-844cc7cc47-7nnp6"
Jul 27 21:15:09.076593 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:15:09.076568    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3329e9db-83a8-474d-bccd-d08bcafe20a7-model-cache\") pod \"e2e-same-id-b-fd2c4ccc-kserve-844cc7cc47-7nnp6\" (UID: \"3329e9db-83a8-474d-bccd-d08bcafe20a7\") " pod="llm/e2e-same-id-b-fd2c4ccc-kserve-844cc7cc47-7nnp6"
Jul 27 21:15:09.076868 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:15:09.076686    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3329e9db-83a8-474d-bccd-d08bcafe20a7-home\") pod \"e2e-same-id-b-fd2c4ccc-kserve-844cc7cc47-7nnp6\" (UID: \"3329e9db-83a8-474d-bccd-d08bcafe20a7\") " pod="llm/e2e-same-id-b-fd2c4ccc-kserve-844cc7cc47-7nnp6"
Jul 27 21:15:09.076868 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:15:09.076734    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3329e9db-83a8-474d-bccd-d08bcafe20a7-tmp-dir\") pod \"e2e-same-id-b-fd2c4ccc-kserve-844cc7cc47-7nnp6\" (UID: \"3329e9db-83a8-474d-bccd-d08bcafe20a7\") " pod="llm/e2e-same-id-b-fd2c4ccc-kserve-844cc7cc47-7nnp6"
Jul 27 21:15:09.076868 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:15:09.076799    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3329e9db-83a8-474d-bccd-d08bcafe20a7-tls-certs\") pod \"e2e-same-id-b-fd2c4ccc-kserve-844cc7cc47-7nnp6\" (UID: \"3329e9db-83a8-474d-bccd-d08bcafe20a7\") " pod="llm/e2e-same-id-b-fd2c4ccc-kserve-844cc7cc47-7nnp6"
Jul 27 21:15:09.077035 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:15:09.077014    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3329e9db-83a8-474d-bccd-d08bcafe20a7-model-cache\") pod \"e2e-same-id-b-fd2c4ccc-kserve-844cc7cc47-7nnp6\" (UID: \"3329e9db-83a8-474d-bccd-d08bcafe20a7\") " pod="llm/e2e-same-id-b-fd2c4ccc-kserve-844cc7cc47-7nnp6"
Jul 27 21:15:09.077130 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:15:09.077109    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3329e9db-83a8-474d-bccd-d08bcafe20a7-home\") pod \"e2e-same-id-b-fd2c4ccc-kserve-844cc7cc47-7nnp6\" (UID: \"3329e9db-83a8-474d-bccd-d08bcafe20a7\") " pod="llm/e2e-same-id-b-fd2c4ccc-kserve-844cc7cc47-7nnp6"
Jul 27 21:15:09.077182 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:15:09.077133    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3329e9db-83a8-474d-bccd-d08bcafe20a7-tmp-dir\") pod \"e2e-same-id-b-fd2c4ccc-kserve-844cc7cc47-7nnp6\" (UID: \"3329e9db-83a8-474d-bccd-d08bcafe20a7\") " pod="llm/e2e-same-id-b-fd2c4ccc-kserve-844cc7cc47-7nnp6"
Jul 27 21:15:09.078941 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:15:09.078916    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3329e9db-83a8-474d-bccd-d08bcafe20a7-dshm\") pod \"e2e-same-id-b-fd2c4ccc-kserve-844cc7cc47-7nnp6\" (UID: \"3329e9db-83a8-474d-bccd-d08bcafe20a7\") " pod="llm/e2e-same-id-b-fd2c4ccc-kserve-844cc7cc47-7nnp6"
Jul 27 21:15:09.079454 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:15:09.079434    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3329e9db-83a8-474d-bccd-d08bcafe20a7-tls-certs\") pod \"e2e-same-id-b-fd2c4ccc-kserve-844cc7cc47-7nnp6\" (UID: \"3329e9db-83a8-474d-bccd-d08bcafe20a7\") " pod="llm/e2e-same-id-b-fd2c4ccc-kserve-844cc7cc47-7nnp6"
Jul 27 21:15:09.084645 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:15:09.084618    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mv47g\" (UniqueName: \"kubernetes.io/projected/3329e9db-83a8-474d-bccd-d08bcafe20a7-kube-api-access-mv47g\") pod \"e2e-same-id-b-fd2c4ccc-kserve-844cc7cc47-7nnp6\" (UID: \"3329e9db-83a8-474d-bccd-d08bcafe20a7\") " pod="llm/e2e-same-id-b-fd2c4ccc-kserve-844cc7cc47-7nnp6"
Jul 27 21:15:09.115656 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:15:09.115613    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-fd2c4ccc-kserve-844cc7cc47-7nnp6"
Jul 27 21:15:09.254564 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:15:09.254509    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-fd2c4ccc-kserve-844cc7cc47-7nnp6"]
Jul 27 21:15:09.257494 ip-10-0-141-64 kubenswrapper[2560]: W0727 21:15:09.257455    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3329e9db_83a8_474d_bccd_d08bcafe20a7.slice/crio-c2d666c8cd54580b10798d9d2246e7bc294d1a5612ba35c1de98d022d2d0e053 WatchSource:0}: Error finding container c2d666c8cd54580b10798d9d2246e7bc294d1a5612ba35c1de98d022d2d0e053: Status 404 returned error can't find the container with id c2d666c8cd54580b10798d9d2246e7bc294d1a5612ba35c1de98d022d2d0e053
Jul 27 21:15:10.017042 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:15:10.016999    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-fd2c4ccc-kserve-844cc7cc47-7nnp6" event={"ID":"3329e9db-83a8-474d-bccd-d08bcafe20a7","Type":"ContainerStarted","Data":"53fba4c4ad66fe2a88d70054b66fcd7cb966a54e695d94baef0158a8c37697a1"}
Jul 27 21:15:10.017502 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:15:10.017052    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-fd2c4ccc-kserve-844cc7cc47-7nnp6" event={"ID":"3329e9db-83a8-474d-bccd-d08bcafe20a7","Type":"ContainerStarted","Data":"c2d666c8cd54580b10798d9d2246e7bc294d1a5612ba35c1de98d022d2d0e053"}
Jul 27 21:15:10.033674 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:15:10.033626    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-fd2c4ccc-kserve-844cc7cc47-7nnp6" podStartSLOduration=2.033611994 podStartE2EDuration="2.033611994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:15:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:15:10.03267157 +0000 UTC m=+1426.784533358" watchObservedRunningTime="2026-07-27 21:15:10.033611994 +0000 UTC m=+1426.785473788"
Jul 27 21:15:14.808975 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:15:14.808900    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:15:19.116217 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:15:19.116187    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-fd2c4ccc-kserve-844cc7cc47-7nnp6"
Jul 27 21:15:19.116217 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:15:19.116228    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-fd2c4ccc-kserve-844cc7cc47-7nnp6"
Jul 27 21:15:19.129225 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:15:19.129200    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-fd2c4ccc-kserve-844cc7cc47-7nnp6"
Jul 27 21:15:20.076603 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:15:20.076571    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-fd2c4ccc-kserve-844cc7cc47-7nnp6"
Jul 27 21:15:40.697062 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:15:40.697022    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-fd2c4ccc-kserve-844cc7cc47-7nnp6"]
Jul 27 21:15:44.000604 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:15:44.000569    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:15:55.716868 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:15:55.716800    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-fd2c4ccc-kserve-844cc7cc47-7nnp6" podUID="3329e9db-83a8-474d-bccd-d08bcafe20a7" containerName="main" containerID="cri-o://53fba4c4ad66fe2a88d70054b66fcd7cb966a54e695d94baef0158a8c37697a1" gracePeriod=45
Jul 27 21:15:55.977135 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:15:55.977063    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-fd2c4ccc-kserve-844cc7cc47-7nnp6"
Jul 27 21:15:56.077021 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:15:56.076983    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3329e9db-83a8-474d-bccd-d08bcafe20a7-tmp-dir\") pod \"3329e9db-83a8-474d-bccd-d08bcafe20a7\" (UID: \"3329e9db-83a8-474d-bccd-d08bcafe20a7\") "
Jul 27 21:15:56.077021 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:15:56.077031    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3329e9db-83a8-474d-bccd-d08bcafe20a7-home\") pod \"3329e9db-83a8-474d-bccd-d08bcafe20a7\" (UID: \"3329e9db-83a8-474d-bccd-d08bcafe20a7\") "
Jul 27 21:15:56.077292 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:15:56.077055    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3329e9db-83a8-474d-bccd-d08bcafe20a7-tls-certs\") pod \"3329e9db-83a8-474d-bccd-d08bcafe20a7\" (UID: \"3329e9db-83a8-474d-bccd-d08bcafe20a7\") "
Jul 27 21:15:56.077292 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:15:56.077130    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3329e9db-83a8-474d-bccd-d08bcafe20a7-model-cache\") pod \"3329e9db-83a8-474d-bccd-d08bcafe20a7\" (UID: \"3329e9db-83a8-474d-bccd-d08bcafe20a7\") "
Jul 27 21:15:56.077292 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:15:56.077181    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3329e9db-83a8-474d-bccd-d08bcafe20a7-dshm\") pod \"3329e9db-83a8-474d-bccd-d08bcafe20a7\" (UID: \"3329e9db-83a8-474d-bccd-d08bcafe20a7\") "
Jul 27 21:15:56.077292 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:15:56.077257    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mv47g\" (UniqueName: \"kubernetes.io/projected/3329e9db-83a8-474d-bccd-d08bcafe20a7-kube-api-access-mv47g\") pod \"3329e9db-83a8-474d-bccd-d08bcafe20a7\" (UID: \"3329e9db-83a8-474d-bccd-d08bcafe20a7\") "
Jul 27 21:15:56.077471 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:15:56.077335    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3329e9db-83a8-474d-bccd-d08bcafe20a7-home" (OuterVolumeSpecName: "home") pod "3329e9db-83a8-474d-bccd-d08bcafe20a7" (UID: "3329e9db-83a8-474d-bccd-d08bcafe20a7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:15:56.077471 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:15:56.077352    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3329e9db-83a8-474d-bccd-d08bcafe20a7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3329e9db-83a8-474d-bccd-d08bcafe20a7" (UID: "3329e9db-83a8-474d-bccd-d08bcafe20a7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:15:56.077471 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:15:56.077444    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3329e9db-83a8-474d-bccd-d08bcafe20a7-model-cache" (OuterVolumeSpecName: "model-cache") pod "3329e9db-83a8-474d-bccd-d08bcafe20a7" (UID: "3329e9db-83a8-474d-bccd-d08bcafe20a7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:15:56.077604 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:15:56.077582    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3329e9db-83a8-474d-bccd-d08bcafe20a7-home\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:15:56.077604 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:15:56.077602    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3329e9db-83a8-474d-bccd-d08bcafe20a7-model-cache\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:15:56.077719 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:15:56.077613    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3329e9db-83a8-474d-bccd-d08bcafe20a7-tmp-dir\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:15:56.079551 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:15:56.079522    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3329e9db-83a8-474d-bccd-d08bcafe20a7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3329e9db-83a8-474d-bccd-d08bcafe20a7" (UID: "3329e9db-83a8-474d-bccd-d08bcafe20a7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:15:56.079551 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:15:56.079527    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3329e9db-83a8-474d-bccd-d08bcafe20a7-dshm" (OuterVolumeSpecName: "dshm") pod "3329e9db-83a8-474d-bccd-d08bcafe20a7" (UID: "3329e9db-83a8-474d-bccd-d08bcafe20a7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:15:56.079551 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:15:56.079551    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3329e9db-83a8-474d-bccd-d08bcafe20a7-kube-api-access-mv47g" (OuterVolumeSpecName: "kube-api-access-mv47g") pod "3329e9db-83a8-474d-bccd-d08bcafe20a7" (UID: "3329e9db-83a8-474d-bccd-d08bcafe20a7"). InnerVolumeSpecName "kube-api-access-mv47g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:15:56.179031 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:15:56.178995    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mv47g\" (UniqueName: \"kubernetes.io/projected/3329e9db-83a8-474d-bccd-d08bcafe20a7-kube-api-access-mv47g\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:15:56.179031 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:15:56.179025    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3329e9db-83a8-474d-bccd-d08bcafe20a7-tls-certs\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:15:56.179031 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:15:56.179039    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3329e9db-83a8-474d-bccd-d08bcafe20a7-dshm\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:15:56.227842 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:15:56.227755    2560 generic.go:356] "Generic (PLEG): container finished" podID="3329e9db-83a8-474d-bccd-d08bcafe20a7" containerID="53fba4c4ad66fe2a88d70054b66fcd7cb966a54e695d94baef0158a8c37697a1" exitCode=0
Jul 27 21:15:56.227842 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:15:56.227797    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-fd2c4ccc-kserve-844cc7cc47-7nnp6"
Jul 27 21:15:56.228008 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:15:56.227832    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-fd2c4ccc-kserve-844cc7cc47-7nnp6" event={"ID":"3329e9db-83a8-474d-bccd-d08bcafe20a7","Type":"ContainerDied","Data":"53fba4c4ad66fe2a88d70054b66fcd7cb966a54e695d94baef0158a8c37697a1"}
Jul 27 21:15:56.228008 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:15:56.227882    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-fd2c4ccc-kserve-844cc7cc47-7nnp6" event={"ID":"3329e9db-83a8-474d-bccd-d08bcafe20a7","Type":"ContainerDied","Data":"c2d666c8cd54580b10798d9d2246e7bc294d1a5612ba35c1de98d022d2d0e053"}
Jul 27 21:15:56.228008 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:15:56.227904    2560 scope.go:117] "RemoveContainer" containerID="53fba4c4ad66fe2a88d70054b66fcd7cb966a54e695d94baef0158a8c37697a1"
Jul 27 21:15:56.237298 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:15:56.237281    2560 scope.go:117] "RemoveContainer" containerID="53fba4c4ad66fe2a88d70054b66fcd7cb966a54e695d94baef0158a8c37697a1"
Jul 27 21:15:56.237537 ip-10-0-141-64 kubenswrapper[2560]: E0727 21:15:56.237519    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53fba4c4ad66fe2a88d70054b66fcd7cb966a54e695d94baef0158a8c37697a1\": container with ID starting with 53fba4c4ad66fe2a88d70054b66fcd7cb966a54e695d94baef0158a8c37697a1 not found: ID does not exist" containerID="53fba4c4ad66fe2a88d70054b66fcd7cb966a54e695d94baef0158a8c37697a1"
Jul 27 21:15:56.237619 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:15:56.237548    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53fba4c4ad66fe2a88d70054b66fcd7cb966a54e695d94baef0158a8c37697a1"} err="failed to get container status \"53fba4c4ad66fe2a88d70054b66fcd7cb966a54e695d94baef0158a8c37697a1\": rpc error: code = NotFound desc = could not find container \"53fba4c4ad66fe2a88d70054b66fcd7cb966a54e695d94baef0158a8c37697a1\": container with ID starting with 53fba4c4ad66fe2a88d70054b66fcd7cb966a54e695d94baef0158a8c37697a1 not found: ID does not exist"
Jul 27 21:15:56.248879 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:15:56.248857    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-fd2c4ccc-kserve-844cc7cc47-7nnp6"]
Jul 27 21:15:56.252879 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:15:56.252860    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-fd2c4ccc-kserve-844cc7cc47-7nnp6"]
Jul 27 21:15:57.846975 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:15:57.846944    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3329e9db-83a8-474d-bccd-d08bcafe20a7" path="/var/lib/kubelet/pods/3329e9db-83a8-474d-bccd-d08bcafe20a7/volumes"
Jul 27 21:15:58.392110 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:15:58.392055    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:16:04.894591 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:16:04.894547    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:16:20.999644 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:16:20.999610    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:16:23.771490 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:16:23.771457    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-z6zxv_8a6b7ae6-6c21-453d-99b9-303066d19915/console-operator/1.log"
Jul 27 21:16:23.772852 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:16:23.772822    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-z6zxv_8a6b7ae6-6c21-453d-99b9-303066d19915/console-operator/1.log"
Jul 27 21:16:23.778167 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:16:23.778147    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-95hg7_433c431a-b5bf-4734-a03f-6f0a16895ded/ovn-acl-logging/0.log"
Jul 27 21:16:23.779369 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:16:23.779353    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-95hg7_433c431a-b5bf-4734-a03f-6f0a16895ded/ovn-acl-logging/0.log"
Jul 27 21:16:30.090634 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:16:30.090585    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:16:39.291732 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:16:39.291691    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:16:49.701178 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:16:49.701142    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:16:59.290178 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:16:59.290144    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:17:10.199995 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:17:10.199954    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:17:21.495375 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:17:21.495335    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:17:31.485210 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:17:31.485170    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:17:45.991424 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:17:45.991390    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:17:54.001801 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:17:54.001767    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:18:06.498920 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:18:06.498882    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:18:22.810612 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:18:22.810514    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:18:34.399037 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:18:34.398995    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:18:45.789918 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:18:45.789880    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:18:58.692184 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:18:58.692141    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:19:07.101336 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:19:07.101301    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:19:17.075842 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:19:17.075808    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:19:29.987342 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:19:29.987306    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:19:44.874336 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:19:44.874253    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-39c7c327-dss6c"]
Jul 27 21:19:44.874719 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:19:44.874610    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3329e9db-83a8-474d-bccd-d08bcafe20a7" containerName="main"
Jul 27 21:19:44.874719 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:19:44.874623    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="3329e9db-83a8-474d-bccd-d08bcafe20a7" containerName="main"
Jul 27 21:19:44.874804 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:19:44.874722    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="3329e9db-83a8-474d-bccd-d08bcafe20a7" containerName="main"
Jul 27 21:19:44.877953 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:19:44.877937    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-39c7c327-dss6c"
Jul 27 21:19:44.880479 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:19:44.880460    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j5rjm\""
Jul 27 21:19:44.880549 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:19:44.880480    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 27 21:19:44.884036 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:19:44.884003    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-39c7c327-dss6c"]
Jul 27 21:19:45.014030 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:19:45.013995    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa13dace-07ad-450a-ac78-001286a0854c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-39c7c327-dss6c\" (UID: \"fa13dace-07ad-450a-ac78-001286a0854c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-39c7c327-dss6c"
Jul 27 21:19:45.114427 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:19:45.114388    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa13dace-07ad-450a-ac78-001286a0854c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-39c7c327-dss6c\" (UID: \"fa13dace-07ad-450a-ac78-001286a0854c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-39c7c327-dss6c"
Jul 27 21:19:45.115013 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:19:45.114992    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa13dace-07ad-450a-ac78-001286a0854c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-39c7c327-dss6c\" (UID: \"fa13dace-07ad-450a-ac78-001286a0854c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-39c7c327-dss6c"
Jul 27 21:19:45.189554 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:19:45.189472    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-39c7c327-dss6c"
Jul 27 21:19:45.322803 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:19:45.322780    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-39c7c327-dss6c"]
Jul 27 21:19:45.325252 ip-10-0-141-64 kubenswrapper[2560]: W0727 21:19:45.325219    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa13dace_07ad_450a_ac78_001286a0854c.slice/crio-ca3d1c6570a25f1446166662218b125f605827a9ec5e608b1de044836ae5dfaa WatchSource:0}: Error finding container ca3d1c6570a25f1446166662218b125f605827a9ec5e608b1de044836ae5dfaa: Status 404 returned error can't find the container with id ca3d1c6570a25f1446166662218b125f605827a9ec5e608b1de044836ae5dfaa
Jul 27 21:19:46.252443 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:19:46.252353    2560 generic.go:356] "Generic (PLEG): container finished" podID="fa13dace-07ad-450a-ac78-001286a0854c" containerID="8faaf3602efa993953aba23c28319cd2bde2c8ce6e60ec5b1868f9986784c6e1" exitCode=0
Jul 27 21:19:46.252443 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:19:46.252406    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-39c7c327-dss6c" event={"ID":"fa13dace-07ad-450a-ac78-001286a0854c","Type":"ContainerDied","Data":"8faaf3602efa993953aba23c28319cd2bde2c8ce6e60ec5b1868f9986784c6e1"}
Jul 27 21:19:46.252839 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:19:46.252459    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-39c7c327-dss6c" event={"ID":"fa13dace-07ad-450a-ac78-001286a0854c","Type":"ContainerStarted","Data":"ca3d1c6570a25f1446166662218b125f605827a9ec5e608b1de044836ae5dfaa"}
Jul 27 21:19:47.387405 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:19:47.387380    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-39c7c327-dss6c"
Jul 27 21:19:47.535425 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:19:47.535391    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa13dace-07ad-450a-ac78-001286a0854c-maas-api-service-ca\") pod \"fa13dace-07ad-450a-ac78-001286a0854c\" (UID: \"fa13dace-07ad-450a-ac78-001286a0854c\") "
Jul 27 21:19:47.535748 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:19:47.535727    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa13dace-07ad-450a-ac78-001286a0854c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fa13dace-07ad-450a-ac78-001286a0854c" (UID: "fa13dace-07ad-450a-ac78-001286a0854c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:19:47.636167 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:19:47.636142    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa13dace-07ad-450a-ac78-001286a0854c-maas-api-service-ca\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:19:48.262872 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:19:48.262792    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-39c7c327-dss6c"
Jul 27 21:19:48.262872 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:19:48.262830    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-39c7c327-dss6c" event={"ID":"fa13dace-07ad-450a-ac78-001286a0854c","Type":"ContainerDied","Data":"ca3d1c6570a25f1446166662218b125f605827a9ec5e608b1de044836ae5dfaa"}
Jul 27 21:19:48.263042 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:19:48.262909    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca3d1c6570a25f1446166662218b125f605827a9ec5e608b1de044836ae5dfaa"
Jul 27 21:19:57.887967 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:19:57.887933    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:19:59.951028 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:19:59.950991    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-39c7c327-dss6c"]
Jul 27 21:19:59.952565 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:19:59.952543    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-39c7c327-dss6c"]
Jul 27 21:20:01.133369 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:01.133333    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-338bbf37-gw-openshift-default-667d7d4596-9vdcr"]
Jul 27 21:20:01.133810 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:01.133776    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa13dace-07ad-450a-ac78-001286a0854c" containerName="revoke-keys"
Jul 27 21:20:01.133810 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:01.133796    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa13dace-07ad-450a-ac78-001286a0854c" containerName="revoke-keys"
Jul 27 21:20:01.133932 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:01.133917    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa13dace-07ad-450a-ac78-001286a0854c" containerName="revoke-keys"
Jul 27 21:20:01.136920 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:01.136897    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-338bbf37-gw-openshift-default-667d7d4596-9vdcr"
Jul 27 21:20:01.139520 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:01.139496    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-338bbf37-gw-openshift-default-dockercfg-fffpd\""
Jul 27 21:20:01.141667 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:01.141649    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbc7f3f4-9cef-4747-9205-2f08e594ba91-workload-socket\") pod \"e2e-migrate-338bbf37-gw-openshift-default-667d7d4596-9vdcr\" (UID: \"fbc7f3f4-9cef-4747-9205-2f08e594ba91\") " pod="openshift-ingress/e2e-migrate-338bbf37-gw-openshift-default-667d7d4596-9vdcr"
Jul 27 21:20:01.141753 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:01.141677    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbc7f3f4-9cef-4747-9205-2f08e594ba91-credential-socket\") pod \"e2e-migrate-338bbf37-gw-openshift-default-667d7d4596-9vdcr\" (UID: \"fbc7f3f4-9cef-4747-9205-2f08e594ba91\") " pod="openshift-ingress/e2e-migrate-338bbf37-gw-openshift-default-667d7d4596-9vdcr"
Jul 27 21:20:01.141753 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:01.141701    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbc7f3f4-9cef-4747-9205-2f08e594ba91-istio-data\") pod \"e2e-migrate-338bbf37-gw-openshift-default-667d7d4596-9vdcr\" (UID: \"fbc7f3f4-9cef-4747-9205-2f08e594ba91\") " pod="openshift-ingress/e2e-migrate-338bbf37-gw-openshift-default-667d7d4596-9vdcr"
Jul 27 21:20:01.141883 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:01.141813    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbc7f3f4-9cef-4747-9205-2f08e594ba91-workload-certs\") pod \"e2e-migrate-338bbf37-gw-openshift-default-667d7d4596-9vdcr\" (UID: \"fbc7f3f4-9cef-4747-9205-2f08e594ba91\") " pod="openshift-ingress/e2e-migrate-338bbf37-gw-openshift-default-667d7d4596-9vdcr"
Jul 27 21:20:01.141883 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:01.141861    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbc7f3f4-9cef-4747-9205-2f08e594ba91-istio-token\") pod \"e2e-migrate-338bbf37-gw-openshift-default-667d7d4596-9vdcr\" (UID: \"fbc7f3f4-9cef-4747-9205-2f08e594ba91\") " pod="openshift-ingress/e2e-migrate-338bbf37-gw-openshift-default-667d7d4596-9vdcr"
Jul 27 21:20:01.141983 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:01.141919    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbc7f3f4-9cef-4747-9205-2f08e594ba91-istiod-ca-cert\") pod \"e2e-migrate-338bbf37-gw-openshift-default-667d7d4596-9vdcr\" (UID: \"fbc7f3f4-9cef-4747-9205-2f08e594ba91\") " pod="openshift-ingress/e2e-migrate-338bbf37-gw-openshift-default-667d7d4596-9vdcr"
Jul 27 21:20:01.141983 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:01.141949    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scrww\" (UniqueName: \"kubernetes.io/projected/fbc7f3f4-9cef-4747-9205-2f08e594ba91-kube-api-access-scrww\") pod \"e2e-migrate-338bbf37-gw-openshift-default-667d7d4596-9vdcr\" (UID: \"fbc7f3f4-9cef-4747-9205-2f08e594ba91\") " pod="openshift-ingress/e2e-migrate-338bbf37-gw-openshift-default-667d7d4596-9vdcr"
Jul 27 21:20:01.142072 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:01.141994    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbc7f3f4-9cef-4747-9205-2f08e594ba91-istio-envoy\") pod \"e2e-migrate-338bbf37-gw-openshift-default-667d7d4596-9vdcr\" (UID: \"fbc7f3f4-9cef-4747-9205-2f08e594ba91\") " pod="openshift-ingress/e2e-migrate-338bbf37-gw-openshift-default-667d7d4596-9vdcr"
Jul 27 21:20:01.142072 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:01.142028    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbc7f3f4-9cef-4747-9205-2f08e594ba91-istio-podinfo\") pod \"e2e-migrate-338bbf37-gw-openshift-default-667d7d4596-9vdcr\" (UID: \"fbc7f3f4-9cef-4747-9205-2f08e594ba91\") " pod="openshift-ingress/e2e-migrate-338bbf37-gw-openshift-default-667d7d4596-9vdcr"
Jul 27 21:20:01.151861 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:01.151835    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-338bbf37-gw-openshift-default-667d7d4596-9vdcr"]
Jul 27 21:20:01.243007 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:01.242973    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbc7f3f4-9cef-4747-9205-2f08e594ba91-workload-certs\") pod \"e2e-migrate-338bbf37-gw-openshift-default-667d7d4596-9vdcr\" (UID: \"fbc7f3f4-9cef-4747-9205-2f08e594ba91\") " pod="openshift-ingress/e2e-migrate-338bbf37-gw-openshift-default-667d7d4596-9vdcr"
Jul 27 21:20:01.243199 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:01.243024    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbc7f3f4-9cef-4747-9205-2f08e594ba91-istio-token\") pod \"e2e-migrate-338bbf37-gw-openshift-default-667d7d4596-9vdcr\" (UID: \"fbc7f3f4-9cef-4747-9205-2f08e594ba91\") " pod="openshift-ingress/e2e-migrate-338bbf37-gw-openshift-default-667d7d4596-9vdcr"
Jul 27 21:20:01.243199 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:01.243107    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbc7f3f4-9cef-4747-9205-2f08e594ba91-istiod-ca-cert\") pod \"e2e-migrate-338bbf37-gw-openshift-default-667d7d4596-9vdcr\" (UID: \"fbc7f3f4-9cef-4747-9205-2f08e594ba91\") " pod="openshift-ingress/e2e-migrate-338bbf37-gw-openshift-default-667d7d4596-9vdcr"
Jul 27 21:20:01.243199 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:01.243128    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scrww\" (UniqueName: \"kubernetes.io/projected/fbc7f3f4-9cef-4747-9205-2f08e594ba91-kube-api-access-scrww\") pod \"e2e-migrate-338bbf37-gw-openshift-default-667d7d4596-9vdcr\" (UID: \"fbc7f3f4-9cef-4747-9205-2f08e594ba91\") " pod="openshift-ingress/e2e-migrate-338bbf37-gw-openshift-default-667d7d4596-9vdcr"
Jul 27 21:20:01.243199 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:01.243159    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbc7f3f4-9cef-4747-9205-2f08e594ba91-istio-envoy\") pod \"e2e-migrate-338bbf37-gw-openshift-default-667d7d4596-9vdcr\" (UID: \"fbc7f3f4-9cef-4747-9205-2f08e594ba91\") " pod="openshift-ingress/e2e-migrate-338bbf37-gw-openshift-default-667d7d4596-9vdcr"
Jul 27 21:20:01.243199 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:01.243174    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbc7f3f4-9cef-4747-9205-2f08e594ba91-istio-podinfo\") pod \"e2e-migrate-338bbf37-gw-openshift-default-667d7d4596-9vdcr\" (UID: \"fbc7f3f4-9cef-4747-9205-2f08e594ba91\") " pod="openshift-ingress/e2e-migrate-338bbf37-gw-openshift-default-667d7d4596-9vdcr"
Jul 27 21:20:01.243467 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:01.243218    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbc7f3f4-9cef-4747-9205-2f08e594ba91-workload-socket\") pod \"e2e-migrate-338bbf37-gw-openshift-default-667d7d4596-9vdcr\" (UID: \"fbc7f3f4-9cef-4747-9205-2f08e594ba91\") " pod="openshift-ingress/e2e-migrate-338bbf37-gw-openshift-default-667d7d4596-9vdcr"
Jul 27 21:20:01.243467 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:01.243257    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbc7f3f4-9cef-4747-9205-2f08e594ba91-credential-socket\") pod \"e2e-migrate-338bbf37-gw-openshift-default-667d7d4596-9vdcr\" (UID: \"fbc7f3f4-9cef-4747-9205-2f08e594ba91\") " pod="openshift-ingress/e2e-migrate-338bbf37-gw-openshift-default-667d7d4596-9vdcr"
Jul 27 21:20:01.243467 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:01.243287    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbc7f3f4-9cef-4747-9205-2f08e594ba91-istio-data\") pod \"e2e-migrate-338bbf37-gw-openshift-default-667d7d4596-9vdcr\" (UID: \"fbc7f3f4-9cef-4747-9205-2f08e594ba91\") " pod="openshift-ingress/e2e-migrate-338bbf37-gw-openshift-default-667d7d4596-9vdcr"
Jul 27 21:20:01.243618 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:01.243462    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbc7f3f4-9cef-4747-9205-2f08e594ba91-workload-certs\") pod \"e2e-migrate-338bbf37-gw-openshift-default-667d7d4596-9vdcr\" (UID: \"fbc7f3f4-9cef-4747-9205-2f08e594ba91\") " pod="openshift-ingress/e2e-migrate-338bbf37-gw-openshift-default-667d7d4596-9vdcr"
Jul 27 21:20:01.243672 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:01.243625    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbc7f3f4-9cef-4747-9205-2f08e594ba91-istio-data\") pod \"e2e-migrate-338bbf37-gw-openshift-default-667d7d4596-9vdcr\" (UID: \"fbc7f3f4-9cef-4747-9205-2f08e594ba91\") " pod="openshift-ingress/e2e-migrate-338bbf37-gw-openshift-default-667d7d4596-9vdcr"
Jul 27 21:20:01.243672 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:01.243663    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbc7f3f4-9cef-4747-9205-2f08e594ba91-workload-socket\") pod \"e2e-migrate-338bbf37-gw-openshift-default-667d7d4596-9vdcr\" (UID: \"fbc7f3f4-9cef-4747-9205-2f08e594ba91\") " pod="openshift-ingress/e2e-migrate-338bbf37-gw-openshift-default-667d7d4596-9vdcr"
Jul 27 21:20:01.243786 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:01.243692    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbc7f3f4-9cef-4747-9205-2f08e594ba91-credential-socket\") pod \"e2e-migrate-338bbf37-gw-openshift-default-667d7d4596-9vdcr\" (UID: \"fbc7f3f4-9cef-4747-9205-2f08e594ba91\") " pod="openshift-ingress/e2e-migrate-338bbf37-gw-openshift-default-667d7d4596-9vdcr"
Jul 27 21:20:01.243888 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:01.243867    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbc7f3f4-9cef-4747-9205-2f08e594ba91-istiod-ca-cert\") pod \"e2e-migrate-338bbf37-gw-openshift-default-667d7d4596-9vdcr\" (UID: \"fbc7f3f4-9cef-4747-9205-2f08e594ba91\") " pod="openshift-ingress/e2e-migrate-338bbf37-gw-openshift-default-667d7d4596-9vdcr"
Jul 27 21:20:01.245684 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:01.245656    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbc7f3f4-9cef-4747-9205-2f08e594ba91-istio-envoy\") pod \"e2e-migrate-338bbf37-gw-openshift-default-667d7d4596-9vdcr\" (UID: \"fbc7f3f4-9cef-4747-9205-2f08e594ba91\") " pod="openshift-ingress/e2e-migrate-338bbf37-gw-openshift-default-667d7d4596-9vdcr"
Jul 27 21:20:01.246194 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:01.246170    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbc7f3f4-9cef-4747-9205-2f08e594ba91-istio-podinfo\") pod \"e2e-migrate-338bbf37-gw-openshift-default-667d7d4596-9vdcr\" (UID: \"fbc7f3f4-9cef-4747-9205-2f08e594ba91\") " pod="openshift-ingress/e2e-migrate-338bbf37-gw-openshift-default-667d7d4596-9vdcr"
Jul 27 21:20:01.251020 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:01.250992    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbc7f3f4-9cef-4747-9205-2f08e594ba91-istio-token\") pod \"e2e-migrate-338bbf37-gw-openshift-default-667d7d4596-9vdcr\" (UID: \"fbc7f3f4-9cef-4747-9205-2f08e594ba91\") " pod="openshift-ingress/e2e-migrate-338bbf37-gw-openshift-default-667d7d4596-9vdcr"
Jul 27 21:20:01.251139 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:01.251123    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-scrww\" (UniqueName: \"kubernetes.io/projected/fbc7f3f4-9cef-4747-9205-2f08e594ba91-kube-api-access-scrww\") pod \"e2e-migrate-338bbf37-gw-openshift-default-667d7d4596-9vdcr\" (UID: \"fbc7f3f4-9cef-4747-9205-2f08e594ba91\") " pod="openshift-ingress/e2e-migrate-338bbf37-gw-openshift-default-667d7d4596-9vdcr"
Jul 27 21:20:01.448617 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:01.448524    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-338bbf37-gw-openshift-default-667d7d4596-9vdcr"
Jul 27 21:20:01.595336 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:01.595307    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-338bbf37-gw-openshift-default-667d7d4596-9vdcr"]
Jul 27 21:20:01.597924 ip-10-0-141-64 kubenswrapper[2560]: W0727 21:20:01.597897    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbc7f3f4_9cef_4747_9205_2f08e594ba91.slice/crio-945ea452fa57338ff8c17a398379b8eadd304a327bcde3d7ccc9a8baeaa4677b WatchSource:0}: Error finding container 945ea452fa57338ff8c17a398379b8eadd304a327bcde3d7ccc9a8baeaa4677b: Status 404 returned error can't find the container with id 945ea452fa57338ff8c17a398379b8eadd304a327bcde3d7ccc9a8baeaa4677b
Jul 27 21:20:01.600000 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:01.599980    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 21:20:01.848174 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:01.848132    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa13dace-07ad-450a-ac78-001286a0854c" path="/var/lib/kubelet/pods/fa13dace-07ad-450a-ac78-001286a0854c/volumes"
Jul 27 21:20:02.329182 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:02.329148    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-338bbf37-gw-openshift-default-667d7d4596-9vdcr" event={"ID":"fbc7f3f4-9cef-4747-9205-2f08e594ba91","Type":"ContainerStarted","Data":"945ea452fa57338ff8c17a398379b8eadd304a327bcde3d7ccc9a8baeaa4677b"}
Jul 27 21:20:05.526207 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:05.526174    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 27 21:20:05.526464 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:05.526245    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 27 21:20:05.526464 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:05.526274    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 27 21:20:06.349757 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:06.349709    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-338bbf37-gw-openshift-default-667d7d4596-9vdcr" event={"ID":"fbc7f3f4-9cef-4747-9205-2f08e594ba91","Type":"ContainerStarted","Data":"6fde6c2010fa05825bce5cea5101b2d6d584d7d9ab01dde1f4e673170f5e7f57"}
Jul 27 21:20:06.373630 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:06.373580    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-338bbf37-gw-openshift-default-667d7d4596-9vdcr" podStartSLOduration=1.4478902599999999 podStartE2EDuration="5.373564165s" totalImagesPullingTime="3.925673905s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:20:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:20:06.372064412 +0000 UTC m=+1723.123926208" watchObservedRunningTime="2026-07-27 21:20:06.373564165 +0000 UTC m=+1723.125425959"
Jul 27 21:20:06.449146 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:06.449108    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-338bbf37-gw-openshift-default-667d7d4596-9vdcr"
Jul 27 21:20:06.453871 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:06.453845    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-338bbf37-gw-openshift-default-667d7d4596-9vdcr"
Jul 27 21:20:07.354100 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:07.354067    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-338bbf37-gw-openshift-default-667d7d4596-9vdcr"
Jul 27 21:20:07.354994 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:07.354970    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-338bbf37-gw-openshift-default-667d7d4596-9vdcr"
Jul 27 21:20:13.690709 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:13.690673    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:20:21.416619 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:21.416584    2560 generic.go:356] "Generic (PLEG): container finished" podID="fbc7f3f4-9cef-4747-9205-2f08e594ba91" containerID="6fde6c2010fa05825bce5cea5101b2d6d584d7d9ab01dde1f4e673170f5e7f57" exitCode=0
Jul 27 21:20:21.417057 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:21.416659    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-338bbf37-gw-openshift-default-667d7d4596-9vdcr" event={"ID":"fbc7f3f4-9cef-4747-9205-2f08e594ba91","Type":"ContainerDied","Data":"6fde6c2010fa05825bce5cea5101b2d6d584d7d9ab01dde1f4e673170f5e7f57"}
Jul 27 21:20:21.417213 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:21.417066    2560 scope.go:117] "RemoveContainer" containerID="6fde6c2010fa05825bce5cea5101b2d6d584d7d9ab01dde1f4e673170f5e7f57"
Jul 27 21:20:21.419097 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:21.419059    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 27 21:20:21.419167 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:21.419137    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 27 21:20:21.419210 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:21.419171    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 27 21:20:22.354069 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:22.354022    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-338bbf37-gw-openshift-default-667d7d4596-9vdcr"
Jul 27 21:20:22.422690 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:22.422651    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-338bbf37-gw-openshift-default-667d7d4596-9vdcr" event={"ID":"fbc7f3f4-9cef-4747-9205-2f08e594ba91","Type":"ContainerStarted","Data":"3b71df12a2ec245fa3a8724e58c94b703e3056ac1f0aee5191bda0f10b59376a"}
Jul 27 21:20:22.449103 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:22.449046    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-338bbf37-gw-openshift-default-667d7d4596-9vdcr"
Jul 27 21:20:22.450143 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:22.450109    2560 patch_prober.go:28] interesting pod/e2e-migrate-338bbf37-gw-openshift-default-667d7d4596-9vdcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Jul 27 21:20:22.450282 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:22.450168    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-338bbf37-gw-openshift-default-667d7d4596-9vdcr" podUID="fbc7f3f4-9cef-4747-9205-2f08e594ba91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Jul 27 21:20:23.449357 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:23.449313    2560 patch_prober.go:28] interesting pod/e2e-migrate-338bbf37-gw-openshift-default-667d7d4596-9vdcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Jul 27 21:20:23.449851 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:23.449381    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-338bbf37-gw-openshift-default-667d7d4596-9vdcr" podUID="fbc7f3f4-9cef-4747-9205-2f08e594ba91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Jul 27 21:20:24.449519 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:24.449472    2560 patch_prober.go:28] interesting pod/e2e-migrate-338bbf37-gw-openshift-default-667d7d4596-9vdcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Jul 27 21:20:24.450002 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:24.449539    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-338bbf37-gw-openshift-default-667d7d4596-9vdcr" podUID="fbc7f3f4-9cef-4747-9205-2f08e594ba91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Jul 27 21:20:25.449499 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:25.449452    2560 patch_prober.go:28] interesting pod/e2e-migrate-338bbf37-gw-openshift-default-667d7d4596-9vdcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Jul 27 21:20:25.449713 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:25.449525    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-338bbf37-gw-openshift-default-667d7d4596-9vdcr" podUID="fbc7f3f4-9cef-4747-9205-2f08e594ba91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Jul 27 21:20:26.453811 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:26.453778    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-338bbf37-gw-openshift-default-667d7d4596-9vdcr"
Jul 27 21:20:26.454179 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:26.454039    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-338bbf37-gw-openshift-default-667d7d4596-9vdcr"
Jul 27 21:20:26.454827 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:26.454810    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-338bbf37-gw-openshift-default-667d7d4596-9vdcr"
Jul 27 21:20:46.197277 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:46.197236    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:20:48.063552 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:48.063510    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-338bbf37-gw-openshift-default-667d7d4596-9vdcr"]
Jul 27 21:20:48.063943 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:48.063758    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-338bbf37-gw-openshift-default-667d7d4596-9vdcr" podUID="fbc7f3f4-9cef-4747-9205-2f08e594ba91" containerName="istio-proxy" containerID="cri-o://3b71df12a2ec245fa3a8724e58c94b703e3056ac1f0aee5191bda0f10b59376a" gracePeriod=30
Jul 27 21:20:48.888142 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:48.888103    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-54ece799-openshift-default-698ff75858-pjsk4"]
Jul 27 21:20:48.894886 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:48.894867    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-54ece799-openshift-default-698ff75858-pjsk4"
Jul 27 21:20:48.900827 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:48.900796    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-54ece799-openshift-default-dockercfg-2b46j\""
Jul 27 21:20:48.922037 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:48.922002    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-54ece799-openshift-default-698ff75858-pjsk4"]
Jul 27 21:20:49.062833 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:49.062788    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee28216e-bfe4-4fff-b1e0-ec23f886ec5f-workload-certs\") pod \"e2e-ait-54ece799-openshift-default-698ff75858-pjsk4\" (UID: \"ee28216e-bfe4-4fff-b1e0-ec23f886ec5f\") " pod="openshift-ingress/e2e-ait-54ece799-openshift-default-698ff75858-pjsk4"
Jul 27 21:20:49.062833 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:49.062835    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee28216e-bfe4-4fff-b1e0-ec23f886ec5f-istio-data\") pod \"e2e-ait-54ece799-openshift-default-698ff75858-pjsk4\" (UID: \"ee28216e-bfe4-4fff-b1e0-ec23f886ec5f\") " pod="openshift-ingress/e2e-ait-54ece799-openshift-default-698ff75858-pjsk4"
Jul 27 21:20:49.063051 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:49.062859    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee28216e-bfe4-4fff-b1e0-ec23f886ec5f-workload-socket\") pod \"e2e-ait-54ece799-openshift-default-698ff75858-pjsk4\" (UID: \"ee28216e-bfe4-4fff-b1e0-ec23f886ec5f\") " pod="openshift-ingress/e2e-ait-54ece799-openshift-default-698ff75858-pjsk4"
Jul 27 21:20:49.063051 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:49.062931    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee28216e-bfe4-4fff-b1e0-ec23f886ec5f-credential-socket\") pod \"e2e-ait-54ece799-openshift-default-698ff75858-pjsk4\" (UID: \"ee28216e-bfe4-4fff-b1e0-ec23f886ec5f\") " pod="openshift-ingress/e2e-ait-54ece799-openshift-default-698ff75858-pjsk4"
Jul 27 21:20:49.063051 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:49.062960    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbnhv\" (UniqueName: \"kubernetes.io/projected/ee28216e-bfe4-4fff-b1e0-ec23f886ec5f-kube-api-access-bbnhv\") pod \"e2e-ait-54ece799-openshift-default-698ff75858-pjsk4\" (UID: \"ee28216e-bfe4-4fff-b1e0-ec23f886ec5f\") " pod="openshift-ingress/e2e-ait-54ece799-openshift-default-698ff75858-pjsk4"
Jul 27 21:20:49.063051 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:49.062987    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee28216e-bfe4-4fff-b1e0-ec23f886ec5f-istiod-ca-cert\") pod \"e2e-ait-54ece799-openshift-default-698ff75858-pjsk4\" (UID: \"ee28216e-bfe4-4fff-b1e0-ec23f886ec5f\") " pod="openshift-ingress/e2e-ait-54ece799-openshift-default-698ff75858-pjsk4"
Jul 27 21:20:49.063051 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:49.063011    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee28216e-bfe4-4fff-b1e0-ec23f886ec5f-istio-envoy\") pod \"e2e-ait-54ece799-openshift-default-698ff75858-pjsk4\" (UID: \"ee28216e-bfe4-4fff-b1e0-ec23f886ec5f\") " pod="openshift-ingress/e2e-ait-54ece799-openshift-default-698ff75858-pjsk4"
Jul 27 21:20:49.063051 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:49.063037    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee28216e-bfe4-4fff-b1e0-ec23f886ec5f-istio-token\") pod \"e2e-ait-54ece799-openshift-default-698ff75858-pjsk4\" (UID: \"ee28216e-bfe4-4fff-b1e0-ec23f886ec5f\") " pod="openshift-ingress/e2e-ait-54ece799-openshift-default-698ff75858-pjsk4"
Jul 27 21:20:49.063289 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:49.063128    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee28216e-bfe4-4fff-b1e0-ec23f886ec5f-istio-podinfo\") pod \"e2e-ait-54ece799-openshift-default-698ff75858-pjsk4\" (UID: \"ee28216e-bfe4-4fff-b1e0-ec23f886ec5f\") " pod="openshift-ingress/e2e-ait-54ece799-openshift-default-698ff75858-pjsk4"
Jul 27 21:20:49.164506 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:49.164422    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee28216e-bfe4-4fff-b1e0-ec23f886ec5f-credential-socket\") pod \"e2e-ait-54ece799-openshift-default-698ff75858-pjsk4\" (UID: \"ee28216e-bfe4-4fff-b1e0-ec23f886ec5f\") " pod="openshift-ingress/e2e-ait-54ece799-openshift-default-698ff75858-pjsk4"
Jul 27 21:20:49.164506 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:49.164462    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbnhv\" (UniqueName: \"kubernetes.io/projected/ee28216e-bfe4-4fff-b1e0-ec23f886ec5f-kube-api-access-bbnhv\") pod \"e2e-ait-54ece799-openshift-default-698ff75858-pjsk4\" (UID: \"ee28216e-bfe4-4fff-b1e0-ec23f886ec5f\") " pod="openshift-ingress/e2e-ait-54ece799-openshift-default-698ff75858-pjsk4"
Jul 27 21:20:49.164506 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:49.164484    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee28216e-bfe4-4fff-b1e0-ec23f886ec5f-istiod-ca-cert\") pod \"e2e-ait-54ece799-openshift-default-698ff75858-pjsk4\" (UID: \"ee28216e-bfe4-4fff-b1e0-ec23f886ec5f\") " pod="openshift-ingress/e2e-ait-54ece799-openshift-default-698ff75858-pjsk4"
Jul 27 21:20:49.165052 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:49.164605    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee28216e-bfe4-4fff-b1e0-ec23f886ec5f-istio-envoy\") pod \"e2e-ait-54ece799-openshift-default-698ff75858-pjsk4\" (UID: \"ee28216e-bfe4-4fff-b1e0-ec23f886ec5f\") " pod="openshift-ingress/e2e-ait-54ece799-openshift-default-698ff75858-pjsk4"
Jul 27 21:20:49.165052 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:49.164651    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee28216e-bfe4-4fff-b1e0-ec23f886ec5f-istio-token\") pod \"e2e-ait-54ece799-openshift-default-698ff75858-pjsk4\" (UID: \"ee28216e-bfe4-4fff-b1e0-ec23f886ec5f\") " pod="openshift-ingress/e2e-ait-54ece799-openshift-default-698ff75858-pjsk4"
Jul 27 21:20:49.165052 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:49.164680    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee28216e-bfe4-4fff-b1e0-ec23f886ec5f-istio-podinfo\") pod \"e2e-ait-54ece799-openshift-default-698ff75858-pjsk4\" (UID: \"ee28216e-bfe4-4fff-b1e0-ec23f886ec5f\") " pod="openshift-ingress/e2e-ait-54ece799-openshift-default-698ff75858-pjsk4"
Jul 27 21:20:49.165052 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:49.164750    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee28216e-bfe4-4fff-b1e0-ec23f886ec5f-workload-certs\") pod \"e2e-ait-54ece799-openshift-default-698ff75858-pjsk4\" (UID: \"ee28216e-bfe4-4fff-b1e0-ec23f886ec5f\") " pod="openshift-ingress/e2e-ait-54ece799-openshift-default-698ff75858-pjsk4"
Jul 27 21:20:49.165052 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:49.164870    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee28216e-bfe4-4fff-b1e0-ec23f886ec5f-credential-socket\") pod \"e2e-ait-54ece799-openshift-default-698ff75858-pjsk4\" (UID: \"ee28216e-bfe4-4fff-b1e0-ec23f886ec5f\") " pod="openshift-ingress/e2e-ait-54ece799-openshift-default-698ff75858-pjsk4"
Jul 27 21:20:49.165052 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:49.164874    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee28216e-bfe4-4fff-b1e0-ec23f886ec5f-istio-data\") pod \"e2e-ait-54ece799-openshift-default-698ff75858-pjsk4\" (UID: \"ee28216e-bfe4-4fff-b1e0-ec23f886ec5f\") " pod="openshift-ingress/e2e-ait-54ece799-openshift-default-698ff75858-pjsk4"
Jul 27 21:20:49.165052 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:49.164944    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee28216e-bfe4-4fff-b1e0-ec23f886ec5f-workload-socket\") pod \"e2e-ait-54ece799-openshift-default-698ff75858-pjsk4\" (UID: \"ee28216e-bfe4-4fff-b1e0-ec23f886ec5f\") " pod="openshift-ingress/e2e-ait-54ece799-openshift-default-698ff75858-pjsk4"
Jul 27 21:20:49.165601 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:49.165115    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee28216e-bfe4-4fff-b1e0-ec23f886ec5f-workload-certs\") pod \"e2e-ait-54ece799-openshift-default-698ff75858-pjsk4\" (UID: \"ee28216e-bfe4-4fff-b1e0-ec23f886ec5f\") " pod="openshift-ingress/e2e-ait-54ece799-openshift-default-698ff75858-pjsk4"
Jul 27 21:20:49.165601 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:49.165162    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee28216e-bfe4-4fff-b1e0-ec23f886ec5f-istiod-ca-cert\") pod \"e2e-ait-54ece799-openshift-default-698ff75858-pjsk4\" (UID: \"ee28216e-bfe4-4fff-b1e0-ec23f886ec5f\") " pod="openshift-ingress/e2e-ait-54ece799-openshift-default-698ff75858-pjsk4"
Jul 27 21:20:49.165601 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:49.165181    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee28216e-bfe4-4fff-b1e0-ec23f886ec5f-istio-data\") pod \"e2e-ait-54ece799-openshift-default-698ff75858-pjsk4\" (UID: \"ee28216e-bfe4-4fff-b1e0-ec23f886ec5f\") " pod="openshift-ingress/e2e-ait-54ece799-openshift-default-698ff75858-pjsk4"
Jul 27 21:20:49.165601 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:49.165239    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee28216e-bfe4-4fff-b1e0-ec23f886ec5f-workload-socket\") pod \"e2e-ait-54ece799-openshift-default-698ff75858-pjsk4\" (UID: \"ee28216e-bfe4-4fff-b1e0-ec23f886ec5f\") " pod="openshift-ingress/e2e-ait-54ece799-openshift-default-698ff75858-pjsk4"
Jul 27 21:20:49.167098 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:49.167061    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee28216e-bfe4-4fff-b1e0-ec23f886ec5f-istio-envoy\") pod \"e2e-ait-54ece799-openshift-default-698ff75858-pjsk4\" (UID: \"ee28216e-bfe4-4fff-b1e0-ec23f886ec5f\") " pod="openshift-ingress/e2e-ait-54ece799-openshift-default-698ff75858-pjsk4"
Jul 27 21:20:49.167401 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:49.167379    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee28216e-bfe4-4fff-b1e0-ec23f886ec5f-istio-podinfo\") pod \"e2e-ait-54ece799-openshift-default-698ff75858-pjsk4\" (UID: \"ee28216e-bfe4-4fff-b1e0-ec23f886ec5f\") " pod="openshift-ingress/e2e-ait-54ece799-openshift-default-698ff75858-pjsk4"
Jul 27 21:20:49.172653 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:49.172631    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee28216e-bfe4-4fff-b1e0-ec23f886ec5f-istio-token\") pod \"e2e-ait-54ece799-openshift-default-698ff75858-pjsk4\" (UID: \"ee28216e-bfe4-4fff-b1e0-ec23f886ec5f\") " pod="openshift-ingress/e2e-ait-54ece799-openshift-default-698ff75858-pjsk4"
Jul 27 21:20:49.172883 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:49.172865    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbnhv\" (UniqueName: \"kubernetes.io/projected/ee28216e-bfe4-4fff-b1e0-ec23f886ec5f-kube-api-access-bbnhv\") pod \"e2e-ait-54ece799-openshift-default-698ff75858-pjsk4\" (UID: \"ee28216e-bfe4-4fff-b1e0-ec23f886ec5f\") " pod="openshift-ingress/e2e-ait-54ece799-openshift-default-698ff75858-pjsk4"
Jul 27 21:20:49.205752 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:49.205729    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-54ece799-openshift-default-698ff75858-pjsk4"
Jul 27 21:20:49.343523 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:49.343495    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-54ece799-openshift-default-698ff75858-pjsk4"]
Jul 27 21:20:49.345628 ip-10-0-141-64 kubenswrapper[2560]: W0727 21:20:49.345595    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee28216e_bfe4_4fff_b1e0_ec23f886ec5f.slice/crio-bed8a5009d83a8ddfa984c601dd09030369cad456444c0ff9ca53ce251804f6d WatchSource:0}: Error finding container bed8a5009d83a8ddfa984c601dd09030369cad456444c0ff9ca53ce251804f6d: Status 404 returned error can't find the container with id bed8a5009d83a8ddfa984c601dd09030369cad456444c0ff9ca53ce251804f6d
Jul 27 21:20:49.347766 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:49.347731    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 27 21:20:49.347838 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:49.347800    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 27 21:20:49.347838 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:49.347834    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 27 21:20:49.552813 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:49.552768    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-54ece799-openshift-default-698ff75858-pjsk4" event={"ID":"ee28216e-bfe4-4fff-b1e0-ec23f886ec5f","Type":"ContainerStarted","Data":"828647e86c79daf695831de097e16195a7df7dc0c035434bd341a527da4f514b"}
Jul 27 21:20:49.552980 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:49.552836    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-54ece799-openshift-default-698ff75858-pjsk4" event={"ID":"ee28216e-bfe4-4fff-b1e0-ec23f886ec5f","Type":"ContainerStarted","Data":"bed8a5009d83a8ddfa984c601dd09030369cad456444c0ff9ca53ce251804f6d"}
Jul 27 21:20:49.573004 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:49.572958    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-54ece799-openshift-default-698ff75858-pjsk4" podStartSLOduration=1.572940674 podStartE2EDuration="1.572940674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:20:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:20:49.571821358 +0000 UTC m=+1766.323683169" watchObservedRunningTime="2026-07-27 21:20:49.572940674 +0000 UTC m=+1766.324802470"
Jul 27 21:20:50.206865 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:50.206829    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-54ece799-openshift-default-698ff75858-pjsk4"
Jul 27 21:20:50.211685 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:50.211662    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-54ece799-openshift-default-698ff75858-pjsk4"
Jul 27 21:20:50.558547 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:50.558518    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-54ece799-openshift-default-698ff75858-pjsk4"
Jul 27 21:20:50.559641 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:50.559621    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-54ece799-openshift-default-698ff75858-pjsk4"
Jul 27 21:20:53.318257 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:53.318231    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-338bbf37-gw-openshift-default-667d7d4596-9vdcr"
Jul 27 21:20:53.397017 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:53.396944    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbc7f3f4-9cef-4747-9205-2f08e594ba91-istio-data\") pod \"fbc7f3f4-9cef-4747-9205-2f08e594ba91\" (UID: \"fbc7f3f4-9cef-4747-9205-2f08e594ba91\") "
Jul 27 21:20:53.397017 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:53.396978    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbc7f3f4-9cef-4747-9205-2f08e594ba91-istio-token\") pod \"fbc7f3f4-9cef-4747-9205-2f08e594ba91\" (UID: \"fbc7f3f4-9cef-4747-9205-2f08e594ba91\") "
Jul 27 21:20:53.397256 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:53.397024    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbc7f3f4-9cef-4747-9205-2f08e594ba91-workload-socket\") pod \"fbc7f3f4-9cef-4747-9205-2f08e594ba91\" (UID: \"fbc7f3f4-9cef-4747-9205-2f08e594ba91\") "
Jul 27 21:20:53.397256 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:53.397050    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbc7f3f4-9cef-4747-9205-2f08e594ba91-istio-podinfo\") pod \"fbc7f3f4-9cef-4747-9205-2f08e594ba91\" (UID: \"fbc7f3f4-9cef-4747-9205-2f08e594ba91\") "
Jul 27 21:20:53.397256 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:53.397076    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-scrww\" (UniqueName: \"kubernetes.io/projected/fbc7f3f4-9cef-4747-9205-2f08e594ba91-kube-api-access-scrww\") pod \"fbc7f3f4-9cef-4747-9205-2f08e594ba91\" (UID: \"fbc7f3f4-9cef-4747-9205-2f08e594ba91\") "
Jul 27 21:20:53.397256 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:53.397131    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbc7f3f4-9cef-4747-9205-2f08e594ba91-istiod-ca-cert\") pod \"fbc7f3f4-9cef-4747-9205-2f08e594ba91\" (UID: \"fbc7f3f4-9cef-4747-9205-2f08e594ba91\") "
Jul 27 21:20:53.397256 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:53.397192    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbc7f3f4-9cef-4747-9205-2f08e594ba91-credential-socket\") pod \"fbc7f3f4-9cef-4747-9205-2f08e594ba91\" (UID: \"fbc7f3f4-9cef-4747-9205-2f08e594ba91\") "
Jul 27 21:20:53.397256 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:53.397215    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbc7f3f4-9cef-4747-9205-2f08e594ba91-istio-envoy\") pod \"fbc7f3f4-9cef-4747-9205-2f08e594ba91\" (UID: \"fbc7f3f4-9cef-4747-9205-2f08e594ba91\") "
Jul 27 21:20:53.397461 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:53.397264    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbc7f3f4-9cef-4747-9205-2f08e594ba91-workload-certs\") pod \"fbc7f3f4-9cef-4747-9205-2f08e594ba91\" (UID: \"fbc7f3f4-9cef-4747-9205-2f08e594ba91\") "
Jul 27 21:20:53.397461 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:53.397351    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbc7f3f4-9cef-4747-9205-2f08e594ba91-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "fbc7f3f4-9cef-4747-9205-2f08e594ba91" (UID: "fbc7f3f4-9cef-4747-9205-2f08e594ba91"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:20:53.397553 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:53.397496    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbc7f3f4-9cef-4747-9205-2f08e594ba91-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "fbc7f3f4-9cef-4747-9205-2f08e594ba91" (UID: "fbc7f3f4-9cef-4747-9205-2f08e594ba91"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:20:53.397608 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:53.397566    2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbc7f3f4-9cef-4747-9205-2f08e594ba91-workload-socket\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:20:53.397608 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:53.397582    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbc7f3f4-9cef-4747-9205-2f08e594ba91-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "fbc7f3f4-9cef-4747-9205-2f08e594ba91" (UID: "fbc7f3f4-9cef-4747-9205-2f08e594ba91"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:20:53.397608 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:53.397593    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbc7f3f4-9cef-4747-9205-2f08e594ba91-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "fbc7f3f4-9cef-4747-9205-2f08e594ba91" (UID: "fbc7f3f4-9cef-4747-9205-2f08e594ba91"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:20:53.398058 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:53.398035    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbc7f3f4-9cef-4747-9205-2f08e594ba91-istio-data" (OuterVolumeSpecName: "istio-data") pod "fbc7f3f4-9cef-4747-9205-2f08e594ba91" (UID: "fbc7f3f4-9cef-4747-9205-2f08e594ba91"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:20:53.399806 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:53.399788    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbc7f3f4-9cef-4747-9205-2f08e594ba91-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "fbc7f3f4-9cef-4747-9205-2f08e594ba91" (UID: "fbc7f3f4-9cef-4747-9205-2f08e594ba91"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:20:53.399886 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:53.399812    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbc7f3f4-9cef-4747-9205-2f08e594ba91-istio-token" (OuterVolumeSpecName: "istio-token") pod "fbc7f3f4-9cef-4747-9205-2f08e594ba91" (UID: "fbc7f3f4-9cef-4747-9205-2f08e594ba91"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:20:53.399975 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:53.399905    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbc7f3f4-9cef-4747-9205-2f08e594ba91-kube-api-access-scrww" (OuterVolumeSpecName: "kube-api-access-scrww") pod "fbc7f3f4-9cef-4747-9205-2f08e594ba91" (UID: "fbc7f3f4-9cef-4747-9205-2f08e594ba91"). InnerVolumeSpecName "kube-api-access-scrww". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:20:53.399975 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:53.399953    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fbc7f3f4-9cef-4747-9205-2f08e594ba91-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "fbc7f3f4-9cef-4747-9205-2f08e594ba91" (UID: "fbc7f3f4-9cef-4747-9205-2f08e594ba91"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 27 21:20:53.498655 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:53.498622    2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbc7f3f4-9cef-4747-9205-2f08e594ba91-istio-podinfo\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:20:53.498655 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:53.498651    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scrww\" (UniqueName: \"kubernetes.io/projected/fbc7f3f4-9cef-4747-9205-2f08e594ba91-kube-api-access-scrww\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:20:53.498655 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:53.498661    2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbc7f3f4-9cef-4747-9205-2f08e594ba91-istiod-ca-cert\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:20:53.498848 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:53.498670    2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbc7f3f4-9cef-4747-9205-2f08e594ba91-credential-socket\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:20:53.498848 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:53.498680    2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbc7f3f4-9cef-4747-9205-2f08e594ba91-istio-envoy\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:20:53.498848 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:53.498689    2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbc7f3f4-9cef-4747-9205-2f08e594ba91-workload-certs\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:20:53.498848 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:53.498697    2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbc7f3f4-9cef-4747-9205-2f08e594ba91-istio-data\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:20:53.498848 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:53.498704    2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbc7f3f4-9cef-4747-9205-2f08e594ba91-istio-token\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:20:53.574248 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:53.574221    2560 generic.go:356] "Generic (PLEG): container finished" podID="fbc7f3f4-9cef-4747-9205-2f08e594ba91" containerID="3b71df12a2ec245fa3a8724e58c94b703e3056ac1f0aee5191bda0f10b59376a" exitCode=0
Jul 27 21:20:53.574399 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:53.574271    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-338bbf37-gw-openshift-default-667d7d4596-9vdcr"
Jul 27 21:20:53.574399 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:53.574315    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-338bbf37-gw-openshift-default-667d7d4596-9vdcr" event={"ID":"fbc7f3f4-9cef-4747-9205-2f08e594ba91","Type":"ContainerDied","Data":"3b71df12a2ec245fa3a8724e58c94b703e3056ac1f0aee5191bda0f10b59376a"}
Jul 27 21:20:53.574399 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:53.574383    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-338bbf37-gw-openshift-default-667d7d4596-9vdcr" event={"ID":"fbc7f3f4-9cef-4747-9205-2f08e594ba91","Type":"ContainerDied","Data":"945ea452fa57338ff8c17a398379b8eadd304a327bcde3d7ccc9a8baeaa4677b"}
Jul 27 21:20:53.574560 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:53.574414    2560 scope.go:117] "RemoveContainer" containerID="3b71df12a2ec245fa3a8724e58c94b703e3056ac1f0aee5191bda0f10b59376a"
Jul 27 21:20:53.583869 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:53.583854    2560 scope.go:117] "RemoveContainer" containerID="6fde6c2010fa05825bce5cea5101b2d6d584d7d9ab01dde1f4e673170f5e7f57"
Jul 27 21:20:53.591772 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:53.591756    2560 scope.go:117] "RemoveContainer" containerID="3b71df12a2ec245fa3a8724e58c94b703e3056ac1f0aee5191bda0f10b59376a"
Jul 27 21:20:53.592014 ip-10-0-141-64 kubenswrapper[2560]: E0727 21:20:53.591995    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b71df12a2ec245fa3a8724e58c94b703e3056ac1f0aee5191bda0f10b59376a\": container with ID starting with 3b71df12a2ec245fa3a8724e58c94b703e3056ac1f0aee5191bda0f10b59376a not found: ID does not exist" containerID="3b71df12a2ec245fa3a8724e58c94b703e3056ac1f0aee5191bda0f10b59376a"
Jul 27 21:20:53.592106 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:53.592026    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b71df12a2ec245fa3a8724e58c94b703e3056ac1f0aee5191bda0f10b59376a"} err="failed to get container status \"3b71df12a2ec245fa3a8724e58c94b703e3056ac1f0aee5191bda0f10b59376a\": rpc error: code = NotFound desc = could not find container \"3b71df12a2ec245fa3a8724e58c94b703e3056ac1f0aee5191bda0f10b59376a\": container with ID starting with 3b71df12a2ec245fa3a8724e58c94b703e3056ac1f0aee5191bda0f10b59376a not found: ID does not exist"
Jul 27 21:20:53.592106 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:53.592062    2560 scope.go:117] "RemoveContainer" containerID="6fde6c2010fa05825bce5cea5101b2d6d584d7d9ab01dde1f4e673170f5e7f57"
Jul 27 21:20:53.592321 ip-10-0-141-64 kubenswrapper[2560]: E0727 21:20:53.592303    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fde6c2010fa05825bce5cea5101b2d6d584d7d9ab01dde1f4e673170f5e7f57\": container with ID starting with 6fde6c2010fa05825bce5cea5101b2d6d584d7d9ab01dde1f4e673170f5e7f57 not found: ID does not exist" containerID="6fde6c2010fa05825bce5cea5101b2d6d584d7d9ab01dde1f4e673170f5e7f57"
Jul 27 21:20:53.592359 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:53.592327    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fde6c2010fa05825bce5cea5101b2d6d584d7d9ab01dde1f4e673170f5e7f57"} err="failed to get container status \"6fde6c2010fa05825bce5cea5101b2d6d584d7d9ab01dde1f4e673170f5e7f57\": rpc error: code = NotFound desc = could not find container \"6fde6c2010fa05825bce5cea5101b2d6d584d7d9ab01dde1f4e673170f5e7f57\": container with ID starting with 6fde6c2010fa05825bce5cea5101b2d6d584d7d9ab01dde1f4e673170f5e7f57 not found: ID does not exist"
Jul 27 21:20:53.603010 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:53.602983    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-338bbf37-gw-openshift-default-667d7d4596-9vdcr"]
Jul 27 21:20:53.609460 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:53.609442    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-338bbf37-gw-openshift-default-667d7d4596-9vdcr"]
Jul 27 21:20:53.848446 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:53.848412    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fbc7f3f4-9cef-4747-9205-2f08e594ba91" path="/var/lib/kubelet/pods/fbc7f3f4-9cef-4747-9205-2f08e594ba91/volumes"
Jul 27 21:20:57.793722 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:20:57.793683    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:21:05.630589 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:05.629931    2560 patch_prober.go:28] interesting pod/e2e-ait-54ece799-openshift-default-698ff75858-pjsk4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": read tcp 10.133.0.2:45330->10.133.0.37:15021: read: connection reset by peer" start-of-body=
Jul 27 21:21:05.630589 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:05.630010    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-54ece799-openshift-default-698ff75858-pjsk4" podUID="ee28216e-bfe4-4fff-b1e0-ec23f886ec5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": read tcp 10.133.0.2:45330->10.133.0.37:15021: read: connection reset by peer"
Jul 27 21:21:06.645226 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:06.645190    2560 generic.go:356] "Generic (PLEG): container finished" podID="ee28216e-bfe4-4fff-b1e0-ec23f886ec5f" containerID="828647e86c79daf695831de097e16195a7df7dc0c035434bd341a527da4f514b" exitCode=0
Jul 27 21:21:06.645627 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:06.645258    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-54ece799-openshift-default-698ff75858-pjsk4" event={"ID":"ee28216e-bfe4-4fff-b1e0-ec23f886ec5f","Type":"ContainerDied","Data":"828647e86c79daf695831de097e16195a7df7dc0c035434bd341a527da4f514b"}
Jul 27 21:21:06.645680 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:06.645647    2560 scope.go:117] "RemoveContainer" containerID="828647e86c79daf695831de097e16195a7df7dc0c035434bd341a527da4f514b"
Jul 27 21:21:06.646694 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:06.646656    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 27 21:21:06.646793 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:06.646743    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 27 21:21:06.646793 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:06.646779    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 27 21:21:07.655107 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:07.655046    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-54ece799-openshift-default-698ff75858-pjsk4" event={"ID":"ee28216e-bfe4-4fff-b1e0-ec23f886ec5f","Type":"ContainerStarted","Data":"13867d03fb66f9e19fc44d56b373f8585ff204d5df26dc2992b4525b38d6dc00"}
Jul 27 21:21:08.206641 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:08.206607    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-54ece799-openshift-default-698ff75858-pjsk4"
Jul 27 21:21:08.206846 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:08.206826    2560 patch_prober.go:28] interesting pod/e2e-ait-54ece799-openshift-default-698ff75858-pjsk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Jul 27 21:21:08.206890 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:08.206872    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-54ece799-openshift-default-698ff75858-pjsk4" podUID="ee28216e-bfe4-4fff-b1e0-ec23f886ec5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Jul 27 21:21:09.206898 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:09.206856    2560 patch_prober.go:28] interesting pod/e2e-ait-54ece799-openshift-default-698ff75858-pjsk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Jul 27 21:21:09.207316 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:09.206914    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-54ece799-openshift-default-698ff75858-pjsk4" podUID="ee28216e-bfe4-4fff-b1e0-ec23f886ec5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Jul 27 21:21:10.206185 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:10.206148    2560 patch_prober.go:28] interesting pod/e2e-ait-54ece799-openshift-default-698ff75858-pjsk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Jul 27 21:21:10.206369 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:10.206207    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-54ece799-openshift-default-698ff75858-pjsk4" podUID="ee28216e-bfe4-4fff-b1e0-ec23f886ec5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Jul 27 21:21:11.206845 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:11.206804    2560 patch_prober.go:28] interesting pod/e2e-ait-54ece799-openshift-default-698ff75858-pjsk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Jul 27 21:21:11.207256 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:11.206878    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-54ece799-openshift-default-698ff75858-pjsk4" podUID="ee28216e-bfe4-4fff-b1e0-ec23f886ec5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Jul 27 21:21:12.209729 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:12.209698    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-54ece799-openshift-default-698ff75858-pjsk4"
Jul 27 21:21:12.210177 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:12.209959    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-54ece799-openshift-default-698ff75858-pjsk4"
Jul 27 21:21:12.210773 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:12.210757    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-54ece799-openshift-default-698ff75858-pjsk4"
Jul 27 21:21:23.804007 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:23.803981    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-z6zxv_8a6b7ae6-6c21-453d-99b9-303066d19915/console-operator/1.log"
Jul 27 21:21:23.806076 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:23.806056    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-z6zxv_8a6b7ae6-6c21-453d-99b9-303066d19915/console-operator/1.log"
Jul 27 21:21:23.810343 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:23.810324    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-95hg7_433c431a-b5bf-4734-a03f-6f0a16895ded/ovn-acl-logging/0.log"
Jul 27 21:21:23.812230 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:23.812212    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-95hg7_433c431a-b5bf-4734-a03f-6f0a16895ded/ovn-acl-logging/0.log"
Jul 27 21:21:30.289495 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:30.289453    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:21:33.131021 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:33.130987    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-54ece799-openshift-default-698ff75858-pjsk4"]
Jul 27 21:21:33.131500 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:33.131230    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-54ece799-openshift-default-698ff75858-pjsk4" podUID="ee28216e-bfe4-4fff-b1e0-ec23f886ec5f" containerName="istio-proxy" containerID="cri-o://13867d03fb66f9e19fc44d56b373f8585ff204d5df26dc2992b4525b38d6dc00" gracePeriod=30
Jul 27 21:21:33.879981 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:33.879942    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-f154b44a-openshift-default-7cdfd86cfb-79s78"]
Jul 27 21:21:33.880327 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:33.880313    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbc7f3f4-9cef-4747-9205-2f08e594ba91" containerName="istio-proxy"
Jul 27 21:21:33.880376 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:33.880329    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbc7f3f4-9cef-4747-9205-2f08e594ba91" containerName="istio-proxy"
Jul 27 21:21:33.880376 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:33.880338    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbc7f3f4-9cef-4747-9205-2f08e594ba91" containerName="istio-proxy"
Jul 27 21:21:33.880376 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:33.880344    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbc7f3f4-9cef-4747-9205-2f08e594ba91" containerName="istio-proxy"
Jul 27 21:21:33.880466 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:33.880416    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="fbc7f3f4-9cef-4747-9205-2f08e594ba91" containerName="istio-proxy"
Jul 27 21:21:33.880466 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:33.880425    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="fbc7f3f4-9cef-4747-9205-2f08e594ba91" containerName="istio-proxy"
Jul 27 21:21:33.884419 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:33.884400    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-f154b44a-openshift-default-7cdfd86cfb-79s78"
Jul 27 21:21:33.886914 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:33.886889    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-f154b44a-openshift-default-dockercfg-2glh2\""
Jul 27 21:21:33.903407 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:33.903376    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-f154b44a-openshift-default-7cdfd86cfb-79s78"]
Jul 27 21:21:34.056531 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:34.056485    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/456e5d77-7483-438e-b365-9f8753d5d441-istio-envoy\") pod \"e2e-derive-f154b44a-openshift-default-7cdfd86cfb-79s78\" (UID: \"456e5d77-7483-438e-b365-9f8753d5d441\") " pod="openshift-ingress/e2e-derive-f154b44a-openshift-default-7cdfd86cfb-79s78"
Jul 27 21:21:34.056531 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:34.056530    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/456e5d77-7483-438e-b365-9f8753d5d441-istiod-ca-cert\") pod \"e2e-derive-f154b44a-openshift-default-7cdfd86cfb-79s78\" (UID: \"456e5d77-7483-438e-b365-9f8753d5d441\") " pod="openshift-ingress/e2e-derive-f154b44a-openshift-default-7cdfd86cfb-79s78"
Jul 27 21:21:34.056798 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:34.056610    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/456e5d77-7483-438e-b365-9f8753d5d441-credential-socket\") pod \"e2e-derive-f154b44a-openshift-default-7cdfd86cfb-79s78\" (UID: \"456e5d77-7483-438e-b365-9f8753d5d441\") " pod="openshift-ingress/e2e-derive-f154b44a-openshift-default-7cdfd86cfb-79s78"
Jul 27 21:21:34.056798 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:34.056710    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/456e5d77-7483-438e-b365-9f8753d5d441-workload-socket\") pod \"e2e-derive-f154b44a-openshift-default-7cdfd86cfb-79s78\" (UID: \"456e5d77-7483-438e-b365-9f8753d5d441\") " pod="openshift-ingress/e2e-derive-f154b44a-openshift-default-7cdfd86cfb-79s78"
Jul 27 21:21:34.056798 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:34.056737    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/456e5d77-7483-438e-b365-9f8753d5d441-istio-token\") pod \"e2e-derive-f154b44a-openshift-default-7cdfd86cfb-79s78\" (UID: \"456e5d77-7483-438e-b365-9f8753d5d441\") " pod="openshift-ingress/e2e-derive-f154b44a-openshift-default-7cdfd86cfb-79s78"
Jul 27 21:21:34.056798 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:34.056776    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4754v\" (UniqueName: \"kubernetes.io/projected/456e5d77-7483-438e-b365-9f8753d5d441-kube-api-access-4754v\") pod \"e2e-derive-f154b44a-openshift-default-7cdfd86cfb-79s78\" (UID: \"456e5d77-7483-438e-b365-9f8753d5d441\") " pod="openshift-ingress/e2e-derive-f154b44a-openshift-default-7cdfd86cfb-79s78"
Jul 27 21:21:34.056952 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:34.056829    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/456e5d77-7483-438e-b365-9f8753d5d441-workload-certs\") pod \"e2e-derive-f154b44a-openshift-default-7cdfd86cfb-79s78\" (UID: \"456e5d77-7483-438e-b365-9f8753d5d441\") " pod="openshift-ingress/e2e-derive-f154b44a-openshift-default-7cdfd86cfb-79s78"
Jul 27 21:21:34.056952 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:34.056861    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/456e5d77-7483-438e-b365-9f8753d5d441-istio-data\") pod \"e2e-derive-f154b44a-openshift-default-7cdfd86cfb-79s78\" (UID: \"456e5d77-7483-438e-b365-9f8753d5d441\") " pod="openshift-ingress/e2e-derive-f154b44a-openshift-default-7cdfd86cfb-79s78"
Jul 27 21:21:34.056952 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:34.056878    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/456e5d77-7483-438e-b365-9f8753d5d441-istio-podinfo\") pod \"e2e-derive-f154b44a-openshift-default-7cdfd86cfb-79s78\" (UID: \"456e5d77-7483-438e-b365-9f8753d5d441\") " pod="openshift-ingress/e2e-derive-f154b44a-openshift-default-7cdfd86cfb-79s78"
Jul 27 21:21:34.158124 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:34.158000    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/456e5d77-7483-438e-b365-9f8753d5d441-workload-socket\") pod \"e2e-derive-f154b44a-openshift-default-7cdfd86cfb-79s78\" (UID: \"456e5d77-7483-438e-b365-9f8753d5d441\") " pod="openshift-ingress/e2e-derive-f154b44a-openshift-default-7cdfd86cfb-79s78"
Jul 27 21:21:34.158124 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:34.158041    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/456e5d77-7483-438e-b365-9f8753d5d441-istio-token\") pod \"e2e-derive-f154b44a-openshift-default-7cdfd86cfb-79s78\" (UID: \"456e5d77-7483-438e-b365-9f8753d5d441\") " pod="openshift-ingress/e2e-derive-f154b44a-openshift-default-7cdfd86cfb-79s78"
Jul 27 21:21:34.158124 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:34.158078    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4754v\" (UniqueName: \"kubernetes.io/projected/456e5d77-7483-438e-b365-9f8753d5d441-kube-api-access-4754v\") pod \"e2e-derive-f154b44a-openshift-default-7cdfd86cfb-79s78\" (UID: \"456e5d77-7483-438e-b365-9f8753d5d441\") " pod="openshift-ingress/e2e-derive-f154b44a-openshift-default-7cdfd86cfb-79s78"
Jul 27 21:21:34.158712 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:34.158146    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/456e5d77-7483-438e-b365-9f8753d5d441-workload-certs\") pod \"e2e-derive-f154b44a-openshift-default-7cdfd86cfb-79s78\" (UID: \"456e5d77-7483-438e-b365-9f8753d5d441\") " pod="openshift-ingress/e2e-derive-f154b44a-openshift-default-7cdfd86cfb-79s78"
Jul 27 21:21:34.158712 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:34.158175    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/456e5d77-7483-438e-b365-9f8753d5d441-istio-data\") pod \"e2e-derive-f154b44a-openshift-default-7cdfd86cfb-79s78\" (UID: \"456e5d77-7483-438e-b365-9f8753d5d441\") " pod="openshift-ingress/e2e-derive-f154b44a-openshift-default-7cdfd86cfb-79s78"
Jul 27 21:21:34.158712 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:34.158202    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/456e5d77-7483-438e-b365-9f8753d5d441-istio-podinfo\") pod \"e2e-derive-f154b44a-openshift-default-7cdfd86cfb-79s78\" (UID: \"456e5d77-7483-438e-b365-9f8753d5d441\") " pod="openshift-ingress/e2e-derive-f154b44a-openshift-default-7cdfd86cfb-79s78"
Jul 27 21:21:34.158712 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:34.158232    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/456e5d77-7483-438e-b365-9f8753d5d441-istio-envoy\") pod \"e2e-derive-f154b44a-openshift-default-7cdfd86cfb-79s78\" (UID: \"456e5d77-7483-438e-b365-9f8753d5d441\") " pod="openshift-ingress/e2e-derive-f154b44a-openshift-default-7cdfd86cfb-79s78"
Jul 27 21:21:34.158712 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:34.158268    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/456e5d77-7483-438e-b365-9f8753d5d441-istiod-ca-cert\") pod \"e2e-derive-f154b44a-openshift-default-7cdfd86cfb-79s78\" (UID: \"456e5d77-7483-438e-b365-9f8753d5d441\") " pod="openshift-ingress/e2e-derive-f154b44a-openshift-default-7cdfd86cfb-79s78"
Jul 27 21:21:34.158712 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:34.158322    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/456e5d77-7483-438e-b365-9f8753d5d441-credential-socket\") pod \"e2e-derive-f154b44a-openshift-default-7cdfd86cfb-79s78\" (UID: \"456e5d77-7483-438e-b365-9f8753d5d441\") " pod="openshift-ingress/e2e-derive-f154b44a-openshift-default-7cdfd86cfb-79s78"
Jul 27 21:21:34.158712 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:34.158468    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/456e5d77-7483-438e-b365-9f8753d5d441-workload-socket\") pod \"e2e-derive-f154b44a-openshift-default-7cdfd86cfb-79s78\" (UID: \"456e5d77-7483-438e-b365-9f8753d5d441\") " pod="openshift-ingress/e2e-derive-f154b44a-openshift-default-7cdfd86cfb-79s78"
Jul 27 21:21:34.158712 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:34.158663    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/456e5d77-7483-438e-b365-9f8753d5d441-credential-socket\") pod \"e2e-derive-f154b44a-openshift-default-7cdfd86cfb-79s78\" (UID: \"456e5d77-7483-438e-b365-9f8753d5d441\") " pod="openshift-ingress/e2e-derive-f154b44a-openshift-default-7cdfd86cfb-79s78"
Jul 27 21:21:34.158712 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:34.158696    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/456e5d77-7483-438e-b365-9f8753d5d441-istio-data\") pod \"e2e-derive-f154b44a-openshift-default-7cdfd86cfb-79s78\" (UID: \"456e5d77-7483-438e-b365-9f8753d5d441\") " pod="openshift-ingress/e2e-derive-f154b44a-openshift-default-7cdfd86cfb-79s78"
Jul 27 21:21:34.159265 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:34.159204    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/456e5d77-7483-438e-b365-9f8753d5d441-workload-certs\") pod \"e2e-derive-f154b44a-openshift-default-7cdfd86cfb-79s78\" (UID: \"456e5d77-7483-438e-b365-9f8753d5d441\") " pod="openshift-ingress/e2e-derive-f154b44a-openshift-default-7cdfd86cfb-79s78"
Jul 27 21:21:34.159375 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:34.159349    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/456e5d77-7483-438e-b365-9f8753d5d441-istiod-ca-cert\") pod \"e2e-derive-f154b44a-openshift-default-7cdfd86cfb-79s78\" (UID: \"456e5d77-7483-438e-b365-9f8753d5d441\") " pod="openshift-ingress/e2e-derive-f154b44a-openshift-default-7cdfd86cfb-79s78"
Jul 27 21:21:34.161121 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:34.161080    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/456e5d77-7483-438e-b365-9f8753d5d441-istio-envoy\") pod \"e2e-derive-f154b44a-openshift-default-7cdfd86cfb-79s78\" (UID: \"456e5d77-7483-438e-b365-9f8753d5d441\") " pod="openshift-ingress/e2e-derive-f154b44a-openshift-default-7cdfd86cfb-79s78"
Jul 27 21:21:34.162124 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:34.162072    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/456e5d77-7483-438e-b365-9f8753d5d441-istio-podinfo\") pod \"e2e-derive-f154b44a-openshift-default-7cdfd86cfb-79s78\" (UID: \"456e5d77-7483-438e-b365-9f8753d5d441\") " pod="openshift-ingress/e2e-derive-f154b44a-openshift-default-7cdfd86cfb-79s78"
Jul 27 21:21:34.168359 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:34.168335    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4754v\" (UniqueName: \"kubernetes.io/projected/456e5d77-7483-438e-b365-9f8753d5d441-kube-api-access-4754v\") pod \"e2e-derive-f154b44a-openshift-default-7cdfd86cfb-79s78\" (UID: \"456e5d77-7483-438e-b365-9f8753d5d441\") " pod="openshift-ingress/e2e-derive-f154b44a-openshift-default-7cdfd86cfb-79s78"
Jul 27 21:21:34.168473 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:34.168404    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/456e5d77-7483-438e-b365-9f8753d5d441-istio-token\") pod \"e2e-derive-f154b44a-openshift-default-7cdfd86cfb-79s78\" (UID: \"456e5d77-7483-438e-b365-9f8753d5d441\") " pod="openshift-ingress/e2e-derive-f154b44a-openshift-default-7cdfd86cfb-79s78"
Jul 27 21:21:34.194501 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:34.194463    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-f154b44a-openshift-default-7cdfd86cfb-79s78"
Jul 27 21:21:34.351469 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:34.351440    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-f154b44a-openshift-default-7cdfd86cfb-79s78"]
Jul 27 21:21:34.353796 ip-10-0-141-64 kubenswrapper[2560]: W0727 21:21:34.353761    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod456e5d77_7483_438e_b365_9f8753d5d441.slice/crio-a6d9b5a1bb883ca2fd0ec5d8fd27de7409ee5c4d1fab60d927115c60035d5319 WatchSource:0}: Error finding container a6d9b5a1bb883ca2fd0ec5d8fd27de7409ee5c4d1fab60d927115c60035d5319: Status 404 returned error can't find the container with id a6d9b5a1bb883ca2fd0ec5d8fd27de7409ee5c4d1fab60d927115c60035d5319
Jul 27 21:21:34.356278 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:34.356246    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 27 21:21:34.356360 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:34.356310    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 27 21:21:34.356360 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:34.356341    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 27 21:21:34.787327 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:34.787288    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-f154b44a-openshift-default-7cdfd86cfb-79s78" event={"ID":"456e5d77-7483-438e-b365-9f8753d5d441","Type":"ContainerStarted","Data":"fd86285a3e1c51b9ba524978fd003ca646b42114284feb1dc2565e3096a4648f"}
Jul 27 21:21:34.787548 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:34.787342    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-f154b44a-openshift-default-7cdfd86cfb-79s78" event={"ID":"456e5d77-7483-438e-b365-9f8753d5d441","Type":"ContainerStarted","Data":"a6d9b5a1bb883ca2fd0ec5d8fd27de7409ee5c4d1fab60d927115c60035d5319"}
Jul 27 21:21:34.809332 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:34.809271    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-f154b44a-openshift-default-7cdfd86cfb-79s78" podStartSLOduration=1.809254527 podStartE2EDuration="1.809254527s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:21:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:21:34.807437286 +0000 UTC m=+1811.559299102" watchObservedRunningTime="2026-07-27 21:21:34.809254527 +0000 UTC m=+1811.561116300"
Jul 27 21:21:35.195197 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:35.195078    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-f154b44a-openshift-default-7cdfd86cfb-79s78"
Jul 27 21:21:35.199892 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:35.199864    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-f154b44a-openshift-default-7cdfd86cfb-79s78"
Jul 27 21:21:35.792438 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:35.792408    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-f154b44a-openshift-default-7cdfd86cfb-79s78"
Jul 27 21:21:35.793317 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:35.793295    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-f154b44a-openshift-default-7cdfd86cfb-79s78"
Jul 27 21:21:38.393607 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:38.393581    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-54ece799-openshift-default-698ff75858-pjsk4"
Jul 27 21:21:38.505013 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:38.504842    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee28216e-bfe4-4fff-b1e0-ec23f886ec5f-istio-podinfo\") pod \"ee28216e-bfe4-4fff-b1e0-ec23f886ec5f\" (UID: \"ee28216e-bfe4-4fff-b1e0-ec23f886ec5f\") "
Jul 27 21:21:38.505013 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:38.504911    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee28216e-bfe4-4fff-b1e0-ec23f886ec5f-workload-certs\") pod \"ee28216e-bfe4-4fff-b1e0-ec23f886ec5f\" (UID: \"ee28216e-bfe4-4fff-b1e0-ec23f886ec5f\") "
Jul 27 21:21:38.505013 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:38.504951    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee28216e-bfe4-4fff-b1e0-ec23f886ec5f-workload-socket\") pod \"ee28216e-bfe4-4fff-b1e0-ec23f886ec5f\" (UID: \"ee28216e-bfe4-4fff-b1e0-ec23f886ec5f\") "
Jul 27 21:21:38.505013 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:38.505005    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bbnhv\" (UniqueName: \"kubernetes.io/projected/ee28216e-bfe4-4fff-b1e0-ec23f886ec5f-kube-api-access-bbnhv\") pod \"ee28216e-bfe4-4fff-b1e0-ec23f886ec5f\" (UID: \"ee28216e-bfe4-4fff-b1e0-ec23f886ec5f\") "
Jul 27 21:21:38.505391 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:38.505032    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee28216e-bfe4-4fff-b1e0-ec23f886ec5f-istio-envoy\") pod \"ee28216e-bfe4-4fff-b1e0-ec23f886ec5f\" (UID: \"ee28216e-bfe4-4fff-b1e0-ec23f886ec5f\") "
Jul 27 21:21:38.505391 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:38.505080    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee28216e-bfe4-4fff-b1e0-ec23f886ec5f-istio-data\") pod \"ee28216e-bfe4-4fff-b1e0-ec23f886ec5f\" (UID: \"ee28216e-bfe4-4fff-b1e0-ec23f886ec5f\") "
Jul 27 21:21:38.505391 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:38.505138    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee28216e-bfe4-4fff-b1e0-ec23f886ec5f-credential-socket\") pod \"ee28216e-bfe4-4fff-b1e0-ec23f886ec5f\" (UID: \"ee28216e-bfe4-4fff-b1e0-ec23f886ec5f\") "
Jul 27 21:21:38.505391 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:38.505223    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee28216e-bfe4-4fff-b1e0-ec23f886ec5f-istiod-ca-cert\") pod \"ee28216e-bfe4-4fff-b1e0-ec23f886ec5f\" (UID: \"ee28216e-bfe4-4fff-b1e0-ec23f886ec5f\") "
Jul 27 21:21:38.505391 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:38.505284    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee28216e-bfe4-4fff-b1e0-ec23f886ec5f-istio-token\") pod \"ee28216e-bfe4-4fff-b1e0-ec23f886ec5f\" (UID: \"ee28216e-bfe4-4fff-b1e0-ec23f886ec5f\") "
Jul 27 21:21:38.505629 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:38.505446    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee28216e-bfe4-4fff-b1e0-ec23f886ec5f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ee28216e-bfe4-4fff-b1e0-ec23f886ec5f" (UID: "ee28216e-bfe4-4fff-b1e0-ec23f886ec5f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:21:38.505629 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:38.505613    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee28216e-bfe4-4fff-b1e0-ec23f886ec5f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ee28216e-bfe4-4fff-b1e0-ec23f886ec5f" (UID: "ee28216e-bfe4-4fff-b1e0-ec23f886ec5f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:21:38.505747 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:38.505630    2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee28216e-bfe4-4fff-b1e0-ec23f886ec5f-workload-socket\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:21:38.505747 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:38.505630    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee28216e-bfe4-4fff-b1e0-ec23f886ec5f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ee28216e-bfe4-4fff-b1e0-ec23f886ec5f" (UID: "ee28216e-bfe4-4fff-b1e0-ec23f886ec5f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:21:38.506206 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:38.506178    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee28216e-bfe4-4fff-b1e0-ec23f886ec5f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ee28216e-bfe4-4fff-b1e0-ec23f886ec5f" (UID: "ee28216e-bfe4-4fff-b1e0-ec23f886ec5f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:21:38.506642 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:38.506602    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee28216e-bfe4-4fff-b1e0-ec23f886ec5f-istio-data" (OuterVolumeSpecName: "istio-data") pod "ee28216e-bfe4-4fff-b1e0-ec23f886ec5f" (UID: "ee28216e-bfe4-4fff-b1e0-ec23f886ec5f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:21:38.507508 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:38.507478    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ee28216e-bfe4-4fff-b1e0-ec23f886ec5f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ee28216e-bfe4-4fff-b1e0-ec23f886ec5f" (UID: "ee28216e-bfe4-4fff-b1e0-ec23f886ec5f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 27 21:21:38.507605 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:38.507506    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee28216e-bfe4-4fff-b1e0-ec23f886ec5f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ee28216e-bfe4-4fff-b1e0-ec23f886ec5f" (UID: "ee28216e-bfe4-4fff-b1e0-ec23f886ec5f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:21:38.507668 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:38.507640    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee28216e-bfe4-4fff-b1e0-ec23f886ec5f-kube-api-access-bbnhv" (OuterVolumeSpecName: "kube-api-access-bbnhv") pod "ee28216e-bfe4-4fff-b1e0-ec23f886ec5f" (UID: "ee28216e-bfe4-4fff-b1e0-ec23f886ec5f"). InnerVolumeSpecName "kube-api-access-bbnhv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:21:38.507753 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:38.507731    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee28216e-bfe4-4fff-b1e0-ec23f886ec5f-istio-token" (OuterVolumeSpecName: "istio-token") pod "ee28216e-bfe4-4fff-b1e0-ec23f886ec5f" (UID: "ee28216e-bfe4-4fff-b1e0-ec23f886ec5f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:21:38.606337 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:38.606293    2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee28216e-bfe4-4fff-b1e0-ec23f886ec5f-istiod-ca-cert\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:21:38.606337 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:38.606328    2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee28216e-bfe4-4fff-b1e0-ec23f886ec5f-istio-token\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:21:38.606337 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:38.606337    2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee28216e-bfe4-4fff-b1e0-ec23f886ec5f-istio-podinfo\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:21:38.606337 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:38.606348    2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee28216e-bfe4-4fff-b1e0-ec23f886ec5f-workload-certs\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:21:38.606613 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:38.606356    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbnhv\" (UniqueName: \"kubernetes.io/projected/ee28216e-bfe4-4fff-b1e0-ec23f886ec5f-kube-api-access-bbnhv\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:21:38.606613 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:38.606366    2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee28216e-bfe4-4fff-b1e0-ec23f886ec5f-istio-envoy\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:21:38.606613 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:38.606374    2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee28216e-bfe4-4fff-b1e0-ec23f886ec5f-istio-data\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:21:38.606613 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:38.606382    2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee28216e-bfe4-4fff-b1e0-ec23f886ec5f-credential-socket\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:21:38.807518 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:38.807479    2560 generic.go:356] "Generic (PLEG): container finished" podID="ee28216e-bfe4-4fff-b1e0-ec23f886ec5f" containerID="13867d03fb66f9e19fc44d56b373f8585ff204d5df26dc2992b4525b38d6dc00" exitCode=0
Jul 27 21:21:38.807692 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:38.807554    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-54ece799-openshift-default-698ff75858-pjsk4"
Jul 27 21:21:38.807692 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:38.807563    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-54ece799-openshift-default-698ff75858-pjsk4" event={"ID":"ee28216e-bfe4-4fff-b1e0-ec23f886ec5f","Type":"ContainerDied","Data":"13867d03fb66f9e19fc44d56b373f8585ff204d5df26dc2992b4525b38d6dc00"}
Jul 27 21:21:38.807692 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:38.807650    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-54ece799-openshift-default-698ff75858-pjsk4" event={"ID":"ee28216e-bfe4-4fff-b1e0-ec23f886ec5f","Type":"ContainerDied","Data":"bed8a5009d83a8ddfa984c601dd09030369cad456444c0ff9ca53ce251804f6d"}
Jul 27 21:21:38.807692 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:38.807673    2560 scope.go:117] "RemoveContainer" containerID="13867d03fb66f9e19fc44d56b373f8585ff204d5df26dc2992b4525b38d6dc00"
Jul 27 21:21:38.817886 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:38.817863    2560 scope.go:117] "RemoveContainer" containerID="828647e86c79daf695831de097e16195a7df7dc0c035434bd341a527da4f514b"
Jul 27 21:21:38.829208 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:38.829184    2560 scope.go:117] "RemoveContainer" containerID="13867d03fb66f9e19fc44d56b373f8585ff204d5df26dc2992b4525b38d6dc00"
Jul 27 21:21:38.829590 ip-10-0-141-64 kubenswrapper[2560]: E0727 21:21:38.829560    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13867d03fb66f9e19fc44d56b373f8585ff204d5df26dc2992b4525b38d6dc00\": container with ID starting with 13867d03fb66f9e19fc44d56b373f8585ff204d5df26dc2992b4525b38d6dc00 not found: ID does not exist" containerID="13867d03fb66f9e19fc44d56b373f8585ff204d5df26dc2992b4525b38d6dc00"
Jul 27 21:21:38.829680 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:38.829600    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13867d03fb66f9e19fc44d56b373f8585ff204d5df26dc2992b4525b38d6dc00"} err="failed to get container status \"13867d03fb66f9e19fc44d56b373f8585ff204d5df26dc2992b4525b38d6dc00\": rpc error: code = NotFound desc = could not find container \"13867d03fb66f9e19fc44d56b373f8585ff204d5df26dc2992b4525b38d6dc00\": container with ID starting with 13867d03fb66f9e19fc44d56b373f8585ff204d5df26dc2992b4525b38d6dc00 not found: ID does not exist"
Jul 27 21:21:38.829680 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:38.829628    2560 scope.go:117] "RemoveContainer" containerID="828647e86c79daf695831de097e16195a7df7dc0c035434bd341a527da4f514b"
Jul 27 21:21:38.830080 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:38.830055    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-54ece799-openshift-default-698ff75858-pjsk4"]
Jul 27 21:21:38.830203 ip-10-0-141-64 kubenswrapper[2560]: E0727 21:21:38.830046    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"828647e86c79daf695831de097e16195a7df7dc0c035434bd341a527da4f514b\": container with ID starting with 828647e86c79daf695831de097e16195a7df7dc0c035434bd341a527da4f514b not found: ID does not exist" containerID="828647e86c79daf695831de097e16195a7df7dc0c035434bd341a527da4f514b"
Jul 27 21:21:38.830203 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:38.830152    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"828647e86c79daf695831de097e16195a7df7dc0c035434bd341a527da4f514b"} err="failed to get container status \"828647e86c79daf695831de097e16195a7df7dc0c035434bd341a527da4f514b\": rpc error: code = NotFound desc = could not find container \"828647e86c79daf695831de097e16195a7df7dc0c035434bd341a527da4f514b\": container with ID starting with 828647e86c79daf695831de097e16195a7df7dc0c035434bd341a527da4f514b not found: ID does not exist"
Jul 27 21:21:38.832340 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:38.832317    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-54ece799-openshift-default-698ff75858-pjsk4"]
Jul 27 21:21:39.597036 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:39.596997    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:21:39.846700 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:39.846660    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee28216e-bfe4-4fff-b1e0-ec23f886ec5f" path="/var/lib/kubelet/pods/ee28216e-bfe4-4fff-b1e0-ec23f886ec5f/volumes"
Jul 27 21:21:47.851181 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:47.851071    2560 generic.go:356] "Generic (PLEG): container finished" podID="456e5d77-7483-438e-b365-9f8753d5d441" containerID="fd86285a3e1c51b9ba524978fd003ca646b42114284feb1dc2565e3096a4648f" exitCode=0
Jul 27 21:21:47.851181 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:47.851128    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-f154b44a-openshift-default-7cdfd86cfb-79s78" event={"ID":"456e5d77-7483-438e-b365-9f8753d5d441","Type":"ContainerDied","Data":"fd86285a3e1c51b9ba524978fd003ca646b42114284feb1dc2565e3096a4648f"}
Jul 27 21:21:47.851720 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:47.851593    2560 scope.go:117] "RemoveContainer" containerID="fd86285a3e1c51b9ba524978fd003ca646b42114284feb1dc2565e3096a4648f"
Jul 27 21:21:47.852663 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:47.852599    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 27 21:21:47.852770 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:47.852678    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 27 21:21:47.852770 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:47.852725    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 27 21:21:48.860642 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:48.860583    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-f154b44a-openshift-default-7cdfd86cfb-79s78" event={"ID":"456e5d77-7483-438e-b365-9f8753d5d441","Type":"ContainerStarted","Data":"030a01a4377e2041f949b7eed65e6cdff4a4c957dfc9a9b3509e3cbfa5c746fc"}
Jul 27 21:21:49.195036 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:49.194947    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-f154b44a-openshift-default-7cdfd86cfb-79s78"
Jul 27 21:21:49.196147 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:49.196118    2560 patch_prober.go:28] interesting pod/e2e-derive-f154b44a-openshift-default-7cdfd86cfb-79s78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Jul 27 21:21:49.196273 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:49.196173    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-f154b44a-openshift-default-7cdfd86cfb-79s78" podUID="456e5d77-7483-438e-b365-9f8753d5d441" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Jul 27 21:21:50.195403 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:50.195364    2560 patch_prober.go:28] interesting pod/e2e-derive-f154b44a-openshift-default-7cdfd86cfb-79s78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Jul 27 21:21:50.195801 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:50.195445    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-f154b44a-openshift-default-7cdfd86cfb-79s78" podUID="456e5d77-7483-438e-b365-9f8753d5d441" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Jul 27 21:21:50.792806 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:50.792771    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-f154b44a-openshift-default-7cdfd86cfb-79s78"
Jul 27 21:21:51.195251 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:51.195157    2560 patch_prober.go:28] interesting pod/e2e-derive-f154b44a-openshift-default-7cdfd86cfb-79s78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Jul 27 21:21:51.195251 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:51.195227    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-f154b44a-openshift-default-7cdfd86cfb-79s78" podUID="456e5d77-7483-438e-b365-9f8753d5d441" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Jul 27 21:21:52.195746 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:52.195710    2560 patch_prober.go:28] interesting pod/e2e-derive-f154b44a-openshift-default-7cdfd86cfb-79s78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Jul 27 21:21:52.196182 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:52.195764    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-f154b44a-openshift-default-7cdfd86cfb-79s78" podUID="456e5d77-7483-438e-b365-9f8753d5d441" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Jul 27 21:21:53.199045 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:53.199011    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-f154b44a-openshift-default-7cdfd86cfb-79s78"
Jul 27 21:21:53.200124 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:21:53.200105    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-f154b44a-openshift-default-7cdfd86cfb-79s78"
Jul 27 21:22:04.497068 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:22:04.497028    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:22:06.015057 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:22:06.015021    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-f154b44a-openshift-default-7cdfd86cfb-79s78"]
Jul 27 21:22:06.015458 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:22:06.015300    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-f154b44a-openshift-default-7cdfd86cfb-79s78" podUID="456e5d77-7483-438e-b365-9f8753d5d441" containerName="istio-proxy" containerID="cri-o://030a01a4377e2041f949b7eed65e6cdff4a4c957dfc9a9b3509e3cbfa5c746fc" gracePeriod=30
Jul 27 21:22:08.201388 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:22:08.201344    2560 patch_prober.go:28] interesting pod/e2e-derive-f154b44a-openshift-default-7cdfd86cfb-79s78 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused
Jul 27 21:22:08.201783 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:22:08.201417    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-f154b44a-openshift-default-7cdfd86cfb-79s78" podUID="456e5d77-7483-438e-b365-9f8753d5d441" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 21:22:11.272783 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:22:11.272755    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-f154b44a-openshift-default-7cdfd86cfb-79s78"
Jul 27 21:22:11.398222 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:22:11.398136    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/456e5d77-7483-438e-b365-9f8753d5d441-workload-socket\") pod \"456e5d77-7483-438e-b365-9f8753d5d441\" (UID: \"456e5d77-7483-438e-b365-9f8753d5d441\") "
Jul 27 21:22:11.398222 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:22:11.398184    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/456e5d77-7483-438e-b365-9f8753d5d441-workload-certs\") pod \"456e5d77-7483-438e-b365-9f8753d5d441\" (UID: \"456e5d77-7483-438e-b365-9f8753d5d441\") "
Jul 27 21:22:11.398222 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:22:11.398223    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4754v\" (UniqueName: \"kubernetes.io/projected/456e5d77-7483-438e-b365-9f8753d5d441-kube-api-access-4754v\") pod \"456e5d77-7483-438e-b365-9f8753d5d441\" (UID: \"456e5d77-7483-438e-b365-9f8753d5d441\") "
Jul 27 21:22:11.398499 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:22:11.398241    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/456e5d77-7483-438e-b365-9f8753d5d441-istio-data\") pod \"456e5d77-7483-438e-b365-9f8753d5d441\" (UID: \"456e5d77-7483-438e-b365-9f8753d5d441\") "
Jul 27 21:22:11.398499 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:22:11.398273    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/456e5d77-7483-438e-b365-9f8753d5d441-istio-envoy\") pod \"456e5d77-7483-438e-b365-9f8753d5d441\" (UID: \"456e5d77-7483-438e-b365-9f8753d5d441\") "
Jul 27 21:22:11.398499 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:22:11.398298    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/456e5d77-7483-438e-b365-9f8753d5d441-istio-podinfo\") pod \"456e5d77-7483-438e-b365-9f8753d5d441\" (UID: \"456e5d77-7483-438e-b365-9f8753d5d441\") "
Jul 27 21:22:11.398499 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:22:11.398354    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/456e5d77-7483-438e-b365-9f8753d5d441-istio-token\") pod \"456e5d77-7483-438e-b365-9f8753d5d441\" (UID: \"456e5d77-7483-438e-b365-9f8753d5d441\") "
Jul 27 21:22:11.398499 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:22:11.398376    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/456e5d77-7483-438e-b365-9f8753d5d441-credential-socket\") pod \"456e5d77-7483-438e-b365-9f8753d5d441\" (UID: \"456e5d77-7483-438e-b365-9f8753d5d441\") "
Jul 27 21:22:11.398499 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:22:11.398419    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/456e5d77-7483-438e-b365-9f8753d5d441-istiod-ca-cert\") pod \"456e5d77-7483-438e-b365-9f8753d5d441\" (UID: \"456e5d77-7483-438e-b365-9f8753d5d441\") "
Jul 27 21:22:11.398499 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:22:11.398446    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/456e5d77-7483-438e-b365-9f8753d5d441-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "456e5d77-7483-438e-b365-9f8753d5d441" (UID: "456e5d77-7483-438e-b365-9f8753d5d441"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:22:11.398845 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:22:11.398596    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/456e5d77-7483-438e-b365-9f8753d5d441-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "456e5d77-7483-438e-b365-9f8753d5d441" (UID: "456e5d77-7483-438e-b365-9f8753d5d441"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:22:11.398845 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:22:11.398652    2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/456e5d77-7483-438e-b365-9f8753d5d441-workload-socket\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:22:11.398845 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:22:11.398709    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/456e5d77-7483-438e-b365-9f8753d5d441-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "456e5d77-7483-438e-b365-9f8753d5d441" (UID: "456e5d77-7483-438e-b365-9f8753d5d441"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:22:11.398990 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:22:11.398947    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/456e5d77-7483-438e-b365-9f8753d5d441-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "456e5d77-7483-438e-b365-9f8753d5d441" (UID: "456e5d77-7483-438e-b365-9f8753d5d441"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:22:11.399305 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:22:11.399282    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/456e5d77-7483-438e-b365-9f8753d5d441-istio-data" (OuterVolumeSpecName: "istio-data") pod "456e5d77-7483-438e-b365-9f8753d5d441" (UID: "456e5d77-7483-438e-b365-9f8753d5d441"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:22:11.400645 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:22:11.400617    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/456e5d77-7483-438e-b365-9f8753d5d441-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "456e5d77-7483-438e-b365-9f8753d5d441" (UID: "456e5d77-7483-438e-b365-9f8753d5d441"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 27 21:22:11.400968 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:22:11.400943    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/456e5d77-7483-438e-b365-9f8753d5d441-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "456e5d77-7483-438e-b365-9f8753d5d441" (UID: "456e5d77-7483-438e-b365-9f8753d5d441"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:22:11.401202 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:22:11.401173    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/456e5d77-7483-438e-b365-9f8753d5d441-istio-token" (OuterVolumeSpecName: "istio-token") pod "456e5d77-7483-438e-b365-9f8753d5d441" (UID: "456e5d77-7483-438e-b365-9f8753d5d441"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:22:11.401202 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:22:11.401188    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/456e5d77-7483-438e-b365-9f8753d5d441-kube-api-access-4754v" (OuterVolumeSpecName: "kube-api-access-4754v") pod "456e5d77-7483-438e-b365-9f8753d5d441" (UID: "456e5d77-7483-438e-b365-9f8753d5d441"). InnerVolumeSpecName "kube-api-access-4754v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:22:11.499792 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:22:11.499749    2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/456e5d77-7483-438e-b365-9f8753d5d441-istio-token\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:22:11.499792 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:22:11.499785    2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/456e5d77-7483-438e-b365-9f8753d5d441-credential-socket\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:22:11.499792 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:22:11.499795    2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/456e5d77-7483-438e-b365-9f8753d5d441-istiod-ca-cert\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:22:11.500030 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:22:11.499808    2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/456e5d77-7483-438e-b365-9f8753d5d441-workload-certs\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:22:11.500030 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:22:11.499817    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4754v\" (UniqueName: \"kubernetes.io/projected/456e5d77-7483-438e-b365-9f8753d5d441-kube-api-access-4754v\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:22:11.500030 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:22:11.499827    2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/456e5d77-7483-438e-b365-9f8753d5d441-istio-data\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:22:11.500030 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:22:11.499835    2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/456e5d77-7483-438e-b365-9f8753d5d441-istio-envoy\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:22:11.500030 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:22:11.499842    2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/456e5d77-7483-438e-b365-9f8753d5d441-istio-podinfo\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:22:11.975017 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:22:11.974983    2560 generic.go:356] "Generic (PLEG): container finished" podID="456e5d77-7483-438e-b365-9f8753d5d441" containerID="030a01a4377e2041f949b7eed65e6cdff4a4c957dfc9a9b3509e3cbfa5c746fc" exitCode=0
Jul 27 21:22:11.975209 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:22:11.975036    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-f154b44a-openshift-default-7cdfd86cfb-79s78"
Jul 27 21:22:11.975209 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:22:11.975073    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-f154b44a-openshift-default-7cdfd86cfb-79s78" event={"ID":"456e5d77-7483-438e-b365-9f8753d5d441","Type":"ContainerDied","Data":"030a01a4377e2041f949b7eed65e6cdff4a4c957dfc9a9b3509e3cbfa5c746fc"}
Jul 27 21:22:11.975209 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:22:11.975155    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-f154b44a-openshift-default-7cdfd86cfb-79s78" event={"ID":"456e5d77-7483-438e-b365-9f8753d5d441","Type":"ContainerDied","Data":"a6d9b5a1bb883ca2fd0ec5d8fd27de7409ee5c4d1fab60d927115c60035d5319"}
Jul 27 21:22:11.975209 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:22:11.975179    2560 scope.go:117] "RemoveContainer" containerID="030a01a4377e2041f949b7eed65e6cdff4a4c957dfc9a9b3509e3cbfa5c746fc"
Jul 27 21:22:11.990567 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:22:11.990537    2560 scope.go:117] "RemoveContainer" containerID="fd86285a3e1c51b9ba524978fd003ca646b42114284feb1dc2565e3096a4648f"
Jul 27 21:22:11.998191 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:22:11.998174    2560 scope.go:117] "RemoveContainer" containerID="030a01a4377e2041f949b7eed65e6cdff4a4c957dfc9a9b3509e3cbfa5c746fc"
Jul 27 21:22:11.998425 ip-10-0-141-64 kubenswrapper[2560]: E0727 21:22:11.998408    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"030a01a4377e2041f949b7eed65e6cdff4a4c957dfc9a9b3509e3cbfa5c746fc\": container with ID starting with 030a01a4377e2041f949b7eed65e6cdff4a4c957dfc9a9b3509e3cbfa5c746fc not found: ID does not exist" containerID="030a01a4377e2041f949b7eed65e6cdff4a4c957dfc9a9b3509e3cbfa5c746fc"
Jul 27 21:22:11.998473 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:22:11.998432    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"030a01a4377e2041f949b7eed65e6cdff4a4c957dfc9a9b3509e3cbfa5c746fc"} err="failed to get container status \"030a01a4377e2041f949b7eed65e6cdff4a4c957dfc9a9b3509e3cbfa5c746fc\": rpc error: code = NotFound desc = could not find container \"030a01a4377e2041f949b7eed65e6cdff4a4c957dfc9a9b3509e3cbfa5c746fc\": container with ID starting with 030a01a4377e2041f949b7eed65e6cdff4a4c957dfc9a9b3509e3cbfa5c746fc not found: ID does not exist"
Jul 27 21:22:11.998473 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:22:11.998448    2560 scope.go:117] "RemoveContainer" containerID="fd86285a3e1c51b9ba524978fd003ca646b42114284feb1dc2565e3096a4648f"
Jul 27 21:22:11.998770 ip-10-0-141-64 kubenswrapper[2560]: E0727 21:22:11.998739    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd86285a3e1c51b9ba524978fd003ca646b42114284feb1dc2565e3096a4648f\": container with ID starting with fd86285a3e1c51b9ba524978fd003ca646b42114284feb1dc2565e3096a4648f not found: ID does not exist" containerID="fd86285a3e1c51b9ba524978fd003ca646b42114284feb1dc2565e3096a4648f"
Jul 27 21:22:11.998936 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:22:11.998778    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd86285a3e1c51b9ba524978fd003ca646b42114284feb1dc2565e3096a4648f"} err="failed to get container status \"fd86285a3e1c51b9ba524978fd003ca646b42114284feb1dc2565e3096a4648f\": rpc error: code = NotFound desc = could not find container \"fd86285a3e1c51b9ba524978fd003ca646b42114284feb1dc2565e3096a4648f\": container with ID starting with fd86285a3e1c51b9ba524978fd003ca646b42114284feb1dc2565e3096a4648f not found: ID does not exist"
Jul 27 21:22:12.000988 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:22:12.000965    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-f154b44a-openshift-default-7cdfd86cfb-79s78"]
Jul 27 21:22:12.004979 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:22:12.004953    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-f154b44a-openshift-default-7cdfd86cfb-79s78"]
Jul 27 21:22:13.847648 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:22:13.847618    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="456e5d77-7483-438e-b365-9f8753d5d441" path="/var/lib/kubelet/pods/456e5d77-7483-438e-b365-9f8753d5d441/volumes"
Jul 27 21:23:32.924508 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:23:32.924460    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:23:41.695063 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:23:41.695025    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:23:43.465564 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:23:43.465523    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-c5876a31-openshift-default-56c95b9584-nt787"]
Jul 27 21:23:43.465938 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:23:43.465882    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="456e5d77-7483-438e-b365-9f8753d5d441" containerName="istio-proxy"
Jul 27 21:23:43.465938 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:23:43.465893    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="456e5d77-7483-438e-b365-9f8753d5d441" containerName="istio-proxy"
Jul 27 21:23:43.465938 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:23:43.465915    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee28216e-bfe4-4fff-b1e0-ec23f886ec5f" containerName="istio-proxy"
Jul 27 21:23:43.465938 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:23:43.465921    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee28216e-bfe4-4fff-b1e0-ec23f886ec5f" containerName="istio-proxy"
Jul 27 21:23:43.465938 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:23:43.465933    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee28216e-bfe4-4fff-b1e0-ec23f886ec5f" containerName="istio-proxy"
Jul 27 21:23:43.465938 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:23:43.465939    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee28216e-bfe4-4fff-b1e0-ec23f886ec5f" containerName="istio-proxy"
Jul 27 21:23:43.466160 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:23:43.465991    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee28216e-bfe4-4fff-b1e0-ec23f886ec5f" containerName="istio-proxy"
Jul 27 21:23:43.466160 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:23:43.465997    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="456e5d77-7483-438e-b365-9f8753d5d441" containerName="istio-proxy"
Jul 27 21:23:43.466160 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:23:43.466004    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee28216e-bfe4-4fff-b1e0-ec23f886ec5f" containerName="istio-proxy"
Jul 27 21:23:43.469065 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:23:43.469047    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c5876a31-openshift-default-56c95b9584-nt787"
Jul 27 21:23:43.471463 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:23:43.471440    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-c5876a31-openshift-default-dockercfg-7v8hx\""
Jul 27 21:23:43.491305 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:23:43.491280    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c5876a31-openshift-default-56c95b9584-nt787"]
Jul 27 21:23:43.626077 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:23:43.626031    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eddaa0ae-1af7-4c61-a660-d805971e6d43-credential-socket\") pod \"e2e-mt-c5876a31-openshift-default-56c95b9584-nt787\" (UID: \"eddaa0ae-1af7-4c61-a660-d805971e6d43\") " pod="openshift-ingress/e2e-mt-c5876a31-openshift-default-56c95b9584-nt787"
Jul 27 21:23:43.626077 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:23:43.626082    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eddaa0ae-1af7-4c61-a660-d805971e6d43-istio-envoy\") pod \"e2e-mt-c5876a31-openshift-default-56c95b9584-nt787\" (UID: \"eddaa0ae-1af7-4c61-a660-d805971e6d43\") " pod="openshift-ingress/e2e-mt-c5876a31-openshift-default-56c95b9584-nt787"
Jul 27 21:23:43.626336 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:23:43.626130    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eddaa0ae-1af7-4c61-a660-d805971e6d43-workload-certs\") pod \"e2e-mt-c5876a31-openshift-default-56c95b9584-nt787\" (UID: \"eddaa0ae-1af7-4c61-a660-d805971e6d43\") " pod="openshift-ingress/e2e-mt-c5876a31-openshift-default-56c95b9584-nt787"
Jul 27 21:23:43.626336 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:23:43.626147    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eddaa0ae-1af7-4c61-a660-d805971e6d43-istiod-ca-cert\") pod \"e2e-mt-c5876a31-openshift-default-56c95b9584-nt787\" (UID: \"eddaa0ae-1af7-4c61-a660-d805971e6d43\") " pod="openshift-ingress/e2e-mt-c5876a31-openshift-default-56c95b9584-nt787"
Jul 27 21:23:43.626336 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:23:43.626167    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eddaa0ae-1af7-4c61-a660-d805971e6d43-istio-podinfo\") pod \"e2e-mt-c5876a31-openshift-default-56c95b9584-nt787\" (UID: \"eddaa0ae-1af7-4c61-a660-d805971e6d43\") " pod="openshift-ingress/e2e-mt-c5876a31-openshift-default-56c95b9584-nt787"
Jul 27 21:23:43.626336 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:23:43.626233    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eddaa0ae-1af7-4c61-a660-d805971e6d43-istio-token\") pod \"e2e-mt-c5876a31-openshift-default-56c95b9584-nt787\" (UID: \"eddaa0ae-1af7-4c61-a660-d805971e6d43\") " pod="openshift-ingress/e2e-mt-c5876a31-openshift-default-56c95b9584-nt787"
Jul 27 21:23:43.626336 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:23:43.626253    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grfgb\" (UniqueName: \"kubernetes.io/projected/eddaa0ae-1af7-4c61-a660-d805971e6d43-kube-api-access-grfgb\") pod \"e2e-mt-c5876a31-openshift-default-56c95b9584-nt787\" (UID: \"eddaa0ae-1af7-4c61-a660-d805971e6d43\") " pod="openshift-ingress/e2e-mt-c5876a31-openshift-default-56c95b9584-nt787"
Jul 27 21:23:43.626336 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:23:43.626287    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eddaa0ae-1af7-4c61-a660-d805971e6d43-workload-socket\") pod \"e2e-mt-c5876a31-openshift-default-56c95b9584-nt787\" (UID: \"eddaa0ae-1af7-4c61-a660-d805971e6d43\") " pod="openshift-ingress/e2e-mt-c5876a31-openshift-default-56c95b9584-nt787"
Jul 27 21:23:43.626336 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:23:43.626302    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eddaa0ae-1af7-4c61-a660-d805971e6d43-istio-data\") pod \"e2e-mt-c5876a31-openshift-default-56c95b9584-nt787\" (UID: \"eddaa0ae-1af7-4c61-a660-d805971e6d43\") " pod="openshift-ingress/e2e-mt-c5876a31-openshift-default-56c95b9584-nt787"
Jul 27 21:23:43.727800 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:23:43.727698    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eddaa0ae-1af7-4c61-a660-d805971e6d43-credential-socket\") pod \"e2e-mt-c5876a31-openshift-default-56c95b9584-nt787\" (UID: \"eddaa0ae-1af7-4c61-a660-d805971e6d43\") " pod="openshift-ingress/e2e-mt-c5876a31-openshift-default-56c95b9584-nt787"
Jul 27 21:23:43.727800 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:23:43.727746    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eddaa0ae-1af7-4c61-a660-d805971e6d43-istio-envoy\") pod \"e2e-mt-c5876a31-openshift-default-56c95b9584-nt787\" (UID: \"eddaa0ae-1af7-4c61-a660-d805971e6d43\") " pod="openshift-ingress/e2e-mt-c5876a31-openshift-default-56c95b9584-nt787"
Jul 27 21:23:43.727800 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:23:43.727764    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eddaa0ae-1af7-4c61-a660-d805971e6d43-workload-certs\") pod \"e2e-mt-c5876a31-openshift-default-56c95b9584-nt787\" (UID: \"eddaa0ae-1af7-4c61-a660-d805971e6d43\") " pod="openshift-ingress/e2e-mt-c5876a31-openshift-default-56c95b9584-nt787"
Jul 27 21:23:43.728236 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:23:43.727881    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eddaa0ae-1af7-4c61-a660-d805971e6d43-istiod-ca-cert\") pod \"e2e-mt-c5876a31-openshift-default-56c95b9584-nt787\" (UID: \"eddaa0ae-1af7-4c61-a660-d805971e6d43\") " pod="openshift-ingress/e2e-mt-c5876a31-openshift-default-56c95b9584-nt787"
Jul 27 21:23:43.728236 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:23:43.727926    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eddaa0ae-1af7-4c61-a660-d805971e6d43-istio-podinfo\") pod \"e2e-mt-c5876a31-openshift-default-56c95b9584-nt787\" (UID: \"eddaa0ae-1af7-4c61-a660-d805971e6d43\") " pod="openshift-ingress/e2e-mt-c5876a31-openshift-default-56c95b9584-nt787"
Jul 27 21:23:43.728236 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:23:43.727978    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eddaa0ae-1af7-4c61-a660-d805971e6d43-istio-token\") pod \"e2e-mt-c5876a31-openshift-default-56c95b9584-nt787\" (UID: \"eddaa0ae-1af7-4c61-a660-d805971e6d43\") " pod="openshift-ingress/e2e-mt-c5876a31-openshift-default-56c95b9584-nt787"
Jul 27 21:23:43.728236 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:23:43.728005    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grfgb\" (UniqueName: \"kubernetes.io/projected/eddaa0ae-1af7-4c61-a660-d805971e6d43-kube-api-access-grfgb\") pod \"e2e-mt-c5876a31-openshift-default-56c95b9584-nt787\" (UID: \"eddaa0ae-1af7-4c61-a660-d805971e6d43\") " pod="openshift-ingress/e2e-mt-c5876a31-openshift-default-56c95b9584-nt787"
Jul 27 21:23:43.728236 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:23:43.728118    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eddaa0ae-1af7-4c61-a660-d805971e6d43-workload-socket\") pod \"e2e-mt-c5876a31-openshift-default-56c95b9584-nt787\" (UID: \"eddaa0ae-1af7-4c61-a660-d805971e6d43\") " pod="openshift-ingress/e2e-mt-c5876a31-openshift-default-56c95b9584-nt787"
Jul 27 21:23:43.728236 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:23:43.728148    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eddaa0ae-1af7-4c61-a660-d805971e6d43-istio-data\") pod \"e2e-mt-c5876a31-openshift-default-56c95b9584-nt787\" (UID: \"eddaa0ae-1af7-4c61-a660-d805971e6d43\") " pod="openshift-ingress/e2e-mt-c5876a31-openshift-default-56c95b9584-nt787"
Jul 27 21:23:43.728554 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:23:43.728267    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eddaa0ae-1af7-4c61-a660-d805971e6d43-credential-socket\") pod \"e2e-mt-c5876a31-openshift-default-56c95b9584-nt787\" (UID: \"eddaa0ae-1af7-4c61-a660-d805971e6d43\") " pod="openshift-ingress/e2e-mt-c5876a31-openshift-default-56c95b9584-nt787"
Jul 27 21:23:43.728554 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:23:43.728435    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eddaa0ae-1af7-4c61-a660-d805971e6d43-workload-certs\") pod \"e2e-mt-c5876a31-openshift-default-56c95b9584-nt787\" (UID: \"eddaa0ae-1af7-4c61-a660-d805971e6d43\") " pod="openshift-ingress/e2e-mt-c5876a31-openshift-default-56c95b9584-nt787"
Jul 27 21:23:43.728662 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:23:43.728589    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eddaa0ae-1af7-4c61-a660-d805971e6d43-istio-data\") pod \"e2e-mt-c5876a31-openshift-default-56c95b9584-nt787\" (UID: \"eddaa0ae-1af7-4c61-a660-d805971e6d43\") " pod="openshift-ingress/e2e-mt-c5876a31-openshift-default-56c95b9584-nt787"
Jul 27 21:23:43.728662 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:23:43.728594    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eddaa0ae-1af7-4c61-a660-d805971e6d43-workload-socket\") pod \"e2e-mt-c5876a31-openshift-default-56c95b9584-nt787\" (UID: \"eddaa0ae-1af7-4c61-a660-d805971e6d43\") " pod="openshift-ingress/e2e-mt-c5876a31-openshift-default-56c95b9584-nt787"
Jul 27 21:23:43.728942 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:23:43.728919    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eddaa0ae-1af7-4c61-a660-d805971e6d43-istiod-ca-cert\") pod \"e2e-mt-c5876a31-openshift-default-56c95b9584-nt787\" (UID: \"eddaa0ae-1af7-4c61-a660-d805971e6d43\") " pod="openshift-ingress/e2e-mt-c5876a31-openshift-default-56c95b9584-nt787"
Jul 27 21:23:43.730217 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:23:43.730192    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eddaa0ae-1af7-4c61-a660-d805971e6d43-istio-envoy\") pod \"e2e-mt-c5876a31-openshift-default-56c95b9584-nt787\" (UID: \"eddaa0ae-1af7-4c61-a660-d805971e6d43\") " pod="openshift-ingress/e2e-mt-c5876a31-openshift-default-56c95b9584-nt787"
Jul 27 21:23:43.730536 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:23:43.730516    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eddaa0ae-1af7-4c61-a660-d805971e6d43-istio-podinfo\") pod \"e2e-mt-c5876a31-openshift-default-56c95b9584-nt787\" (UID: \"eddaa0ae-1af7-4c61-a660-d805971e6d43\") " pod="openshift-ingress/e2e-mt-c5876a31-openshift-default-56c95b9584-nt787"
Jul 27 21:23:43.736641 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:23:43.736620    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eddaa0ae-1af7-4c61-a660-d805971e6d43-istio-token\") pod \"e2e-mt-c5876a31-openshift-default-56c95b9584-nt787\" (UID: \"eddaa0ae-1af7-4c61-a660-d805971e6d43\") " pod="openshift-ingress/e2e-mt-c5876a31-openshift-default-56c95b9584-nt787"
Jul 27 21:23:43.737158 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:23:43.737135    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-grfgb\" (UniqueName: \"kubernetes.io/projected/eddaa0ae-1af7-4c61-a660-d805971e6d43-kube-api-access-grfgb\") pod \"e2e-mt-c5876a31-openshift-default-56c95b9584-nt787\" (UID: \"eddaa0ae-1af7-4c61-a660-d805971e6d43\") " pod="openshift-ingress/e2e-mt-c5876a31-openshift-default-56c95b9584-nt787"
Jul 27 21:23:43.779732 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:23:43.779691    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c5876a31-openshift-default-56c95b9584-nt787"
Jul 27 21:23:43.925655 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:23:43.925606    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c5876a31-openshift-default-56c95b9584-nt787"]
Jul 27 21:23:43.928594 ip-10-0-141-64 kubenswrapper[2560]: W0727 21:23:43.928563    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeddaa0ae_1af7_4c61_a660_d805971e6d43.slice/crio-44379f70995515c44a55f45ea66d934845b8ce370bed8397bf43ae4bf917d901 WatchSource:0}: Error finding container 44379f70995515c44a55f45ea66d934845b8ce370bed8397bf43ae4bf917d901: Status 404 returned error can't find the container with id 44379f70995515c44a55f45ea66d934845b8ce370bed8397bf43ae4bf917d901
Jul 27 21:23:43.930726 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:23:43.930693    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 27 21:23:43.930832 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:23:43.930758    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 27 21:23:43.930832 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:23:43.930787    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 27 21:23:44.400632 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:23:44.400596    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c5876a31-openshift-default-56c95b9584-nt787" event={"ID":"eddaa0ae-1af7-4c61-a660-d805971e6d43","Type":"ContainerStarted","Data":"034bc212770a13e425f308ada6cfbdfdd54d3a5f87407a7be555272219656c82"}
Jul 27 21:23:44.400838 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:23:44.400653    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c5876a31-openshift-default-56c95b9584-nt787" event={"ID":"eddaa0ae-1af7-4c61-a660-d805971e6d43","Type":"ContainerStarted","Data":"44379f70995515c44a55f45ea66d934845b8ce370bed8397bf43ae4bf917d901"}
Jul 27 21:23:44.422460 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:23:44.422405    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-c5876a31-openshift-default-56c95b9584-nt787" podStartSLOduration=1.422389437 podStartE2EDuration="1.422389437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:23:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:23:44.420786877 +0000 UTC m=+1941.172648683" watchObservedRunningTime="2026-07-27 21:23:44.422389437 +0000 UTC m=+1941.174251232"
Jul 27 21:23:44.780188 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:23:44.780156    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-c5876a31-openshift-default-56c95b9584-nt787"
Jul 27 21:23:44.784841 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:23:44.784818    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-c5876a31-openshift-default-56c95b9584-nt787"
Jul 27 21:23:45.405932 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:23:45.405903    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c5876a31-openshift-default-56c95b9584-nt787"
Jul 27 21:23:45.406823 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:23:45.406803    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-c5876a31-openshift-default-56c95b9584-nt787"
Jul 27 21:23:54.890254 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:23:54.890210    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:23:56.464669 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:23:56.464633    2560 generic.go:356] "Generic (PLEG): container finished" podID="eddaa0ae-1af7-4c61-a660-d805971e6d43" containerID="034bc212770a13e425f308ada6cfbdfdd54d3a5f87407a7be555272219656c82" exitCode=0
Jul 27 21:23:56.465237 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:23:56.464712    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c5876a31-openshift-default-56c95b9584-nt787" event={"ID":"eddaa0ae-1af7-4c61-a660-d805971e6d43","Type":"ContainerDied","Data":"034bc212770a13e425f308ada6cfbdfdd54d3a5f87407a7be555272219656c82"}
Jul 27 21:23:56.465237 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:23:56.465219    2560 scope.go:117] "RemoveContainer" containerID="034bc212770a13e425f308ada6cfbdfdd54d3a5f87407a7be555272219656c82"
Jul 27 21:23:56.466289 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:23:56.466256    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 27 21:23:56.466405 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:23:56.466332    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 27 21:23:56.466405 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:23:56.466366    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 27 21:23:57.471697 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:23:57.471662    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c5876a31-openshift-default-56c95b9584-nt787" event={"ID":"eddaa0ae-1af7-4c61-a660-d805971e6d43","Type":"ContainerStarted","Data":"f7c1f9a53fc61da7947b124de8276b69545ea3692f2fe9c0232ff4f58aa0c5de"}
Jul 27 21:23:57.780959 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:23:57.780866    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-c5876a31-openshift-default-56c95b9584-nt787"
Jul 27 21:23:57.782465 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:23:57.782080    2560 patch_prober.go:28] interesting pod/e2e-mt-c5876a31-openshift-default-56c95b9584-nt787 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Jul 27 21:23:57.782465 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:23:57.782174    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c5876a31-openshift-default-56c95b9584-nt787" podUID="eddaa0ae-1af7-4c61-a660-d805971e6d43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Jul 27 21:23:58.780200 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:23:58.780156    2560 patch_prober.go:28] interesting pod/e2e-mt-c5876a31-openshift-default-56c95b9584-nt787 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Jul 27 21:23:58.780702 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:23:58.780226    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c5876a31-openshift-default-56c95b9584-nt787" podUID="eddaa0ae-1af7-4c61-a660-d805971e6d43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Jul 27 21:23:59.780681 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:23:59.780640    2560 patch_prober.go:28] interesting pod/e2e-mt-c5876a31-openshift-default-56c95b9584-nt787 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Jul 27 21:23:59.781130 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:23:59.780702    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c5876a31-openshift-default-56c95b9584-nt787" podUID="eddaa0ae-1af7-4c61-a660-d805971e6d43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Jul 27 21:24:00.406025 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:24:00.405986    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c5876a31-openshift-default-56c95b9584-nt787"
Jul 27 21:24:00.780688 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:24:00.780658    2560 patch_prober.go:28] interesting pod/e2e-mt-c5876a31-openshift-default-56c95b9584-nt787 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Jul 27 21:24:00.780848 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:24:00.780714    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c5876a31-openshift-default-56c95b9584-nt787" podUID="eddaa0ae-1af7-4c61-a660-d805971e6d43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Jul 27 21:24:01.780789 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:24:01.780746    2560 patch_prober.go:28] interesting pod/e2e-mt-c5876a31-openshift-default-56c95b9584-nt787 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Jul 27 21:24:01.780997 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:24:01.780815    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c5876a31-openshift-default-56c95b9584-nt787" podUID="eddaa0ae-1af7-4c61-a660-d805971e6d43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Jul 27 21:24:02.784392 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:24:02.784356    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-c5876a31-openshift-default-56c95b9584-nt787"
Jul 27 21:24:02.785401 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:24:02.785382    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-c5876a31-openshift-default-56c95b9584-nt787"
Jul 27 21:24:11.691400 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:24:11.691360    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:24:19.711482 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:24:19.711402    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c5876a31-openshift-default-56c95b9584-nt787"]
Jul 27 21:24:19.711959 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:24:19.711648    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-c5876a31-openshift-default-56c95b9584-nt787" podUID="eddaa0ae-1af7-4c61-a660-d805971e6d43" containerName="istio-proxy" containerID="cri-o://f7c1f9a53fc61da7947b124de8276b69545ea3692f2fe9c0232ff4f58aa0c5de" gracePeriod=30
Jul 27 21:24:24.969252 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:24:24.969227    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c5876a31-openshift-default-56c95b9584-nt787"
Jul 27 21:24:25.097324 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:24:25.097290    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eddaa0ae-1af7-4c61-a660-d805971e6d43-istiod-ca-cert\") pod \"eddaa0ae-1af7-4c61-a660-d805971e6d43\" (UID: \"eddaa0ae-1af7-4c61-a660-d805971e6d43\") "
Jul 27 21:24:25.097324 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:24:25.097323    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-grfgb\" (UniqueName: \"kubernetes.io/projected/eddaa0ae-1af7-4c61-a660-d805971e6d43-kube-api-access-grfgb\") pod \"eddaa0ae-1af7-4c61-a660-d805971e6d43\" (UID: \"eddaa0ae-1af7-4c61-a660-d805971e6d43\") "
Jul 27 21:24:25.097577 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:24:25.097387    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eddaa0ae-1af7-4c61-a660-d805971e6d43-workload-socket\") pod \"eddaa0ae-1af7-4c61-a660-d805971e6d43\" (UID: \"eddaa0ae-1af7-4c61-a660-d805971e6d43\") "
Jul 27 21:24:25.097577 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:24:25.097519    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eddaa0ae-1af7-4c61-a660-d805971e6d43-istio-token\") pod \"eddaa0ae-1af7-4c61-a660-d805971e6d43\" (UID: \"eddaa0ae-1af7-4c61-a660-d805971e6d43\") "
Jul 27 21:24:25.097577 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:24:25.097570    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eddaa0ae-1af7-4c61-a660-d805971e6d43-istio-podinfo\") pod \"eddaa0ae-1af7-4c61-a660-d805971e6d43\" (UID: \"eddaa0ae-1af7-4c61-a660-d805971e6d43\") "
Jul 27 21:24:25.097712 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:24:25.097668    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eddaa0ae-1af7-4c61-a660-d805971e6d43-istio-data\") pod \"eddaa0ae-1af7-4c61-a660-d805971e6d43\" (UID: \"eddaa0ae-1af7-4c61-a660-d805971e6d43\") "
Jul 27 21:24:25.097749 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:24:25.097711    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eddaa0ae-1af7-4c61-a660-d805971e6d43-credential-socket\") pod \"eddaa0ae-1af7-4c61-a660-d805971e6d43\" (UID: \"eddaa0ae-1af7-4c61-a660-d805971e6d43\") "
Jul 27 21:24:25.097749 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:24:25.097667    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eddaa0ae-1af7-4c61-a660-d805971e6d43-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "eddaa0ae-1af7-4c61-a660-d805971e6d43" (UID: "eddaa0ae-1af7-4c61-a660-d805971e6d43"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:24:25.097856 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:24:25.097719    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eddaa0ae-1af7-4c61-a660-d805971e6d43-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "eddaa0ae-1af7-4c61-a660-d805971e6d43" (UID: "eddaa0ae-1af7-4c61-a660-d805971e6d43"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:24:25.097856 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:24:25.097771    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eddaa0ae-1af7-4c61-a660-d805971e6d43-istio-envoy\") pod \"eddaa0ae-1af7-4c61-a660-d805971e6d43\" (UID: \"eddaa0ae-1af7-4c61-a660-d805971e6d43\") "
Jul 27 21:24:25.097856 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:24:25.097805    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eddaa0ae-1af7-4c61-a660-d805971e6d43-workload-certs\") pod \"eddaa0ae-1af7-4c61-a660-d805971e6d43\" (UID: \"eddaa0ae-1af7-4c61-a660-d805971e6d43\") "
Jul 27 21:24:25.098155 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:24:25.098060    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eddaa0ae-1af7-4c61-a660-d805971e6d43-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "eddaa0ae-1af7-4c61-a660-d805971e6d43" (UID: "eddaa0ae-1af7-4c61-a660-d805971e6d43"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:24:25.098155 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:24:25.098075    2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eddaa0ae-1af7-4c61-a660-d805971e6d43-workload-socket\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:24:25.098155 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:24:25.098113    2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eddaa0ae-1af7-4c61-a660-d805971e6d43-istiod-ca-cert\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:24:25.098382 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:24:25.098297    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eddaa0ae-1af7-4c61-a660-d805971e6d43-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "eddaa0ae-1af7-4c61-a660-d805971e6d43" (UID: "eddaa0ae-1af7-4c61-a660-d805971e6d43"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:24:25.098537 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:24:25.098513    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eddaa0ae-1af7-4c61-a660-d805971e6d43-istio-data" (OuterVolumeSpecName: "istio-data") pod "eddaa0ae-1af7-4c61-a660-d805971e6d43" (UID: "eddaa0ae-1af7-4c61-a660-d805971e6d43"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:24:25.099635 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:24:25.099611    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eddaa0ae-1af7-4c61-a660-d805971e6d43-kube-api-access-grfgb" (OuterVolumeSpecName: "kube-api-access-grfgb") pod "eddaa0ae-1af7-4c61-a660-d805971e6d43" (UID: "eddaa0ae-1af7-4c61-a660-d805971e6d43"). InnerVolumeSpecName "kube-api-access-grfgb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:24:25.099901 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:24:25.099883    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eddaa0ae-1af7-4c61-a660-d805971e6d43-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "eddaa0ae-1af7-4c61-a660-d805971e6d43" (UID: "eddaa0ae-1af7-4c61-a660-d805971e6d43"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:24:25.100249 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:24:25.100230    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eddaa0ae-1af7-4c61-a660-d805971e6d43-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "eddaa0ae-1af7-4c61-a660-d805971e6d43" (UID: "eddaa0ae-1af7-4c61-a660-d805971e6d43"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 27 21:24:25.100249 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:24:25.100236    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eddaa0ae-1af7-4c61-a660-d805971e6d43-istio-token" (OuterVolumeSpecName: "istio-token") pod "eddaa0ae-1af7-4c61-a660-d805971e6d43" (UID: "eddaa0ae-1af7-4c61-a660-d805971e6d43"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:24:25.199076 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:24:25.199040    2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eddaa0ae-1af7-4c61-a660-d805971e6d43-istio-podinfo\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:24:25.199076 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:24:25.199068    2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eddaa0ae-1af7-4c61-a660-d805971e6d43-istio-data\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:24:25.199076 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:24:25.199077    2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eddaa0ae-1af7-4c61-a660-d805971e6d43-credential-socket\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:24:25.199076 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:24:25.199110    2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eddaa0ae-1af7-4c61-a660-d805971e6d43-istio-envoy\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:24:25.199076 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:24:25.199120    2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eddaa0ae-1af7-4c61-a660-d805971e6d43-workload-certs\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:24:25.199405 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:24:25.199128    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grfgb\" (UniqueName: \"kubernetes.io/projected/eddaa0ae-1af7-4c61-a660-d805971e6d43-kube-api-access-grfgb\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:24:25.199405 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:24:25.199136    2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eddaa0ae-1af7-4c61-a660-d805971e6d43-istio-token\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:24:25.611047 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:24:25.611008    2560 generic.go:356] "Generic (PLEG): container finished" podID="eddaa0ae-1af7-4c61-a660-d805971e6d43" containerID="f7c1f9a53fc61da7947b124de8276b69545ea3692f2fe9c0232ff4f58aa0c5de" exitCode=0
Jul 27 21:24:25.611291 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:24:25.611065    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c5876a31-openshift-default-56c95b9584-nt787"
Jul 27 21:24:25.611291 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:24:25.611127    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c5876a31-openshift-default-56c95b9584-nt787" event={"ID":"eddaa0ae-1af7-4c61-a660-d805971e6d43","Type":"ContainerDied","Data":"f7c1f9a53fc61da7947b124de8276b69545ea3692f2fe9c0232ff4f58aa0c5de"}
Jul 27 21:24:25.611291 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:24:25.611194    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c5876a31-openshift-default-56c95b9584-nt787" event={"ID":"eddaa0ae-1af7-4c61-a660-d805971e6d43","Type":"ContainerDied","Data":"44379f70995515c44a55f45ea66d934845b8ce370bed8397bf43ae4bf917d901"}
Jul 27 21:24:25.611291 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:24:25.611226    2560 scope.go:117] "RemoveContainer" containerID="f7c1f9a53fc61da7947b124de8276b69545ea3692f2fe9c0232ff4f58aa0c5de"
Jul 27 21:24:25.620572 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:24:25.620554    2560 scope.go:117] "RemoveContainer" containerID="034bc212770a13e425f308ada6cfbdfdd54d3a5f87407a7be555272219656c82"
Jul 27 21:24:25.629336 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:24:25.629314    2560 scope.go:117] "RemoveContainer" containerID="f7c1f9a53fc61da7947b124de8276b69545ea3692f2fe9c0232ff4f58aa0c5de"
Jul 27 21:24:25.629720 ip-10-0-141-64 kubenswrapper[2560]: E0727 21:24:25.629675    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7c1f9a53fc61da7947b124de8276b69545ea3692f2fe9c0232ff4f58aa0c5de\": container with ID starting with f7c1f9a53fc61da7947b124de8276b69545ea3692f2fe9c0232ff4f58aa0c5de not found: ID does not exist" containerID="f7c1f9a53fc61da7947b124de8276b69545ea3692f2fe9c0232ff4f58aa0c5de"
Jul 27 21:24:25.629720 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:24:25.629708    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7c1f9a53fc61da7947b124de8276b69545ea3692f2fe9c0232ff4f58aa0c5de"} err="failed to get container status \"f7c1f9a53fc61da7947b124de8276b69545ea3692f2fe9c0232ff4f58aa0c5de\": rpc error: code = NotFound desc = could not find container \"f7c1f9a53fc61da7947b124de8276b69545ea3692f2fe9c0232ff4f58aa0c5de\": container with ID starting with f7c1f9a53fc61da7947b124de8276b69545ea3692f2fe9c0232ff4f58aa0c5de not found: ID does not exist"
Jul 27 21:24:25.629906 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:24:25.629726    2560 scope.go:117] "RemoveContainer" containerID="034bc212770a13e425f308ada6cfbdfdd54d3a5f87407a7be555272219656c82"
Jul 27 21:24:25.630011 ip-10-0-141-64 kubenswrapper[2560]: E0727 21:24:25.629989    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"034bc212770a13e425f308ada6cfbdfdd54d3a5f87407a7be555272219656c82\": container with ID starting with 034bc212770a13e425f308ada6cfbdfdd54d3a5f87407a7be555272219656c82 not found: ID does not exist" containerID="034bc212770a13e425f308ada6cfbdfdd54d3a5f87407a7be555272219656c82"
Jul 27 21:24:25.630113 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:24:25.630019    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"034bc212770a13e425f308ada6cfbdfdd54d3a5f87407a7be555272219656c82"} err="failed to get container status \"034bc212770a13e425f308ada6cfbdfdd54d3a5f87407a7be555272219656c82\": rpc error: code = NotFound desc = could not find container \"034bc212770a13e425f308ada6cfbdfdd54d3a5f87407a7be555272219656c82\": container with ID starting with 034bc212770a13e425f308ada6cfbdfdd54d3a5f87407a7be555272219656c82 not found: ID does not exist"
Jul 27 21:24:25.632298 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:24:25.632270    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c5876a31-openshift-default-56c95b9584-nt787"]
Jul 27 21:24:25.636244 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:24:25.636218    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-c5876a31-openshift-default-56c95b9584-nt787"]
Jul 27 21:24:25.848879 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:24:25.848840    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eddaa0ae-1af7-4c61-a660-d805971e6d43" path="/var/lib/kubelet/pods/eddaa0ae-1af7-4c61-a660-d805971e6d43/volumes"
Jul 27 21:24:39.165489 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:24:39.165451    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-3499910e-openshift-default-8555448976-qz77j"]
Jul 27 21:24:39.165974 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:24:39.165954    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eddaa0ae-1af7-4c61-a660-d805971e6d43" containerName="istio-proxy"
Jul 27 21:24:39.166066 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:24:39.165976    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="eddaa0ae-1af7-4c61-a660-d805971e6d43" containerName="istio-proxy"
Jul 27 21:24:39.166066 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:24:39.165989    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eddaa0ae-1af7-4c61-a660-d805971e6d43" containerName="istio-proxy"
Jul 27 21:24:39.166066 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:24:39.165998    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="eddaa0ae-1af7-4c61-a660-d805971e6d43" containerName="istio-proxy"
Jul 27 21:24:39.166066 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:24:39.166039    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="456e5d77-7483-438e-b365-9f8753d5d441" containerName="istio-proxy"
Jul 27 21:24:39.166066 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:24:39.166048    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="456e5d77-7483-438e-b365-9f8753d5d441" containerName="istio-proxy"
Jul 27 21:24:39.166338 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:24:39.166159    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="eddaa0ae-1af7-4c61-a660-d805971e6d43" containerName="istio-proxy"
Jul 27 21:24:39.166338 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:24:39.166171    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="456e5d77-7483-438e-b365-9f8753d5d441" containerName="istio-proxy"
Jul 27 21:24:39.170591 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:24:39.170572    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3499910e-openshift-default-8555448976-qz77j"
Jul 27 21:24:39.173545 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:24:39.173524    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-3499910e-openshift-default-dockercfg-zb4fb\""
Jul 27 21:24:39.180829 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:24:39.180803    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-3499910e-openshift-default-8555448976-qz77j"]
Jul 27 21:24:39.320159 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:24:39.320114    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2a9322c-2637-44f1-b0c0-3b651487ad13-istiod-ca-cert\") pod \"e2e-mt-3499910e-openshift-default-8555448976-qz77j\" (UID: \"e2a9322c-2637-44f1-b0c0-3b651487ad13\") " pod="openshift-ingress/e2e-mt-3499910e-openshift-default-8555448976-qz77j"
Jul 27 21:24:39.320159 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:24:39.320166    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2a9322c-2637-44f1-b0c0-3b651487ad13-workload-socket\") pod \"e2e-mt-3499910e-openshift-default-8555448976-qz77j\" (UID: \"e2a9322c-2637-44f1-b0c0-3b651487ad13\") " pod="openshift-ingress/e2e-mt-3499910e-openshift-default-8555448976-qz77j"
Jul 27 21:24:39.320367 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:24:39.320184    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2a9322c-2637-44f1-b0c0-3b651487ad13-istio-token\") pod \"e2e-mt-3499910e-openshift-default-8555448976-qz77j\" (UID: \"e2a9322c-2637-44f1-b0c0-3b651487ad13\") " pod="openshift-ingress/e2e-mt-3499910e-openshift-default-8555448976-qz77j"
Jul 27 21:24:39.320367 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:24:39.320281    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2a9322c-2637-44f1-b0c0-3b651487ad13-credential-socket\") pod \"e2e-mt-3499910e-openshift-default-8555448976-qz77j\" (UID: \"e2a9322c-2637-44f1-b0c0-3b651487ad13\") " pod="openshift-ingress/e2e-mt-3499910e-openshift-default-8555448976-qz77j"
Jul 27 21:24:39.320367 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:24:39.320313    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npgkp\" (UniqueName: \"kubernetes.io/projected/e2a9322c-2637-44f1-b0c0-3b651487ad13-kube-api-access-npgkp\") pod \"e2e-mt-3499910e-openshift-default-8555448976-qz77j\" (UID: \"e2a9322c-2637-44f1-b0c0-3b651487ad13\") " pod="openshift-ingress/e2e-mt-3499910e-openshift-default-8555448976-qz77j"
Jul 27 21:24:39.320367 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:24:39.320332    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2a9322c-2637-44f1-b0c0-3b651487ad13-istio-podinfo\") pod \"e2e-mt-3499910e-openshift-default-8555448976-qz77j\" (UID: \"e2a9322c-2637-44f1-b0c0-3b651487ad13\") " pod="openshift-ingress/e2e-mt-3499910e-openshift-default-8555448976-qz77j"
Jul 27 21:24:39.320367 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:24:39.320352    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2a9322c-2637-44f1-b0c0-3b651487ad13-istio-envoy\") pod \"e2e-mt-3499910e-openshift-default-8555448976-qz77j\" (UID: \"e2a9322c-2637-44f1-b0c0-3b651487ad13\") " pod="openshift-ingress/e2e-mt-3499910e-openshift-default-8555448976-qz77j"
Jul 27 21:24:39.320367 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:24:39.320367    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2a9322c-2637-44f1-b0c0-3b651487ad13-istio-data\") pod \"e2e-mt-3499910e-openshift-default-8555448976-qz77j\" (UID: \"e2a9322c-2637-44f1-b0c0-3b651487ad13\") " pod="openshift-ingress/e2e-mt-3499910e-openshift-default-8555448976-qz77j"
Jul 27 21:24:39.320638 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:24:39.320482    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2a9322c-2637-44f1-b0c0-3b651487ad13-workload-certs\") pod \"e2e-mt-3499910e-openshift-default-8555448976-qz77j\" (UID: \"e2a9322c-2637-44f1-b0c0-3b651487ad13\") " pod="openshift-ingress/e2e-mt-3499910e-openshift-default-8555448976-qz77j"
Jul 27 21:24:39.421789 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:24:39.421686    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2a9322c-2637-44f1-b0c0-3b651487ad13-workload-certs\") pod \"e2e-mt-3499910e-openshift-default-8555448976-qz77j\" (UID: \"e2a9322c-2637-44f1-b0c0-3b651487ad13\") " pod="openshift-ingress/e2e-mt-3499910e-openshift-default-8555448976-qz77j"
Jul 27 21:24:39.421789 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:24:39.421741    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2a9322c-2637-44f1-b0c0-3b651487ad13-istiod-ca-cert\") pod \"e2e-mt-3499910e-openshift-default-8555448976-qz77j\" (UID: \"e2a9322c-2637-44f1-b0c0-3b651487ad13\") " pod="openshift-ingress/e2e-mt-3499910e-openshift-default-8555448976-qz77j"
Jul 27 21:24:39.421789 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:24:39.421766    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2a9322c-2637-44f1-b0c0-3b651487ad13-workload-socket\") pod \"e2e-mt-3499910e-openshift-default-8555448976-qz77j\" (UID: \"e2a9322c-2637-44f1-b0c0-3b651487ad13\") " pod="openshift-ingress/e2e-mt-3499910e-openshift-default-8555448976-qz77j"
Jul 27 21:24:39.421789 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:24:39.421785    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2a9322c-2637-44f1-b0c0-3b651487ad13-istio-token\") pod \"e2e-mt-3499910e-openshift-default-8555448976-qz77j\" (UID: \"e2a9322c-2637-44f1-b0c0-3b651487ad13\") " pod="openshift-ingress/e2e-mt-3499910e-openshift-default-8555448976-qz77j"
Jul 27 21:24:39.422177 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:24:39.421815    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2a9322c-2637-44f1-b0c0-3b651487ad13-credential-socket\") pod \"e2e-mt-3499910e-openshift-default-8555448976-qz77j\" (UID: \"e2a9322c-2637-44f1-b0c0-3b651487ad13\") " pod="openshift-ingress/e2e-mt-3499910e-openshift-default-8555448976-qz77j"
Jul 27 21:24:39.422177 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:24:39.421832    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npgkp\" (UniqueName: \"kubernetes.io/projected/e2a9322c-2637-44f1-b0c0-3b651487ad13-kube-api-access-npgkp\") pod \"e2e-mt-3499910e-openshift-default-8555448976-qz77j\" (UID: \"e2a9322c-2637-44f1-b0c0-3b651487ad13\") " pod="openshift-ingress/e2e-mt-3499910e-openshift-default-8555448976-qz77j"
Jul 27 21:24:39.422177 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:24:39.421849    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2a9322c-2637-44f1-b0c0-3b651487ad13-istio-podinfo\") pod \"e2e-mt-3499910e-openshift-default-8555448976-qz77j\" (UID: \"e2a9322c-2637-44f1-b0c0-3b651487ad13\") " pod="openshift-ingress/e2e-mt-3499910e-openshift-default-8555448976-qz77j"
Jul 27 21:24:39.422177 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:24:39.421910    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2a9322c-2637-44f1-b0c0-3b651487ad13-istio-envoy\") pod \"e2e-mt-3499910e-openshift-default-8555448976-qz77j\" (UID: \"e2a9322c-2637-44f1-b0c0-3b651487ad13\") " pod="openshift-ingress/e2e-mt-3499910e-openshift-default-8555448976-qz77j"
Jul 27 21:24:39.422177 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:24:39.421944    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2a9322c-2637-44f1-b0c0-3b651487ad13-istio-data\") pod \"e2e-mt-3499910e-openshift-default-8555448976-qz77j\" (UID: \"e2a9322c-2637-44f1-b0c0-3b651487ad13\") " pod="openshift-ingress/e2e-mt-3499910e-openshift-default-8555448976-qz77j"
Jul 27 21:24:39.422428 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:24:39.422221    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2a9322c-2637-44f1-b0c0-3b651487ad13-workload-certs\") pod \"e2e-mt-3499910e-openshift-default-8555448976-qz77j\" (UID: \"e2a9322c-2637-44f1-b0c0-3b651487ad13\") " pod="openshift-ingress/e2e-mt-3499910e-openshift-default-8555448976-qz77j"
Jul 27 21:24:39.422428 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:24:39.422259    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2a9322c-2637-44f1-b0c0-3b651487ad13-credential-socket\") pod \"e2e-mt-3499910e-openshift-default-8555448976-qz77j\" (UID: \"e2a9322c-2637-44f1-b0c0-3b651487ad13\") " pod="openshift-ingress/e2e-mt-3499910e-openshift-default-8555448976-qz77j"
Jul 27 21:24:39.422428 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:24:39.422342    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2a9322c-2637-44f1-b0c0-3b651487ad13-istio-data\") pod \"e2e-mt-3499910e-openshift-default-8555448976-qz77j\" (UID: \"e2a9322c-2637-44f1-b0c0-3b651487ad13\") " pod="openshift-ingress/e2e-mt-3499910e-openshift-default-8555448976-qz77j"
Jul 27 21:24:39.422428 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:24:39.422423    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2a9322c-2637-44f1-b0c0-3b651487ad13-workload-socket\") pod \"e2e-mt-3499910e-openshift-default-8555448976-qz77j\" (UID: \"e2a9322c-2637-44f1-b0c0-3b651487ad13\") " pod="openshift-ingress/e2e-mt-3499910e-openshift-default-8555448976-qz77j"
Jul 27 21:24:39.422793 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:24:39.422577    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2a9322c-2637-44f1-b0c0-3b651487ad13-istiod-ca-cert\") pod \"e2e-mt-3499910e-openshift-default-8555448976-qz77j\" (UID: \"e2a9322c-2637-44f1-b0c0-3b651487ad13\") " pod="openshift-ingress/e2e-mt-3499910e-openshift-default-8555448976-qz77j"
Jul 27 21:24:39.424404 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:24:39.424378    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2a9322c-2637-44f1-b0c0-3b651487ad13-istio-envoy\") pod \"e2e-mt-3499910e-openshift-default-8555448976-qz77j\" (UID: \"e2a9322c-2637-44f1-b0c0-3b651487ad13\") " pod="openshift-ingress/e2e-mt-3499910e-openshift-default-8555448976-qz77j"
Jul 27 21:24:39.424632 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:24:39.424612    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2a9322c-2637-44f1-b0c0-3b651487ad13-istio-podinfo\") pod \"e2e-mt-3499910e-openshift-default-8555448976-qz77j\" (UID: \"e2a9322c-2637-44f1-b0c0-3b651487ad13\") " pod="openshift-ingress/e2e-mt-3499910e-openshift-default-8555448976-qz77j"
Jul 27 21:24:39.429771 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:24:39.429745    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2a9322c-2637-44f1-b0c0-3b651487ad13-istio-token\") pod \"e2e-mt-3499910e-openshift-default-8555448976-qz77j\" (UID: \"e2a9322c-2637-44f1-b0c0-3b651487ad13\") " pod="openshift-ingress/e2e-mt-3499910e-openshift-default-8555448976-qz77j"
Jul 27 21:24:39.429869 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:24:39.429793    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-npgkp\" (UniqueName: \"kubernetes.io/projected/e2a9322c-2637-44f1-b0c0-3b651487ad13-kube-api-access-npgkp\") pod \"e2e-mt-3499910e-openshift-default-8555448976-qz77j\" (UID: \"e2a9322c-2637-44f1-b0c0-3b651487ad13\") " pod="openshift-ingress/e2e-mt-3499910e-openshift-default-8555448976-qz77j"
Jul 27 21:24:39.481543 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:24:39.481509    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3499910e-openshift-default-8555448976-qz77j"
Jul 27 21:24:39.647849 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:24:39.647821    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-3499910e-openshift-default-8555448976-qz77j"]
Jul 27 21:24:39.650359 ip-10-0-141-64 kubenswrapper[2560]: W0727 21:24:39.650328    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2a9322c_2637_44f1_b0c0_3b651487ad13.slice/crio-cdbec2be29f05950141995db6c4a1bffe7363d584aca7ef2d943ba624c2f8a56 WatchSource:0}: Error finding container cdbec2be29f05950141995db6c4a1bffe7363d584aca7ef2d943ba624c2f8a56: Status 404 returned error can't find the container with id cdbec2be29f05950141995db6c4a1bffe7363d584aca7ef2d943ba624c2f8a56
Jul 27 21:24:39.652754 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:24:39.652712    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 27 21:24:39.652861 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:24:39.652795    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 27 21:24:39.652861 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:24:39.652841    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 27 21:24:39.680365 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:24:39.680301    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3499910e-openshift-default-8555448976-qz77j" event={"ID":"e2a9322c-2637-44f1-b0c0-3b651487ad13","Type":"ContainerStarted","Data":"cdbec2be29f05950141995db6c4a1bffe7363d584aca7ef2d943ba624c2f8a56"}
Jul 27 21:24:40.692674 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:24:40.692633    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3499910e-openshift-default-8555448976-qz77j" event={"ID":"e2a9322c-2637-44f1-b0c0-3b651487ad13","Type":"ContainerStarted","Data":"1e1eed41fe3001903353ee31f8e07931990bd0611f1faca067af47a28ffeb269"}
Jul 27 21:24:40.711569 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:24:40.711513    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-3499910e-openshift-default-8555448976-qz77j" podStartSLOduration=1.711498658 podStartE2EDuration="1.711498658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:24:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:24:40.71018547 +0000 UTC m=+1997.462047283" watchObservedRunningTime="2026-07-27 21:24:40.711498658 +0000 UTC m=+1997.463360453"
Jul 27 21:24:41.481960 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:24:41.481917    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-3499910e-openshift-default-8555448976-qz77j"
Jul 27 21:24:41.486741 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:24:41.486715    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-3499910e-openshift-default-8555448976-qz77j"
Jul 27 21:24:41.697020 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:24:41.696992    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-3499910e-openshift-default-8555448976-qz77j"
Jul 27 21:24:41.698059 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:24:41.698038    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-3499910e-openshift-default-8555448976-qz77j"
Jul 27 21:24:48.891426 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:24:48.891377    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:24:56.699005 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:24:56.698920    2560 patch_prober.go:28] interesting pod/e2e-mt-3499910e-openshift-default-8555448976-qz77j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Jul 27 21:24:56.699005 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:24:56.698990    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-3499910e-openshift-default-8555448976-qz77j" podUID="e2a9322c-2637-44f1-b0c0-3b651487ad13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Jul 27 21:24:56.772167 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:24:56.772138    2560 generic.go:356] "Generic (PLEG): container finished" podID="e2a9322c-2637-44f1-b0c0-3b651487ad13" containerID="1e1eed41fe3001903353ee31f8e07931990bd0611f1faca067af47a28ffeb269" exitCode=0
Jul 27 21:24:56.772321 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:24:56.772219    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3499910e-openshift-default-8555448976-qz77j" event={"ID":"e2a9322c-2637-44f1-b0c0-3b651487ad13","Type":"ContainerDied","Data":"1e1eed41fe3001903353ee31f8e07931990bd0611f1faca067af47a28ffeb269"}
Jul 27 21:24:56.772628 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:24:56.772613    2560 scope.go:117] "RemoveContainer" containerID="1e1eed41fe3001903353ee31f8e07931990bd0611f1faca067af47a28ffeb269"
Jul 27 21:24:56.773684 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:24:56.773660    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 27 21:24:56.773760 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:24:56.773722    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 27 21:24:56.773760 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:24:56.773753    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 27 21:24:57.779497 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:24:57.779460    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3499910e-openshift-default-8555448976-qz77j" event={"ID":"e2a9322c-2637-44f1-b0c0-3b651487ad13","Type":"ContainerStarted","Data":"0d13909d6956cbe70ef1a3088dae18f57ff6bda763d7e22f4c7849fd72ac4de4"}
Jul 27 21:24:58.482239 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:24:58.482200    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-3499910e-openshift-default-8555448976-qz77j"
Jul 27 21:24:58.482469 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:24:58.482439    2560 patch_prober.go:28] interesting pod/e2e-mt-3499910e-openshift-default-8555448976-qz77j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Jul 27 21:24:58.482550 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:24:58.482507    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3499910e-openshift-default-8555448976-qz77j" podUID="e2a9322c-2637-44f1-b0c0-3b651487ad13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Jul 27 21:24:59.482849 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:24:59.482809    2560 patch_prober.go:28] interesting pod/e2e-mt-3499910e-openshift-default-8555448976-qz77j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Jul 27 21:24:59.483293 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:24:59.482880    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3499910e-openshift-default-8555448976-qz77j" podUID="e2a9322c-2637-44f1-b0c0-3b651487ad13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Jul 27 21:25:00.482586 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:25:00.482545    2560 patch_prober.go:28] interesting pod/e2e-mt-3499910e-openshift-default-8555448976-qz77j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Jul 27 21:25:00.482777 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:25:00.482612    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3499910e-openshift-default-8555448976-qz77j" podUID="e2a9322c-2637-44f1-b0c0-3b651487ad13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Jul 27 21:25:00.900436 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:25:00.900395    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:25:01.482385 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:25:01.482344    2560 patch_prober.go:28] interesting pod/e2e-mt-3499910e-openshift-default-8555448976-qz77j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Jul 27 21:25:01.482577 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:25:01.482403    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3499910e-openshift-default-8555448976-qz77j" podUID="e2a9322c-2637-44f1-b0c0-3b651487ad13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Jul 27 21:25:02.486106 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:25:02.486068    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-3499910e-openshift-default-8555448976-qz77j"
Jul 27 21:25:02.486548 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:25:02.486366    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-3499910e-openshift-default-8555448976-qz77j"
Jul 27 21:25:02.486959 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:25:02.486944    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-3499910e-openshift-default-8555448976-qz77j"
Jul 27 21:25:11.798442 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:25:11.798407    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:25:30.287443 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:25:30.287408    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:25:38.795331 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:25:38.795292    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:26:23.832822 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:26:23.832792    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-z6zxv_8a6b7ae6-6c21-453d-99b9-303066d19915/console-operator/1.log"
Jul 27 21:26:23.835196 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:26:23.835172    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-z6zxv_8a6b7ae6-6c21-453d-99b9-303066d19915/console-operator/1.log"
Jul 27 21:26:23.839634 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:26:23.839614    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-95hg7_433c431a-b5bf-4734-a03f-6f0a16895ded/ovn-acl-logging/0.log"
Jul 27 21:26:23.841796 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:26:23.841779    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-95hg7_433c431a-b5bf-4734-a03f-6f0a16895ded/ovn-acl-logging/0.log"
Jul 27 21:26:23.921741 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:26:23.921714    2560 scope.go:117] "RemoveContainer" containerID="8faaf3602efa993953aba23c28319cd2bde2c8ce6e60ec5b1868f9986784c6e1"
Jul 27 21:26:27.641540 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:26:27.641501    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-3499910e-openshift-default-8555448976-qz77j"]
Jul 27 21:26:27.642040 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:26:27.641720    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-3499910e-openshift-default-8555448976-qz77j" podUID="e2a9322c-2637-44f1-b0c0-3b651487ad13" containerName="istio-proxy" containerID="cri-o://0d13909d6956cbe70ef1a3088dae18f57ff6bda763d7e22f4c7849fd72ac4de4" gracePeriod=30
Jul 27 21:26:31.396355 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:26:31.396322    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:26:32.488466 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:26:32.488426    2560 patch_prober.go:28] interesting pod/e2e-mt-3499910e-openshift-default-8555448976-qz77j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 27 21:26:32.488843 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:26:32.488498    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-3499910e-openshift-default-8555448976-qz77j" podUID="e2a9322c-2637-44f1-b0c0-3b651487ad13" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 21:26:32.888614 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:26:32.888588    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3499910e-openshift-default-8555448976-qz77j"
Jul 27 21:26:32.959784 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:26:32.959747    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2a9322c-2637-44f1-b0c0-3b651487ad13-istiod-ca-cert\") pod \"e2a9322c-2637-44f1-b0c0-3b651487ad13\" (UID: \"e2a9322c-2637-44f1-b0c0-3b651487ad13\") "
Jul 27 21:26:32.959784 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:26:32.959792    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-npgkp\" (UniqueName: \"kubernetes.io/projected/e2a9322c-2637-44f1-b0c0-3b651487ad13-kube-api-access-npgkp\") pod \"e2a9322c-2637-44f1-b0c0-3b651487ad13\" (UID: \"e2a9322c-2637-44f1-b0c0-3b651487ad13\") "
Jul 27 21:26:32.960027 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:26:32.959822    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2a9322c-2637-44f1-b0c0-3b651487ad13-istio-token\") pod \"e2a9322c-2637-44f1-b0c0-3b651487ad13\" (UID: \"e2a9322c-2637-44f1-b0c0-3b651487ad13\") "
Jul 27 21:26:32.960027 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:26:32.959862    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2a9322c-2637-44f1-b0c0-3b651487ad13-istio-podinfo\") pod \"e2a9322c-2637-44f1-b0c0-3b651487ad13\" (UID: \"e2a9322c-2637-44f1-b0c0-3b651487ad13\") "
Jul 27 21:26:32.960027 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:26:32.959928    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2a9322c-2637-44f1-b0c0-3b651487ad13-istio-envoy\") pod \"e2a9322c-2637-44f1-b0c0-3b651487ad13\" (UID: \"e2a9322c-2637-44f1-b0c0-3b651487ad13\") "
Jul 27 21:26:32.960027 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:26:32.959953    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2a9322c-2637-44f1-b0c0-3b651487ad13-workload-certs\") pod \"e2a9322c-2637-44f1-b0c0-3b651487ad13\" (UID: \"e2a9322c-2637-44f1-b0c0-3b651487ad13\") "
Jul 27 21:26:32.960027 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:26:32.960015    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2a9322c-2637-44f1-b0c0-3b651487ad13-istio-data\") pod \"e2a9322c-2637-44f1-b0c0-3b651487ad13\" (UID: \"e2a9322c-2637-44f1-b0c0-3b651487ad13\") "
Jul 27 21:26:32.960324 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:26:32.960060    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2a9322c-2637-44f1-b0c0-3b651487ad13-workload-socket\") pod \"e2a9322c-2637-44f1-b0c0-3b651487ad13\" (UID: \"e2a9322c-2637-44f1-b0c0-3b651487ad13\") "
Jul 27 21:26:32.960324 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:26:32.960115    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2a9322c-2637-44f1-b0c0-3b651487ad13-credential-socket\") pod \"e2a9322c-2637-44f1-b0c0-3b651487ad13\" (UID: \"e2a9322c-2637-44f1-b0c0-3b651487ad13\") "
Jul 27 21:26:32.960324 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:26:32.960284    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2a9322c-2637-44f1-b0c0-3b651487ad13-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e2a9322c-2637-44f1-b0c0-3b651487ad13" (UID: "e2a9322c-2637-44f1-b0c0-3b651487ad13"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:26:32.960489 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:26:32.960314    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2a9322c-2637-44f1-b0c0-3b651487ad13-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e2a9322c-2637-44f1-b0c0-3b651487ad13" (UID: "e2a9322c-2637-44f1-b0c0-3b651487ad13"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:26:32.960548 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:26:32.960508    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2a9322c-2637-44f1-b0c0-3b651487ad13-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e2a9322c-2637-44f1-b0c0-3b651487ad13" (UID: "e2a9322c-2637-44f1-b0c0-3b651487ad13"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:26:32.960786 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:26:32.960686    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2a9322c-2637-44f1-b0c0-3b651487ad13-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e2a9322c-2637-44f1-b0c0-3b651487ad13" (UID: "e2a9322c-2637-44f1-b0c0-3b651487ad13"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:26:32.960946 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:26:32.960747    2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2a9322c-2637-44f1-b0c0-3b651487ad13-istiod-ca-cert\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:26:32.960946 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:26:32.960901    2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2a9322c-2637-44f1-b0c0-3b651487ad13-workload-certs\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:26:32.961496 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:26:32.961473    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2a9322c-2637-44f1-b0c0-3b651487ad13-istio-data" (OuterVolumeSpecName: "istio-data") pod "e2a9322c-2637-44f1-b0c0-3b651487ad13" (UID: "e2a9322c-2637-44f1-b0c0-3b651487ad13"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:26:32.962435 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:26:32.962406    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e2a9322c-2637-44f1-b0c0-3b651487ad13-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e2a9322c-2637-44f1-b0c0-3b651487ad13" (UID: "e2a9322c-2637-44f1-b0c0-3b651487ad13"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 27 21:26:32.962542 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:26:32.962416    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2a9322c-2637-44f1-b0c0-3b651487ad13-kube-api-access-npgkp" (OuterVolumeSpecName: "kube-api-access-npgkp") pod "e2a9322c-2637-44f1-b0c0-3b651487ad13" (UID: "e2a9322c-2637-44f1-b0c0-3b651487ad13"). InnerVolumeSpecName "kube-api-access-npgkp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:26:32.962705 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:26:32.962686    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2a9322c-2637-44f1-b0c0-3b651487ad13-istio-token" (OuterVolumeSpecName: "istio-token") pod "e2a9322c-2637-44f1-b0c0-3b651487ad13" (UID: "e2a9322c-2637-44f1-b0c0-3b651487ad13"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:26:32.962772 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:26:32.962687    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2a9322c-2637-44f1-b0c0-3b651487ad13-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e2a9322c-2637-44f1-b0c0-3b651487ad13" (UID: "e2a9322c-2637-44f1-b0c0-3b651487ad13"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:26:33.061700 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:26:33.061666    2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2a9322c-2637-44f1-b0c0-3b651487ad13-istio-data\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:26:33.061700 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:26:33.061696    2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2a9322c-2637-44f1-b0c0-3b651487ad13-workload-socket\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:26:33.061700 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:26:33.061705    2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2a9322c-2637-44f1-b0c0-3b651487ad13-credential-socket\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:26:33.061936 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:26:33.061715    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npgkp\" (UniqueName: \"kubernetes.io/projected/e2a9322c-2637-44f1-b0c0-3b651487ad13-kube-api-access-npgkp\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:26:33.061936 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:26:33.061724    2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2a9322c-2637-44f1-b0c0-3b651487ad13-istio-token\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:26:33.061936 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:26:33.061731    2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2a9322c-2637-44f1-b0c0-3b651487ad13-istio-podinfo\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:26:33.061936 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:26:33.061739    2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2a9322c-2637-44f1-b0c0-3b651487ad13-istio-envoy\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:26:33.224042 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:26:33.224009    2560 generic.go:356] "Generic (PLEG): container finished" podID="e2a9322c-2637-44f1-b0c0-3b651487ad13" containerID="0d13909d6956cbe70ef1a3088dae18f57ff6bda763d7e22f4c7849fd72ac4de4" exitCode=0
Jul 27 21:26:33.224242 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:26:33.224062    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3499910e-openshift-default-8555448976-qz77j"
Jul 27 21:26:33.224242 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:26:33.224121    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3499910e-openshift-default-8555448976-qz77j" event={"ID":"e2a9322c-2637-44f1-b0c0-3b651487ad13","Type":"ContainerDied","Data":"0d13909d6956cbe70ef1a3088dae18f57ff6bda763d7e22f4c7849fd72ac4de4"}
Jul 27 21:26:33.224242 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:26:33.224177    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3499910e-openshift-default-8555448976-qz77j" event={"ID":"e2a9322c-2637-44f1-b0c0-3b651487ad13","Type":"ContainerDied","Data":"cdbec2be29f05950141995db6c4a1bffe7363d584aca7ef2d943ba624c2f8a56"}
Jul 27 21:26:33.224242 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:26:33.224201    2560 scope.go:117] "RemoveContainer" containerID="0d13909d6956cbe70ef1a3088dae18f57ff6bda763d7e22f4c7849fd72ac4de4"
Jul 27 21:26:33.234011 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:26:33.233994    2560 scope.go:117] "RemoveContainer" containerID="1e1eed41fe3001903353ee31f8e07931990bd0611f1faca067af47a28ffeb269"
Jul 27 21:26:33.241798 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:26:33.241779    2560 scope.go:117] "RemoveContainer" containerID="0d13909d6956cbe70ef1a3088dae18f57ff6bda763d7e22f4c7849fd72ac4de4"
Jul 27 21:26:33.242072 ip-10-0-141-64 kubenswrapper[2560]: E0727 21:26:33.242053    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d13909d6956cbe70ef1a3088dae18f57ff6bda763d7e22f4c7849fd72ac4de4\": container with ID starting with 0d13909d6956cbe70ef1a3088dae18f57ff6bda763d7e22f4c7849fd72ac4de4 not found: ID does not exist" containerID="0d13909d6956cbe70ef1a3088dae18f57ff6bda763d7e22f4c7849fd72ac4de4"
Jul 27 21:26:33.242150 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:26:33.242082    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d13909d6956cbe70ef1a3088dae18f57ff6bda763d7e22f4c7849fd72ac4de4"} err="failed to get container status \"0d13909d6956cbe70ef1a3088dae18f57ff6bda763d7e22f4c7849fd72ac4de4\": rpc error: code = NotFound desc = could not find container \"0d13909d6956cbe70ef1a3088dae18f57ff6bda763d7e22f4c7849fd72ac4de4\": container with ID starting with 0d13909d6956cbe70ef1a3088dae18f57ff6bda763d7e22f4c7849fd72ac4de4 not found: ID does not exist"
Jul 27 21:26:33.242150 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:26:33.242118    2560 scope.go:117] "RemoveContainer" containerID="1e1eed41fe3001903353ee31f8e07931990bd0611f1faca067af47a28ffeb269"
Jul 27 21:26:33.242355 ip-10-0-141-64 kubenswrapper[2560]: E0727 21:26:33.242340    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e1eed41fe3001903353ee31f8e07931990bd0611f1faca067af47a28ffeb269\": container with ID starting with 1e1eed41fe3001903353ee31f8e07931990bd0611f1faca067af47a28ffeb269 not found: ID does not exist" containerID="1e1eed41fe3001903353ee31f8e07931990bd0611f1faca067af47a28ffeb269"
Jul 27 21:26:33.242408 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:26:33.242359    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e1eed41fe3001903353ee31f8e07931990bd0611f1faca067af47a28ffeb269"} err="failed to get container status \"1e1eed41fe3001903353ee31f8e07931990bd0611f1faca067af47a28ffeb269\": rpc error: code = NotFound desc = could not find container \"1e1eed41fe3001903353ee31f8e07931990bd0611f1faca067af47a28ffeb269\": container with ID starting with 1e1eed41fe3001903353ee31f8e07931990bd0611f1faca067af47a28ffeb269 not found: ID does not exist"
Jul 27 21:26:33.248425 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:26:33.248400    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-3499910e-openshift-default-8555448976-qz77j"]
Jul 27 21:26:33.254941 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:26:33.254921    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-3499910e-openshift-default-8555448976-qz77j"]
Jul 27 21:26:33.847406 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:26:33.847373    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2a9322c-2637-44f1-b0c0-3b651487ad13" path="/var/lib/kubelet/pods/e2a9322c-2637-44f1-b0c0-3b651487ad13/volumes"
Jul 27 21:26:45.077053 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:26:45.077018    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:27:23.887324 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:23.887239    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-029d89-openshift-default-58d5f8678-kgp6f"]
Jul 27 21:27:23.887877 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:23.887733    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2a9322c-2637-44f1-b0c0-3b651487ad13" containerName="istio-proxy"
Jul 27 21:27:23.887877 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:23.887755    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2a9322c-2637-44f1-b0c0-3b651487ad13" containerName="istio-proxy"
Jul 27 21:27:23.887877 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:23.887779    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2a9322c-2637-44f1-b0c0-3b651487ad13" containerName="istio-proxy"
Jul 27 21:27:23.887877 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:23.887788    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2a9322c-2637-44f1-b0c0-3b651487ad13" containerName="istio-proxy"
Jul 27 21:27:23.887877 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:23.887866    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2a9322c-2637-44f1-b0c0-3b651487ad13" containerName="istio-proxy"
Jul 27 21:27:23.888164 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:23.887879    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2a9322c-2637-44f1-b0c0-3b651487ad13" containerName="istio-proxy"
Jul 27 21:27:23.888164 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:23.887893    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="eddaa0ae-1af7-4c61-a660-d805971e6d43" containerName="istio-proxy"
Jul 27 21:27:23.890982 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:23.890962    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-029d89-openshift-default-58d5f8678-kgp6f"
Jul 27 21:27:23.895992 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:23.895957    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-029d89-openshift-default-dockercfg-rxnpz\""
Jul 27 21:27:23.897992 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:23.897973    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c082ecb-b11c-4454-90b9-6f0f4917fbd9-workload-socket\") pod \"e2e-inf-a-029d89-openshift-default-58d5f8678-kgp6f\" (UID: \"3c082ecb-b11c-4454-90b9-6f0f4917fbd9\") " pod="openshift-ingress/e2e-inf-a-029d89-openshift-default-58d5f8678-kgp6f"
Jul 27 21:27:23.898069 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:23.898001    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c082ecb-b11c-4454-90b9-6f0f4917fbd9-istio-data\") pod \"e2e-inf-a-029d89-openshift-default-58d5f8678-kgp6f\" (UID: \"3c082ecb-b11c-4454-90b9-6f0f4917fbd9\") " pod="openshift-ingress/e2e-inf-a-029d89-openshift-default-58d5f8678-kgp6f"
Jul 27 21:27:23.898069 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:23.898030    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c082ecb-b11c-4454-90b9-6f0f4917fbd9-istio-podinfo\") pod \"e2e-inf-a-029d89-openshift-default-58d5f8678-kgp6f\" (UID: \"3c082ecb-b11c-4454-90b9-6f0f4917fbd9\") " pod="openshift-ingress/e2e-inf-a-029d89-openshift-default-58d5f8678-kgp6f"
Jul 27 21:27:23.898196 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:23.898067    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c082ecb-b11c-4454-90b9-6f0f4917fbd9-credential-socket\") pod \"e2e-inf-a-029d89-openshift-default-58d5f8678-kgp6f\" (UID: \"3c082ecb-b11c-4454-90b9-6f0f4917fbd9\") " pod="openshift-ingress/e2e-inf-a-029d89-openshift-default-58d5f8678-kgp6f"
Jul 27 21:27:23.898196 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:23.898130    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c082ecb-b11c-4454-90b9-6f0f4917fbd9-workload-certs\") pod \"e2e-inf-a-029d89-openshift-default-58d5f8678-kgp6f\" (UID: \"3c082ecb-b11c-4454-90b9-6f0f4917fbd9\") " pod="openshift-ingress/e2e-inf-a-029d89-openshift-default-58d5f8678-kgp6f"
Jul 27 21:27:23.898196 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:23.898147    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l74br\" (UniqueName: \"kubernetes.io/projected/3c082ecb-b11c-4454-90b9-6f0f4917fbd9-kube-api-access-l74br\") pod \"e2e-inf-a-029d89-openshift-default-58d5f8678-kgp6f\" (UID: \"3c082ecb-b11c-4454-90b9-6f0f4917fbd9\") " pod="openshift-ingress/e2e-inf-a-029d89-openshift-default-58d5f8678-kgp6f"
Jul 27 21:27:23.898196 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:23.898173    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c082ecb-b11c-4454-90b9-6f0f4917fbd9-istio-envoy\") pod \"e2e-inf-a-029d89-openshift-default-58d5f8678-kgp6f\" (UID: \"3c082ecb-b11c-4454-90b9-6f0f4917fbd9\") " pod="openshift-ingress/e2e-inf-a-029d89-openshift-default-58d5f8678-kgp6f"
Jul 27 21:27:23.898333 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:23.898197    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c082ecb-b11c-4454-90b9-6f0f4917fbd9-istio-token\") pod \"e2e-inf-a-029d89-openshift-default-58d5f8678-kgp6f\" (UID: \"3c082ecb-b11c-4454-90b9-6f0f4917fbd9\") " pod="openshift-ingress/e2e-inf-a-029d89-openshift-default-58d5f8678-kgp6f"
Jul 27 21:27:23.898333 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:23.898252    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c082ecb-b11c-4454-90b9-6f0f4917fbd9-istiod-ca-cert\") pod \"e2e-inf-a-029d89-openshift-default-58d5f8678-kgp6f\" (UID: \"3c082ecb-b11c-4454-90b9-6f0f4917fbd9\") " pod="openshift-ingress/e2e-inf-a-029d89-openshift-default-58d5f8678-kgp6f"
Jul 27 21:27:23.906438 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:23.906413    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-029d89-openshift-default-58d5f8678-kgp6f"]
Jul 27 21:27:23.999451 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:23.999412    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c082ecb-b11c-4454-90b9-6f0f4917fbd9-istio-token\") pod \"e2e-inf-a-029d89-openshift-default-58d5f8678-kgp6f\" (UID: \"3c082ecb-b11c-4454-90b9-6f0f4917fbd9\") " pod="openshift-ingress/e2e-inf-a-029d89-openshift-default-58d5f8678-kgp6f"
Jul 27 21:27:23.999596 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:23.999470    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c082ecb-b11c-4454-90b9-6f0f4917fbd9-istiod-ca-cert\") pod \"e2e-inf-a-029d89-openshift-default-58d5f8678-kgp6f\" (UID: \"3c082ecb-b11c-4454-90b9-6f0f4917fbd9\") " pod="openshift-ingress/e2e-inf-a-029d89-openshift-default-58d5f8678-kgp6f"
Jul 27 21:27:23.999596 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:23.999496    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c082ecb-b11c-4454-90b9-6f0f4917fbd9-workload-socket\") pod \"e2e-inf-a-029d89-openshift-default-58d5f8678-kgp6f\" (UID: \"3c082ecb-b11c-4454-90b9-6f0f4917fbd9\") " pod="openshift-ingress/e2e-inf-a-029d89-openshift-default-58d5f8678-kgp6f"
Jul 27 21:27:23.999596 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:23.999511    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c082ecb-b11c-4454-90b9-6f0f4917fbd9-istio-data\") pod \"e2e-inf-a-029d89-openshift-default-58d5f8678-kgp6f\" (UID: \"3c082ecb-b11c-4454-90b9-6f0f4917fbd9\") " pod="openshift-ingress/e2e-inf-a-029d89-openshift-default-58d5f8678-kgp6f"
Jul 27 21:27:23.999596 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:23.999538    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c082ecb-b11c-4454-90b9-6f0f4917fbd9-istio-podinfo\") pod \"e2e-inf-a-029d89-openshift-default-58d5f8678-kgp6f\" (UID: \"3c082ecb-b11c-4454-90b9-6f0f4917fbd9\") " pod="openshift-ingress/e2e-inf-a-029d89-openshift-default-58d5f8678-kgp6f"
Jul 27 21:27:23.999596 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:23.999562    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c082ecb-b11c-4454-90b9-6f0f4917fbd9-credential-socket\") pod \"e2e-inf-a-029d89-openshift-default-58d5f8678-kgp6f\" (UID: \"3c082ecb-b11c-4454-90b9-6f0f4917fbd9\") " pod="openshift-ingress/e2e-inf-a-029d89-openshift-default-58d5f8678-kgp6f"
Jul 27 21:27:23.999596 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:23.999579    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c082ecb-b11c-4454-90b9-6f0f4917fbd9-workload-certs\") pod \"e2e-inf-a-029d89-openshift-default-58d5f8678-kgp6f\" (UID: \"3c082ecb-b11c-4454-90b9-6f0f4917fbd9\") " pod="openshift-ingress/e2e-inf-a-029d89-openshift-default-58d5f8678-kgp6f"
Jul 27 21:27:23.999891 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:23.999603    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l74br\" (UniqueName: \"kubernetes.io/projected/3c082ecb-b11c-4454-90b9-6f0f4917fbd9-kube-api-access-l74br\") pod \"e2e-inf-a-029d89-openshift-default-58d5f8678-kgp6f\" (UID: \"3c082ecb-b11c-4454-90b9-6f0f4917fbd9\") " pod="openshift-ingress/e2e-inf-a-029d89-openshift-default-58d5f8678-kgp6f"
Jul 27 21:27:23.999891 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:23.999637    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c082ecb-b11c-4454-90b9-6f0f4917fbd9-istio-envoy\") pod \"e2e-inf-a-029d89-openshift-default-58d5f8678-kgp6f\" (UID: \"3c082ecb-b11c-4454-90b9-6f0f4917fbd9\") " pod="openshift-ingress/e2e-inf-a-029d89-openshift-default-58d5f8678-kgp6f"
Jul 27 21:27:24.000005 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:23.999929    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c082ecb-b11c-4454-90b9-6f0f4917fbd9-workload-socket\") pod \"e2e-inf-a-029d89-openshift-default-58d5f8678-kgp6f\" (UID: \"3c082ecb-b11c-4454-90b9-6f0f4917fbd9\") " pod="openshift-ingress/e2e-inf-a-029d89-openshift-default-58d5f8678-kgp6f"
Jul 27 21:27:24.000005 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:23.999959    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c082ecb-b11c-4454-90b9-6f0f4917fbd9-istio-data\") pod \"e2e-inf-a-029d89-openshift-default-58d5f8678-kgp6f\" (UID: \"3c082ecb-b11c-4454-90b9-6f0f4917fbd9\") " pod="openshift-ingress/e2e-inf-a-029d89-openshift-default-58d5f8678-kgp6f"
Jul 27 21:27:24.000149 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:24.000009    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c082ecb-b11c-4454-90b9-6f0f4917fbd9-credential-socket\") pod \"e2e-inf-a-029d89-openshift-default-58d5f8678-kgp6f\" (UID: \"3c082ecb-b11c-4454-90b9-6f0f4917fbd9\") " pod="openshift-ingress/e2e-inf-a-029d89-openshift-default-58d5f8678-kgp6f"
Jul 27 21:27:24.000218 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:24.000198    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c082ecb-b11c-4454-90b9-6f0f4917fbd9-workload-certs\") pod \"e2e-inf-a-029d89-openshift-default-58d5f8678-kgp6f\" (UID: \"3c082ecb-b11c-4454-90b9-6f0f4917fbd9\") " pod="openshift-ingress/e2e-inf-a-029d89-openshift-default-58d5f8678-kgp6f"
Jul 27 21:27:24.000402 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:24.000329    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c082ecb-b11c-4454-90b9-6f0f4917fbd9-istiod-ca-cert\") pod \"e2e-inf-a-029d89-openshift-default-58d5f8678-kgp6f\" (UID: \"3c082ecb-b11c-4454-90b9-6f0f4917fbd9\") " pod="openshift-ingress/e2e-inf-a-029d89-openshift-default-58d5f8678-kgp6f"
Jul 27 21:27:24.002099 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:24.002063    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c082ecb-b11c-4454-90b9-6f0f4917fbd9-istio-envoy\") pod \"e2e-inf-a-029d89-openshift-default-58d5f8678-kgp6f\" (UID: \"3c082ecb-b11c-4454-90b9-6f0f4917fbd9\") " pod="openshift-ingress/e2e-inf-a-029d89-openshift-default-58d5f8678-kgp6f"
Jul 27 21:27:24.002251 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:24.002236    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c082ecb-b11c-4454-90b9-6f0f4917fbd9-istio-podinfo\") pod \"e2e-inf-a-029d89-openshift-default-58d5f8678-kgp6f\" (UID: \"3c082ecb-b11c-4454-90b9-6f0f4917fbd9\") " pod="openshift-ingress/e2e-inf-a-029d89-openshift-default-58d5f8678-kgp6f"
Jul 27 21:27:24.007593 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:24.007572    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c082ecb-b11c-4454-90b9-6f0f4917fbd9-istio-token\") pod \"e2e-inf-a-029d89-openshift-default-58d5f8678-kgp6f\" (UID: \"3c082ecb-b11c-4454-90b9-6f0f4917fbd9\") " pod="openshift-ingress/e2e-inf-a-029d89-openshift-default-58d5f8678-kgp6f"
Jul 27 21:27:24.007737 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:24.007716    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l74br\" (UniqueName: \"kubernetes.io/projected/3c082ecb-b11c-4454-90b9-6f0f4917fbd9-kube-api-access-l74br\") pod \"e2e-inf-a-029d89-openshift-default-58d5f8678-kgp6f\" (UID: \"3c082ecb-b11c-4454-90b9-6f0f4917fbd9\") " pod="openshift-ingress/e2e-inf-a-029d89-openshift-default-58d5f8678-kgp6f"
Jul 27 21:27:24.200558 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:24.200475    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-029d89-openshift-default-58d5f8678-kgp6f"
Jul 27 21:27:24.347213 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:24.347168    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-029d89-openshift-default-58d5f8678-kgp6f"]
Jul 27 21:27:24.350138 ip-10-0-141-64 kubenswrapper[2560]: W0727 21:27:24.350110    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c082ecb_b11c_4454_90b9_6f0f4917fbd9.slice/crio-bf1f85c766e96903eac6909f915f1a597148829fdfeddd4e5cb02aaeff8ef468 WatchSource:0}: Error finding container bf1f85c766e96903eac6909f915f1a597148829fdfeddd4e5cb02aaeff8ef468: Status 404 returned error can't find the container with id bf1f85c766e96903eac6909f915f1a597148829fdfeddd4e5cb02aaeff8ef468
Jul 27 21:27:24.352160 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:24.352128    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 27 21:27:24.352235 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:24.352202    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 27 21:27:24.352274 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:24.352235    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 27 21:27:24.456822 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:24.456736    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-029d89-openshift-default-58d5f8678-kgp6f" event={"ID":"3c082ecb-b11c-4454-90b9-6f0f4917fbd9","Type":"ContainerStarted","Data":"e7319a0c7ddfe37c63309fc95ded212dd7d6b3deddddf4480d9fc1559a21b246"}
Jul 27 21:27:24.456822 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:24.456788    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-029d89-openshift-default-58d5f8678-kgp6f" event={"ID":"3c082ecb-b11c-4454-90b9-6f0f4917fbd9","Type":"ContainerStarted","Data":"bf1f85c766e96903eac6909f915f1a597148829fdfeddd4e5cb02aaeff8ef468"}
Jul 27 21:27:24.488254 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:24.488190    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-029d89-openshift-default-58d5f8678-kgp6f" podStartSLOduration=1.488174539 podStartE2EDuration="1.488174539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:27:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:27:24.486462077 +0000 UTC m=+2161.238323892" watchObservedRunningTime="2026-07-27 21:27:24.488174539 +0000 UTC m=+2161.240036333"
Jul 27 21:27:25.201511 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:25.201473    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-029d89-openshift-default-58d5f8678-kgp6f"
Jul 27 21:27:25.206376 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:25.206349    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-029d89-openshift-default-58d5f8678-kgp6f"
Jul 27 21:27:25.462133 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:25.462037    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-029d89-openshift-default-58d5f8678-kgp6f"
Jul 27 21:27:25.462946 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:25.462929    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-029d89-openshift-default-58d5f8678-kgp6f"
Jul 27 21:27:31.709008 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:31.708967    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-ccc682-openshift-default-858b64fd74-fz6mr"]
Jul 27 21:27:31.712630 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:31.712606    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ccc682-openshift-default-858b64fd74-fz6mr"
Jul 27 21:27:31.715438 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:31.715408    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-ccc682-openshift-default-dockercfg-5m27h\""
Jul 27 21:27:31.736027 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:31.735995    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-ccc682-openshift-default-858b64fd74-fz6mr"]
Jul 27 21:27:31.766208 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:31.766176    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92cacf44-a1b3-4ee5-93e2-fa46a7cee818-istiod-ca-cert\") pod \"e2e-inf-b-ccc682-openshift-default-858b64fd74-fz6mr\" (UID: \"92cacf44-a1b3-4ee5-93e2-fa46a7cee818\") " pod="openshift-ingress/e2e-inf-b-ccc682-openshift-default-858b64fd74-fz6mr"
Jul 27 21:27:31.766363 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:31.766229    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92cacf44-a1b3-4ee5-93e2-fa46a7cee818-credential-socket\") pod \"e2e-inf-b-ccc682-openshift-default-858b64fd74-fz6mr\" (UID: \"92cacf44-a1b3-4ee5-93e2-fa46a7cee818\") " pod="openshift-ingress/e2e-inf-b-ccc682-openshift-default-858b64fd74-fz6mr"
Jul 27 21:27:31.766363 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:31.766247    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bq85w\" (UniqueName: \"kubernetes.io/projected/92cacf44-a1b3-4ee5-93e2-fa46a7cee818-kube-api-access-bq85w\") pod \"e2e-inf-b-ccc682-openshift-default-858b64fd74-fz6mr\" (UID: \"92cacf44-a1b3-4ee5-93e2-fa46a7cee818\") " pod="openshift-ingress/e2e-inf-b-ccc682-openshift-default-858b64fd74-fz6mr"
Jul 27 21:27:31.766363 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:31.766307    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92cacf44-a1b3-4ee5-93e2-fa46a7cee818-istio-podinfo\") pod \"e2e-inf-b-ccc682-openshift-default-858b64fd74-fz6mr\" (UID: \"92cacf44-a1b3-4ee5-93e2-fa46a7cee818\") " pod="openshift-ingress/e2e-inf-b-ccc682-openshift-default-858b64fd74-fz6mr"
Jul 27 21:27:31.766363 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:31.766335    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92cacf44-a1b3-4ee5-93e2-fa46a7cee818-istio-token\") pod \"e2e-inf-b-ccc682-openshift-default-858b64fd74-fz6mr\" (UID: \"92cacf44-a1b3-4ee5-93e2-fa46a7cee818\") " pod="openshift-ingress/e2e-inf-b-ccc682-openshift-default-858b64fd74-fz6mr"
Jul 27 21:27:31.766363 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:31.766354    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92cacf44-a1b3-4ee5-93e2-fa46a7cee818-workload-certs\") pod \"e2e-inf-b-ccc682-openshift-default-858b64fd74-fz6mr\" (UID: \"92cacf44-a1b3-4ee5-93e2-fa46a7cee818\") " pod="openshift-ingress/e2e-inf-b-ccc682-openshift-default-858b64fd74-fz6mr"
Jul 27 21:27:31.766549 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:31.766376    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92cacf44-a1b3-4ee5-93e2-fa46a7cee818-istio-envoy\") pod \"e2e-inf-b-ccc682-openshift-default-858b64fd74-fz6mr\" (UID: \"92cacf44-a1b3-4ee5-93e2-fa46a7cee818\") " pod="openshift-ingress/e2e-inf-b-ccc682-openshift-default-858b64fd74-fz6mr"
Jul 27 21:27:31.766549 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:31.766424    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92cacf44-a1b3-4ee5-93e2-fa46a7cee818-istio-data\") pod \"e2e-inf-b-ccc682-openshift-default-858b64fd74-fz6mr\" (UID: \"92cacf44-a1b3-4ee5-93e2-fa46a7cee818\") " pod="openshift-ingress/e2e-inf-b-ccc682-openshift-default-858b64fd74-fz6mr"
Jul 27 21:27:31.766549 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:31.766456    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92cacf44-a1b3-4ee5-93e2-fa46a7cee818-workload-socket\") pod \"e2e-inf-b-ccc682-openshift-default-858b64fd74-fz6mr\" (UID: \"92cacf44-a1b3-4ee5-93e2-fa46a7cee818\") " pod="openshift-ingress/e2e-inf-b-ccc682-openshift-default-858b64fd74-fz6mr"
Jul 27 21:27:31.866884 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:31.866844    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92cacf44-a1b3-4ee5-93e2-fa46a7cee818-istio-podinfo\") pod \"e2e-inf-b-ccc682-openshift-default-858b64fd74-fz6mr\" (UID: \"92cacf44-a1b3-4ee5-93e2-fa46a7cee818\") " pod="openshift-ingress/e2e-inf-b-ccc682-openshift-default-858b64fd74-fz6mr"
Jul 27 21:27:31.866884 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:31.866883    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92cacf44-a1b3-4ee5-93e2-fa46a7cee818-istio-token\") pod \"e2e-inf-b-ccc682-openshift-default-858b64fd74-fz6mr\" (UID: \"92cacf44-a1b3-4ee5-93e2-fa46a7cee818\") " pod="openshift-ingress/e2e-inf-b-ccc682-openshift-default-858b64fd74-fz6mr"
Jul 27 21:27:31.867183 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:31.866903    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92cacf44-a1b3-4ee5-93e2-fa46a7cee818-workload-certs\") pod \"e2e-inf-b-ccc682-openshift-default-858b64fd74-fz6mr\" (UID: \"92cacf44-a1b3-4ee5-93e2-fa46a7cee818\") " pod="openshift-ingress/e2e-inf-b-ccc682-openshift-default-858b64fd74-fz6mr"
Jul 27 21:27:31.867183 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:31.866925    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92cacf44-a1b3-4ee5-93e2-fa46a7cee818-istio-envoy\") pod \"e2e-inf-b-ccc682-openshift-default-858b64fd74-fz6mr\" (UID: \"92cacf44-a1b3-4ee5-93e2-fa46a7cee818\") " pod="openshift-ingress/e2e-inf-b-ccc682-openshift-default-858b64fd74-fz6mr"
Jul 27 21:27:31.867183 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:31.866951    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92cacf44-a1b3-4ee5-93e2-fa46a7cee818-istio-data\") pod \"e2e-inf-b-ccc682-openshift-default-858b64fd74-fz6mr\" (UID: \"92cacf44-a1b3-4ee5-93e2-fa46a7cee818\") " pod="openshift-ingress/e2e-inf-b-ccc682-openshift-default-858b64fd74-fz6mr"
Jul 27 21:27:31.867183 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:31.866975    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92cacf44-a1b3-4ee5-93e2-fa46a7cee818-workload-socket\") pod \"e2e-inf-b-ccc682-openshift-default-858b64fd74-fz6mr\" (UID: \"92cacf44-a1b3-4ee5-93e2-fa46a7cee818\") " pod="openshift-ingress/e2e-inf-b-ccc682-openshift-default-858b64fd74-fz6mr"
Jul 27 21:27:31.867183 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:31.867011    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92cacf44-a1b3-4ee5-93e2-fa46a7cee818-istiod-ca-cert\") pod \"e2e-inf-b-ccc682-openshift-default-858b64fd74-fz6mr\" (UID: \"92cacf44-a1b3-4ee5-93e2-fa46a7cee818\") " pod="openshift-ingress/e2e-inf-b-ccc682-openshift-default-858b64fd74-fz6mr"
Jul 27 21:27:31.867183 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:31.867113    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92cacf44-a1b3-4ee5-93e2-fa46a7cee818-credential-socket\") pod \"e2e-inf-b-ccc682-openshift-default-858b64fd74-fz6mr\" (UID: \"92cacf44-a1b3-4ee5-93e2-fa46a7cee818\") " pod="openshift-ingress/e2e-inf-b-ccc682-openshift-default-858b64fd74-fz6mr"
Jul 27 21:27:31.867183 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:31.867137    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bq85w\" (UniqueName: \"kubernetes.io/projected/92cacf44-a1b3-4ee5-93e2-fa46a7cee818-kube-api-access-bq85w\") pod \"e2e-inf-b-ccc682-openshift-default-858b64fd74-fz6mr\" (UID: \"92cacf44-a1b3-4ee5-93e2-fa46a7cee818\") " pod="openshift-ingress/e2e-inf-b-ccc682-openshift-default-858b64fd74-fz6mr"
Jul 27 21:27:31.867541 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:31.867337    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92cacf44-a1b3-4ee5-93e2-fa46a7cee818-workload-certs\") pod \"e2e-inf-b-ccc682-openshift-default-858b64fd74-fz6mr\" (UID: \"92cacf44-a1b3-4ee5-93e2-fa46a7cee818\") " pod="openshift-ingress/e2e-inf-b-ccc682-openshift-default-858b64fd74-fz6mr"
Jul 27 21:27:31.867541 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:31.867388    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92cacf44-a1b3-4ee5-93e2-fa46a7cee818-istio-data\") pod \"e2e-inf-b-ccc682-openshift-default-858b64fd74-fz6mr\" (UID: \"92cacf44-a1b3-4ee5-93e2-fa46a7cee818\") " pod="openshift-ingress/e2e-inf-b-ccc682-openshift-default-858b64fd74-fz6mr"
Jul 27 21:27:31.867648 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:31.867569    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92cacf44-a1b3-4ee5-93e2-fa46a7cee818-workload-socket\") pod \"e2e-inf-b-ccc682-openshift-default-858b64fd74-fz6mr\" (UID: \"92cacf44-a1b3-4ee5-93e2-fa46a7cee818\") " pod="openshift-ingress/e2e-inf-b-ccc682-openshift-default-858b64fd74-fz6mr"
Jul 27 21:27:31.867648 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:31.867635    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92cacf44-a1b3-4ee5-93e2-fa46a7cee818-credential-socket\") pod \"e2e-inf-b-ccc682-openshift-default-858b64fd74-fz6mr\" (UID: \"92cacf44-a1b3-4ee5-93e2-fa46a7cee818\") " pod="openshift-ingress/e2e-inf-b-ccc682-openshift-default-858b64fd74-fz6mr"
Jul 27 21:27:31.868029 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:31.868005    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92cacf44-a1b3-4ee5-93e2-fa46a7cee818-istiod-ca-cert\") pod \"e2e-inf-b-ccc682-openshift-default-858b64fd74-fz6mr\" (UID: \"92cacf44-a1b3-4ee5-93e2-fa46a7cee818\") " pod="openshift-ingress/e2e-inf-b-ccc682-openshift-default-858b64fd74-fz6mr"
Jul 27 21:27:31.869588 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:31.869562    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92cacf44-a1b3-4ee5-93e2-fa46a7cee818-istio-envoy\") pod \"e2e-inf-b-ccc682-openshift-default-858b64fd74-fz6mr\" (UID: \"92cacf44-a1b3-4ee5-93e2-fa46a7cee818\") " pod="openshift-ingress/e2e-inf-b-ccc682-openshift-default-858b64fd74-fz6mr"
Jul 27 21:27:31.869676 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:31.869650    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92cacf44-a1b3-4ee5-93e2-fa46a7cee818-istio-podinfo\") pod \"e2e-inf-b-ccc682-openshift-default-858b64fd74-fz6mr\" (UID: \"92cacf44-a1b3-4ee5-93e2-fa46a7cee818\") " pod="openshift-ingress/e2e-inf-b-ccc682-openshift-default-858b64fd74-fz6mr"
Jul 27 21:27:31.890695 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:31.890666    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92cacf44-a1b3-4ee5-93e2-fa46a7cee818-istio-token\") pod \"e2e-inf-b-ccc682-openshift-default-858b64fd74-fz6mr\" (UID: \"92cacf44-a1b3-4ee5-93e2-fa46a7cee818\") " pod="openshift-ingress/e2e-inf-b-ccc682-openshift-default-858b64fd74-fz6mr"
Jul 27 21:27:31.890813 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:31.890703    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bq85w\" (UniqueName: \"kubernetes.io/projected/92cacf44-a1b3-4ee5-93e2-fa46a7cee818-kube-api-access-bq85w\") pod \"e2e-inf-b-ccc682-openshift-default-858b64fd74-fz6mr\" (UID: \"92cacf44-a1b3-4ee5-93e2-fa46a7cee818\") " pod="openshift-ingress/e2e-inf-b-ccc682-openshift-default-858b64fd74-fz6mr"
Jul 27 21:27:32.022580 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:32.022545    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ccc682-openshift-default-858b64fd74-fz6mr"
Jul 27 21:27:32.167643 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:32.167610    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-ccc682-openshift-default-858b64fd74-fz6mr"]
Jul 27 21:27:32.171342 ip-10-0-141-64 kubenswrapper[2560]: W0727 21:27:32.171316    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92cacf44_a1b3_4ee5_93e2_fa46a7cee818.slice/crio-9cb9a580aa2829be104bb8f2d4562c3dcd88f84e1dc540e430f93d0c2db6e789 WatchSource:0}: Error finding container 9cb9a580aa2829be104bb8f2d4562c3dcd88f84e1dc540e430f93d0c2db6e789: Status 404 returned error can't find the container with id 9cb9a580aa2829be104bb8f2d4562c3dcd88f84e1dc540e430f93d0c2db6e789
Jul 27 21:27:32.173452 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:32.173418    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 27 21:27:32.173548 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:32.173497    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 27 21:27:32.173615 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:32.173545    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 27 21:27:32.333881 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:32.333793    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:27:32.495994 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:32.495934    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ccc682-openshift-default-858b64fd74-fz6mr" event={"ID":"92cacf44-a1b3-4ee5-93e2-fa46a7cee818","Type":"ContainerStarted","Data":"9829a29269c1d03198abe002f624599214b8bf3db74030b82d0192d48d23ab9b"}
Jul 27 21:27:32.496202 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:32.496019    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ccc682-openshift-default-858b64fd74-fz6mr" event={"ID":"92cacf44-a1b3-4ee5-93e2-fa46a7cee818","Type":"ContainerStarted","Data":"9cb9a580aa2829be104bb8f2d4562c3dcd88f84e1dc540e430f93d0c2db6e789"}
Jul 27 21:27:32.537526 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:32.537458    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-ccc682-openshift-default-858b64fd74-fz6mr" podStartSLOduration=1.537441421 podStartE2EDuration="1.537441421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:27:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:27:32.535866346 +0000 UTC m=+2169.287728141" watchObservedRunningTime="2026-07-27 21:27:32.537441421 +0000 UTC m=+2169.289303216"
Jul 27 21:27:33.023461 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:33.023422    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-ccc682-openshift-default-858b64fd74-fz6mr"
Jul 27 21:27:33.027900 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:33.027873    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-ccc682-openshift-default-858b64fd74-fz6mr"
Jul 27 21:27:33.500743 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:33.500663    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-ccc682-openshift-default-858b64fd74-fz6mr"
Jul 27 21:27:33.501554 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:33.501535    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-ccc682-openshift-default-858b64fd74-fz6mr"
Jul 27 21:27:40.463528 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:40.463488    2560 patch_prober.go:28] interesting pod/e2e-inf-a-029d89-openshift-default-58d5f8678-kgp6f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Jul 27 21:27:40.463981 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:40.463570    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-029d89-openshift-default-58d5f8678-kgp6f" podUID="3c082ecb-b11c-4454-90b9-6f0f4917fbd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Jul 27 21:27:40.534273 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:40.534239    2560 generic.go:356] "Generic (PLEG): container finished" podID="3c082ecb-b11c-4454-90b9-6f0f4917fbd9" containerID="e7319a0c7ddfe37c63309fc95ded212dd7d6b3deddddf4480d9fc1559a21b246" exitCode=0
Jul 27 21:27:40.534441 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:40.534294    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-029d89-openshift-default-58d5f8678-kgp6f" event={"ID":"3c082ecb-b11c-4454-90b9-6f0f4917fbd9","Type":"ContainerDied","Data":"e7319a0c7ddfe37c63309fc95ded212dd7d6b3deddddf4480d9fc1559a21b246"}
Jul 27 21:27:40.534703 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:40.534684    2560 scope.go:117] "RemoveContainer" containerID="e7319a0c7ddfe37c63309fc95ded212dd7d6b3deddddf4480d9fc1559a21b246"
Jul 27 21:27:40.535836 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:40.535808    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 27 21:27:40.535896 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:40.535875    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 27 21:27:40.535930 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:40.535909    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 27 21:27:41.541473 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:41.541428    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-029d89-openshift-default-58d5f8678-kgp6f" event={"ID":"3c082ecb-b11c-4454-90b9-6f0f4917fbd9","Type":"ContainerStarted","Data":"30a34654cf17f387b1ab7f97e211f1dd34d4fb19ca09aae3692d65efda955c6e"}
Jul 27 21:27:41.607413 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:41.607365    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:27:42.200966 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:42.200922    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-029d89-openshift-default-58d5f8678-kgp6f"
Jul 27 21:27:42.201170 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:42.201118    2560 patch_prober.go:28] interesting pod/e2e-inf-a-029d89-openshift-default-58d5f8678-kgp6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Jul 27 21:27:42.201232 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:42.201170    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-029d89-openshift-default-58d5f8678-kgp6f" podUID="3c082ecb-b11c-4454-90b9-6f0f4917fbd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Jul 27 21:27:43.201673 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:43.201625    2560 patch_prober.go:28] interesting pod/e2e-inf-a-029d89-openshift-default-58d5f8678-kgp6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Jul 27 21:27:43.202117 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:43.201685    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-029d89-openshift-default-58d5f8678-kgp6f" podUID="3c082ecb-b11c-4454-90b9-6f0f4917fbd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Jul 27 21:27:44.201170 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:44.201133    2560 patch_prober.go:28] interesting pod/e2e-inf-a-029d89-openshift-default-58d5f8678-kgp6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Jul 27 21:27:44.201356 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:44.201190    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-029d89-openshift-default-58d5f8678-kgp6f" podUID="3c082ecb-b11c-4454-90b9-6f0f4917fbd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Jul 27 21:27:45.201013 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:45.200972    2560 patch_prober.go:28] interesting pod/e2e-inf-a-029d89-openshift-default-58d5f8678-kgp6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Jul 27 21:27:45.201437 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:45.201028    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-029d89-openshift-default-58d5f8678-kgp6f" podUID="3c082ecb-b11c-4454-90b9-6f0f4917fbd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Jul 27 21:27:46.205268 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:46.205232    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-029d89-openshift-default-58d5f8678-kgp6f"
Jul 27 21:27:46.205691 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:46.205505    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-029d89-openshift-default-58d5f8678-kgp6f"
Jul 27 21:27:46.206134 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:46.206118    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-029d89-openshift-default-58d5f8678-kgp6f"
Jul 27 21:27:49.543256 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:49.543211    2560 patch_prober.go:28] interesting pod/e2e-inf-b-ccc682-openshift-default-858b64fd74-fz6mr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 27 21:27:49.544529 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:49.543337    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-ccc682-openshift-default-858b64fd74-fz6mr" podUID="92cacf44-a1b3-4ee5-93e2-fa46a7cee818" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 27 21:27:50.603914 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:50.603878    2560 generic.go:356] "Generic (PLEG): container finished" podID="92cacf44-a1b3-4ee5-93e2-fa46a7cee818" containerID="9829a29269c1d03198abe002f624599214b8bf3db74030b82d0192d48d23ab9b" exitCode=0
Jul 27 21:27:50.604378 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:50.603950    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ccc682-openshift-default-858b64fd74-fz6mr" event={"ID":"92cacf44-a1b3-4ee5-93e2-fa46a7cee818","Type":"ContainerDied","Data":"9829a29269c1d03198abe002f624599214b8bf3db74030b82d0192d48d23ab9b"}
Jul 27 21:27:50.604378 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:50.604349    2560 scope.go:117] "RemoveContainer" containerID="9829a29269c1d03198abe002f624599214b8bf3db74030b82d0192d48d23ab9b"
Jul 27 21:27:50.605385 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:50.605359    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 27 21:27:50.605440 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:50.605425    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 27 21:27:50.605475 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:50.605459    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 27 21:27:51.610909 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:51.610875    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ccc682-openshift-default-858b64fd74-fz6mr" event={"ID":"92cacf44-a1b3-4ee5-93e2-fa46a7cee818","Type":"ContainerStarted","Data":"3bf33ad78a44947407607ed1b7f927a21895abe7d9e26519914113deefff1212"}
Jul 27 21:27:52.023392 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:52.023360    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-ccc682-openshift-default-858b64fd74-fz6mr"
Jul 27 21:27:52.023605 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:52.023582    2560 patch_prober.go:28] interesting pod/e2e-inf-b-ccc682-openshift-default-858b64fd74-fz6mr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Jul 27 21:27:52.023652 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:52.023630    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ccc682-openshift-default-858b64fd74-fz6mr" podUID="92cacf44-a1b3-4ee5-93e2-fa46a7cee818" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Jul 27 21:27:53.023699 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:53.023658    2560 patch_prober.go:28] interesting pod/e2e-inf-b-ccc682-openshift-default-858b64fd74-fz6mr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Jul 27 21:27:53.024144 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:53.023717    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ccc682-openshift-default-858b64fd74-fz6mr" podUID="92cacf44-a1b3-4ee5-93e2-fa46a7cee818" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Jul 27 21:27:54.023909 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:54.023868    2560 patch_prober.go:28] interesting pod/e2e-inf-b-ccc682-openshift-default-858b64fd74-fz6mr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Jul 27 21:27:54.024360 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:54.023933    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ccc682-openshift-default-858b64fd74-fz6mr" podUID="92cacf44-a1b3-4ee5-93e2-fa46a7cee818" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Jul 27 21:27:54.604506 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:54.604471    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:27:55.023458 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:55.023427    2560 patch_prober.go:28] interesting pod/e2e-inf-b-ccc682-openshift-default-858b64fd74-fz6mr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Jul 27 21:27:55.023692 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:55.023482    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ccc682-openshift-default-858b64fd74-fz6mr" podUID="92cacf44-a1b3-4ee5-93e2-fa46a7cee818" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Jul 27 21:27:56.026519 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:56.026483    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-ccc682-openshift-default-858b64fd74-fz6mr"
Jul 27 21:27:56.026902 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:56.026807    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-ccc682-openshift-default-858b64fd74-fz6mr"
Jul 27 21:27:56.027534 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:27:56.027514    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-ccc682-openshift-default-858b64fd74-fz6mr"
Jul 27 21:28:10.860489 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:28:10.860451    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-ccc682/test-model-ccc682-kserve-778b6d5d4c-dvxkq"]
Jul 27 21:28:10.864378 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:28:10.864351    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ccc682/test-model-ccc682-kserve-778b6d5d4c-dvxkq"
Jul 27 21:28:10.868002 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:28:10.867982    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-ccc682\"/\"test-model-ccc682-kserve-self-signed-certs\""
Jul 27 21:28:10.869022 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:28:10.868997    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-ccc682\"/\"kube-root-ca.crt\""
Jul 27 21:28:10.869022 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:28:10.869015    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-ccc682\"/\"default-dockercfg-kjlcd\""
Jul 27 21:28:10.869234 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:28:10.868998    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-ccc682\"/\"openshift-service-ca.crt\""
Jul 27 21:28:10.892806 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:28:10.892779    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-ccc682/test-model-ccc682-kserve-778b6d5d4c-dvxkq"]
Jul 27 21:28:10.938867 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:28:10.938830    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5558544-69ac-4444-8e81-56782ead58ca-model-cache\") pod \"test-model-ccc682-kserve-778b6d5d4c-dvxkq\" (UID: \"f5558544-69ac-4444-8e81-56782ead58ca\") " pod="ai-tenant-e2e-inf-b-ccc682/test-model-ccc682-kserve-778b6d5d4c-dvxkq"
Jul 27 21:28:10.939029 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:28:10.938891    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5558544-69ac-4444-8e81-56782ead58ca-tls-certs\") pod \"test-model-ccc682-kserve-778b6d5d4c-dvxkq\" (UID: \"f5558544-69ac-4444-8e81-56782ead58ca\") " pod="ai-tenant-e2e-inf-b-ccc682/test-model-ccc682-kserve-778b6d5d4c-dvxkq"
Jul 27 21:28:10.939029 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:28:10.938962    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x46ks\" (UniqueName: \"kubernetes.io/projected/f5558544-69ac-4444-8e81-56782ead58ca-kube-api-access-x46ks\") pod \"test-model-ccc682-kserve-778b6d5d4c-dvxkq\" (UID: \"f5558544-69ac-4444-8e81-56782ead58ca\") " pod="ai-tenant-e2e-inf-b-ccc682/test-model-ccc682-kserve-778b6d5d4c-dvxkq"
Jul 27 21:28:10.939029 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:28:10.939022    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5558544-69ac-4444-8e81-56782ead58ca-dshm\") pod \"test-model-ccc682-kserve-778b6d5d4c-dvxkq\" (UID: \"f5558544-69ac-4444-8e81-56782ead58ca\") " pod="ai-tenant-e2e-inf-b-ccc682/test-model-ccc682-kserve-778b6d5d4c-dvxkq"
Jul 27 21:28:10.939179 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:28:10.939047    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5558544-69ac-4444-8e81-56782ead58ca-home\") pod \"test-model-ccc682-kserve-778b6d5d4c-dvxkq\" (UID: \"f5558544-69ac-4444-8e81-56782ead58ca\") " pod="ai-tenant-e2e-inf-b-ccc682/test-model-ccc682-kserve-778b6d5d4c-dvxkq"
Jul 27 21:28:10.939179 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:28:10.939068    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5558544-69ac-4444-8e81-56782ead58ca-tmp-dir\") pod \"test-model-ccc682-kserve-778b6d5d4c-dvxkq\" (UID: \"f5558544-69ac-4444-8e81-56782ead58ca\") " pod="ai-tenant-e2e-inf-b-ccc682/test-model-ccc682-kserve-778b6d5d4c-dvxkq"
Jul 27 21:28:11.040672 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:28:11.040625    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5558544-69ac-4444-8e81-56782ead58ca-tls-certs\") pod \"test-model-ccc682-kserve-778b6d5d4c-dvxkq\" (UID: \"f5558544-69ac-4444-8e81-56782ead58ca\") " pod="ai-tenant-e2e-inf-b-ccc682/test-model-ccc682-kserve-778b6d5d4c-dvxkq"
Jul 27 21:28:11.040858 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:28:11.040704    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x46ks\" (UniqueName: \"kubernetes.io/projected/f5558544-69ac-4444-8e81-56782ead58ca-kube-api-access-x46ks\") pod \"test-model-ccc682-kserve-778b6d5d4c-dvxkq\" (UID: \"f5558544-69ac-4444-8e81-56782ead58ca\") " pod="ai-tenant-e2e-inf-b-ccc682/test-model-ccc682-kserve-778b6d5d4c-dvxkq"
Jul 27 21:28:11.040858 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:28:11.040771    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5558544-69ac-4444-8e81-56782ead58ca-dshm\") pod \"test-model-ccc682-kserve-778b6d5d4c-dvxkq\" (UID: \"f5558544-69ac-4444-8e81-56782ead58ca\") " pod="ai-tenant-e2e-inf-b-ccc682/test-model-ccc682-kserve-778b6d5d4c-dvxkq"
Jul 27 21:28:11.040858 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:28:11.040798    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5558544-69ac-4444-8e81-56782ead58ca-home\") pod \"test-model-ccc682-kserve-778b6d5d4c-dvxkq\" (UID: \"f5558544-69ac-4444-8e81-56782ead58ca\") " pod="ai-tenant-e2e-inf-b-ccc682/test-model-ccc682-kserve-778b6d5d4c-dvxkq"
Jul 27 21:28:11.040858 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:28:11.040830    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5558544-69ac-4444-8e81-56782ead58ca-tmp-dir\") pod \"test-model-ccc682-kserve-778b6d5d4c-dvxkq\" (UID: \"f5558544-69ac-4444-8e81-56782ead58ca\") " pod="ai-tenant-e2e-inf-b-ccc682/test-model-ccc682-kserve-778b6d5d4c-dvxkq"
Jul 27 21:28:11.041077 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:28:11.040863    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5558544-69ac-4444-8e81-56782ead58ca-model-cache\") pod \"test-model-ccc682-kserve-778b6d5d4c-dvxkq\" (UID: \"f5558544-69ac-4444-8e81-56782ead58ca\") " pod="ai-tenant-e2e-inf-b-ccc682/test-model-ccc682-kserve-778b6d5d4c-dvxkq"
Jul 27 21:28:11.041345 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:28:11.041316    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5558544-69ac-4444-8e81-56782ead58ca-tmp-dir\") pod \"test-model-ccc682-kserve-778b6d5d4c-dvxkq\" (UID: \"f5558544-69ac-4444-8e81-56782ead58ca\") " pod="ai-tenant-e2e-inf-b-ccc682/test-model-ccc682-kserve-778b6d5d4c-dvxkq"
Jul 27 21:28:11.041618 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:28:11.041353    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5558544-69ac-4444-8e81-56782ead58ca-model-cache\") pod \"test-model-ccc682-kserve-778b6d5d4c-dvxkq\" (UID: \"f5558544-69ac-4444-8e81-56782ead58ca\") " pod="ai-tenant-e2e-inf-b-ccc682/test-model-ccc682-kserve-778b6d5d4c-dvxkq"
Jul 27 21:28:11.041618 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:28:11.041398    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5558544-69ac-4444-8e81-56782ead58ca-home\") pod \"test-model-ccc682-kserve-778b6d5d4c-dvxkq\" (UID: \"f5558544-69ac-4444-8e81-56782ead58ca\") " pod="ai-tenant-e2e-inf-b-ccc682/test-model-ccc682-kserve-778b6d5d4c-dvxkq"
Jul 27 21:28:11.043504 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:28:11.043476    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5558544-69ac-4444-8e81-56782ead58ca-tls-certs\") pod \"test-model-ccc682-kserve-778b6d5d4c-dvxkq\" (UID: \"f5558544-69ac-4444-8e81-56782ead58ca\") " pod="ai-tenant-e2e-inf-b-ccc682/test-model-ccc682-kserve-778b6d5d4c-dvxkq"
Jul 27 21:28:11.043624 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:28:11.043613    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5558544-69ac-4444-8e81-56782ead58ca-dshm\") pod \"test-model-ccc682-kserve-778b6d5d4c-dvxkq\" (UID: \"f5558544-69ac-4444-8e81-56782ead58ca\") " pod="ai-tenant-e2e-inf-b-ccc682/test-model-ccc682-kserve-778b6d5d4c-dvxkq"
Jul 27 21:28:11.057899 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:28:11.057870    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x46ks\" (UniqueName: \"kubernetes.io/projected/f5558544-69ac-4444-8e81-56782ead58ca-kube-api-access-x46ks\") pod \"test-model-ccc682-kserve-778b6d5d4c-dvxkq\" (UID: \"f5558544-69ac-4444-8e81-56782ead58ca\") " pod="ai-tenant-e2e-inf-b-ccc682/test-model-ccc682-kserve-778b6d5d4c-dvxkq"
Jul 27 21:28:11.173313 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:28:11.173215    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ccc682/test-model-ccc682-kserve-778b6d5d4c-dvxkq"
Jul 27 21:28:11.314814 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:28:11.314783    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-ccc682/test-model-ccc682-kserve-778b6d5d4c-dvxkq"]
Jul 27 21:28:11.317932 ip-10-0-141-64 kubenswrapper[2560]: W0727 21:28:11.317889    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf5558544_69ac_4444_8e81_56782ead58ca.slice/crio-73da15f6c36ebdf7882a4e4cca15aa31d9d0bd35ebcc4ba0359485554d3c78b8 WatchSource:0}: Error finding container 73da15f6c36ebdf7882a4e4cca15aa31d9d0bd35ebcc4ba0359485554d3c78b8: Status 404 returned error can't find the container with id 73da15f6c36ebdf7882a4e4cca15aa31d9d0bd35ebcc4ba0359485554d3c78b8
Jul 27 21:28:11.710429 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:28:11.710392    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ccc682/test-model-ccc682-kserve-778b6d5d4c-dvxkq" event={"ID":"f5558544-69ac-4444-8e81-56782ead58ca","Type":"ContainerStarted","Data":"5ffaffb3efd9e71f00be10b1ac5f2aacae02b3c94cfbdb139e1be2c304acb4f7"}
Jul 27 21:28:11.710429 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:28:11.710435    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ccc682/test-model-ccc682-kserve-778b6d5d4c-dvxkq" event={"ID":"f5558544-69ac-4444-8e81-56782ead58ca","Type":"ContainerStarted","Data":"73da15f6c36ebdf7882a4e4cca15aa31d9d0bd35ebcc4ba0359485554d3c78b8"}
Jul 27 21:28:16.554922 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:28:16.554858    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-ccc682/test-model-ccc682-kserve-778b6d5d4c-dvxkq" podStartSLOduration=6.55483943 podStartE2EDuration="6.55483943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:28:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:28:11.740242624 +0000 UTC m=+2208.492104421" watchObservedRunningTime="2026-07-27 21:28:16.55483943 +0000 UTC m=+2213.306701229"
Jul 27 21:28:16.555362 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:28:16.555343    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:28:21.173961 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:28:21.173920    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-ccc682/test-model-ccc682-kserve-778b6d5d4c-dvxkq"
Jul 27 21:28:21.174433 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:28:21.173976    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-ccc682/test-model-ccc682-kserve-778b6d5d4c-dvxkq"
Jul 27 21:28:21.186695 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:28:21.186671    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-ccc682/test-model-ccc682-kserve-778b6d5d4c-dvxkq"
Jul 27 21:28:21.766312 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:28:21.766273    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-ccc682/test-model-ccc682-kserve-778b6d5d4c-dvxkq"
Jul 27 21:28:54.855316 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:28:54.855277    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-ccc682/test-model-ccc682-kserve-778b6d5d4c-dvxkq"]
Jul 27 21:28:58.842996 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:28:58.842958    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:29:09.874282 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:09.874217    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-ccc682/test-model-ccc682-kserve-778b6d5d4c-dvxkq" podUID="f5558544-69ac-4444-8e81-56782ead58ca" containerName="main" containerID="cri-o://5ffaffb3efd9e71f00be10b1ac5f2aacae02b3c94cfbdb139e1be2c304acb4f7" gracePeriod=45
Jul 27 21:29:09.990948 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:09.990913    2560 generic.go:356] "Generic (PLEG): container finished" podID="f5558544-69ac-4444-8e81-56782ead58ca" containerID="5ffaffb3efd9e71f00be10b1ac5f2aacae02b3c94cfbdb139e1be2c304acb4f7" exitCode=0
Jul 27 21:29:09.991135 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:09.990982    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ccc682/test-model-ccc682-kserve-778b6d5d4c-dvxkq" event={"ID":"f5558544-69ac-4444-8e81-56782ead58ca","Type":"ContainerDied","Data":"5ffaffb3efd9e71f00be10b1ac5f2aacae02b3c94cfbdb139e1be2c304acb4f7"}
Jul 27 21:29:10.127500 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:10.127442    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ccc682/test-model-ccc682-kserve-778b6d5d4c-dvxkq"
Jul 27 21:29:10.273785 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:10.273742    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5558544-69ac-4444-8e81-56782ead58ca-dshm\") pod \"f5558544-69ac-4444-8e81-56782ead58ca\" (UID: \"f5558544-69ac-4444-8e81-56782ead58ca\") "
Jul 27 21:29:10.273977 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:10.273800    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x46ks\" (UniqueName: \"kubernetes.io/projected/f5558544-69ac-4444-8e81-56782ead58ca-kube-api-access-x46ks\") pod \"f5558544-69ac-4444-8e81-56782ead58ca\" (UID: \"f5558544-69ac-4444-8e81-56782ead58ca\") "
Jul 27 21:29:10.273977 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:10.273815    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5558544-69ac-4444-8e81-56782ead58ca-home\") pod \"f5558544-69ac-4444-8e81-56782ead58ca\" (UID: \"f5558544-69ac-4444-8e81-56782ead58ca\") "
Jul 27 21:29:10.273977 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:10.273843    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5558544-69ac-4444-8e81-56782ead58ca-tls-certs\") pod \"f5558544-69ac-4444-8e81-56782ead58ca\" (UID: \"f5558544-69ac-4444-8e81-56782ead58ca\") "
Jul 27 21:29:10.273977 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:10.273875    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5558544-69ac-4444-8e81-56782ead58ca-tmp-dir\") pod \"f5558544-69ac-4444-8e81-56782ead58ca\" (UID: \"f5558544-69ac-4444-8e81-56782ead58ca\") "
Jul 27 21:29:10.273977 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:10.273909    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5558544-69ac-4444-8e81-56782ead58ca-model-cache\") pod \"f5558544-69ac-4444-8e81-56782ead58ca\" (UID: \"f5558544-69ac-4444-8e81-56782ead58ca\") "
Jul 27 21:29:10.274375 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:10.274199    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5558544-69ac-4444-8e81-56782ead58ca-home" (OuterVolumeSpecName: "home") pod "f5558544-69ac-4444-8e81-56782ead58ca" (UID: "f5558544-69ac-4444-8e81-56782ead58ca"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:29:10.274375 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:10.274207    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5558544-69ac-4444-8e81-56782ead58ca-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f5558544-69ac-4444-8e81-56782ead58ca" (UID: "f5558544-69ac-4444-8e81-56782ead58ca"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:29:10.274375 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:10.274361    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5558544-69ac-4444-8e81-56782ead58ca-model-cache" (OuterVolumeSpecName: "model-cache") pod "f5558544-69ac-4444-8e81-56782ead58ca" (UID: "f5558544-69ac-4444-8e81-56782ead58ca"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:29:10.276312 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:10.276288    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5558544-69ac-4444-8e81-56782ead58ca-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f5558544-69ac-4444-8e81-56782ead58ca" (UID: "f5558544-69ac-4444-8e81-56782ead58ca"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:29:10.276564 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:10.276546    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5558544-69ac-4444-8e81-56782ead58ca-dshm" (OuterVolumeSpecName: "dshm") pod "f5558544-69ac-4444-8e81-56782ead58ca" (UID: "f5558544-69ac-4444-8e81-56782ead58ca"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:29:10.276634 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:10.276562    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5558544-69ac-4444-8e81-56782ead58ca-kube-api-access-x46ks" (OuterVolumeSpecName: "kube-api-access-x46ks") pod "f5558544-69ac-4444-8e81-56782ead58ca" (UID: "f5558544-69ac-4444-8e81-56782ead58ca"). InnerVolumeSpecName "kube-api-access-x46ks". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:29:10.375162 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:10.375125    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x46ks\" (UniqueName: \"kubernetes.io/projected/f5558544-69ac-4444-8e81-56782ead58ca-kube-api-access-x46ks\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:29:10.375162 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:10.375156    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5558544-69ac-4444-8e81-56782ead58ca-home\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:29:10.375162 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:10.375165    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5558544-69ac-4444-8e81-56782ead58ca-tls-certs\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:29:10.375393 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:10.375195    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5558544-69ac-4444-8e81-56782ead58ca-tmp-dir\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:29:10.375393 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:10.375208    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5558544-69ac-4444-8e81-56782ead58ca-model-cache\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:29:10.375393 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:10.375221    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5558544-69ac-4444-8e81-56782ead58ca-dshm\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:29:10.996468 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:10.996432    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ccc682/test-model-ccc682-kserve-778b6d5d4c-dvxkq" event={"ID":"f5558544-69ac-4444-8e81-56782ead58ca","Type":"ContainerDied","Data":"73da15f6c36ebdf7882a4e4cca15aa31d9d0bd35ebcc4ba0359485554d3c78b8"}
Jul 27 21:29:10.996468 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:10.996467    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ccc682/test-model-ccc682-kserve-778b6d5d4c-dvxkq"
Jul 27 21:29:10.996931 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:10.996481    2560 scope.go:117] "RemoveContainer" containerID="5ffaffb3efd9e71f00be10b1ac5f2aacae02b3c94cfbdb139e1be2c304acb4f7"
Jul 27 21:29:11.034873 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:11.034845    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-ccc682/test-model-ccc682-kserve-778b6d5d4c-dvxkq"]
Jul 27 21:29:11.049671 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:11.049643    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-ccc682/test-model-ccc682-kserve-778b6d5d4c-dvxkq"]
Jul 27 21:29:11.698453 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:11.698416    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:29:11.847420 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:11.847390    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5558544-69ac-4444-8e81-56782ead58ca" path="/var/lib/kubelet/pods/f5558544-69ac-4444-8e81-56782ead58ca/volumes"
Jul 27 21:29:18.266300 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:18.266264    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-029d89-openshift-default-58d5f8678-kgp6f"]
Jul 27 21:29:18.266738 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:18.266478    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-029d89-openshift-default-58d5f8678-kgp6f" podUID="3c082ecb-b11c-4454-90b9-6f0f4917fbd9" containerName="istio-proxy" containerID="cri-o://30a34654cf17f387b1ab7f97e211f1dd34d4fb19ca09aae3692d65efda955c6e" gracePeriod=30
Jul 27 21:29:18.593860 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:18.593827    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ccc682-9k4v9"]
Jul 27 21:29:18.594245 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:18.594230    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5558544-69ac-4444-8e81-56782ead58ca" containerName="main"
Jul 27 21:29:18.594289 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:18.594247    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5558544-69ac-4444-8e81-56782ead58ca" containerName="main"
Jul 27 21:29:18.594328 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:18.594304    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5558544-69ac-4444-8e81-56782ead58ca" containerName="main"
Jul 27 21:29:18.598746 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:18.598729    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ccc682-9k4v9"
Jul 27 21:29:18.601779 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:18.601754    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j5rjm\""
Jul 27 21:29:18.601907 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:18.601834    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 27 21:29:18.619523 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:18.619500    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ccc682-9k4v9"]
Jul 27 21:29:18.750988 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:18.750952    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/214fdede-e5b1-4545-8f18-4e570c3e494d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ccc682-9k4v9\" (UID: \"214fdede-e5b1-4545-8f18-4e570c3e494d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ccc682-9k4v9"
Jul 27 21:29:18.851885 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:18.851792    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/214fdede-e5b1-4545-8f18-4e570c3e494d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ccc682-9k4v9\" (UID: \"214fdede-e5b1-4545-8f18-4e570c3e494d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ccc682-9k4v9"
Jul 27 21:29:18.852415 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:18.852396    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/214fdede-e5b1-4545-8f18-4e570c3e494d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ccc682-9k4v9\" (UID: \"214fdede-e5b1-4545-8f18-4e570c3e494d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ccc682-9k4v9"
Jul 27 21:29:18.908884 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:18.908850    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ccc682-9k4v9"
Jul 27 21:29:19.066266 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:19.066235    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ccc682-9k4v9"]
Jul 27 21:29:19.069323 ip-10-0-141-64 kubenswrapper[2560]: W0727 21:29:19.069297    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod214fdede_e5b1_4545_8f18_4e570c3e494d.slice/crio-8d00ae33b23bfc565042f15dc55454715b336a62cced66f934df963aaecc32b6 WatchSource:0}: Error finding container 8d00ae33b23bfc565042f15dc55454715b336a62cced66f934df963aaecc32b6: Status 404 returned error can't find the container with id 8d00ae33b23bfc565042f15dc55454715b336a62cced66f934df963aaecc32b6
Jul 27 21:29:20.042356 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:20.042321    2560 generic.go:356] "Generic (PLEG): container finished" podID="214fdede-e5b1-4545-8f18-4e570c3e494d" containerID="35af375f87bf6be52fda4341a6aa2cb1a33fbdb020fc890f99c63bf4f5a2e9cd" exitCode=0
Jul 27 21:29:20.042769 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:20.042398    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ccc682-9k4v9" event={"ID":"214fdede-e5b1-4545-8f18-4e570c3e494d","Type":"ContainerDied","Data":"35af375f87bf6be52fda4341a6aa2cb1a33fbdb020fc890f99c63bf4f5a2e9cd"}
Jul 27 21:29:20.042769 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:20.042454    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ccc682-9k4v9" event={"ID":"214fdede-e5b1-4545-8f18-4e570c3e494d","Type":"ContainerStarted","Data":"8d00ae33b23bfc565042f15dc55454715b336a62cced66f934df963aaecc32b6"}
Jul 27 21:29:21.190358 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:21.190324    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ccc682-9k4v9"
Jul 27 21:29:21.274796 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:21.274763    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/214fdede-e5b1-4545-8f18-4e570c3e494d-maas-api-service-ca\") pod \"214fdede-e5b1-4545-8f18-4e570c3e494d\" (UID: \"214fdede-e5b1-4545-8f18-4e570c3e494d\") "
Jul 27 21:29:21.275207 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:21.275181    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/214fdede-e5b1-4545-8f18-4e570c3e494d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "214fdede-e5b1-4545-8f18-4e570c3e494d" (UID: "214fdede-e5b1-4545-8f18-4e570c3e494d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:29:21.376380 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:21.376282    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/214fdede-e5b1-4545-8f18-4e570c3e494d-maas-api-service-ca\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:29:22.053220 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:22.053184    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ccc682-9k4v9" event={"ID":"214fdede-e5b1-4545-8f18-4e570c3e494d","Type":"ContainerDied","Data":"8d00ae33b23bfc565042f15dc55454715b336a62cced66f934df963aaecc32b6"}
Jul 27 21:29:22.053220 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:22.053203    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ccc682-9k4v9"
Jul 27 21:29:22.053424 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:22.053233    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8d00ae33b23bfc565042f15dc55454715b336a62cced66f934df963aaecc32b6"
Jul 27 21:29:23.532956 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:23.532888    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-029d89-openshift-default-58d5f8678-kgp6f"
Jul 27 21:29:23.697170 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:23.697135    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c082ecb-b11c-4454-90b9-6f0f4917fbd9-istio-token\") pod \"3c082ecb-b11c-4454-90b9-6f0f4917fbd9\" (UID: \"3c082ecb-b11c-4454-90b9-6f0f4917fbd9\") "
Jul 27 21:29:23.697332 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:23.697175    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c082ecb-b11c-4454-90b9-6f0f4917fbd9-istio-envoy\") pod \"3c082ecb-b11c-4454-90b9-6f0f4917fbd9\" (UID: \"3c082ecb-b11c-4454-90b9-6f0f4917fbd9\") "
Jul 27 21:29:23.697332 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:23.697203    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c082ecb-b11c-4454-90b9-6f0f4917fbd9-workload-socket\") pod \"3c082ecb-b11c-4454-90b9-6f0f4917fbd9\" (UID: \"3c082ecb-b11c-4454-90b9-6f0f4917fbd9\") "
Jul 27 21:29:23.697332 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:23.697252    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c082ecb-b11c-4454-90b9-6f0f4917fbd9-workload-certs\") pod \"3c082ecb-b11c-4454-90b9-6f0f4917fbd9\" (UID: \"3c082ecb-b11c-4454-90b9-6f0f4917fbd9\") "
Jul 27 21:29:23.697332 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:23.697276    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c082ecb-b11c-4454-90b9-6f0f4917fbd9-istiod-ca-cert\") pod \"3c082ecb-b11c-4454-90b9-6f0f4917fbd9\" (UID: \"3c082ecb-b11c-4454-90b9-6f0f4917fbd9\") "
Jul 27 21:29:23.697332 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:23.697303    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c082ecb-b11c-4454-90b9-6f0f4917fbd9-credential-socket\") pod \"3c082ecb-b11c-4454-90b9-6f0f4917fbd9\" (UID: \"3c082ecb-b11c-4454-90b9-6f0f4917fbd9\") "
Jul 27 21:29:23.697332 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:23.697331    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l74br\" (UniqueName: \"kubernetes.io/projected/3c082ecb-b11c-4454-90b9-6f0f4917fbd9-kube-api-access-l74br\") pod \"3c082ecb-b11c-4454-90b9-6f0f4917fbd9\" (UID: \"3c082ecb-b11c-4454-90b9-6f0f4917fbd9\") "
Jul 27 21:29:23.697674 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:23.697410    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c082ecb-b11c-4454-90b9-6f0f4917fbd9-istio-podinfo\") pod \"3c082ecb-b11c-4454-90b9-6f0f4917fbd9\" (UID: \"3c082ecb-b11c-4454-90b9-6f0f4917fbd9\") "
Jul 27 21:29:23.697674 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:23.697474    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c082ecb-b11c-4454-90b9-6f0f4917fbd9-istio-data\") pod \"3c082ecb-b11c-4454-90b9-6f0f4917fbd9\" (UID: \"3c082ecb-b11c-4454-90b9-6f0f4917fbd9\") "
Jul 27 21:29:23.697674 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:23.697560    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c082ecb-b11c-4454-90b9-6f0f4917fbd9-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3c082ecb-b11c-4454-90b9-6f0f4917fbd9" (UID: "3c082ecb-b11c-4454-90b9-6f0f4917fbd9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:29:23.697674 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:23.697631    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c082ecb-b11c-4454-90b9-6f0f4917fbd9-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3c082ecb-b11c-4454-90b9-6f0f4917fbd9" (UID: "3c082ecb-b11c-4454-90b9-6f0f4917fbd9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:29:23.697883 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:23.697712    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c082ecb-b11c-4454-90b9-6f0f4917fbd9-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3c082ecb-b11c-4454-90b9-6f0f4917fbd9" (UID: "3c082ecb-b11c-4454-90b9-6f0f4917fbd9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:29:23.697883 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:23.697835    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c082ecb-b11c-4454-90b9-6f0f4917fbd9-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3c082ecb-b11c-4454-90b9-6f0f4917fbd9" (UID: "3c082ecb-b11c-4454-90b9-6f0f4917fbd9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:29:23.698001 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:23.697898    2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c082ecb-b11c-4454-90b9-6f0f4917fbd9-workload-socket\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:29:23.698001 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:23.697920    2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c082ecb-b11c-4454-90b9-6f0f4917fbd9-istiod-ca-cert\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:29:23.698001 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:23.697947    2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c082ecb-b11c-4454-90b9-6f0f4917fbd9-credential-socket\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:29:23.698391 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:23.698371    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c082ecb-b11c-4454-90b9-6f0f4917fbd9-istio-data" (OuterVolumeSpecName: "istio-data") pod "3c082ecb-b11c-4454-90b9-6f0f4917fbd9" (UID: "3c082ecb-b11c-4454-90b9-6f0f4917fbd9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:29:23.699553 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:23.699530    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c082ecb-b11c-4454-90b9-6f0f4917fbd9-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3c082ecb-b11c-4454-90b9-6f0f4917fbd9" (UID: "3c082ecb-b11c-4454-90b9-6f0f4917fbd9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:29:23.699809 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:23.699792    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3c082ecb-b11c-4454-90b9-6f0f4917fbd9-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3c082ecb-b11c-4454-90b9-6f0f4917fbd9" (UID: "3c082ecb-b11c-4454-90b9-6f0f4917fbd9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 27 21:29:23.700032 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:23.700007    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c082ecb-b11c-4454-90b9-6f0f4917fbd9-kube-api-access-l74br" (OuterVolumeSpecName: "kube-api-access-l74br") pod "3c082ecb-b11c-4454-90b9-6f0f4917fbd9" (UID: "3c082ecb-b11c-4454-90b9-6f0f4917fbd9"). InnerVolumeSpecName "kube-api-access-l74br". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:29:23.700159 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:23.700032    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c082ecb-b11c-4454-90b9-6f0f4917fbd9-istio-token" (OuterVolumeSpecName: "istio-token") pod "3c082ecb-b11c-4454-90b9-6f0f4917fbd9" (UID: "3c082ecb-b11c-4454-90b9-6f0f4917fbd9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:29:23.798756 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:23.798673    2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c082ecb-b11c-4454-90b9-6f0f4917fbd9-istio-token\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:29:23.798756 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:23.798702    2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c082ecb-b11c-4454-90b9-6f0f4917fbd9-istio-envoy\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:29:23.798756 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:23.798712    2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c082ecb-b11c-4454-90b9-6f0f4917fbd9-workload-certs\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:29:23.798756 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:23.798721    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l74br\" (UniqueName: \"kubernetes.io/projected/3c082ecb-b11c-4454-90b9-6f0f4917fbd9-kube-api-access-l74br\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:29:23.798756 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:23.798732    2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c082ecb-b11c-4454-90b9-6f0f4917fbd9-istio-podinfo\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:29:23.798756 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:23.798740    2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c082ecb-b11c-4454-90b9-6f0f4917fbd9-istio-data\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:29:24.064888 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:24.064793    2560 generic.go:356] "Generic (PLEG): container finished" podID="3c082ecb-b11c-4454-90b9-6f0f4917fbd9" containerID="30a34654cf17f387b1ab7f97e211f1dd34d4fb19ca09aae3692d65efda955c6e" exitCode=0
Jul 27 21:29:24.064888 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:24.064847    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-029d89-openshift-default-58d5f8678-kgp6f"
Jul 27 21:29:24.064888 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:24.064877    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-029d89-openshift-default-58d5f8678-kgp6f" event={"ID":"3c082ecb-b11c-4454-90b9-6f0f4917fbd9","Type":"ContainerDied","Data":"30a34654cf17f387b1ab7f97e211f1dd34d4fb19ca09aae3692d65efda955c6e"}
Jul 27 21:29:24.065209 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:24.064934    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-029d89-openshift-default-58d5f8678-kgp6f" event={"ID":"3c082ecb-b11c-4454-90b9-6f0f4917fbd9","Type":"ContainerDied","Data":"bf1f85c766e96903eac6909f915f1a597148829fdfeddd4e5cb02aaeff8ef468"}
Jul 27 21:29:24.065209 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:24.064959    2560 scope.go:117] "RemoveContainer" containerID="30a34654cf17f387b1ab7f97e211f1dd34d4fb19ca09aae3692d65efda955c6e"
Jul 27 21:29:24.073412 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:24.073394    2560 scope.go:117] "RemoveContainer" containerID="e7319a0c7ddfe37c63309fc95ded212dd7d6b3deddddf4480d9fc1559a21b246"
Jul 27 21:29:24.081044 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:24.081027    2560 scope.go:117] "RemoveContainer" containerID="30a34654cf17f387b1ab7f97e211f1dd34d4fb19ca09aae3692d65efda955c6e"
Jul 27 21:29:24.081302 ip-10-0-141-64 kubenswrapper[2560]: E0727 21:29:24.081279    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30a34654cf17f387b1ab7f97e211f1dd34d4fb19ca09aae3692d65efda955c6e\": container with ID starting with 30a34654cf17f387b1ab7f97e211f1dd34d4fb19ca09aae3692d65efda955c6e not found: ID does not exist" containerID="30a34654cf17f387b1ab7f97e211f1dd34d4fb19ca09aae3692d65efda955c6e"
Jul 27 21:29:24.081360 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:24.081310    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30a34654cf17f387b1ab7f97e211f1dd34d4fb19ca09aae3692d65efda955c6e"} err="failed to get container status \"30a34654cf17f387b1ab7f97e211f1dd34d4fb19ca09aae3692d65efda955c6e\": rpc error: code = NotFound desc = could not find container \"30a34654cf17f387b1ab7f97e211f1dd34d4fb19ca09aae3692d65efda955c6e\": container with ID starting with 30a34654cf17f387b1ab7f97e211f1dd34d4fb19ca09aae3692d65efda955c6e not found: ID does not exist"
Jul 27 21:29:24.081360 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:24.081328    2560 scope.go:117] "RemoveContainer" containerID="e7319a0c7ddfe37c63309fc95ded212dd7d6b3deddddf4480d9fc1559a21b246"
Jul 27 21:29:24.081578 ip-10-0-141-64 kubenswrapper[2560]: E0727 21:29:24.081559    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7319a0c7ddfe37c63309fc95ded212dd7d6b3deddddf4480d9fc1559a21b246\": container with ID starting with e7319a0c7ddfe37c63309fc95ded212dd7d6b3deddddf4480d9fc1559a21b246 not found: ID does not exist" containerID="e7319a0c7ddfe37c63309fc95ded212dd7d6b3deddddf4480d9fc1559a21b246"
Jul 27 21:29:24.081619 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:24.081584    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7319a0c7ddfe37c63309fc95ded212dd7d6b3deddddf4480d9fc1559a21b246"} err="failed to get container status \"e7319a0c7ddfe37c63309fc95ded212dd7d6b3deddddf4480d9fc1559a21b246\": rpc error: code = NotFound desc = could not find container \"e7319a0c7ddfe37c63309fc95ded212dd7d6b3deddddf4480d9fc1559a21b246\": container with ID starting with e7319a0c7ddfe37c63309fc95ded212dd7d6b3deddddf4480d9fc1559a21b246 not found: ID does not exist"
Jul 27 21:29:24.100378 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:24.100349    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-029d89-openshift-default-58d5f8678-kgp6f"]
Jul 27 21:29:24.105969 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:24.105946    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-029d89-openshift-default-58d5f8678-kgp6f"]
Jul 27 21:29:25.847306 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:25.847269    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c082ecb-b11c-4454-90b9-6f0f4917fbd9" path="/var/lib/kubelet/pods/3c082ecb-b11c-4454-90b9-6f0f4917fbd9/volumes"
Jul 27 21:29:32.330599 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:32.330564    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:29:33.689531 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:33.689491    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ccc682-9k4v9"]
Jul 27 21:29:33.694990 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:33.694964    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ccc682-9k4v9"]
Jul 27 21:29:33.847387 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:33.847350    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="214fdede-e5b1-4545-8f18-4e570c3e494d" path="/var/lib/kubelet/pods/214fdede-e5b1-4545-8f18-4e570c3e494d/volumes"
Jul 27 21:29:40.677124 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:40.677073    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-ccc682-openshift-default-858b64fd74-fz6mr"]
Jul 27 21:29:40.677534 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:40.677302    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-ccc682-openshift-default-858b64fd74-fz6mr" podUID="92cacf44-a1b3-4ee5-93e2-fa46a7cee818" containerName="istio-proxy" containerID="cri-o://3bf33ad78a44947407607ed1b7f927a21895abe7d9e26519914113deefff1212" gracePeriod=30
Jul 27 21:29:41.029398 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:41.029362    2560 patch_prober.go:28] interesting pod/e2e-inf-b-ccc682-openshift-default-858b64fd74-fz6mr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 27 21:29:41.029571 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:41.029436    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-ccc682-openshift-default-858b64fd74-fz6mr" podUID="92cacf44-a1b3-4ee5-93e2-fa46a7cee818" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 21:29:41.685854 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:41.685820    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-9d44e2-openshift-default-8584f86688-vvz9k"]
Jul 27 21:29:41.686265 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:41.686190    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c082ecb-b11c-4454-90b9-6f0f4917fbd9" containerName="istio-proxy"
Jul 27 21:29:41.686265 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:41.686203    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c082ecb-b11c-4454-90b9-6f0f4917fbd9" containerName="istio-proxy"
Jul 27 21:29:41.686265 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:41.686217    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="214fdede-e5b1-4545-8f18-4e570c3e494d" containerName="revoke-keys"
Jul 27 21:29:41.686265 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:41.686222    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="214fdede-e5b1-4545-8f18-4e570c3e494d" containerName="revoke-keys"
Jul 27 21:29:41.686399 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:41.686279    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c082ecb-b11c-4454-90b9-6f0f4917fbd9" containerName="istio-proxy"
Jul 27 21:29:41.686399 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:41.686287    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c082ecb-b11c-4454-90b9-6f0f4917fbd9" containerName="istio-proxy"
Jul 27 21:29:41.686399 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:41.686296    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="214fdede-e5b1-4545-8f18-4e570c3e494d" containerName="revoke-keys"
Jul 27 21:29:41.689406 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:41.689389    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-9d44e2-openshift-default-8584f86688-vvz9k"
Jul 27 21:29:41.694099 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:41.694071    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-9d44e2-openshift-default-dockercfg-mzql6\""
Jul 27 21:29:41.707079 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:41.707052    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-9d44e2-openshift-default-8584f86688-vvz9k"]
Jul 27 21:29:41.743804 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:41.743773    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2193cf4-8865-4e4e-bd7c-eb2393be1914-workload-socket\") pod \"e2e-api-a-9d44e2-openshift-default-8584f86688-vvz9k\" (UID: \"f2193cf4-8865-4e4e-bd7c-eb2393be1914\") " pod="openshift-ingress/e2e-api-a-9d44e2-openshift-default-8584f86688-vvz9k"
Jul 27 21:29:41.743804 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:41.743808    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2193cf4-8865-4e4e-bd7c-eb2393be1914-credential-socket\") pod \"e2e-api-a-9d44e2-openshift-default-8584f86688-vvz9k\" (UID: \"f2193cf4-8865-4e4e-bd7c-eb2393be1914\") " pod="openshift-ingress/e2e-api-a-9d44e2-openshift-default-8584f86688-vvz9k"
Jul 27 21:29:41.744012 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:41.743861    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2193cf4-8865-4e4e-bd7c-eb2393be1914-istio-envoy\") pod \"e2e-api-a-9d44e2-openshift-default-8584f86688-vvz9k\" (UID: \"f2193cf4-8865-4e4e-bd7c-eb2393be1914\") " pod="openshift-ingress/e2e-api-a-9d44e2-openshift-default-8584f86688-vvz9k"
Jul 27 21:29:41.744012 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:41.743887    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2193cf4-8865-4e4e-bd7c-eb2393be1914-istio-data\") pod \"e2e-api-a-9d44e2-openshift-default-8584f86688-vvz9k\" (UID: \"f2193cf4-8865-4e4e-bd7c-eb2393be1914\") " pod="openshift-ingress/e2e-api-a-9d44e2-openshift-default-8584f86688-vvz9k"
Jul 27 21:29:41.744012 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:41.743909    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sp662\" (UniqueName: \"kubernetes.io/projected/f2193cf4-8865-4e4e-bd7c-eb2393be1914-kube-api-access-sp662\") pod \"e2e-api-a-9d44e2-openshift-default-8584f86688-vvz9k\" (UID: \"f2193cf4-8865-4e4e-bd7c-eb2393be1914\") " pod="openshift-ingress/e2e-api-a-9d44e2-openshift-default-8584f86688-vvz9k"
Jul 27 21:29:41.744012 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:41.743970    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2193cf4-8865-4e4e-bd7c-eb2393be1914-workload-certs\") pod \"e2e-api-a-9d44e2-openshift-default-8584f86688-vvz9k\" (UID: \"f2193cf4-8865-4e4e-bd7c-eb2393be1914\") " pod="openshift-ingress/e2e-api-a-9d44e2-openshift-default-8584f86688-vvz9k"
Jul 27 21:29:41.744012 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:41.743999    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2193cf4-8865-4e4e-bd7c-eb2393be1914-istio-token\") pod \"e2e-api-a-9d44e2-openshift-default-8584f86688-vvz9k\" (UID: \"f2193cf4-8865-4e4e-bd7c-eb2393be1914\") " pod="openshift-ingress/e2e-api-a-9d44e2-openshift-default-8584f86688-vvz9k"
Jul 27 21:29:41.744315 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:41.744018    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2193cf4-8865-4e4e-bd7c-eb2393be1914-istiod-ca-cert\") pod \"e2e-api-a-9d44e2-openshift-default-8584f86688-vvz9k\" (UID: \"f2193cf4-8865-4e4e-bd7c-eb2393be1914\") " pod="openshift-ingress/e2e-api-a-9d44e2-openshift-default-8584f86688-vvz9k"
Jul 27 21:29:41.744315 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:41.744109    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2193cf4-8865-4e4e-bd7c-eb2393be1914-istio-podinfo\") pod \"e2e-api-a-9d44e2-openshift-default-8584f86688-vvz9k\" (UID: \"f2193cf4-8865-4e4e-bd7c-eb2393be1914\") " pod="openshift-ingress/e2e-api-a-9d44e2-openshift-default-8584f86688-vvz9k"
Jul 27 21:29:41.849631 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:41.847114    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2193cf4-8865-4e4e-bd7c-eb2393be1914-workload-socket\") pod \"e2e-api-a-9d44e2-openshift-default-8584f86688-vvz9k\" (UID: \"f2193cf4-8865-4e4e-bd7c-eb2393be1914\") " pod="openshift-ingress/e2e-api-a-9d44e2-openshift-default-8584f86688-vvz9k"
Jul 27 21:29:41.849631 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:41.847172    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2193cf4-8865-4e4e-bd7c-eb2393be1914-credential-socket\") pod \"e2e-api-a-9d44e2-openshift-default-8584f86688-vvz9k\" (UID: \"f2193cf4-8865-4e4e-bd7c-eb2393be1914\") " pod="openshift-ingress/e2e-api-a-9d44e2-openshift-default-8584f86688-vvz9k"
Jul 27 21:29:41.849631 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:41.847222    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2193cf4-8865-4e4e-bd7c-eb2393be1914-istio-envoy\") pod \"e2e-api-a-9d44e2-openshift-default-8584f86688-vvz9k\" (UID: \"f2193cf4-8865-4e4e-bd7c-eb2393be1914\") " pod="openshift-ingress/e2e-api-a-9d44e2-openshift-default-8584f86688-vvz9k"
Jul 27 21:29:41.849631 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:41.847312    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2193cf4-8865-4e4e-bd7c-eb2393be1914-istio-data\") pod \"e2e-api-a-9d44e2-openshift-default-8584f86688-vvz9k\" (UID: \"f2193cf4-8865-4e4e-bd7c-eb2393be1914\") " pod="openshift-ingress/e2e-api-a-9d44e2-openshift-default-8584f86688-vvz9k"
Jul 27 21:29:41.849631 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:41.847364    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sp662\" (UniqueName: \"kubernetes.io/projected/f2193cf4-8865-4e4e-bd7c-eb2393be1914-kube-api-access-sp662\") pod \"e2e-api-a-9d44e2-openshift-default-8584f86688-vvz9k\" (UID: \"f2193cf4-8865-4e4e-bd7c-eb2393be1914\") " pod="openshift-ingress/e2e-api-a-9d44e2-openshift-default-8584f86688-vvz9k"
Jul 27 21:29:41.849631 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:41.847432    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2193cf4-8865-4e4e-bd7c-eb2393be1914-workload-certs\") pod \"e2e-api-a-9d44e2-openshift-default-8584f86688-vvz9k\" (UID: \"f2193cf4-8865-4e4e-bd7c-eb2393be1914\") " pod="openshift-ingress/e2e-api-a-9d44e2-openshift-default-8584f86688-vvz9k"
Jul 27 21:29:41.849631 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:41.847466    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2193cf4-8865-4e4e-bd7c-eb2393be1914-istio-token\") pod \"e2e-api-a-9d44e2-openshift-default-8584f86688-vvz9k\" (UID: \"f2193cf4-8865-4e4e-bd7c-eb2393be1914\") " pod="openshift-ingress/e2e-api-a-9d44e2-openshift-default-8584f86688-vvz9k"
Jul 27 21:29:41.849631 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:41.847540    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2193cf4-8865-4e4e-bd7c-eb2393be1914-istiod-ca-cert\") pod \"e2e-api-a-9d44e2-openshift-default-8584f86688-vvz9k\" (UID: \"f2193cf4-8865-4e4e-bd7c-eb2393be1914\") " pod="openshift-ingress/e2e-api-a-9d44e2-openshift-default-8584f86688-vvz9k"
Jul 27 21:29:41.849631 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:41.847619    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2193cf4-8865-4e4e-bd7c-eb2393be1914-istio-podinfo\") pod \"e2e-api-a-9d44e2-openshift-default-8584f86688-vvz9k\" (UID: \"f2193cf4-8865-4e4e-bd7c-eb2393be1914\") " pod="openshift-ingress/e2e-api-a-9d44e2-openshift-default-8584f86688-vvz9k"
Jul 27 21:29:41.849631 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:41.848167    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2193cf4-8865-4e4e-bd7c-eb2393be1914-workload-socket\") pod \"e2e-api-a-9d44e2-openshift-default-8584f86688-vvz9k\" (UID: \"f2193cf4-8865-4e4e-bd7c-eb2393be1914\") " pod="openshift-ingress/e2e-api-a-9d44e2-openshift-default-8584f86688-vvz9k"
Jul 27 21:29:41.849631 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:41.849619    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2193cf4-8865-4e4e-bd7c-eb2393be1914-workload-certs\") pod \"e2e-api-a-9d44e2-openshift-default-8584f86688-vvz9k\" (UID: \"f2193cf4-8865-4e4e-bd7c-eb2393be1914\") " pod="openshift-ingress/e2e-api-a-9d44e2-openshift-default-8584f86688-vvz9k"
Jul 27 21:29:41.850322 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:41.849911    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2193cf4-8865-4e4e-bd7c-eb2393be1914-istiod-ca-cert\") pod \"e2e-api-a-9d44e2-openshift-default-8584f86688-vvz9k\" (UID: \"f2193cf4-8865-4e4e-bd7c-eb2393be1914\") " pod="openshift-ingress/e2e-api-a-9d44e2-openshift-default-8584f86688-vvz9k"
Jul 27 21:29:41.850381 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:41.850342    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2193cf4-8865-4e4e-bd7c-eb2393be1914-credential-socket\") pod \"e2e-api-a-9d44e2-openshift-default-8584f86688-vvz9k\" (UID: \"f2193cf4-8865-4e4e-bd7c-eb2393be1914\") " pod="openshift-ingress/e2e-api-a-9d44e2-openshift-default-8584f86688-vvz9k"
Jul 27 21:29:41.851287 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:41.851214    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2193cf4-8865-4e4e-bd7c-eb2393be1914-istio-data\") pod \"e2e-api-a-9d44e2-openshift-default-8584f86688-vvz9k\" (UID: \"f2193cf4-8865-4e4e-bd7c-eb2393be1914\") " pod="openshift-ingress/e2e-api-a-9d44e2-openshift-default-8584f86688-vvz9k"
Jul 27 21:29:41.854473 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:41.853528    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2193cf4-8865-4e4e-bd7c-eb2393be1914-istio-envoy\") pod \"e2e-api-a-9d44e2-openshift-default-8584f86688-vvz9k\" (UID: \"f2193cf4-8865-4e4e-bd7c-eb2393be1914\") " pod="openshift-ingress/e2e-api-a-9d44e2-openshift-default-8584f86688-vvz9k"
Jul 27 21:29:41.854473 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:41.853637    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2193cf4-8865-4e4e-bd7c-eb2393be1914-istio-podinfo\") pod \"e2e-api-a-9d44e2-openshift-default-8584f86688-vvz9k\" (UID: \"f2193cf4-8865-4e4e-bd7c-eb2393be1914\") " pod="openshift-ingress/e2e-api-a-9d44e2-openshift-default-8584f86688-vvz9k"
Jul 27 21:29:41.862903 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:41.862876    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2193cf4-8865-4e4e-bd7c-eb2393be1914-istio-token\") pod \"e2e-api-a-9d44e2-openshift-default-8584f86688-vvz9k\" (UID: \"f2193cf4-8865-4e4e-bd7c-eb2393be1914\") " pod="openshift-ingress/e2e-api-a-9d44e2-openshift-default-8584f86688-vvz9k"
Jul 27 21:29:41.864026 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:41.864002    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sp662\" (UniqueName: \"kubernetes.io/projected/f2193cf4-8865-4e4e-bd7c-eb2393be1914-kube-api-access-sp662\") pod \"e2e-api-a-9d44e2-openshift-default-8584f86688-vvz9k\" (UID: \"f2193cf4-8865-4e4e-bd7c-eb2393be1914\") " pod="openshift-ingress/e2e-api-a-9d44e2-openshift-default-8584f86688-vvz9k"
Jul 27 21:29:41.999030 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:41.998922    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-9d44e2-openshift-default-8584f86688-vvz9k"
Jul 27 21:29:42.147907 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:42.147885    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-9d44e2-openshift-default-8584f86688-vvz9k"]
Jul 27 21:29:42.150019 ip-10-0-141-64 kubenswrapper[2560]: W0727 21:29:42.149990    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2193cf4_8865_4e4e_bd7c_eb2393be1914.slice/crio-6aabdcb70380ad2a52d9f1650862b774faf3560a9638788ae6ac0d2a847d3aae WatchSource:0}: Error finding container 6aabdcb70380ad2a52d9f1650862b774faf3560a9638788ae6ac0d2a847d3aae: Status 404 returned error can't find the container with id 6aabdcb70380ad2a52d9f1650862b774faf3560a9638788ae6ac0d2a847d3aae
Jul 27 21:29:42.151901 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:42.151866    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 27 21:29:42.151996 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:42.151937    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 27 21:29:42.151996 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:42.151968    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 27 21:29:43.156578 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:43.156532    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9d44e2-openshift-default-8584f86688-vvz9k" event={"ID":"f2193cf4-8865-4e4e-bd7c-eb2393be1914","Type":"ContainerStarted","Data":"51e520270f16cbe7473ec4e04cd35ac94f4ff799637daabb41d8897f724f5475"}
Jul 27 21:29:43.156980 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:43.156600    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9d44e2-openshift-default-8584f86688-vvz9k" event={"ID":"f2193cf4-8865-4e4e-bd7c-eb2393be1914","Type":"ContainerStarted","Data":"6aabdcb70380ad2a52d9f1650862b774faf3560a9638788ae6ac0d2a847d3aae"}
Jul 27 21:29:43.183101 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:43.183042    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-9d44e2-openshift-default-8584f86688-vvz9k" podStartSLOduration=2.183026146 podStartE2EDuration="2.183026146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:29:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:29:43.181195753 +0000 UTC m=+2299.933057559" watchObservedRunningTime="2026-07-27 21:29:43.183026146 +0000 UTC m=+2299.934888019"
Jul 27 21:29:43.999443 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:43.999394    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-9d44e2-openshift-default-8584f86688-vvz9k"
Jul 27 21:29:44.004612 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:44.004580    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-9d44e2-openshift-default-8584f86688-vvz9k"
Jul 27 21:29:44.162525 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:44.162493    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-9d44e2-openshift-default-8584f86688-vvz9k"
Jul 27 21:29:44.163548 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:44.163525    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-9d44e2-openshift-default-8584f86688-vvz9k"
Jul 27 21:29:45.950347 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:45.950273    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ccc682-openshift-default-858b64fd74-fz6mr"
Jul 27 21:29:45.983168 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:45.983139    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92cacf44-a1b3-4ee5-93e2-fa46a7cee818-istio-envoy\") pod \"92cacf44-a1b3-4ee5-93e2-fa46a7cee818\" (UID: \"92cacf44-a1b3-4ee5-93e2-fa46a7cee818\") "
Jul 27 21:29:45.983343 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:45.983182    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92cacf44-a1b3-4ee5-93e2-fa46a7cee818-istio-token\") pod \"92cacf44-a1b3-4ee5-93e2-fa46a7cee818\" (UID: \"92cacf44-a1b3-4ee5-93e2-fa46a7cee818\") "
Jul 27 21:29:45.983343 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:45.983209    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92cacf44-a1b3-4ee5-93e2-fa46a7cee818-istio-data\") pod \"92cacf44-a1b3-4ee5-93e2-fa46a7cee818\" (UID: \"92cacf44-a1b3-4ee5-93e2-fa46a7cee818\") "
Jul 27 21:29:45.983343 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:45.983243    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bq85w\" (UniqueName: \"kubernetes.io/projected/92cacf44-a1b3-4ee5-93e2-fa46a7cee818-kube-api-access-bq85w\") pod \"92cacf44-a1b3-4ee5-93e2-fa46a7cee818\" (UID: \"92cacf44-a1b3-4ee5-93e2-fa46a7cee818\") "
Jul 27 21:29:45.983343 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:45.983265    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92cacf44-a1b3-4ee5-93e2-fa46a7cee818-workload-certs\") pod \"92cacf44-a1b3-4ee5-93e2-fa46a7cee818\" (UID: \"92cacf44-a1b3-4ee5-93e2-fa46a7cee818\") "
Jul 27 21:29:45.983571 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:45.983343    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92cacf44-a1b3-4ee5-93e2-fa46a7cee818-istiod-ca-cert\") pod \"92cacf44-a1b3-4ee5-93e2-fa46a7cee818\" (UID: \"92cacf44-a1b3-4ee5-93e2-fa46a7cee818\") "
Jul 27 21:29:45.983571 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:45.983432    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92cacf44-a1b3-4ee5-93e2-fa46a7cee818-credential-socket\") pod \"92cacf44-a1b3-4ee5-93e2-fa46a7cee818\" (UID: \"92cacf44-a1b3-4ee5-93e2-fa46a7cee818\") "
Jul 27 21:29:45.983571 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:45.983494    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92cacf44-a1b3-4ee5-93e2-fa46a7cee818-istio-podinfo\") pod \"92cacf44-a1b3-4ee5-93e2-fa46a7cee818\" (UID: \"92cacf44-a1b3-4ee5-93e2-fa46a7cee818\") "
Jul 27 21:29:45.983571 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:45.983539    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92cacf44-a1b3-4ee5-93e2-fa46a7cee818-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "92cacf44-a1b3-4ee5-93e2-fa46a7cee818" (UID: "92cacf44-a1b3-4ee5-93e2-fa46a7cee818"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:29:45.983571 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:45.983547    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92cacf44-a1b3-4ee5-93e2-fa46a7cee818-workload-socket\") pod \"92cacf44-a1b3-4ee5-93e2-fa46a7cee818\" (UID: \"92cacf44-a1b3-4ee5-93e2-fa46a7cee818\") "
Jul 27 21:29:45.983814 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:45.983732    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92cacf44-a1b3-4ee5-93e2-fa46a7cee818-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "92cacf44-a1b3-4ee5-93e2-fa46a7cee818" (UID: "92cacf44-a1b3-4ee5-93e2-fa46a7cee818"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:29:45.983952 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:45.983882    2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92cacf44-a1b3-4ee5-93e2-fa46a7cee818-workload-certs\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:29:45.984005 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:45.983926    2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92cacf44-a1b3-4ee5-93e2-fa46a7cee818-credential-socket\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:29:45.984005 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:45.983989    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92cacf44-a1b3-4ee5-93e2-fa46a7cee818-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "92cacf44-a1b3-4ee5-93e2-fa46a7cee818" (UID: "92cacf44-a1b3-4ee5-93e2-fa46a7cee818"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:29:45.984150 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:45.984130    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92cacf44-a1b3-4ee5-93e2-fa46a7cee818-istio-data" (OuterVolumeSpecName: "istio-data") pod "92cacf44-a1b3-4ee5-93e2-fa46a7cee818" (UID: "92cacf44-a1b3-4ee5-93e2-fa46a7cee818"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:29:45.984420 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:45.984392    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92cacf44-a1b3-4ee5-93e2-fa46a7cee818-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "92cacf44-a1b3-4ee5-93e2-fa46a7cee818" (UID: "92cacf44-a1b3-4ee5-93e2-fa46a7cee818"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:29:45.985889 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:45.985857    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92cacf44-a1b3-4ee5-93e2-fa46a7cee818-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "92cacf44-a1b3-4ee5-93e2-fa46a7cee818" (UID: "92cacf44-a1b3-4ee5-93e2-fa46a7cee818"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:29:45.985981 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:45.985904    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92cacf44-a1b3-4ee5-93e2-fa46a7cee818-istio-token" (OuterVolumeSpecName: "istio-token") pod "92cacf44-a1b3-4ee5-93e2-fa46a7cee818" (UID: "92cacf44-a1b3-4ee5-93e2-fa46a7cee818"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:29:45.986241 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:45.986219    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92cacf44-a1b3-4ee5-93e2-fa46a7cee818-kube-api-access-bq85w" (OuterVolumeSpecName: "kube-api-access-bq85w") pod "92cacf44-a1b3-4ee5-93e2-fa46a7cee818" (UID: "92cacf44-a1b3-4ee5-93e2-fa46a7cee818"). InnerVolumeSpecName "kube-api-access-bq85w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:29:45.986241 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:45.986233    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/92cacf44-a1b3-4ee5-93e2-fa46a7cee818-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "92cacf44-a1b3-4ee5-93e2-fa46a7cee818" (UID: "92cacf44-a1b3-4ee5-93e2-fa46a7cee818"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 27 21:29:46.084675 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:46.084636    2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92cacf44-a1b3-4ee5-93e2-fa46a7cee818-workload-socket\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:29:46.084675 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:46.084664    2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92cacf44-a1b3-4ee5-93e2-fa46a7cee818-istio-envoy\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:29:46.084675 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:46.084677    2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92cacf44-a1b3-4ee5-93e2-fa46a7cee818-istio-token\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:29:46.084901 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:46.084689    2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92cacf44-a1b3-4ee5-93e2-fa46a7cee818-istio-data\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:29:46.084901 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:46.084697    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bq85w\" (UniqueName: \"kubernetes.io/projected/92cacf44-a1b3-4ee5-93e2-fa46a7cee818-kube-api-access-bq85w\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:29:46.084901 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:46.084706    2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92cacf44-a1b3-4ee5-93e2-fa46a7cee818-istiod-ca-cert\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:29:46.084901 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:46.084714    2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92cacf44-a1b3-4ee5-93e2-fa46a7cee818-istio-podinfo\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:29:46.174316 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:46.174281    2560 generic.go:356] "Generic (PLEG): container finished" podID="92cacf44-a1b3-4ee5-93e2-fa46a7cee818" containerID="3bf33ad78a44947407607ed1b7f927a21895abe7d9e26519914113deefff1212" exitCode=0
Jul 27 21:29:46.174482 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:46.174355    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ccc682-openshift-default-858b64fd74-fz6mr"
Jul 27 21:29:46.174482 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:46.174375    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ccc682-openshift-default-858b64fd74-fz6mr" event={"ID":"92cacf44-a1b3-4ee5-93e2-fa46a7cee818","Type":"ContainerDied","Data":"3bf33ad78a44947407607ed1b7f927a21895abe7d9e26519914113deefff1212"}
Jul 27 21:29:46.174482 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:46.174444    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ccc682-openshift-default-858b64fd74-fz6mr" event={"ID":"92cacf44-a1b3-4ee5-93e2-fa46a7cee818","Type":"ContainerDied","Data":"9cb9a580aa2829be104bb8f2d4562c3dcd88f84e1dc540e430f93d0c2db6e789"}
Jul 27 21:29:46.174482 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:46.174480    2560 scope.go:117] "RemoveContainer" containerID="3bf33ad78a44947407607ed1b7f927a21895abe7d9e26519914113deefff1212"
Jul 27 21:29:46.185799 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:46.185782    2560 scope.go:117] "RemoveContainer" containerID="9829a29269c1d03198abe002f624599214b8bf3db74030b82d0192d48d23ab9b"
Jul 27 21:29:46.194362 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:46.194343    2560 scope.go:117] "RemoveContainer" containerID="3bf33ad78a44947407607ed1b7f927a21895abe7d9e26519914113deefff1212"
Jul 27 21:29:46.194626 ip-10-0-141-64 kubenswrapper[2560]: E0727 21:29:46.194596    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3bf33ad78a44947407607ed1b7f927a21895abe7d9e26519914113deefff1212\": container with ID starting with 3bf33ad78a44947407607ed1b7f927a21895abe7d9e26519914113deefff1212 not found: ID does not exist" containerID="3bf33ad78a44947407607ed1b7f927a21895abe7d9e26519914113deefff1212"
Jul 27 21:29:46.194736 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:46.194634    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bf33ad78a44947407607ed1b7f927a21895abe7d9e26519914113deefff1212"} err="failed to get container status \"3bf33ad78a44947407607ed1b7f927a21895abe7d9e26519914113deefff1212\": rpc error: code = NotFound desc = could not find container \"3bf33ad78a44947407607ed1b7f927a21895abe7d9e26519914113deefff1212\": container with ID starting with 3bf33ad78a44947407607ed1b7f927a21895abe7d9e26519914113deefff1212 not found: ID does not exist"
Jul 27 21:29:46.194736 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:46.194652    2560 scope.go:117] "RemoveContainer" containerID="9829a29269c1d03198abe002f624599214b8bf3db74030b82d0192d48d23ab9b"
Jul 27 21:29:46.194913 ip-10-0-141-64 kubenswrapper[2560]: E0727 21:29:46.194886    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9829a29269c1d03198abe002f624599214b8bf3db74030b82d0192d48d23ab9b\": container with ID starting with 9829a29269c1d03198abe002f624599214b8bf3db74030b82d0192d48d23ab9b not found: ID does not exist" containerID="9829a29269c1d03198abe002f624599214b8bf3db74030b82d0192d48d23ab9b"
Jul 27 21:29:46.194952 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:46.194917    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9829a29269c1d03198abe002f624599214b8bf3db74030b82d0192d48d23ab9b"} err="failed to get container status \"9829a29269c1d03198abe002f624599214b8bf3db74030b82d0192d48d23ab9b\": rpc error: code = NotFound desc = could not find container \"9829a29269c1d03198abe002f624599214b8bf3db74030b82d0192d48d23ab9b\": container with ID starting with 9829a29269c1d03198abe002f624599214b8bf3db74030b82d0192d48d23ab9b not found: ID does not exist"
Jul 27 21:29:46.222460 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:46.222402    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-ccc682-openshift-default-858b64fd74-fz6mr"]
Jul 27 21:29:46.236227 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:46.236204    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-ccc682-openshift-default-858b64fd74-fz6mr"]
Jul 27 21:29:47.847901 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:47.847865    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="92cacf44-a1b3-4ee5-93e2-fa46a7cee818" path="/var/lib/kubelet/pods/92cacf44-a1b3-4ee5-93e2-fa46a7cee818/volumes"
Jul 27 21:29:50.518825 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:50.518789    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-55137c-openshift-default-866bbfc94-clccr"]
Jul 27 21:29:50.519258 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:50.519183    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c082ecb-b11c-4454-90b9-6f0f4917fbd9" containerName="istio-proxy"
Jul 27 21:29:50.519258 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:50.519197    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c082ecb-b11c-4454-90b9-6f0f4917fbd9" containerName="istio-proxy"
Jul 27 21:29:50.519258 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:50.519209    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92cacf44-a1b3-4ee5-93e2-fa46a7cee818" containerName="istio-proxy"
Jul 27 21:29:50.519258 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:50.519215    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="92cacf44-a1b3-4ee5-93e2-fa46a7cee818" containerName="istio-proxy"
Jul 27 21:29:50.519258 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:50.519221    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92cacf44-a1b3-4ee5-93e2-fa46a7cee818" containerName="istio-proxy"
Jul 27 21:29:50.519258 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:50.519227    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="92cacf44-a1b3-4ee5-93e2-fa46a7cee818" containerName="istio-proxy"
Jul 27 21:29:50.519446 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:50.519287    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="92cacf44-a1b3-4ee5-93e2-fa46a7cee818" containerName="istio-proxy"
Jul 27 21:29:50.519446 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:50.519296    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="92cacf44-a1b3-4ee5-93e2-fa46a7cee818" containerName="istio-proxy"
Jul 27 21:29:50.523810 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:50.523786    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-55137c-openshift-default-866bbfc94-clccr"
Jul 27 21:29:50.526961 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:50.526942    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-55137c-openshift-default-dockercfg-q479j\""
Jul 27 21:29:50.543394 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:50.543370    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-55137c-openshift-default-866bbfc94-clccr"]
Jul 27 21:29:50.557384 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:50.557356    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:29:50.623593 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:50.623552    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce51fc53-45c9-4c8a-81a9-65c5407a7b0d-istio-podinfo\") pod \"e2e-api-b-55137c-openshift-default-866bbfc94-clccr\" (UID: \"ce51fc53-45c9-4c8a-81a9-65c5407a7b0d\") " pod="openshift-ingress/e2e-api-b-55137c-openshift-default-866bbfc94-clccr"
Jul 27 21:29:50.623780 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:50.623610    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlr7h\" (UniqueName: \"kubernetes.io/projected/ce51fc53-45c9-4c8a-81a9-65c5407a7b0d-kube-api-access-qlr7h\") pod \"e2e-api-b-55137c-openshift-default-866bbfc94-clccr\" (UID: \"ce51fc53-45c9-4c8a-81a9-65c5407a7b0d\") " pod="openshift-ingress/e2e-api-b-55137c-openshift-default-866bbfc94-clccr"
Jul 27 21:29:50.623780 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:50.623657    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce51fc53-45c9-4c8a-81a9-65c5407a7b0d-istiod-ca-cert\") pod \"e2e-api-b-55137c-openshift-default-866bbfc94-clccr\" (UID: \"ce51fc53-45c9-4c8a-81a9-65c5407a7b0d\") " pod="openshift-ingress/e2e-api-b-55137c-openshift-default-866bbfc94-clccr"
Jul 27 21:29:50.623780 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:50.623685    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce51fc53-45c9-4c8a-81a9-65c5407a7b0d-credential-socket\") pod \"e2e-api-b-55137c-openshift-default-866bbfc94-clccr\" (UID: \"ce51fc53-45c9-4c8a-81a9-65c5407a7b0d\") " pod="openshift-ingress/e2e-api-b-55137c-openshift-default-866bbfc94-clccr"
Jul 27 21:29:50.623780 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:50.623704    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce51fc53-45c9-4c8a-81a9-65c5407a7b0d-istio-envoy\") pod \"e2e-api-b-55137c-openshift-default-866bbfc94-clccr\" (UID: \"ce51fc53-45c9-4c8a-81a9-65c5407a7b0d\") " pod="openshift-ingress/e2e-api-b-55137c-openshift-default-866bbfc94-clccr"
Jul 27 21:29:50.623780 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:50.623722    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce51fc53-45c9-4c8a-81a9-65c5407a7b0d-istio-token\") pod \"e2e-api-b-55137c-openshift-default-866bbfc94-clccr\" (UID: \"ce51fc53-45c9-4c8a-81a9-65c5407a7b0d\") " pod="openshift-ingress/e2e-api-b-55137c-openshift-default-866bbfc94-clccr"
Jul 27 21:29:50.623780 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:50.623750    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce51fc53-45c9-4c8a-81a9-65c5407a7b0d-workload-socket\") pod \"e2e-api-b-55137c-openshift-default-866bbfc94-clccr\" (UID: \"ce51fc53-45c9-4c8a-81a9-65c5407a7b0d\") " pod="openshift-ingress/e2e-api-b-55137c-openshift-default-866bbfc94-clccr"
Jul 27 21:29:50.624019 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:50.623788    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce51fc53-45c9-4c8a-81a9-65c5407a7b0d-istio-data\") pod \"e2e-api-b-55137c-openshift-default-866bbfc94-clccr\" (UID: \"ce51fc53-45c9-4c8a-81a9-65c5407a7b0d\") " pod="openshift-ingress/e2e-api-b-55137c-openshift-default-866bbfc94-clccr"
Jul 27 21:29:50.624019 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:50.623820    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce51fc53-45c9-4c8a-81a9-65c5407a7b0d-workload-certs\") pod \"e2e-api-b-55137c-openshift-default-866bbfc94-clccr\" (UID: \"ce51fc53-45c9-4c8a-81a9-65c5407a7b0d\") " pod="openshift-ingress/e2e-api-b-55137c-openshift-default-866bbfc94-clccr"
Jul 27 21:29:50.724874 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:50.724834    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce51fc53-45c9-4c8a-81a9-65c5407a7b0d-istio-podinfo\") pod \"e2e-api-b-55137c-openshift-default-866bbfc94-clccr\" (UID: \"ce51fc53-45c9-4c8a-81a9-65c5407a7b0d\") " pod="openshift-ingress/e2e-api-b-55137c-openshift-default-866bbfc94-clccr"
Jul 27 21:29:50.725042 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:50.724885    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlr7h\" (UniqueName: \"kubernetes.io/projected/ce51fc53-45c9-4c8a-81a9-65c5407a7b0d-kube-api-access-qlr7h\") pod \"e2e-api-b-55137c-openshift-default-866bbfc94-clccr\" (UID: \"ce51fc53-45c9-4c8a-81a9-65c5407a7b0d\") " pod="openshift-ingress/e2e-api-b-55137c-openshift-default-866bbfc94-clccr"
Jul 27 21:29:50.725042 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:50.724932    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce51fc53-45c9-4c8a-81a9-65c5407a7b0d-istiod-ca-cert\") pod \"e2e-api-b-55137c-openshift-default-866bbfc94-clccr\" (UID: \"ce51fc53-45c9-4c8a-81a9-65c5407a7b0d\") " pod="openshift-ingress/e2e-api-b-55137c-openshift-default-866bbfc94-clccr"
Jul 27 21:29:50.725042 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:50.724963    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce51fc53-45c9-4c8a-81a9-65c5407a7b0d-credential-socket\") pod \"e2e-api-b-55137c-openshift-default-866bbfc94-clccr\" (UID: \"ce51fc53-45c9-4c8a-81a9-65c5407a7b0d\") " pod="openshift-ingress/e2e-api-b-55137c-openshift-default-866bbfc94-clccr"
Jul 27 21:29:50.725042 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:50.724984    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce51fc53-45c9-4c8a-81a9-65c5407a7b0d-istio-envoy\") pod \"e2e-api-b-55137c-openshift-default-866bbfc94-clccr\" (UID: \"ce51fc53-45c9-4c8a-81a9-65c5407a7b0d\") " pod="openshift-ingress/e2e-api-b-55137c-openshift-default-866bbfc94-clccr"
Jul 27 21:29:50.725042 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:50.725007    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce51fc53-45c9-4c8a-81a9-65c5407a7b0d-istio-token\") pod \"e2e-api-b-55137c-openshift-default-866bbfc94-clccr\" (UID: \"ce51fc53-45c9-4c8a-81a9-65c5407a7b0d\") " pod="openshift-ingress/e2e-api-b-55137c-openshift-default-866bbfc94-clccr"
Jul 27 21:29:50.725400 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:50.725145    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce51fc53-45c9-4c8a-81a9-65c5407a7b0d-workload-socket\") pod \"e2e-api-b-55137c-openshift-default-866bbfc94-clccr\" (UID: \"ce51fc53-45c9-4c8a-81a9-65c5407a7b0d\") " pod="openshift-ingress/e2e-api-b-55137c-openshift-default-866bbfc94-clccr"
Jul 27 21:29:50.725400 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:50.725207    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce51fc53-45c9-4c8a-81a9-65c5407a7b0d-istio-data\") pod \"e2e-api-b-55137c-openshift-default-866bbfc94-clccr\" (UID: \"ce51fc53-45c9-4c8a-81a9-65c5407a7b0d\") " pod="openshift-ingress/e2e-api-b-55137c-openshift-default-866bbfc94-clccr"
Jul 27 21:29:50.725400 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:50.725249    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce51fc53-45c9-4c8a-81a9-65c5407a7b0d-workload-certs\") pod \"e2e-api-b-55137c-openshift-default-866bbfc94-clccr\" (UID: \"ce51fc53-45c9-4c8a-81a9-65c5407a7b0d\") " pod="openshift-ingress/e2e-api-b-55137c-openshift-default-866bbfc94-clccr"
Jul 27 21:29:50.725571 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:50.725439    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce51fc53-45c9-4c8a-81a9-65c5407a7b0d-credential-socket\") pod \"e2e-api-b-55137c-openshift-default-866bbfc94-clccr\" (UID: \"ce51fc53-45c9-4c8a-81a9-65c5407a7b0d\") " pod="openshift-ingress/e2e-api-b-55137c-openshift-default-866bbfc94-clccr"
Jul 27 21:29:50.725571 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:50.725536    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce51fc53-45c9-4c8a-81a9-65c5407a7b0d-workload-socket\") pod \"e2e-api-b-55137c-openshift-default-866bbfc94-clccr\" (UID: \"ce51fc53-45c9-4c8a-81a9-65c5407a7b0d\") " pod="openshift-ingress/e2e-api-b-55137c-openshift-default-866bbfc94-clccr"
Jul 27 21:29:50.725648 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:50.725636    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce51fc53-45c9-4c8a-81a9-65c5407a7b0d-workload-certs\") pod \"e2e-api-b-55137c-openshift-default-866bbfc94-clccr\" (UID: \"ce51fc53-45c9-4c8a-81a9-65c5407a7b0d\") " pod="openshift-ingress/e2e-api-b-55137c-openshift-default-866bbfc94-clccr"
Jul 27 21:29:50.725696 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:50.725633    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce51fc53-45c9-4c8a-81a9-65c5407a7b0d-istio-data\") pod \"e2e-api-b-55137c-openshift-default-866bbfc94-clccr\" (UID: \"ce51fc53-45c9-4c8a-81a9-65c5407a7b0d\") " pod="openshift-ingress/e2e-api-b-55137c-openshift-default-866bbfc94-clccr"
Jul 27 21:29:50.726014 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:50.725994    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce51fc53-45c9-4c8a-81a9-65c5407a7b0d-istiod-ca-cert\") pod \"e2e-api-b-55137c-openshift-default-866bbfc94-clccr\" (UID: \"ce51fc53-45c9-4c8a-81a9-65c5407a7b0d\") " pod="openshift-ingress/e2e-api-b-55137c-openshift-default-866bbfc94-clccr"
Jul 27 21:29:50.727411 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:50.727384    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce51fc53-45c9-4c8a-81a9-65c5407a7b0d-istio-envoy\") pod \"e2e-api-b-55137c-openshift-default-866bbfc94-clccr\" (UID: \"ce51fc53-45c9-4c8a-81a9-65c5407a7b0d\") " pod="openshift-ingress/e2e-api-b-55137c-openshift-default-866bbfc94-clccr"
Jul 27 21:29:50.727608 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:50.727593    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce51fc53-45c9-4c8a-81a9-65c5407a7b0d-istio-podinfo\") pod \"e2e-api-b-55137c-openshift-default-866bbfc94-clccr\" (UID: \"ce51fc53-45c9-4c8a-81a9-65c5407a7b0d\") " pod="openshift-ingress/e2e-api-b-55137c-openshift-default-866bbfc94-clccr"
Jul 27 21:29:50.735379 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:50.735354    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce51fc53-45c9-4c8a-81a9-65c5407a7b0d-istio-token\") pod \"e2e-api-b-55137c-openshift-default-866bbfc94-clccr\" (UID: \"ce51fc53-45c9-4c8a-81a9-65c5407a7b0d\") " pod="openshift-ingress/e2e-api-b-55137c-openshift-default-866bbfc94-clccr"
Jul 27 21:29:50.737364 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:50.737342    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlr7h\" (UniqueName: \"kubernetes.io/projected/ce51fc53-45c9-4c8a-81a9-65c5407a7b0d-kube-api-access-qlr7h\") pod \"e2e-api-b-55137c-openshift-default-866bbfc94-clccr\" (UID: \"ce51fc53-45c9-4c8a-81a9-65c5407a7b0d\") " pod="openshift-ingress/e2e-api-b-55137c-openshift-default-866bbfc94-clccr"
Jul 27 21:29:50.833710 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:50.833621    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-55137c-openshift-default-866bbfc94-clccr"
Jul 27 21:29:50.981866 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:50.981742    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-55137c-openshift-default-866bbfc94-clccr"]
Jul 27 21:29:50.984692 ip-10-0-141-64 kubenswrapper[2560]: W0727 21:29:50.984643    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce51fc53_45c9_4c8a_81a9_65c5407a7b0d.slice/crio-84eb79c99ff446ff148cf7cd2bd1e90bb45701ac843eee34e2c8a62f3223c312 WatchSource:0}: Error finding container 84eb79c99ff446ff148cf7cd2bd1e90bb45701ac843eee34e2c8a62f3223c312: Status 404 returned error can't find the container with id 84eb79c99ff446ff148cf7cd2bd1e90bb45701ac843eee34e2c8a62f3223c312
Jul 27 21:29:50.987011 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:50.986973    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 27 21:29:50.987149 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:50.987053    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 27 21:29:50.987149 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:50.987115    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 27 21:29:51.201526 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:51.201413    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-55137c-openshift-default-866bbfc94-clccr" event={"ID":"ce51fc53-45c9-4c8a-81a9-65c5407a7b0d","Type":"ContainerStarted","Data":"eab37c5c6c4303bb2b21fc9a5a569b649e5479160a1ffa93bd91a5bc0e3a3542"}
Jul 27 21:29:51.201526 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:51.201521    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-55137c-openshift-default-866bbfc94-clccr" event={"ID":"ce51fc53-45c9-4c8a-81a9-65c5407a7b0d","Type":"ContainerStarted","Data":"84eb79c99ff446ff148cf7cd2bd1e90bb45701ac843eee34e2c8a62f3223c312"}
Jul 27 21:29:51.233756 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:51.233679    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-55137c-openshift-default-866bbfc94-clccr" podStartSLOduration=1.233656257 podStartE2EDuration="1.233656257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:29:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:29:51.231275084 +0000 UTC m=+2307.983136880" watchObservedRunningTime="2026-07-27 21:29:51.233656257 +0000 UTC m=+2307.985518053"
Jul 27 21:29:51.834043 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:51.834005    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-55137c-openshift-default-866bbfc94-clccr"
Jul 27 21:29:52.843607 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:52.843569    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-55137c-openshift-default-866bbfc94-clccr"
Jul 27 21:29:53.212770 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:53.212685    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-55137c-openshift-default-866bbfc94-clccr"
Jul 27 21:29:53.213805 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:53.213785    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-55137c-openshift-default-866bbfc94-clccr"
Jul 27 21:29:58.238897 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:58.238793    2560 generic.go:356] "Generic (PLEG): container finished" podID="f2193cf4-8865-4e4e-bd7c-eb2393be1914" containerID="51e520270f16cbe7473ec4e04cd35ac94f4ff799637daabb41d8897f724f5475" exitCode=0
Jul 27 21:29:58.238897 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:58.238862    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9d44e2-openshift-default-8584f86688-vvz9k" event={"ID":"f2193cf4-8865-4e4e-bd7c-eb2393be1914","Type":"ContainerDied","Data":"51e520270f16cbe7473ec4e04cd35ac94f4ff799637daabb41d8897f724f5475"}
Jul 27 21:29:58.239393 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:58.239278    2560 scope.go:117] "RemoveContainer" containerID="51e520270f16cbe7473ec4e04cd35ac94f4ff799637daabb41d8897f724f5475"
Jul 27 21:29:58.240343 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:58.240312    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 27 21:29:58.240445 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:58.240380    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 27 21:29:58.240445 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:58.240412    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 27 21:29:59.163242 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:59.163195    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-9d44e2-openshift-default-8584f86688-vvz9k"
Jul 27 21:29:59.252969 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:59.252916    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9d44e2-openshift-default-8584f86688-vvz9k" event={"ID":"f2193cf4-8865-4e4e-bd7c-eb2393be1914","Type":"ContainerStarted","Data":"e51afcd1d5b59d62fc973567226205ebecda48b1b3498ae5e5b0607c4785fa92"}
Jul 27 21:29:59.999804 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:29:59.999762    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-9d44e2-openshift-default-8584f86688-vvz9k"
Jul 27 21:30:00.000995 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:30:00.000966    2560 patch_prober.go:28] interesting pod/e2e-api-a-9d44e2-openshift-default-8584f86688-vvz9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Jul 27 21:30:00.001158 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:30:00.001021    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9d44e2-openshift-default-8584f86688-vvz9k" podUID="f2193cf4-8865-4e4e-bd7c-eb2393be1914" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Jul 27 21:30:00.137822 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:30:00.137779    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29753130-54ln4"]
Jul 27 21:30:00.140506 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:30:00.140473    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753130-54ln4"
Jul 27 21:30:00.143473 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:30:00.143446    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j5rjm\""
Jul 27 21:30:00.148238 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:30:00.148209    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-55137c-29753130-blkhc"]
Jul 27 21:30:00.158129 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:30:00.158062    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29753130-54ln4"]
Jul 27 21:30:00.158297 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:30:00.158251    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-55137c-29753130-blkhc"
Jul 27 21:30:00.158368 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:30:00.158305    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753130-54ln4"
Jul 27 21:30:00.180925 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:30:00.180885    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-55137c-29753130-blkhc"]
Jul 27 21:30:00.198520 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:30:00.198491    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-55137c-29753130-blkhc"
Jul 27 21:30:00.366853 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:30:00.366613    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29753130-54ln4"]
Jul 27 21:30:00.377738 ip-10-0-141-64 kubenswrapper[2560]: W0727 21:30:00.377682    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda88707a8_fd98_4349_b1e6_d3fabc7c5c78.slice/crio-0f2f1d8f19779ed641e83107067c6b875359aa5dcbfa77a73207a958dee42068 WatchSource:0}: Error finding container 0f2f1d8f19779ed641e83107067c6b875359aa5dcbfa77a73207a958dee42068: Status 404 returned error can't find the container with id 0f2f1d8f19779ed641e83107067c6b875359aa5dcbfa77a73207a958dee42068
Jul 27 21:30:00.410788 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:30:00.410758    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-55137c-29753130-blkhc"]
Jul 27 21:30:00.413407 ip-10-0-141-64 kubenswrapper[2560]: W0727 21:30:00.413379    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48e0e9ca_5236_4288_bba5_c9b873cec6c9.slice/crio-1d914d15802f3d5e75456097519524ef041300e204348badb5ac72dcdbd46087 WatchSource:0}: Error finding container 1d914d15802f3d5e75456097519524ef041300e204348badb5ac72dcdbd46087: Status 404 returned error can't find the container with id 1d914d15802f3d5e75456097519524ef041300e204348badb5ac72dcdbd46087
Jul 27 21:30:00.428969 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:30:00.428943    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:30:00.999814 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:30:00.999434    2560 patch_prober.go:28] interesting pod/e2e-api-a-9d44e2-openshift-default-8584f86688-vvz9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Jul 27 21:30:00.999814 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:30:00.999482    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9d44e2-openshift-default-8584f86688-vvz9k" podUID="f2193cf4-8865-4e4e-bd7c-eb2393be1914" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Jul 27 21:30:01.278269 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:30:01.278229    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-55137c-29753130-blkhc" event={"ID":"48e0e9ca-5236-4288-bba5-c9b873cec6c9","Type":"ContainerStarted","Data":"6b439afa4f3adc71c5b1a9ba16d058c14af9e26cd5302e72fa7ddf38137c5991"}
Jul 27 21:30:01.278470 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:30:01.278285    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-55137c-29753130-blkhc" event={"ID":"48e0e9ca-5236-4288-bba5-c9b873cec6c9","Type":"ContainerStarted","Data":"1d914d15802f3d5e75456097519524ef041300e204348badb5ac72dcdbd46087"}
Jul 27 21:30:01.279883 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:30:01.279856    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753130-54ln4" event={"ID":"a88707a8-fd98-4349-b1e6-d3fabc7c5c78","Type":"ContainerStarted","Data":"959aa11671b59a54470edb66b2df4e51fd9aa28e71156fbc26af9b9e45d42157"}
Jul 27 21:30:01.280070 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:30:01.279901    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753130-54ln4" event={"ID":"a88707a8-fd98-4349-b1e6-d3fabc7c5c78","Type":"ContainerStarted","Data":"0f2f1d8f19779ed641e83107067c6b875359aa5dcbfa77a73207a958dee42068"}
Jul 27 21:30:01.294657 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:30:01.294602    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-55137c-29753130-blkhc" podStartSLOduration=1.294581601 podStartE2EDuration="1.294581601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:30:01.292863752 +0000 UTC m=+2318.044725547" watchObservedRunningTime="2026-07-27 21:30:01.294581601 +0000 UTC m=+2318.046443396"
Jul 27 21:30:01.310672 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:30:01.310623    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753130-54ln4" podStartSLOduration=1.310609396 podStartE2EDuration="1.310609396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:30:01.308751164 +0000 UTC m=+2318.060612973" watchObservedRunningTime="2026-07-27 21:30:01.310609396 +0000 UTC m=+2318.062471185"
Jul 27 21:30:01.999782 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:30:01.999697    2560 patch_prober.go:28] interesting pod/e2e-api-a-9d44e2-openshift-default-8584f86688-vvz9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Jul 27 21:30:01.999782 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:30:01.999751    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9d44e2-openshift-default-8584f86688-vvz9k" podUID="f2193cf4-8865-4e4e-bd7c-eb2393be1914" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Jul 27 21:30:02.291104 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:30:02.291038    2560 generic.go:356] "Generic (PLEG): container finished" podID="48e0e9ca-5236-4288-bba5-c9b873cec6c9" containerID="6b439afa4f3adc71c5b1a9ba16d058c14af9e26cd5302e72fa7ddf38137c5991" exitCode=7
Jul 27 21:30:02.291424 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:30:02.291403    2560 scope.go:117] "RemoveContainer" containerID="6b439afa4f3adc71c5b1a9ba16d058c14af9e26cd5302e72fa7ddf38137c5991"
Jul 27 21:30:02.291651 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:30:02.291561    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-55137c-29753130-blkhc" event={"ID":"48e0e9ca-5236-4288-bba5-c9b873cec6c9","Type":"ContainerDied","Data":"6b439afa4f3adc71c5b1a9ba16d058c14af9e26cd5302e72fa7ddf38137c5991"}
Jul 27 21:30:02.294254 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:30:02.294227    2560 generic.go:356] "Generic (PLEG): container finished" podID="a88707a8-fd98-4349-b1e6-d3fabc7c5c78" containerID="959aa11671b59a54470edb66b2df4e51fd9aa28e71156fbc26af9b9e45d42157" exitCode=0
Jul 27 21:30:02.294339 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:30:02.294288    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753130-54ln4" event={"ID":"a88707a8-fd98-4349-b1e6-d3fabc7c5c78","Type":"ContainerDied","Data":"959aa11671b59a54470edb66b2df4e51fd9aa28e71156fbc26af9b9e45d42157"}
Jul 27 21:30:03.000158 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:30:03.000021    2560 patch_prober.go:28] interesting pod/e2e-api-a-9d44e2-openshift-default-8584f86688-vvz9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Jul 27 21:30:03.000526 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:30:03.000219    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9d44e2-openshift-default-8584f86688-vvz9k" podUID="f2193cf4-8865-4e4e-bd7c-eb2393be1914" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Jul 27 21:30:03.301171 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:30:03.301133    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-55137c-29753130-blkhc" event={"ID":"48e0e9ca-5236-4288-bba5-c9b873cec6c9","Type":"ContainerStarted","Data":"3fc6a8ed5ba08383b634cc3220d24e7772cb644005370b08a7ed9b9f1099a20d"}
Jul 27 21:30:03.474995 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:30:03.474965    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753130-54ln4"
Jul 27 21:30:04.004615 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:30:04.004578    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-9d44e2-openshift-default-8584f86688-vvz9k"
Jul 27 21:30:04.004976 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:30:04.004941    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-9d44e2-openshift-default-8584f86688-vvz9k"
Jul 27 21:30:04.005671 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:30:04.005651    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-9d44e2-openshift-default-8584f86688-vvz9k"
Jul 27 21:30:04.307036 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:30:04.306998    2560 generic.go:356] "Generic (PLEG): container finished" podID="48e0e9ca-5236-4288-bba5-c9b873cec6c9" containerID="3fc6a8ed5ba08383b634cc3220d24e7772cb644005370b08a7ed9b9f1099a20d" exitCode=0
Jul 27 21:30:04.307036 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:30:04.307032    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-55137c-29753130-blkhc" event={"ID":"48e0e9ca-5236-4288-bba5-c9b873cec6c9","Type":"ContainerDied","Data":"3fc6a8ed5ba08383b634cc3220d24e7772cb644005370b08a7ed9b9f1099a20d"}
Jul 27 21:30:04.307328 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:30:04.307109    2560 scope.go:117] "RemoveContainer" containerID="6b439afa4f3adc71c5b1a9ba16d058c14af9e26cd5302e72fa7ddf38137c5991"
Jul 27 21:30:04.308592 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:30:04.308567    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753130-54ln4"
Jul 27 21:30:04.308719 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:30:04.308601    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753130-54ln4" event={"ID":"a88707a8-fd98-4349-b1e6-d3fabc7c5c78","Type":"ContainerDied","Data":"0f2f1d8f19779ed641e83107067c6b875359aa5dcbfa77a73207a958dee42068"}
Jul 27 21:30:04.308719 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:30:04.308640    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f2f1d8f19779ed641e83107067c6b875359aa5dcbfa77a73207a958dee42068"
Jul 27 21:30:05.490446 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:30:05.490424    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-55137c-29753130-blkhc"
Jul 27 21:30:06.321098 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:30:06.321049    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-55137c-29753130-blkhc" event={"ID":"48e0e9ca-5236-4288-bba5-c9b873cec6c9","Type":"ContainerDied","Data":"1d914d15802f3d5e75456097519524ef041300e204348badb5ac72dcdbd46087"}
Jul 27 21:30:06.321098 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:30:06.321072    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-55137c-29753130-blkhc"
Jul 27 21:30:06.321319 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:30:06.321126    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1d914d15802f3d5e75456097519524ef041300e204348badb5ac72dcdbd46087"
Jul 27 21:30:08.909796 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:30:08.908642    2560 patch_prober.go:28] interesting pod/e2e-api-b-55137c-openshift-default-866bbfc94-clccr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": read tcp 10.133.0.2:49316->10.133.0.46:15021: read: connection reset by peer" start-of-body=
Jul 27 21:30:08.909796 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:30:08.908710    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-55137c-openshift-default-866bbfc94-clccr" podUID="ce51fc53-45c9-4c8a-81a9-65c5407a7b0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": read tcp 10.133.0.2:49316->10.133.0.46:15021: read: connection reset by peer"
Jul 27 21:30:09.337297 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:30:09.337263    2560 generic.go:356] "Generic (PLEG): container finished" podID="ce51fc53-45c9-4c8a-81a9-65c5407a7b0d" containerID="eab37c5c6c4303bb2b21fc9a5a569b649e5479160a1ffa93bd91a5bc0e3a3542" exitCode=0
Jul 27 21:30:09.337499 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:30:09.337337    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-55137c-openshift-default-866bbfc94-clccr" event={"ID":"ce51fc53-45c9-4c8a-81a9-65c5407a7b0d","Type":"ContainerDied","Data":"eab37c5c6c4303bb2b21fc9a5a569b649e5479160a1ffa93bd91a5bc0e3a3542"}
Jul 27 21:30:09.337848 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:30:09.337832    2560 scope.go:117] "RemoveContainer" containerID="eab37c5c6c4303bb2b21fc9a5a569b649e5479160a1ffa93bd91a5bc0e3a3542"
Jul 27 21:30:09.338825 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:30:09.338800    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 27 21:30:09.338884 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:30:09.338867    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 27 21:30:09.338921 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:30:09.338897    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 27 21:30:10.348046 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:30:10.348008    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-55137c-openshift-default-866bbfc94-clccr" event={"ID":"ce51fc53-45c9-4c8a-81a9-65c5407a7b0d","Type":"ContainerStarted","Data":"71262ccfc51cbd3237c94989117be89e45ba3de39abf9b14747e038ad11c1848"}
Jul 27 21:30:10.834738 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:30:10.834703    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-55137c-openshift-default-866bbfc94-clccr"
Jul 27 21:30:10.834927 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:30:10.834906    2560 patch_prober.go:28] interesting pod/e2e-api-b-55137c-openshift-default-866bbfc94-clccr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Jul 27 21:30:10.834986 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:30:10.834966    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-55137c-openshift-default-866bbfc94-clccr" podUID="ce51fc53-45c9-4c8a-81a9-65c5407a7b0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Jul 27 21:30:11.834847 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:30:11.834807    2560 patch_prober.go:28] interesting pod/e2e-api-b-55137c-openshift-default-866bbfc94-clccr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Jul 27 21:30:11.835258 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:30:11.834869    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-55137c-openshift-default-866bbfc94-clccr" podUID="ce51fc53-45c9-4c8a-81a9-65c5407a7b0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Jul 27 21:30:12.834027 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:30:12.833989    2560 patch_prober.go:28] interesting pod/e2e-api-b-55137c-openshift-default-866bbfc94-clccr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Jul 27 21:30:12.834265 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:30:12.834044    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-55137c-openshift-default-866bbfc94-clccr" podUID="ce51fc53-45c9-4c8a-81a9-65c5407a7b0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Jul 27 21:30:13.834533 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:30:13.834445    2560 patch_prober.go:28] interesting pod/e2e-api-b-55137c-openshift-default-866bbfc94-clccr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Jul 27 21:30:13.834533 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:30:13.834505    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-55137c-openshift-default-866bbfc94-clccr" podUID="ce51fc53-45c9-4c8a-81a9-65c5407a7b0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Jul 27 21:30:14.838139 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:30:14.838105    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-55137c-openshift-default-866bbfc94-clccr"
Jul 27 21:30:14.838528 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:30:14.838444    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-55137c-openshift-default-866bbfc94-clccr"
Jul 27 21:30:14.839123 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:30:14.839105    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-55137c-openshift-default-866bbfc94-clccr"
Jul 27 21:30:28.692523 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:30:28.692486    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:30:32.503212 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:30:32.503171    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-9d44e2-openshift-default-8584f86688-vvz9k"]
Jul 27 21:30:32.503675 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:30:32.503452    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-9d44e2-openshift-default-8584f86688-vvz9k" podUID="f2193cf4-8865-4e4e-bd7c-eb2393be1914" containerName="istio-proxy" containerID="cri-o://e51afcd1d5b59d62fc973567226205ebecda48b1b3498ae5e5b0607c4785fa92" gracePeriod=30
Jul 27 21:30:34.007170 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:30:34.007124    2560 patch_prober.go:28] interesting pod/e2e-api-a-9d44e2-openshift-default-8584f86688-vvz9k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 27 21:30:34.007556 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:30:34.007196    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-9d44e2-openshift-default-8584f86688-vvz9k" podUID="f2193cf4-8865-4e4e-bd7c-eb2393be1914" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 21:30:37.759701 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:30:37.759679    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-9d44e2-openshift-default-8584f86688-vvz9k"
Jul 27 21:30:37.886113 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:30:37.886004    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2193cf4-8865-4e4e-bd7c-eb2393be1914-istio-data\") pod \"f2193cf4-8865-4e4e-bd7c-eb2393be1914\" (UID: \"f2193cf4-8865-4e4e-bd7c-eb2393be1914\") "
Jul 27 21:30:37.886113 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:30:37.886039    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2193cf4-8865-4e4e-bd7c-eb2393be1914-istiod-ca-cert\") pod \"f2193cf4-8865-4e4e-bd7c-eb2393be1914\" (UID: \"f2193cf4-8865-4e4e-bd7c-eb2393be1914\") "
Jul 27 21:30:37.886113 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:30:37.886069    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sp662\" (UniqueName: \"kubernetes.io/projected/f2193cf4-8865-4e4e-bd7c-eb2393be1914-kube-api-access-sp662\") pod \"f2193cf4-8865-4e4e-bd7c-eb2393be1914\" (UID: \"f2193cf4-8865-4e4e-bd7c-eb2393be1914\") "
Jul 27 21:30:37.886404 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:30:37.886114    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2193cf4-8865-4e4e-bd7c-eb2393be1914-istio-podinfo\") pod \"f2193cf4-8865-4e4e-bd7c-eb2393be1914\" (UID: \"f2193cf4-8865-4e4e-bd7c-eb2393be1914\") "
Jul 27 21:30:37.886404 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:30:37.886157    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2193cf4-8865-4e4e-bd7c-eb2393be1914-workload-socket\") pod \"f2193cf4-8865-4e4e-bd7c-eb2393be1914\" (UID: \"f2193cf4-8865-4e4e-bd7c-eb2393be1914\") "
Jul 27 21:30:37.886404 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:30:37.886185    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2193cf4-8865-4e4e-bd7c-eb2393be1914-istio-envoy\") pod \"f2193cf4-8865-4e4e-bd7c-eb2393be1914\" (UID: \"f2193cf4-8865-4e4e-bd7c-eb2393be1914\") "
Jul 27 21:30:37.886404 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:30:37.886208    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2193cf4-8865-4e4e-bd7c-eb2393be1914-credential-socket\") pod \"f2193cf4-8865-4e4e-bd7c-eb2393be1914\" (UID: \"f2193cf4-8865-4e4e-bd7c-eb2393be1914\") "
Jul 27 21:30:37.886404 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:30:37.886233    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2193cf4-8865-4e4e-bd7c-eb2393be1914-workload-certs\") pod \"f2193cf4-8865-4e4e-bd7c-eb2393be1914\" (UID: \"f2193cf4-8865-4e4e-bd7c-eb2393be1914\") "
Jul 27 21:30:37.886404 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:30:37.886291    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2193cf4-8865-4e4e-bd7c-eb2393be1914-istio-token\") pod \"f2193cf4-8865-4e4e-bd7c-eb2393be1914\" (UID: \"f2193cf4-8865-4e4e-bd7c-eb2393be1914\") "
Jul 27 21:30:37.886794 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:30:37.886516    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2193cf4-8865-4e4e-bd7c-eb2393be1914-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f2193cf4-8865-4e4e-bd7c-eb2393be1914" (UID: "f2193cf4-8865-4e4e-bd7c-eb2393be1914"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:30:37.886794 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:30:37.886676    2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2193cf4-8865-4e4e-bd7c-eb2393be1914-credential-socket\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:30:37.886794 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:30:37.886721    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2193cf4-8865-4e4e-bd7c-eb2393be1914-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f2193cf4-8865-4e4e-bd7c-eb2393be1914" (UID: "f2193cf4-8865-4e4e-bd7c-eb2393be1914"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:30:37.886951 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:30:37.886860    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2193cf4-8865-4e4e-bd7c-eb2393be1914-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f2193cf4-8865-4e4e-bd7c-eb2393be1914" (UID: "f2193cf4-8865-4e4e-bd7c-eb2393be1914"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:30:37.886951 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:30:37.886904    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2193cf4-8865-4e4e-bd7c-eb2393be1914-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f2193cf4-8865-4e4e-bd7c-eb2393be1914" (UID: "f2193cf4-8865-4e4e-bd7c-eb2393be1914"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:30:37.887283 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:30:37.887252    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2193cf4-8865-4e4e-bd7c-eb2393be1914-istio-data" (OuterVolumeSpecName: "istio-data") pod "f2193cf4-8865-4e4e-bd7c-eb2393be1914" (UID: "f2193cf4-8865-4e4e-bd7c-eb2393be1914"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:30:37.888547 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:30:37.888516    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2193cf4-8865-4e4e-bd7c-eb2393be1914-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f2193cf4-8865-4e4e-bd7c-eb2393be1914" (UID: "f2193cf4-8865-4e4e-bd7c-eb2393be1914"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:30:37.888547 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:30:37.888533    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f2193cf4-8865-4e4e-bd7c-eb2393be1914-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f2193cf4-8865-4e4e-bd7c-eb2393be1914" (UID: "f2193cf4-8865-4e4e-bd7c-eb2393be1914"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 27 21:30:37.888816 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:30:37.888788    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2193cf4-8865-4e4e-bd7c-eb2393be1914-kube-api-access-sp662" (OuterVolumeSpecName: "kube-api-access-sp662") pod "f2193cf4-8865-4e4e-bd7c-eb2393be1914" (UID: "f2193cf4-8865-4e4e-bd7c-eb2393be1914"). InnerVolumeSpecName "kube-api-access-sp662". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:30:37.888912 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:30:37.888898    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2193cf4-8865-4e4e-bd7c-eb2393be1914-istio-token" (OuterVolumeSpecName: "istio-token") pod "f2193cf4-8865-4e4e-bd7c-eb2393be1914" (UID: "f2193cf4-8865-4e4e-bd7c-eb2393be1914"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:30:37.987294 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:30:37.987265    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sp662\" (UniqueName: \"kubernetes.io/projected/f2193cf4-8865-4e4e-bd7c-eb2393be1914-kube-api-access-sp662\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:30:37.987294 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:30:37.987289    2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2193cf4-8865-4e4e-bd7c-eb2393be1914-istio-podinfo\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:30:37.987294 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:30:37.987300    2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2193cf4-8865-4e4e-bd7c-eb2393be1914-workload-socket\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:30:37.987488 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:30:37.987308    2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2193cf4-8865-4e4e-bd7c-eb2393be1914-istio-envoy\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:30:37.987488 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:30:37.987319    2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2193cf4-8865-4e4e-bd7c-eb2393be1914-workload-certs\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:30:37.987488 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:30:37.987326    2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2193cf4-8865-4e4e-bd7c-eb2393be1914-istio-token\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:30:37.987488 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:30:37.987333    2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2193cf4-8865-4e4e-bd7c-eb2393be1914-istio-data\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:30:37.987488 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:30:37.987341    2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2193cf4-8865-4e4e-bd7c-eb2393be1914-istiod-ca-cert\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:30:38.491102 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:30:38.491060    2560 generic.go:356] "Generic (PLEG): container finished" podID="f2193cf4-8865-4e4e-bd7c-eb2393be1914" containerID="e51afcd1d5b59d62fc973567226205ebecda48b1b3498ae5e5b0607c4785fa92" exitCode=0
Jul 27 21:30:38.491274 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:30:38.491127    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9d44e2-openshift-default-8584f86688-vvz9k" event={"ID":"f2193cf4-8865-4e4e-bd7c-eb2393be1914","Type":"ContainerDied","Data":"e51afcd1d5b59d62fc973567226205ebecda48b1b3498ae5e5b0607c4785fa92"}
Jul 27 21:30:38.491274 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:30:38.491151    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-9d44e2-openshift-default-8584f86688-vvz9k"
Jul 27 21:30:38.491274 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:30:38.491227    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9d44e2-openshift-default-8584f86688-vvz9k" event={"ID":"f2193cf4-8865-4e4e-bd7c-eb2393be1914","Type":"ContainerDied","Data":"6aabdcb70380ad2a52d9f1650862b774faf3560a9638788ae6ac0d2a847d3aae"}
Jul 27 21:30:38.491274 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:30:38.491260    2560 scope.go:117] "RemoveContainer" containerID="e51afcd1d5b59d62fc973567226205ebecda48b1b3498ae5e5b0607c4785fa92"
Jul 27 21:30:38.500520 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:30:38.500501    2560 scope.go:117] "RemoveContainer" containerID="51e520270f16cbe7473ec4e04cd35ac94f4ff799637daabb41d8897f724f5475"
Jul 27 21:30:38.508144 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:30:38.508099    2560 scope.go:117] "RemoveContainer" containerID="e51afcd1d5b59d62fc973567226205ebecda48b1b3498ae5e5b0607c4785fa92"
Jul 27 21:30:38.508355 ip-10-0-141-64 kubenswrapper[2560]: E0727 21:30:38.508335    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e51afcd1d5b59d62fc973567226205ebecda48b1b3498ae5e5b0607c4785fa92\": container with ID starting with e51afcd1d5b59d62fc973567226205ebecda48b1b3498ae5e5b0607c4785fa92 not found: ID does not exist" containerID="e51afcd1d5b59d62fc973567226205ebecda48b1b3498ae5e5b0607c4785fa92"
Jul 27 21:30:38.508421 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:30:38.508363    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e51afcd1d5b59d62fc973567226205ebecda48b1b3498ae5e5b0607c4785fa92"} err="failed to get container status \"e51afcd1d5b59d62fc973567226205ebecda48b1b3498ae5e5b0607c4785fa92\": rpc error: code = NotFound desc = could not find container \"e51afcd1d5b59d62fc973567226205ebecda48b1b3498ae5e5b0607c4785fa92\": container with ID starting with e51afcd1d5b59d62fc973567226205ebecda48b1b3498ae5e5b0607c4785fa92 not found: ID does not exist"
Jul 27 21:30:38.508421 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:30:38.508380    2560 scope.go:117] "RemoveContainer" containerID="51e520270f16cbe7473ec4e04cd35ac94f4ff799637daabb41d8897f724f5475"
Jul 27 21:30:38.508627 ip-10-0-141-64 kubenswrapper[2560]: E0727 21:30:38.508608    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51e520270f16cbe7473ec4e04cd35ac94f4ff799637daabb41d8897f724f5475\": container with ID starting with 51e520270f16cbe7473ec4e04cd35ac94f4ff799637daabb41d8897f724f5475 not found: ID does not exist" containerID="51e520270f16cbe7473ec4e04cd35ac94f4ff799637daabb41d8897f724f5475"
Jul 27 21:30:38.508679 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:30:38.508629    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51e520270f16cbe7473ec4e04cd35ac94f4ff799637daabb41d8897f724f5475"} err="failed to get container status \"51e520270f16cbe7473ec4e04cd35ac94f4ff799637daabb41d8897f724f5475\": rpc error: code = NotFound desc = could not find container \"51e520270f16cbe7473ec4e04cd35ac94f4ff799637daabb41d8897f724f5475\": container with ID starting with 51e520270f16cbe7473ec4e04cd35ac94f4ff799637daabb41d8897f724f5475 not found: ID does not exist"
Jul 27 21:30:38.514352 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:30:38.514330    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-9d44e2-openshift-default-8584f86688-vvz9k"]
Jul 27 21:30:38.519969 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:30:38.519947    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-9d44e2-openshift-default-8584f86688-vvz9k"]
Jul 27 21:30:39.847287 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:30:39.847256    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2193cf4-8865-4e4e-bd7c-eb2393be1914" path="/var/lib/kubelet/pods/f2193cf4-8865-4e4e-bd7c-eb2393be1914/volumes"
Jul 27 21:30:48.319558 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:30:48.319519    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-55137c-29753130-blkhc"]
Jul 27 21:30:48.322017 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:30:48.321981    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-55137c-29753130-blkhc"]
Jul 27 21:30:49.846938 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:30:49.846904    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="48e0e9ca-5236-4288-bba5-c9b873cec6c9" path="/var/lib/kubelet/pods/48e0e9ca-5236-4288-bba5-c9b873cec6c9/volumes"
Jul 27 21:30:53.115326 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:30:53.115293    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:30:59.386939 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:30:59.386898    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-55137c-openshift-default-866bbfc94-clccr"]
Jul 27 21:30:59.387555 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:30:59.387186    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-55137c-openshift-default-866bbfc94-clccr" podUID="ce51fc53-45c9-4c8a-81a9-65c5407a7b0d" containerName="istio-proxy" containerID="cri-o://71262ccfc51cbd3237c94989117be89e45ba3de39abf9b14747e038ad11c1848" gracePeriod=30
Jul 27 21:30:59.840958 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:30:59.840921    2560 patch_prober.go:28] interesting pod/e2e-api-b-55137c-openshift-default-866bbfc94-clccr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 27 21:30:59.841135 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:30:59.840983    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-55137c-openshift-default-866bbfc94-clccr" podUID="ce51fc53-45c9-4c8a-81a9-65c5407a7b0d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 21:31:00.317382 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:00.317349    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-6dd721-openshift-default-55b7fb86b4-622lp"]
Jul 27 21:31:00.317716 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:00.317703    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a88707a8-fd98-4349-b1e6-d3fabc7c5c78" containerName="cleanup"
Jul 27 21:31:00.317761 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:00.317718    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="a88707a8-fd98-4349-b1e6-d3fabc7c5c78" containerName="cleanup"
Jul 27 21:31:00.317761 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:00.317728    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48e0e9ca-5236-4288-bba5-c9b873cec6c9" containerName="cleanup"
Jul 27 21:31:00.317761 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:00.317734    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="48e0e9ca-5236-4288-bba5-c9b873cec6c9" containerName="cleanup"
Jul 27 21:31:00.317761 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:00.317743    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2193cf4-8865-4e4e-bd7c-eb2393be1914" containerName="istio-proxy"
Jul 27 21:31:00.317761 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:00.317748    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2193cf4-8865-4e4e-bd7c-eb2393be1914" containerName="istio-proxy"
Jul 27 21:31:00.317761 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:00.317754    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2193cf4-8865-4e4e-bd7c-eb2393be1914" containerName="istio-proxy"
Jul 27 21:31:00.317761 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:00.317759    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2193cf4-8865-4e4e-bd7c-eb2393be1914" containerName="istio-proxy"
Jul 27 21:31:00.317975 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:00.317823    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="a88707a8-fd98-4349-b1e6-d3fabc7c5c78" containerName="cleanup"
Jul 27 21:31:00.317975 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:00.317831    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="48e0e9ca-5236-4288-bba5-c9b873cec6c9" containerName="cleanup"
Jul 27 21:31:00.317975 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:00.317838    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="48e0e9ca-5236-4288-bba5-c9b873cec6c9" containerName="cleanup"
Jul 27 21:31:00.317975 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:00.317845    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2193cf4-8865-4e4e-bd7c-eb2393be1914" containerName="istio-proxy"
Jul 27 21:31:00.317975 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:00.317853    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2193cf4-8865-4e4e-bd7c-eb2393be1914" containerName="istio-proxy"
Jul 27 21:31:00.320879 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:00.320863    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-6dd721-openshift-default-55b7fb86b4-622lp"
Jul 27 21:31:00.327669 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:00.327652    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-6dd721-openshift-default-dockercfg-5z8wt\""
Jul 27 21:31:00.338397 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:00.338373    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-6dd721-openshift-default-55b7fb86b4-622lp"]
Jul 27 21:31:00.487923 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:00.487882    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b059bf6-3c81-41ed-b200-965fa935795f-workload-socket\") pod \"e2e-shared-a-6dd721-openshift-default-55b7fb86b4-622lp\" (UID: \"8b059bf6-3c81-41ed-b200-965fa935795f\") " pod="openshift-ingress/e2e-shared-a-6dd721-openshift-default-55b7fb86b4-622lp"
Jul 27 21:31:00.488351 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:00.487935    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b059bf6-3c81-41ed-b200-965fa935795f-istio-podinfo\") pod \"e2e-shared-a-6dd721-openshift-default-55b7fb86b4-622lp\" (UID: \"8b059bf6-3c81-41ed-b200-965fa935795f\") " pod="openshift-ingress/e2e-shared-a-6dd721-openshift-default-55b7fb86b4-622lp"
Jul 27 21:31:00.488351 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:00.488006    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b059bf6-3c81-41ed-b200-965fa935795f-credential-socket\") pod \"e2e-shared-a-6dd721-openshift-default-55b7fb86b4-622lp\" (UID: \"8b059bf6-3c81-41ed-b200-965fa935795f\") " pod="openshift-ingress/e2e-shared-a-6dd721-openshift-default-55b7fb86b4-622lp"
Jul 27 21:31:00.488351 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:00.488047    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b059bf6-3c81-41ed-b200-965fa935795f-workload-certs\") pod \"e2e-shared-a-6dd721-openshift-default-55b7fb86b4-622lp\" (UID: \"8b059bf6-3c81-41ed-b200-965fa935795f\") " pod="openshift-ingress/e2e-shared-a-6dd721-openshift-default-55b7fb86b4-622lp"
Jul 27 21:31:00.488351 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:00.488071    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45t4t\" (UniqueName: \"kubernetes.io/projected/8b059bf6-3c81-41ed-b200-965fa935795f-kube-api-access-45t4t\") pod \"e2e-shared-a-6dd721-openshift-default-55b7fb86b4-622lp\" (UID: \"8b059bf6-3c81-41ed-b200-965fa935795f\") " pod="openshift-ingress/e2e-shared-a-6dd721-openshift-default-55b7fb86b4-622lp"
Jul 27 21:31:00.488351 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:00.488115    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b059bf6-3c81-41ed-b200-965fa935795f-istio-data\") pod \"e2e-shared-a-6dd721-openshift-default-55b7fb86b4-622lp\" (UID: \"8b059bf6-3c81-41ed-b200-965fa935795f\") " pod="openshift-ingress/e2e-shared-a-6dd721-openshift-default-55b7fb86b4-622lp"
Jul 27 21:31:00.488351 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:00.488157    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b059bf6-3c81-41ed-b200-965fa935795f-istio-envoy\") pod \"e2e-shared-a-6dd721-openshift-default-55b7fb86b4-622lp\" (UID: \"8b059bf6-3c81-41ed-b200-965fa935795f\") " pod="openshift-ingress/e2e-shared-a-6dd721-openshift-default-55b7fb86b4-622lp"
Jul 27 21:31:00.488351 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:00.488174    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b059bf6-3c81-41ed-b200-965fa935795f-istiod-ca-cert\") pod \"e2e-shared-a-6dd721-openshift-default-55b7fb86b4-622lp\" (UID: \"8b059bf6-3c81-41ed-b200-965fa935795f\") " pod="openshift-ingress/e2e-shared-a-6dd721-openshift-default-55b7fb86b4-622lp"
Jul 27 21:31:00.488351 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:00.488239    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b059bf6-3c81-41ed-b200-965fa935795f-istio-token\") pod \"e2e-shared-a-6dd721-openshift-default-55b7fb86b4-622lp\" (UID: \"8b059bf6-3c81-41ed-b200-965fa935795f\") " pod="openshift-ingress/e2e-shared-a-6dd721-openshift-default-55b7fb86b4-622lp"
Jul 27 21:31:00.588763 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:00.588681    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b059bf6-3c81-41ed-b200-965fa935795f-istio-token\") pod \"e2e-shared-a-6dd721-openshift-default-55b7fb86b4-622lp\" (UID: \"8b059bf6-3c81-41ed-b200-965fa935795f\") " pod="openshift-ingress/e2e-shared-a-6dd721-openshift-default-55b7fb86b4-622lp"
Jul 27 21:31:00.588763 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:00.588720    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b059bf6-3c81-41ed-b200-965fa935795f-workload-socket\") pod \"e2e-shared-a-6dd721-openshift-default-55b7fb86b4-622lp\" (UID: \"8b059bf6-3c81-41ed-b200-965fa935795f\") " pod="openshift-ingress/e2e-shared-a-6dd721-openshift-default-55b7fb86b4-622lp"
Jul 27 21:31:00.588763 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:00.588759    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b059bf6-3c81-41ed-b200-965fa935795f-istio-podinfo\") pod \"e2e-shared-a-6dd721-openshift-default-55b7fb86b4-622lp\" (UID: \"8b059bf6-3c81-41ed-b200-965fa935795f\") " pod="openshift-ingress/e2e-shared-a-6dd721-openshift-default-55b7fb86b4-622lp"
Jul 27 21:31:00.589065 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:00.588788    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b059bf6-3c81-41ed-b200-965fa935795f-credential-socket\") pod \"e2e-shared-a-6dd721-openshift-default-55b7fb86b4-622lp\" (UID: \"8b059bf6-3c81-41ed-b200-965fa935795f\") " pod="openshift-ingress/e2e-shared-a-6dd721-openshift-default-55b7fb86b4-622lp"
Jul 27 21:31:00.589065 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:00.588808    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b059bf6-3c81-41ed-b200-965fa935795f-workload-certs\") pod \"e2e-shared-a-6dd721-openshift-default-55b7fb86b4-622lp\" (UID: \"8b059bf6-3c81-41ed-b200-965fa935795f\") " pod="openshift-ingress/e2e-shared-a-6dd721-openshift-default-55b7fb86b4-622lp"
Jul 27 21:31:00.589065 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:00.588835    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45t4t\" (UniqueName: \"kubernetes.io/projected/8b059bf6-3c81-41ed-b200-965fa935795f-kube-api-access-45t4t\") pod \"e2e-shared-a-6dd721-openshift-default-55b7fb86b4-622lp\" (UID: \"8b059bf6-3c81-41ed-b200-965fa935795f\") " pod="openshift-ingress/e2e-shared-a-6dd721-openshift-default-55b7fb86b4-622lp"
Jul 27 21:31:00.589065 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:00.588854    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b059bf6-3c81-41ed-b200-965fa935795f-istio-data\") pod \"e2e-shared-a-6dd721-openshift-default-55b7fb86b4-622lp\" (UID: \"8b059bf6-3c81-41ed-b200-965fa935795f\") " pod="openshift-ingress/e2e-shared-a-6dd721-openshift-default-55b7fb86b4-622lp"
Jul 27 21:31:00.589065 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:00.588903    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b059bf6-3c81-41ed-b200-965fa935795f-istio-envoy\") pod \"e2e-shared-a-6dd721-openshift-default-55b7fb86b4-622lp\" (UID: \"8b059bf6-3c81-41ed-b200-965fa935795f\") " pod="openshift-ingress/e2e-shared-a-6dd721-openshift-default-55b7fb86b4-622lp"
Jul 27 21:31:00.589065 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:00.588933    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b059bf6-3c81-41ed-b200-965fa935795f-istiod-ca-cert\") pod \"e2e-shared-a-6dd721-openshift-default-55b7fb86b4-622lp\" (UID: \"8b059bf6-3c81-41ed-b200-965fa935795f\") " pod="openshift-ingress/e2e-shared-a-6dd721-openshift-default-55b7fb86b4-622lp"
Jul 27 21:31:00.589509 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:00.589484    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b059bf6-3c81-41ed-b200-965fa935795f-workload-socket\") pod \"e2e-shared-a-6dd721-openshift-default-55b7fb86b4-622lp\" (UID: \"8b059bf6-3c81-41ed-b200-965fa935795f\") " pod="openshift-ingress/e2e-shared-a-6dd721-openshift-default-55b7fb86b4-622lp"
Jul 27 21:31:00.589645 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:00.589619    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b059bf6-3c81-41ed-b200-965fa935795f-credential-socket\") pod \"e2e-shared-a-6dd721-openshift-default-55b7fb86b4-622lp\" (UID: \"8b059bf6-3c81-41ed-b200-965fa935795f\") " pod="openshift-ingress/e2e-shared-a-6dd721-openshift-default-55b7fb86b4-622lp"
Jul 27 21:31:00.589645 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:00.589550    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b059bf6-3c81-41ed-b200-965fa935795f-istio-data\") pod \"e2e-shared-a-6dd721-openshift-default-55b7fb86b4-622lp\" (UID: \"8b059bf6-3c81-41ed-b200-965fa935795f\") " pod="openshift-ingress/e2e-shared-a-6dd721-openshift-default-55b7fb86b4-622lp"
Jul 27 21:31:00.589645 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:00.589501    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b059bf6-3c81-41ed-b200-965fa935795f-workload-certs\") pod \"e2e-shared-a-6dd721-openshift-default-55b7fb86b4-622lp\" (UID: \"8b059bf6-3c81-41ed-b200-965fa935795f\") " pod="openshift-ingress/e2e-shared-a-6dd721-openshift-default-55b7fb86b4-622lp"
Jul 27 21:31:00.589919 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:00.589897    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b059bf6-3c81-41ed-b200-965fa935795f-istiod-ca-cert\") pod \"e2e-shared-a-6dd721-openshift-default-55b7fb86b4-622lp\" (UID: \"8b059bf6-3c81-41ed-b200-965fa935795f\") " pod="openshift-ingress/e2e-shared-a-6dd721-openshift-default-55b7fb86b4-622lp"
Jul 27 21:31:00.591407 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:00.591388    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b059bf6-3c81-41ed-b200-965fa935795f-istio-envoy\") pod \"e2e-shared-a-6dd721-openshift-default-55b7fb86b4-622lp\" (UID: \"8b059bf6-3c81-41ed-b200-965fa935795f\") " pod="openshift-ingress/e2e-shared-a-6dd721-openshift-default-55b7fb86b4-622lp"
Jul 27 21:31:00.591624 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:00.591608    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b059bf6-3c81-41ed-b200-965fa935795f-istio-podinfo\") pod \"e2e-shared-a-6dd721-openshift-default-55b7fb86b4-622lp\" (UID: \"8b059bf6-3c81-41ed-b200-965fa935795f\") " pod="openshift-ingress/e2e-shared-a-6dd721-openshift-default-55b7fb86b4-622lp"
Jul 27 21:31:00.597515 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:00.597490    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b059bf6-3c81-41ed-b200-965fa935795f-istio-token\") pod \"e2e-shared-a-6dd721-openshift-default-55b7fb86b4-622lp\" (UID: \"8b059bf6-3c81-41ed-b200-965fa935795f\") " pod="openshift-ingress/e2e-shared-a-6dd721-openshift-default-55b7fb86b4-622lp"
Jul 27 21:31:00.597618 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:00.597603    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-45t4t\" (UniqueName: \"kubernetes.io/projected/8b059bf6-3c81-41ed-b200-965fa935795f-kube-api-access-45t4t\") pod \"e2e-shared-a-6dd721-openshift-default-55b7fb86b4-622lp\" (UID: \"8b059bf6-3c81-41ed-b200-965fa935795f\") " pod="openshift-ingress/e2e-shared-a-6dd721-openshift-default-55b7fb86b4-622lp"
Jul 27 21:31:00.630853 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:00.630833    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-6dd721-openshift-default-55b7fb86b4-622lp"
Jul 27 21:31:00.808326 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:00.808192    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-6dd721-openshift-default-55b7fb86b4-622lp"]
Jul 27 21:31:00.812337 ip-10-0-141-64 kubenswrapper[2560]: W0727 21:31:00.812311    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b059bf6_3c81_41ed_b200_965fa935795f.slice/crio-5e9697572127cfbd98aee9cd738fc895be189361b8d6daaa57002008c710add2 WatchSource:0}: Error finding container 5e9697572127cfbd98aee9cd738fc895be189361b8d6daaa57002008c710add2: Status 404 returned error can't find the container with id 5e9697572127cfbd98aee9cd738fc895be189361b8d6daaa57002008c710add2
Jul 27 21:31:00.814883 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:00.814853    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 27 21:31:00.814941 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:00.814915    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 27 21:31:00.814976 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:00.814942    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 27 21:31:01.607771 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:01.607733    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-6dd721-openshift-default-55b7fb86b4-622lp" event={"ID":"8b059bf6-3c81-41ed-b200-965fa935795f","Type":"ContainerStarted","Data":"c563cf2cde488be91a55e5dc2f30e67410d582f01d8577329c37fdddd2ce622f"}
Jul 27 21:31:01.608172 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:01.607793    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-6dd721-openshift-default-55b7fb86b4-622lp" event={"ID":"8b059bf6-3c81-41ed-b200-965fa935795f","Type":"ContainerStarted","Data":"5e9697572127cfbd98aee9cd738fc895be189361b8d6daaa57002008c710add2"}
Jul 27 21:31:01.630475 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:01.630425    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-6dd721-openshift-default-55b7fb86b4-622lp" podStartSLOduration=1.630409123 podStartE2EDuration="1.630409123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:31:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:31:01.629642228 +0000 UTC m=+2378.381504059" watchObservedRunningTime="2026-07-27 21:31:01.630409123 +0000 UTC m=+2378.382270919"
Jul 27 21:31:01.631721 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:01.631700    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-6dd721-openshift-default-55b7fb86b4-622lp"
Jul 27 21:31:01.636686 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:01.636664    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-6dd721-openshift-default-55b7fb86b4-622lp"
Jul 27 21:31:02.612724 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:02.612692    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-6dd721-openshift-default-55b7fb86b4-622lp"
Jul 27 21:31:02.613718 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:02.613697    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-6dd721-openshift-default-55b7fb86b4-622lp"
Jul 27 21:31:04.624574 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:04.624547    2560 generic.go:356] "Generic (PLEG): container finished" podID="ce51fc53-45c9-4c8a-81a9-65c5407a7b0d" containerID="71262ccfc51cbd3237c94989117be89e45ba3de39abf9b14747e038ad11c1848" exitCode=0
Jul 27 21:31:04.624899 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:04.624621    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-55137c-openshift-default-866bbfc94-clccr" event={"ID":"ce51fc53-45c9-4c8a-81a9-65c5407a7b0d","Type":"ContainerDied","Data":"71262ccfc51cbd3237c94989117be89e45ba3de39abf9b14747e038ad11c1848"}
Jul 27 21:31:04.624899 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:04.624678    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-55137c-openshift-default-866bbfc94-clccr" event={"ID":"ce51fc53-45c9-4c8a-81a9-65c5407a7b0d","Type":"ContainerDied","Data":"84eb79c99ff446ff148cf7cd2bd1e90bb45701ac843eee34e2c8a62f3223c312"}
Jul 27 21:31:04.624899 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:04.624704    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="84eb79c99ff446ff148cf7cd2bd1e90bb45701ac843eee34e2c8a62f3223c312"
Jul 27 21:31:04.624899 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:04.624731    2560 scope.go:117] "RemoveContainer" containerID="eab37c5c6c4303bb2b21fc9a5a569b649e5479160a1ffa93bd91a5bc0e3a3542"
Jul 27 21:31:04.640493 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:04.640470    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-55137c-openshift-default-866bbfc94-clccr"
Jul 27 21:31:04.824413 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:04.824382    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce51fc53-45c9-4c8a-81a9-65c5407a7b0d-istio-podinfo\") pod \"ce51fc53-45c9-4c8a-81a9-65c5407a7b0d\" (UID: \"ce51fc53-45c9-4c8a-81a9-65c5407a7b0d\") "
Jul 27 21:31:04.824612 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:04.824426    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce51fc53-45c9-4c8a-81a9-65c5407a7b0d-credential-socket\") pod \"ce51fc53-45c9-4c8a-81a9-65c5407a7b0d\" (UID: \"ce51fc53-45c9-4c8a-81a9-65c5407a7b0d\") "
Jul 27 21:31:04.824612 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:04.824444    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce51fc53-45c9-4c8a-81a9-65c5407a7b0d-istio-data\") pod \"ce51fc53-45c9-4c8a-81a9-65c5407a7b0d\" (UID: \"ce51fc53-45c9-4c8a-81a9-65c5407a7b0d\") "
Jul 27 21:31:04.824612 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:04.824468    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce51fc53-45c9-4c8a-81a9-65c5407a7b0d-workload-certs\") pod \"ce51fc53-45c9-4c8a-81a9-65c5407a7b0d\" (UID: \"ce51fc53-45c9-4c8a-81a9-65c5407a7b0d\") "
Jul 27 21:31:04.824612 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:04.824500    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce51fc53-45c9-4c8a-81a9-65c5407a7b0d-istio-token\") pod \"ce51fc53-45c9-4c8a-81a9-65c5407a7b0d\" (UID: \"ce51fc53-45c9-4c8a-81a9-65c5407a7b0d\") "
Jul 27 21:31:04.824612 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:04.824568    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce51fc53-45c9-4c8a-81a9-65c5407a7b0d-istio-envoy\") pod \"ce51fc53-45c9-4c8a-81a9-65c5407a7b0d\" (UID: \"ce51fc53-45c9-4c8a-81a9-65c5407a7b0d\") "
Jul 27 21:31:04.824612 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:04.824600    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qlr7h\" (UniqueName: \"kubernetes.io/projected/ce51fc53-45c9-4c8a-81a9-65c5407a7b0d-kube-api-access-qlr7h\") pod \"ce51fc53-45c9-4c8a-81a9-65c5407a7b0d\" (UID: \"ce51fc53-45c9-4c8a-81a9-65c5407a7b0d\") "
Jul 27 21:31:04.824938 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:04.824655    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce51fc53-45c9-4c8a-81a9-65c5407a7b0d-istiod-ca-cert\") pod \"ce51fc53-45c9-4c8a-81a9-65c5407a7b0d\" (UID: \"ce51fc53-45c9-4c8a-81a9-65c5407a7b0d\") "
Jul 27 21:31:04.824938 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:04.824703    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce51fc53-45c9-4c8a-81a9-65c5407a7b0d-workload-socket\") pod \"ce51fc53-45c9-4c8a-81a9-65c5407a7b0d\" (UID: \"ce51fc53-45c9-4c8a-81a9-65c5407a7b0d\") "
Jul 27 21:31:04.824938 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:04.824760    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce51fc53-45c9-4c8a-81a9-65c5407a7b0d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ce51fc53-45c9-4c8a-81a9-65c5407a7b0d" (UID: "ce51fc53-45c9-4c8a-81a9-65c5407a7b0d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:31:04.824938 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:04.824843    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce51fc53-45c9-4c8a-81a9-65c5407a7b0d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ce51fc53-45c9-4c8a-81a9-65c5407a7b0d" (UID: "ce51fc53-45c9-4c8a-81a9-65c5407a7b0d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:31:04.825171 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:04.824973    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce51fc53-45c9-4c8a-81a9-65c5407a7b0d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ce51fc53-45c9-4c8a-81a9-65c5407a7b0d" (UID: "ce51fc53-45c9-4c8a-81a9-65c5407a7b0d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:31:04.825171 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:04.825021    2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce51fc53-45c9-4c8a-81a9-65c5407a7b0d-credential-socket\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:31:04.825171 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:04.825043    2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce51fc53-45c9-4c8a-81a9-65c5407a7b0d-workload-certs\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:31:04.825318 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:04.825264    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce51fc53-45c9-4c8a-81a9-65c5407a7b0d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ce51fc53-45c9-4c8a-81a9-65c5407a7b0d" (UID: "ce51fc53-45c9-4c8a-81a9-65c5407a7b0d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:31:04.825373 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:04.825357    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce51fc53-45c9-4c8a-81a9-65c5407a7b0d-istio-data" (OuterVolumeSpecName: "istio-data") pod "ce51fc53-45c9-4c8a-81a9-65c5407a7b0d" (UID: "ce51fc53-45c9-4c8a-81a9-65c5407a7b0d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:31:04.827504 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:04.827475    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce51fc53-45c9-4c8a-81a9-65c5407a7b0d-kube-api-access-qlr7h" (OuterVolumeSpecName: "kube-api-access-qlr7h") pod "ce51fc53-45c9-4c8a-81a9-65c5407a7b0d" (UID: "ce51fc53-45c9-4c8a-81a9-65c5407a7b0d"). InnerVolumeSpecName "kube-api-access-qlr7h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:31:04.827504 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:04.827479    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ce51fc53-45c9-4c8a-81a9-65c5407a7b0d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ce51fc53-45c9-4c8a-81a9-65c5407a7b0d" (UID: "ce51fc53-45c9-4c8a-81a9-65c5407a7b0d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 27 21:31:04.827661 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:04.827490    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce51fc53-45c9-4c8a-81a9-65c5407a7b0d-istio-token" (OuterVolumeSpecName: "istio-token") pod "ce51fc53-45c9-4c8a-81a9-65c5407a7b0d" (UID: "ce51fc53-45c9-4c8a-81a9-65c5407a7b0d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:31:04.827661 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:04.827562    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce51fc53-45c9-4c8a-81a9-65c5407a7b0d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ce51fc53-45c9-4c8a-81a9-65c5407a7b0d" (UID: "ce51fc53-45c9-4c8a-81a9-65c5407a7b0d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:31:04.926150 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:04.926113    2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce51fc53-45c9-4c8a-81a9-65c5407a7b0d-istio-envoy\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:31:04.926150 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:04.926145    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlr7h\" (UniqueName: \"kubernetes.io/projected/ce51fc53-45c9-4c8a-81a9-65c5407a7b0d-kube-api-access-qlr7h\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:31:04.926359 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:04.926159    2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce51fc53-45c9-4c8a-81a9-65c5407a7b0d-istiod-ca-cert\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:31:04.926359 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:04.926172    2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce51fc53-45c9-4c8a-81a9-65c5407a7b0d-workload-socket\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:31:04.926359 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:04.926183    2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce51fc53-45c9-4c8a-81a9-65c5407a7b0d-istio-podinfo\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:31:04.926359 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:04.926193    2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce51fc53-45c9-4c8a-81a9-65c5407a7b0d-istio-data\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:31:04.926359 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:04.926203    2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce51fc53-45c9-4c8a-81a9-65c5407a7b0d-istio-token\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:31:05.630646 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:05.630614    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-55137c-openshift-default-866bbfc94-clccr"
Jul 27 21:31:05.664829 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:05.664799    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-55137c-openshift-default-866bbfc94-clccr"]
Jul 27 21:31:05.670029 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:05.670006    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-55137c-openshift-default-866bbfc94-clccr"]
Jul 27 21:31:05.848322 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:05.848292    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce51fc53-45c9-4c8a-81a9-65c5407a7b0d" path="/var/lib/kubelet/pods/ce51fc53-45c9-4c8a-81a9-65c5407a7b0d/volumes"
Jul 27 21:31:08.697548 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:08.697515    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-5f50cb-openshift-default-5fcbf9d69b-tts4q"]
Jul 27 21:31:08.698548 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:08.698518    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce51fc53-45c9-4c8a-81a9-65c5407a7b0d" containerName="istio-proxy"
Jul 27 21:31:08.698548 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:08.698545    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce51fc53-45c9-4c8a-81a9-65c5407a7b0d" containerName="istio-proxy"
Jul 27 21:31:08.698745 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:08.698571    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48e0e9ca-5236-4288-bba5-c9b873cec6c9" containerName="cleanup"
Jul 27 21:31:08.698745 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:08.698588    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="48e0e9ca-5236-4288-bba5-c9b873cec6c9" containerName="cleanup"
Jul 27 21:31:08.698745 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:08.698683    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce51fc53-45c9-4c8a-81a9-65c5407a7b0d" containerName="istio-proxy"
Jul 27 21:31:08.698745 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:08.698694    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce51fc53-45c9-4c8a-81a9-65c5407a7b0d" containerName="istio-proxy"
Jul 27 21:31:08.703487 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:08.703461    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-5f50cb-openshift-default-5fcbf9d69b-tts4q"
Jul 27 21:31:08.706228 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:08.706194    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-5f50cb-openshift-default-dockercfg-b4jmq\""
Jul 27 21:31:08.711253 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:08.711226    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-5f50cb-openshift-default-5fcbf9d69b-tts4q"]
Jul 27 21:31:08.857338 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:08.857291    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00e24a4b-9175-46f4-b80e-42d8694750cb-istio-envoy\") pod \"e2e-shared-b-5f50cb-openshift-default-5fcbf9d69b-tts4q\" (UID: \"00e24a4b-9175-46f4-b80e-42d8694750cb\") " pod="openshift-ingress/e2e-shared-b-5f50cb-openshift-default-5fcbf9d69b-tts4q"
Jul 27 21:31:08.857338 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:08.857349    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00e24a4b-9175-46f4-b80e-42d8694750cb-istio-podinfo\") pod \"e2e-shared-b-5f50cb-openshift-default-5fcbf9d69b-tts4q\" (UID: \"00e24a4b-9175-46f4-b80e-42d8694750cb\") " pod="openshift-ingress/e2e-shared-b-5f50cb-openshift-default-5fcbf9d69b-tts4q"
Jul 27 21:31:08.857560 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:08.857400    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00e24a4b-9175-46f4-b80e-42d8694750cb-workload-socket\") pod \"e2e-shared-b-5f50cb-openshift-default-5fcbf9d69b-tts4q\" (UID: \"00e24a4b-9175-46f4-b80e-42d8694750cb\") " pod="openshift-ingress/e2e-shared-b-5f50cb-openshift-default-5fcbf9d69b-tts4q"
Jul 27 21:31:08.857560 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:08.857419    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00e24a4b-9175-46f4-b80e-42d8694750cb-istio-token\") pod \"e2e-shared-b-5f50cb-openshift-default-5fcbf9d69b-tts4q\" (UID: \"00e24a4b-9175-46f4-b80e-42d8694750cb\") " pod="openshift-ingress/e2e-shared-b-5f50cb-openshift-default-5fcbf9d69b-tts4q"
Jul 27 21:31:08.857560 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:08.857501    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvjlb\" (UniqueName: \"kubernetes.io/projected/00e24a4b-9175-46f4-b80e-42d8694750cb-kube-api-access-qvjlb\") pod \"e2e-shared-b-5f50cb-openshift-default-5fcbf9d69b-tts4q\" (UID: \"00e24a4b-9175-46f4-b80e-42d8694750cb\") " pod="openshift-ingress/e2e-shared-b-5f50cb-openshift-default-5fcbf9d69b-tts4q"
Jul 27 21:31:08.857560 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:08.857537    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00e24a4b-9175-46f4-b80e-42d8694750cb-istio-data\") pod \"e2e-shared-b-5f50cb-openshift-default-5fcbf9d69b-tts4q\" (UID: \"00e24a4b-9175-46f4-b80e-42d8694750cb\") " pod="openshift-ingress/e2e-shared-b-5f50cb-openshift-default-5fcbf9d69b-tts4q"
Jul 27 21:31:08.857560 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:08.857558    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00e24a4b-9175-46f4-b80e-42d8694750cb-istiod-ca-cert\") pod \"e2e-shared-b-5f50cb-openshift-default-5fcbf9d69b-tts4q\" (UID: \"00e24a4b-9175-46f4-b80e-42d8694750cb\") " pod="openshift-ingress/e2e-shared-b-5f50cb-openshift-default-5fcbf9d69b-tts4q"
Jul 27 21:31:08.857793 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:08.857586    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00e24a4b-9175-46f4-b80e-42d8694750cb-workload-certs\") pod \"e2e-shared-b-5f50cb-openshift-default-5fcbf9d69b-tts4q\" (UID: \"00e24a4b-9175-46f4-b80e-42d8694750cb\") " pod="openshift-ingress/e2e-shared-b-5f50cb-openshift-default-5fcbf9d69b-tts4q"
Jul 27 21:31:08.857793 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:08.857664    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00e24a4b-9175-46f4-b80e-42d8694750cb-credential-socket\") pod \"e2e-shared-b-5f50cb-openshift-default-5fcbf9d69b-tts4q\" (UID: \"00e24a4b-9175-46f4-b80e-42d8694750cb\") " pod="openshift-ingress/e2e-shared-b-5f50cb-openshift-default-5fcbf9d69b-tts4q"
Jul 27 21:31:08.958805 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:08.958721    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00e24a4b-9175-46f4-b80e-42d8694750cb-istiod-ca-cert\") pod \"e2e-shared-b-5f50cb-openshift-default-5fcbf9d69b-tts4q\" (UID: \"00e24a4b-9175-46f4-b80e-42d8694750cb\") " pod="openshift-ingress/e2e-shared-b-5f50cb-openshift-default-5fcbf9d69b-tts4q"
Jul 27 21:31:08.958805 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:08.958758    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00e24a4b-9175-46f4-b80e-42d8694750cb-workload-certs\") pod \"e2e-shared-b-5f50cb-openshift-default-5fcbf9d69b-tts4q\" (UID: \"00e24a4b-9175-46f4-b80e-42d8694750cb\") " pod="openshift-ingress/e2e-shared-b-5f50cb-openshift-default-5fcbf9d69b-tts4q"
Jul 27 21:31:08.958805 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:08.958781    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00e24a4b-9175-46f4-b80e-42d8694750cb-credential-socket\") pod \"e2e-shared-b-5f50cb-openshift-default-5fcbf9d69b-tts4q\" (UID: \"00e24a4b-9175-46f4-b80e-42d8694750cb\") " pod="openshift-ingress/e2e-shared-b-5f50cb-openshift-default-5fcbf9d69b-tts4q"
Jul 27 21:31:08.959107 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:08.958813    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00e24a4b-9175-46f4-b80e-42d8694750cb-istio-envoy\") pod \"e2e-shared-b-5f50cb-openshift-default-5fcbf9d69b-tts4q\" (UID: \"00e24a4b-9175-46f4-b80e-42d8694750cb\") " pod="openshift-ingress/e2e-shared-b-5f50cb-openshift-default-5fcbf9d69b-tts4q"
Jul 27 21:31:08.959107 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:08.958838    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00e24a4b-9175-46f4-b80e-42d8694750cb-istio-podinfo\") pod \"e2e-shared-b-5f50cb-openshift-default-5fcbf9d69b-tts4q\" (UID: \"00e24a4b-9175-46f4-b80e-42d8694750cb\") " pod="openshift-ingress/e2e-shared-b-5f50cb-openshift-default-5fcbf9d69b-tts4q"
Jul 27 21:31:08.959107 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:08.958863    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00e24a4b-9175-46f4-b80e-42d8694750cb-workload-socket\") pod \"e2e-shared-b-5f50cb-openshift-default-5fcbf9d69b-tts4q\" (UID: \"00e24a4b-9175-46f4-b80e-42d8694750cb\") " pod="openshift-ingress/e2e-shared-b-5f50cb-openshift-default-5fcbf9d69b-tts4q"
Jul 27 21:31:08.959107 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:08.958880    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00e24a4b-9175-46f4-b80e-42d8694750cb-istio-token\") pod \"e2e-shared-b-5f50cb-openshift-default-5fcbf9d69b-tts4q\" (UID: \"00e24a4b-9175-46f4-b80e-42d8694750cb\") " pod="openshift-ingress/e2e-shared-b-5f50cb-openshift-default-5fcbf9d69b-tts4q"
Jul 27 21:31:08.959107 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:08.958919    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvjlb\" (UniqueName: \"kubernetes.io/projected/00e24a4b-9175-46f4-b80e-42d8694750cb-kube-api-access-qvjlb\") pod \"e2e-shared-b-5f50cb-openshift-default-5fcbf9d69b-tts4q\" (UID: \"00e24a4b-9175-46f4-b80e-42d8694750cb\") " pod="openshift-ingress/e2e-shared-b-5f50cb-openshift-default-5fcbf9d69b-tts4q"
Jul 27 21:31:08.959107 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:08.958944    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00e24a4b-9175-46f4-b80e-42d8694750cb-istio-data\") pod \"e2e-shared-b-5f50cb-openshift-default-5fcbf9d69b-tts4q\" (UID: \"00e24a4b-9175-46f4-b80e-42d8694750cb\") " pod="openshift-ingress/e2e-shared-b-5f50cb-openshift-default-5fcbf9d69b-tts4q"
Jul 27 21:31:08.959427 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:08.959212    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00e24a4b-9175-46f4-b80e-42d8694750cb-workload-certs\") pod \"e2e-shared-b-5f50cb-openshift-default-5fcbf9d69b-tts4q\" (UID: \"00e24a4b-9175-46f4-b80e-42d8694750cb\") " pod="openshift-ingress/e2e-shared-b-5f50cb-openshift-default-5fcbf9d69b-tts4q"
Jul 27 21:31:08.959488 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:08.959420    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00e24a4b-9175-46f4-b80e-42d8694750cb-istio-data\") pod \"e2e-shared-b-5f50cb-openshift-default-5fcbf9d69b-tts4q\" (UID: \"00e24a4b-9175-46f4-b80e-42d8694750cb\") " pod="openshift-ingress/e2e-shared-b-5f50cb-openshift-default-5fcbf9d69b-tts4q"
Jul 27 21:31:08.959548 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:08.959495    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00e24a4b-9175-46f4-b80e-42d8694750cb-credential-socket\") pod \"e2e-shared-b-5f50cb-openshift-default-5fcbf9d69b-tts4q\" (UID: \"00e24a4b-9175-46f4-b80e-42d8694750cb\") " pod="openshift-ingress/e2e-shared-b-5f50cb-openshift-default-5fcbf9d69b-tts4q"
Jul 27 21:31:08.959548 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:08.959541    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00e24a4b-9175-46f4-b80e-42d8694750cb-workload-socket\") pod \"e2e-shared-b-5f50cb-openshift-default-5fcbf9d69b-tts4q\" (UID: \"00e24a4b-9175-46f4-b80e-42d8694750cb\") " pod="openshift-ingress/e2e-shared-b-5f50cb-openshift-default-5fcbf9d69b-tts4q"
Jul 27 21:31:08.959745 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:08.959721    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00e24a4b-9175-46f4-b80e-42d8694750cb-istiod-ca-cert\") pod \"e2e-shared-b-5f50cb-openshift-default-5fcbf9d69b-tts4q\" (UID: \"00e24a4b-9175-46f4-b80e-42d8694750cb\") " pod="openshift-ingress/e2e-shared-b-5f50cb-openshift-default-5fcbf9d69b-tts4q"
Jul 27 21:31:08.961373 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:08.961353    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00e24a4b-9175-46f4-b80e-42d8694750cb-istio-envoy\") pod \"e2e-shared-b-5f50cb-openshift-default-5fcbf9d69b-tts4q\" (UID: \"00e24a4b-9175-46f4-b80e-42d8694750cb\") " pod="openshift-ingress/e2e-shared-b-5f50cb-openshift-default-5fcbf9d69b-tts4q"
Jul 27 21:31:08.961569 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:08.961549    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00e24a4b-9175-46f4-b80e-42d8694750cb-istio-podinfo\") pod \"e2e-shared-b-5f50cb-openshift-default-5fcbf9d69b-tts4q\" (UID: \"00e24a4b-9175-46f4-b80e-42d8694750cb\") " pod="openshift-ingress/e2e-shared-b-5f50cb-openshift-default-5fcbf9d69b-tts4q"
Jul 27 21:31:08.966682 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:08.966659    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00e24a4b-9175-46f4-b80e-42d8694750cb-istio-token\") pod \"e2e-shared-b-5f50cb-openshift-default-5fcbf9d69b-tts4q\" (UID: \"00e24a4b-9175-46f4-b80e-42d8694750cb\") " pod="openshift-ingress/e2e-shared-b-5f50cb-openshift-default-5fcbf9d69b-tts4q"
Jul 27 21:31:08.967217 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:08.967196    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvjlb\" (UniqueName: \"kubernetes.io/projected/00e24a4b-9175-46f4-b80e-42d8694750cb-kube-api-access-qvjlb\") pod \"e2e-shared-b-5f50cb-openshift-default-5fcbf9d69b-tts4q\" (UID: \"00e24a4b-9175-46f4-b80e-42d8694750cb\") " pod="openshift-ingress/e2e-shared-b-5f50cb-openshift-default-5fcbf9d69b-tts4q"
Jul 27 21:31:09.017162 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:09.017136    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-5f50cb-openshift-default-5fcbf9d69b-tts4q"
Jul 27 21:31:09.153857 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:09.153828    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-5f50cb-openshift-default-5fcbf9d69b-tts4q"]
Jul 27 21:31:09.155694 ip-10-0-141-64 kubenswrapper[2560]: W0727 21:31:09.155666    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00e24a4b_9175_46f4_b80e_42d8694750cb.slice/crio-78439aaaf2f610b9999d1e29184b5dad0603c658e2554ecfd6b431a3e438b4e0 WatchSource:0}: Error finding container 78439aaaf2f610b9999d1e29184b5dad0603c658e2554ecfd6b431a3e438b4e0: Status 404 returned error can't find the container with id 78439aaaf2f610b9999d1e29184b5dad0603c658e2554ecfd6b431a3e438b4e0
Jul 27 21:31:09.157812 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:09.157782    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 27 21:31:09.157892 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:09.157848    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 27 21:31:09.157892 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:09.157876    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 27 21:31:09.650166 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:09.650125    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-5f50cb-openshift-default-5fcbf9d69b-tts4q" event={"ID":"00e24a4b-9175-46f4-b80e-42d8694750cb","Type":"ContainerStarted","Data":"f34160be023c9b3a7260d6fefb9db064a552ef2d13b01e86d44aca5de303d56d"}
Jul 27 21:31:09.650343 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:09.650176    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-5f50cb-openshift-default-5fcbf9d69b-tts4q" event={"ID":"00e24a4b-9175-46f4-b80e-42d8694750cb","Type":"ContainerStarted","Data":"78439aaaf2f610b9999d1e29184b5dad0603c658e2554ecfd6b431a3e438b4e0"}
Jul 27 21:31:09.676046 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:09.675986    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-5f50cb-openshift-default-5fcbf9d69b-tts4q" podStartSLOduration=1.675964453 podStartE2EDuration="1.675964453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:31:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:31:09.674913342 +0000 UTC m=+2386.426775153" watchObservedRunningTime="2026-07-27 21:31:09.675964453 +0000 UTC m=+2386.427826251"
Jul 27 21:31:10.018045 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:10.018014    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-5f50cb-openshift-default-5fcbf9d69b-tts4q"
Jul 27 21:31:10.022940 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:10.022915    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-5f50cb-openshift-default-5fcbf9d69b-tts4q"
Jul 27 21:31:10.655507 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:10.655471    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-5f50cb-openshift-default-5fcbf9d69b-tts4q"
Jul 27 21:31:10.656513 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:10.656497    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-5f50cb-openshift-default-5fcbf9d69b-tts4q"
Jul 27 21:31:12.094970 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:12.094933    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:31:18.655324 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:18.655271    2560 patch_prober.go:28] interesting pod/e2e-shared-a-6dd721-openshift-default-55b7fb86b4-622lp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 27 21:31:18.655725 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:18.655371    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-6dd721-openshift-default-55b7fb86b4-622lp" podUID="8b059bf6-3c81-41ed-b200-965fa935795f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 27 21:31:20.715320 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:20.715279    2560 generic.go:356] "Generic (PLEG): container finished" podID="8b059bf6-3c81-41ed-b200-965fa935795f" containerID="c563cf2cde488be91a55e5dc2f30e67410d582f01d8577329c37fdddd2ce622f" exitCode=0
Jul 27 21:31:20.715766 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:20.715352    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-6dd721-openshift-default-55b7fb86b4-622lp" event={"ID":"8b059bf6-3c81-41ed-b200-965fa935795f","Type":"ContainerDied","Data":"c563cf2cde488be91a55e5dc2f30e67410d582f01d8577329c37fdddd2ce622f"}
Jul 27 21:31:20.715766 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:20.715755    2560 scope.go:117] "RemoveContainer" containerID="c563cf2cde488be91a55e5dc2f30e67410d582f01d8577329c37fdddd2ce622f"
Jul 27 21:31:20.716856 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:20.716828    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 27 21:31:20.716931 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:20.716915    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 27 21:31:20.716972 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:20.716958    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 27 21:31:21.722015 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:21.721978    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-6dd721-openshift-default-55b7fb86b4-622lp" event={"ID":"8b059bf6-3c81-41ed-b200-965fa935795f","Type":"ContainerStarted","Data":"4b347a473a46fbab2678a965d81ca9ad462f7faa395fae2387db9e9db434978d"}
Jul 27 21:31:22.632040 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:22.631996    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-6dd721-openshift-default-55b7fb86b4-622lp"
Jul 27 21:31:22.632259 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:22.632227    2560 patch_prober.go:28] interesting pod/e2e-shared-a-6dd721-openshift-default-55b7fb86b4-622lp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Jul 27 21:31:22.632309 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:22.632278    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-6dd721-openshift-default-55b7fb86b4-622lp" podUID="8b059bf6-3c81-41ed-b200-965fa935795f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Jul 27 21:31:23.206340 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:23.206302    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:31:23.631685 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:23.631648    2560 patch_prober.go:28] interesting pod/e2e-shared-a-6dd721-openshift-default-55b7fb86b4-622lp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Jul 27 21:31:23.631869 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:23.631706    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-6dd721-openshift-default-55b7fb86b4-622lp" podUID="8b059bf6-3c81-41ed-b200-965fa935795f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Jul 27 21:31:23.883387 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:23.883292    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-z6zxv_8a6b7ae6-6c21-453d-99b9-303066d19915/console-operator/1.log"
Jul 27 21:31:23.884718 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:23.884694    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-z6zxv_8a6b7ae6-6c21-453d-99b9-303066d19915/console-operator/1.log"
Jul 27 21:31:23.891353 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:23.891327    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-95hg7_433c431a-b5bf-4734-a03f-6f0a16895ded/ovn-acl-logging/0.log"
Jul 27 21:31:23.893239 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:23.893220    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-95hg7_433c431a-b5bf-4734-a03f-6f0a16895ded/ovn-acl-logging/0.log"
Jul 27 21:31:24.631848 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:24.631800    2560 patch_prober.go:28] interesting pod/e2e-shared-a-6dd721-openshift-default-55b7fb86b4-622lp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Jul 27 21:31:24.632276 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:24.631895    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-6dd721-openshift-default-55b7fb86b4-622lp" podUID="8b059bf6-3c81-41ed-b200-965fa935795f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Jul 27 21:31:25.631957 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:25.631922    2560 patch_prober.go:28] interesting pod/e2e-shared-a-6dd721-openshift-default-55b7fb86b4-622lp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Jul 27 21:31:25.632484 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:25.631975    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-6dd721-openshift-default-55b7fb86b4-622lp" podUID="8b059bf6-3c81-41ed-b200-965fa935795f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Jul 27 21:31:26.635431 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:26.635385    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-6dd721-openshift-default-55b7fb86b4-622lp"
Jul 27 21:31:26.635896 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:26.635823    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-6dd721-openshift-default-55b7fb86b4-622lp"
Jul 27 21:31:26.636619 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:26.636592    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-6dd721-openshift-default-55b7fb86b4-622lp"
Jul 27 21:31:26.698338 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:26.698300    2560 patch_prober.go:28] interesting pod/e2e-shared-b-5f50cb-openshift-default-5fcbf9d69b-tts4q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 27 21:31:26.698526 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:26.698369    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-5f50cb-openshift-default-5fcbf9d69b-tts4q" podUID="00e24a4b-9175-46f4-b80e-42d8694750cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 27 21:31:27.755375 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:27.755339    2560 generic.go:356] "Generic (PLEG): container finished" podID="00e24a4b-9175-46f4-b80e-42d8694750cb" containerID="f34160be023c9b3a7260d6fefb9db064a552ef2d13b01e86d44aca5de303d56d" exitCode=0
Jul 27 21:31:27.755837 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:27.755418    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-5f50cb-openshift-default-5fcbf9d69b-tts4q" event={"ID":"00e24a4b-9175-46f4-b80e-42d8694750cb","Type":"ContainerDied","Data":"f34160be023c9b3a7260d6fefb9db064a552ef2d13b01e86d44aca5de303d56d"}
Jul 27 21:31:27.755890 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:27.755876    2560 scope.go:117] "RemoveContainer" containerID="f34160be023c9b3a7260d6fefb9db064a552ef2d13b01e86d44aca5de303d56d"
Jul 27 21:31:27.756751 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:27.756725    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 27 21:31:27.756841 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:27.756791    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 27 21:31:27.756841 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:27.756822    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 27 21:31:28.763337 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:28.763300    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-5f50cb-openshift-default-5fcbf9d69b-tts4q" event={"ID":"00e24a4b-9175-46f4-b80e-42d8694750cb","Type":"ContainerStarted","Data":"550028813a7471e91b053b2c8a0c0e111e47b11bd8ffb7a7990f01b91634e8cd"}
Jul 27 21:31:28.802832 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:28.802796    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-6dd721/auth-test-model-6dd721-kserve-84b66dd459-s8lqk"]
Jul 27 21:31:28.803206 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:28.803193    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce51fc53-45c9-4c8a-81a9-65c5407a7b0d" containerName="istio-proxy"
Jul 27 21:31:28.803264 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:28.803209    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce51fc53-45c9-4c8a-81a9-65c5407a7b0d" containerName="istio-proxy"
Jul 27 21:31:28.805296 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:28.805233    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-6dd721/auth-test-model-6dd721-kserve-84b66dd459-s8lqk"
Jul 27 21:31:28.807950 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:28.807720    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-6dd721\"/\"kube-root-ca.crt\""
Jul 27 21:31:28.807950 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:28.807805    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-6dd721\"/\"openshift-service-ca.crt\""
Jul 27 21:31:28.807950 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:28.807861    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-6dd721\"/\"default-dockercfg-7jnzq\""
Jul 27 21:31:28.808184 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:28.808141    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-6dd721\"/\"auth-test-model-6dd721-kserve-self-signed-certs\""
Jul 27 21:31:28.817333 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:28.817310    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-6dd721/auth-test-model-6dd721-kserve-84b66dd459-s8lqk"]
Jul 27 21:31:28.834676 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:28.834655    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/adbeb9a4-b5aa-40ec-9d34-fb2ce992fda1-tmp-dir\") pod \"auth-test-model-6dd721-kserve-84b66dd459-s8lqk\" (UID: \"adbeb9a4-b5aa-40ec-9d34-fb2ce992fda1\") " pod="ai-tenant-e2e-shared-a-6dd721/auth-test-model-6dd721-kserve-84b66dd459-s8lqk"
Jul 27 21:31:28.834810 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:28.834689    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psz7z\" (UniqueName: \"kubernetes.io/projected/adbeb9a4-b5aa-40ec-9d34-fb2ce992fda1-kube-api-access-psz7z\") pod \"auth-test-model-6dd721-kserve-84b66dd459-s8lqk\" (UID: \"adbeb9a4-b5aa-40ec-9d34-fb2ce992fda1\") " pod="ai-tenant-e2e-shared-a-6dd721/auth-test-model-6dd721-kserve-84b66dd459-s8lqk"
Jul 27 21:31:28.834810 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:28.834782    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/adbeb9a4-b5aa-40ec-9d34-fb2ce992fda1-tls-certs\") pod \"auth-test-model-6dd721-kserve-84b66dd459-s8lqk\" (UID: \"adbeb9a4-b5aa-40ec-9d34-fb2ce992fda1\") " pod="ai-tenant-e2e-shared-a-6dd721/auth-test-model-6dd721-kserve-84b66dd459-s8lqk"
Jul 27 21:31:28.835025 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:28.835000    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/adbeb9a4-b5aa-40ec-9d34-fb2ce992fda1-home\") pod \"auth-test-model-6dd721-kserve-84b66dd459-s8lqk\" (UID: \"adbeb9a4-b5aa-40ec-9d34-fb2ce992fda1\") " pod="ai-tenant-e2e-shared-a-6dd721/auth-test-model-6dd721-kserve-84b66dd459-s8lqk"
Jul 27 21:31:28.835123 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:28.835033    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/adbeb9a4-b5aa-40ec-9d34-fb2ce992fda1-dshm\") pod \"auth-test-model-6dd721-kserve-84b66dd459-s8lqk\" (UID: \"adbeb9a4-b5aa-40ec-9d34-fb2ce992fda1\") " pod="ai-tenant-e2e-shared-a-6dd721/auth-test-model-6dd721-kserve-84b66dd459-s8lqk"
Jul 27 21:31:28.835123 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:28.835062    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/adbeb9a4-b5aa-40ec-9d34-fb2ce992fda1-model-cache\") pod \"auth-test-model-6dd721-kserve-84b66dd459-s8lqk\" (UID: \"adbeb9a4-b5aa-40ec-9d34-fb2ce992fda1\") " pod="ai-tenant-e2e-shared-a-6dd721/auth-test-model-6dd721-kserve-84b66dd459-s8lqk"
Jul 27 21:31:28.936479 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:28.936442    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/adbeb9a4-b5aa-40ec-9d34-fb2ce992fda1-tls-certs\") pod \"auth-test-model-6dd721-kserve-84b66dd459-s8lqk\" (UID: \"adbeb9a4-b5aa-40ec-9d34-fb2ce992fda1\") " pod="ai-tenant-e2e-shared-a-6dd721/auth-test-model-6dd721-kserve-84b66dd459-s8lqk"
Jul 27 21:31:28.936679 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:28.936506    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/adbeb9a4-b5aa-40ec-9d34-fb2ce992fda1-home\") pod \"auth-test-model-6dd721-kserve-84b66dd459-s8lqk\" (UID: \"adbeb9a4-b5aa-40ec-9d34-fb2ce992fda1\") " pod="ai-tenant-e2e-shared-a-6dd721/auth-test-model-6dd721-kserve-84b66dd459-s8lqk"
Jul 27 21:31:28.936679 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:28.936532    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/adbeb9a4-b5aa-40ec-9d34-fb2ce992fda1-dshm\") pod \"auth-test-model-6dd721-kserve-84b66dd459-s8lqk\" (UID: \"adbeb9a4-b5aa-40ec-9d34-fb2ce992fda1\") " pod="ai-tenant-e2e-shared-a-6dd721/auth-test-model-6dd721-kserve-84b66dd459-s8lqk"
Jul 27 21:31:28.936679 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:28.936555    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/adbeb9a4-b5aa-40ec-9d34-fb2ce992fda1-model-cache\") pod \"auth-test-model-6dd721-kserve-84b66dd459-s8lqk\" (UID: \"adbeb9a4-b5aa-40ec-9d34-fb2ce992fda1\") " pod="ai-tenant-e2e-shared-a-6dd721/auth-test-model-6dd721-kserve-84b66dd459-s8lqk"
Jul 27 21:31:28.936679 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:28.936636    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/adbeb9a4-b5aa-40ec-9d34-fb2ce992fda1-tmp-dir\") pod \"auth-test-model-6dd721-kserve-84b66dd459-s8lqk\" (UID: \"adbeb9a4-b5aa-40ec-9d34-fb2ce992fda1\") " pod="ai-tenant-e2e-shared-a-6dd721/auth-test-model-6dd721-kserve-84b66dd459-s8lqk"
Jul 27 21:31:28.936902 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:28.936838    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psz7z\" (UniqueName: \"kubernetes.io/projected/adbeb9a4-b5aa-40ec-9d34-fb2ce992fda1-kube-api-access-psz7z\") pod \"auth-test-model-6dd721-kserve-84b66dd459-s8lqk\" (UID: \"adbeb9a4-b5aa-40ec-9d34-fb2ce992fda1\") " pod="ai-tenant-e2e-shared-a-6dd721/auth-test-model-6dd721-kserve-84b66dd459-s8lqk"
Jul 27 21:31:28.936902 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:28.936880    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/adbeb9a4-b5aa-40ec-9d34-fb2ce992fda1-home\") pod \"auth-test-model-6dd721-kserve-84b66dd459-s8lqk\" (UID: \"adbeb9a4-b5aa-40ec-9d34-fb2ce992fda1\") " pod="ai-tenant-e2e-shared-a-6dd721/auth-test-model-6dd721-kserve-84b66dd459-s8lqk"
Jul 27 21:31:28.937309 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:28.937281    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/adbeb9a4-b5aa-40ec-9d34-fb2ce992fda1-model-cache\") pod \"auth-test-model-6dd721-kserve-84b66dd459-s8lqk\" (UID: \"adbeb9a4-b5aa-40ec-9d34-fb2ce992fda1\") " pod="ai-tenant-e2e-shared-a-6dd721/auth-test-model-6dd721-kserve-84b66dd459-s8lqk"
Jul 27 21:31:28.937409 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:28.937322    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/adbeb9a4-b5aa-40ec-9d34-fb2ce992fda1-tmp-dir\") pod \"auth-test-model-6dd721-kserve-84b66dd459-s8lqk\" (UID: \"adbeb9a4-b5aa-40ec-9d34-fb2ce992fda1\") " pod="ai-tenant-e2e-shared-a-6dd721/auth-test-model-6dd721-kserve-84b66dd459-s8lqk"
Jul 27 21:31:28.939018 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:28.938991    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/adbeb9a4-b5aa-40ec-9d34-fb2ce992fda1-dshm\") pod \"auth-test-model-6dd721-kserve-84b66dd459-s8lqk\" (UID: \"adbeb9a4-b5aa-40ec-9d34-fb2ce992fda1\") " pod="ai-tenant-e2e-shared-a-6dd721/auth-test-model-6dd721-kserve-84b66dd459-s8lqk"
Jul 27 21:31:28.940127 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:28.939218    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/adbeb9a4-b5aa-40ec-9d34-fb2ce992fda1-tls-certs\") pod \"auth-test-model-6dd721-kserve-84b66dd459-s8lqk\" (UID: \"adbeb9a4-b5aa-40ec-9d34-fb2ce992fda1\") " pod="ai-tenant-e2e-shared-a-6dd721/auth-test-model-6dd721-kserve-84b66dd459-s8lqk"
Jul 27 21:31:28.946152 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:28.946121    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-psz7z\" (UniqueName: \"kubernetes.io/projected/adbeb9a4-b5aa-40ec-9d34-fb2ce992fda1-kube-api-access-psz7z\") pod \"auth-test-model-6dd721-kserve-84b66dd459-s8lqk\" (UID: \"adbeb9a4-b5aa-40ec-9d34-fb2ce992fda1\") " pod="ai-tenant-e2e-shared-a-6dd721/auth-test-model-6dd721-kserve-84b66dd459-s8lqk"
Jul 27 21:31:29.018104 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:29.018053    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-5f50cb-openshift-default-5fcbf9d69b-tts4q"
Jul 27 21:31:29.018286 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:29.018262    2560 patch_prober.go:28] interesting pod/e2e-shared-b-5f50cb-openshift-default-5fcbf9d69b-tts4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Jul 27 21:31:29.018347 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:29.018326    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-5f50cb-openshift-default-5fcbf9d69b-tts4q" podUID="00e24a4b-9175-46f4-b80e-42d8694750cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Jul 27 21:31:29.115767 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:29.115725    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-6dd721/auth-test-model-6dd721-kserve-84b66dd459-s8lqk"
Jul 27 21:31:29.280208 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:29.280183    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-6dd721/auth-test-model-6dd721-kserve-84b66dd459-s8lqk"]
Jul 27 21:31:29.282645 ip-10-0-141-64 kubenswrapper[2560]: W0727 21:31:29.282608    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podadbeb9a4_b5aa_40ec_9d34_fb2ce992fda1.slice/crio-68f86ef65d7cae7010f467615599e604eac24611962a6377ec93ae277ef0d9f4 WatchSource:0}: Error finding container 68f86ef65d7cae7010f467615599e604eac24611962a6377ec93ae277ef0d9f4: Status 404 returned error can't find the container with id 68f86ef65d7cae7010f467615599e604eac24611962a6377ec93ae277ef0d9f4
Jul 27 21:31:29.770829 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:29.770792    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-6dd721/auth-test-model-6dd721-kserve-84b66dd459-s8lqk" event={"ID":"adbeb9a4-b5aa-40ec-9d34-fb2ce992fda1","Type":"ContainerStarted","Data":"31bf141b6ccaa0d82c8af3f416f70ea0b94e7ff00ef17a0f438bf51de94aede4"}
Jul 27 21:31:29.771273 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:29.770841    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-6dd721/auth-test-model-6dd721-kserve-84b66dd459-s8lqk" event={"ID":"adbeb9a4-b5aa-40ec-9d34-fb2ce992fda1","Type":"ContainerStarted","Data":"68f86ef65d7cae7010f467615599e604eac24611962a6377ec93ae277ef0d9f4"}
Jul 27 21:31:29.787593 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:29.787552    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-6dd721/auth-test-model-6dd721-kserve-84b66dd459-s8lqk" podStartSLOduration=1.78753758 podStartE2EDuration="1.78753758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:31:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:31:29.786001772 +0000 UTC m=+2406.537863566" watchObservedRunningTime="2026-07-27 21:31:29.78753758 +0000 UTC m=+2406.539399353"
Jul 27 21:31:30.017522 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:30.017481    2560 patch_prober.go:28] interesting pod/e2e-shared-b-5f50cb-openshift-default-5fcbf9d69b-tts4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Jul 27 21:31:30.017700 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:30.017547    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-5f50cb-openshift-default-5fcbf9d69b-tts4q" podUID="00e24a4b-9175-46f4-b80e-42d8694750cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Jul 27 21:31:31.018496 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:31.018451    2560 patch_prober.go:28] interesting pod/e2e-shared-b-5f50cb-openshift-default-5fcbf9d69b-tts4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Jul 27 21:31:31.018942 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:31.018521    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-5f50cb-openshift-default-5fcbf9d69b-tts4q" podUID="00e24a4b-9175-46f4-b80e-42d8694750cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Jul 27 21:31:32.018049 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:32.018007    2560 patch_prober.go:28] interesting pod/e2e-shared-b-5f50cb-openshift-default-5fcbf9d69b-tts4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Jul 27 21:31:32.018254 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:32.018069    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-5f50cb-openshift-default-5fcbf9d69b-tts4q" podUID="00e24a4b-9175-46f4-b80e-42d8694750cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Jul 27 21:31:32.100644 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:32.100601    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:31:33.017554 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:33.017513    2560 patch_prober.go:28] interesting pod/e2e-shared-b-5f50cb-openshift-default-5fcbf9d69b-tts4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Jul 27 21:31:33.017735 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:33.017573    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-5f50cb-openshift-default-5fcbf9d69b-tts4q" podUID="00e24a4b-9175-46f4-b80e-42d8694750cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Jul 27 21:31:34.021398 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:34.021367    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-5f50cb-openshift-default-5fcbf9d69b-tts4q"
Jul 27 21:31:34.021802 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:34.021640    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-5f50cb-openshift-default-5fcbf9d69b-tts4q"
Jul 27 21:31:34.022297 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:34.022278    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-5f50cb-openshift-default-5fcbf9d69b-tts4q"
Jul 27 21:31:39.116649 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:39.116614    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-6dd721/auth-test-model-6dd721-kserve-84b66dd459-s8lqk"
Jul 27 21:31:39.117080 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:39.116664    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-6dd721/auth-test-model-6dd721-kserve-84b66dd459-s8lqk"
Jul 27 21:31:39.129214 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:39.129181    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-6dd721/auth-test-model-6dd721-kserve-84b66dd459-s8lqk"
Jul 27 21:31:39.829511 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:39.829474    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-6dd721/auth-test-model-6dd721-kserve-84b66dd459-s8lqk"
Jul 27 21:31:58.891119 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:31:58.891052    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:32:23.292104 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:32:23.292063    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:32:44.090234 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:32:44.090198    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:33:00.090929 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:33:00.090886    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:33:20.592888 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:33:20.592803    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:33:33.396320 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:33:33.396284    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:33:45.088349 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:33:45.088314    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:33:59.698795 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:33:59.698753    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:34:18.303542 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:34:18.303506    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:34:41.704606 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:34:41.704567    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:34:59.100550 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:34:59.100510    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:35:24.018792 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:35:24.018754    2560 scope.go:117] "RemoveContainer" containerID="35af375f87bf6be52fda4341a6aa2cb1a33fbdb020fc890f99c63bf4f5a2e9cd"
Jul 27 21:35:26.217273 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:35:26.217233    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:35:40.115916 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:35:40.115877    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:35:51.205467 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:35:51.205430    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:36:06.591740 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:36:06.591704    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:36:11.170348 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:36:11.170314    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-6dd721/auth-test-model-6dd721-kserve-84b66dd459-s8lqk"]
Jul 27 21:36:23.914679 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:36:23.914648    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-z6zxv_8a6b7ae6-6c21-453d-99b9-303066d19915/console-operator/1.log"
Jul 27 21:36:23.916844 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:36:23.916825    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-z6zxv_8a6b7ae6-6c21-453d-99b9-303066d19915/console-operator/1.log"
Jul 27 21:36:23.921398 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:36:23.921379    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-95hg7_433c431a-b5bf-4734-a03f-6f0a16895ded/ovn-acl-logging/0.log"
Jul 27 21:36:23.923459 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:36:23.923442    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-95hg7_433c431a-b5bf-4734-a03f-6f0a16895ded/ovn-acl-logging/0.log"
Jul 27 21:36:24.037961 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:36:24.037925    2560 scope.go:117] "RemoveContainer" containerID="71262ccfc51cbd3237c94989117be89e45ba3de39abf9b14747e038ad11c1848"
Jul 27 21:36:24.046685 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:36:24.046666    2560 scope.go:117] "RemoveContainer" containerID="3fc6a8ed5ba08383b634cc3220d24e7772cb644005370b08a7ed9b9f1099a20d"
Jul 27 21:36:24.193005 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:36:24.192929    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:36:26.189750 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:36:26.189685    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-6dd721/auth-test-model-6dd721-kserve-84b66dd459-s8lqk" podUID="adbeb9a4-b5aa-40ec-9d34-fb2ce992fda1" containerName="main" containerID="cri-o://31bf141b6ccaa0d82c8af3f416f70ea0b94e7ff00ef17a0f438bf51de94aede4" gracePeriod=45
Jul 27 21:36:26.443662 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:36:26.443592    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-6dd721/auth-test-model-6dd721-kserve-84b66dd459-s8lqk"
Jul 27 21:36:26.513957 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:36:26.513930    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/adbeb9a4-b5aa-40ec-9d34-fb2ce992fda1-home\") pod \"adbeb9a4-b5aa-40ec-9d34-fb2ce992fda1\" (UID: \"adbeb9a4-b5aa-40ec-9d34-fb2ce992fda1\") "
Jul 27 21:36:26.514142 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:36:26.513992    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/adbeb9a4-b5aa-40ec-9d34-fb2ce992fda1-tmp-dir\") pod \"adbeb9a4-b5aa-40ec-9d34-fb2ce992fda1\" (UID: \"adbeb9a4-b5aa-40ec-9d34-fb2ce992fda1\") "
Jul 27 21:36:26.514142 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:36:26.514057    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/adbeb9a4-b5aa-40ec-9d34-fb2ce992fda1-dshm\") pod \"adbeb9a4-b5aa-40ec-9d34-fb2ce992fda1\" (UID: \"adbeb9a4-b5aa-40ec-9d34-fb2ce992fda1\") "
Jul 27 21:36:26.514142 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:36:26.514113    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-psz7z\" (UniqueName: \"kubernetes.io/projected/adbeb9a4-b5aa-40ec-9d34-fb2ce992fda1-kube-api-access-psz7z\") pod \"adbeb9a4-b5aa-40ec-9d34-fb2ce992fda1\" (UID: \"adbeb9a4-b5aa-40ec-9d34-fb2ce992fda1\") "
Jul 27 21:36:26.514291 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:36:26.514147    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/adbeb9a4-b5aa-40ec-9d34-fb2ce992fda1-tls-certs\") pod \"adbeb9a4-b5aa-40ec-9d34-fb2ce992fda1\" (UID: \"adbeb9a4-b5aa-40ec-9d34-fb2ce992fda1\") "
Jul 27 21:36:26.514291 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:36:26.514193    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/adbeb9a4-b5aa-40ec-9d34-fb2ce992fda1-model-cache\") pod \"adbeb9a4-b5aa-40ec-9d34-fb2ce992fda1\" (UID: \"adbeb9a4-b5aa-40ec-9d34-fb2ce992fda1\") "
Jul 27 21:36:26.514291 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:36:26.514213    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adbeb9a4-b5aa-40ec-9d34-fb2ce992fda1-home" (OuterVolumeSpecName: "home") pod "adbeb9a4-b5aa-40ec-9d34-fb2ce992fda1" (UID: "adbeb9a4-b5aa-40ec-9d34-fb2ce992fda1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:36:26.514447 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:36:26.514329    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adbeb9a4-b5aa-40ec-9d34-fb2ce992fda1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "adbeb9a4-b5aa-40ec-9d34-fb2ce992fda1" (UID: "adbeb9a4-b5aa-40ec-9d34-fb2ce992fda1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:36:26.514501 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:36:26.514454    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adbeb9a4-b5aa-40ec-9d34-fb2ce992fda1-model-cache" (OuterVolumeSpecName: "model-cache") pod "adbeb9a4-b5aa-40ec-9d34-fb2ce992fda1" (UID: "adbeb9a4-b5aa-40ec-9d34-fb2ce992fda1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:36:26.514558 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:36:26.514517    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/adbeb9a4-b5aa-40ec-9d34-fb2ce992fda1-model-cache\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:36:26.514558 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:36:26.514535    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/adbeb9a4-b5aa-40ec-9d34-fb2ce992fda1-home\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:36:26.514558 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:36:26.514546    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/adbeb9a4-b5aa-40ec-9d34-fb2ce992fda1-tmp-dir\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:36:26.516486 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:36:26.516455    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adbeb9a4-b5aa-40ec-9d34-fb2ce992fda1-dshm" (OuterVolumeSpecName: "dshm") pod "adbeb9a4-b5aa-40ec-9d34-fb2ce992fda1" (UID: "adbeb9a4-b5aa-40ec-9d34-fb2ce992fda1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:36:26.516486 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:36:26.516452    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/adbeb9a4-b5aa-40ec-9d34-fb2ce992fda1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "adbeb9a4-b5aa-40ec-9d34-fb2ce992fda1" (UID: "adbeb9a4-b5aa-40ec-9d34-fb2ce992fda1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:36:26.516486 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:36:26.516471    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adbeb9a4-b5aa-40ec-9d34-fb2ce992fda1-kube-api-access-psz7z" (OuterVolumeSpecName: "kube-api-access-psz7z") pod "adbeb9a4-b5aa-40ec-9d34-fb2ce992fda1" (UID: "adbeb9a4-b5aa-40ec-9d34-fb2ce992fda1"). InnerVolumeSpecName "kube-api-access-psz7z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:36:26.614991 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:36:26.614956    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/adbeb9a4-b5aa-40ec-9d34-fb2ce992fda1-dshm\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:36:26.614991 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:36:26.614987    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-psz7z\" (UniqueName: \"kubernetes.io/projected/adbeb9a4-b5aa-40ec-9d34-fb2ce992fda1-kube-api-access-psz7z\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:36:26.614991 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:36:26.614998    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/adbeb9a4-b5aa-40ec-9d34-fb2ce992fda1-tls-certs\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:36:27.223687 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:36:27.223653    2560 generic.go:356] "Generic (PLEG): container finished" podID="adbeb9a4-b5aa-40ec-9d34-fb2ce992fda1" containerID="31bf141b6ccaa0d82c8af3f416f70ea0b94e7ff00ef17a0f438bf51de94aede4" exitCode=0
Jul 27 21:36:27.226047 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:36:27.223697    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-6dd721/auth-test-model-6dd721-kserve-84b66dd459-s8lqk"
Jul 27 21:36:27.226047 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:36:27.223700    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-6dd721/auth-test-model-6dd721-kserve-84b66dd459-s8lqk" event={"ID":"adbeb9a4-b5aa-40ec-9d34-fb2ce992fda1","Type":"ContainerDied","Data":"31bf141b6ccaa0d82c8af3f416f70ea0b94e7ff00ef17a0f438bf51de94aede4"}
Jul 27 21:36:27.226047 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:36:27.223827    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-6dd721/auth-test-model-6dd721-kserve-84b66dd459-s8lqk" event={"ID":"adbeb9a4-b5aa-40ec-9d34-fb2ce992fda1","Type":"ContainerDied","Data":"68f86ef65d7cae7010f467615599e604eac24611962a6377ec93ae277ef0d9f4"}
Jul 27 21:36:27.226047 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:36:27.223852    2560 scope.go:117] "RemoveContainer" containerID="31bf141b6ccaa0d82c8af3f416f70ea0b94e7ff00ef17a0f438bf51de94aede4"
Jul 27 21:36:27.232971 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:36:27.232956    2560 scope.go:117] "RemoveContainer" containerID="31bf141b6ccaa0d82c8af3f416f70ea0b94e7ff00ef17a0f438bf51de94aede4"
Jul 27 21:36:27.233240 ip-10-0-141-64 kubenswrapper[2560]: E0727 21:36:27.233221    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31bf141b6ccaa0d82c8af3f416f70ea0b94e7ff00ef17a0f438bf51de94aede4\": container with ID starting with 31bf141b6ccaa0d82c8af3f416f70ea0b94e7ff00ef17a0f438bf51de94aede4 not found: ID does not exist" containerID="31bf141b6ccaa0d82c8af3f416f70ea0b94e7ff00ef17a0f438bf51de94aede4"
Jul 27 21:36:27.233319 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:36:27.233251    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31bf141b6ccaa0d82c8af3f416f70ea0b94e7ff00ef17a0f438bf51de94aede4"} err="failed to get container status \"31bf141b6ccaa0d82c8af3f416f70ea0b94e7ff00ef17a0f438bf51de94aede4\": rpc error: code = NotFound desc = could not find container \"31bf141b6ccaa0d82c8af3f416f70ea0b94e7ff00ef17a0f438bf51de94aede4\": container with ID starting with 31bf141b6ccaa0d82c8af3f416f70ea0b94e7ff00ef17a0f438bf51de94aede4 not found: ID does not exist"
Jul 27 21:36:27.244141 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:36:27.244111    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-6dd721/auth-test-model-6dd721-kserve-84b66dd459-s8lqk"]
Jul 27 21:36:27.248469 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:36:27.248448    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-6dd721/auth-test-model-6dd721-kserve-84b66dd459-s8lqk"]
Jul 27 21:36:27.848293 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:36:27.848253    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="adbeb9a4-b5aa-40ec-9d34-fb2ce992fda1" path="/var/lib/kubelet/pods/adbeb9a4-b5aa-40ec-9d34-fb2ce992fda1/volumes"
Jul 27 21:36:39.503008 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:36:39.502968    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-5f50cb/sub-test-model-5f50cb-kserve-dd956c885-gcxpr"]
Jul 27 21:36:39.503449 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:36:39.503347    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="adbeb9a4-b5aa-40ec-9d34-fb2ce992fda1" containerName="main"
Jul 27 21:36:39.503449 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:36:39.503358    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="adbeb9a4-b5aa-40ec-9d34-fb2ce992fda1" containerName="main"
Jul 27 21:36:39.503449 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:36:39.503435    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="adbeb9a4-b5aa-40ec-9d34-fb2ce992fda1" containerName="main"
Jul 27 21:36:39.506306 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:36:39.506289    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-5f50cb/sub-test-model-5f50cb-kserve-dd956c885-gcxpr"
Jul 27 21:36:39.508940 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:36:39.508920    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-5f50cb\"/\"kube-root-ca.crt\""
Jul 27 21:36:39.509043 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:36:39.508949    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-5f50cb\"/\"openshift-service-ca.crt\""
Jul 27 21:36:39.509783 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:36:39.509759    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-5f50cb\"/\"default-dockercfg-54kfm\""
Jul 27 21:36:39.509848 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:36:39.509829    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-5f50cb\"/\"sub-test-model-5f50cb-kserve-self-signed-certs\""
Jul 27 21:36:39.516434 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:36:39.516412    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-5f50cb/sub-test-model-5f50cb-kserve-dd956c885-gcxpr"]
Jul 27 21:36:39.630893 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:36:39.630854    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0cf10bb-9d19-4779-9e55-e7c0cae84429-tmp-dir\") pod \"sub-test-model-5f50cb-kserve-dd956c885-gcxpr\" (UID: \"d0cf10bb-9d19-4779-9e55-e7c0cae84429\") " pod="ai-tenant-e2e-shared-b-5f50cb/sub-test-model-5f50cb-kserve-dd956c885-gcxpr"
Jul 27 21:36:39.630893 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:36:39.630898    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbzh4\" (UniqueName: \"kubernetes.io/projected/d0cf10bb-9d19-4779-9e55-e7c0cae84429-kube-api-access-zbzh4\") pod \"sub-test-model-5f50cb-kserve-dd956c885-gcxpr\" (UID: \"d0cf10bb-9d19-4779-9e55-e7c0cae84429\") " pod="ai-tenant-e2e-shared-b-5f50cb/sub-test-model-5f50cb-kserve-dd956c885-gcxpr"
Jul 27 21:36:39.631118 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:36:39.630936    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0cf10bb-9d19-4779-9e55-e7c0cae84429-model-cache\") pod \"sub-test-model-5f50cb-kserve-dd956c885-gcxpr\" (UID: \"d0cf10bb-9d19-4779-9e55-e7c0cae84429\") " pod="ai-tenant-e2e-shared-b-5f50cb/sub-test-model-5f50cb-kserve-dd956c885-gcxpr"
Jul 27 21:36:39.631118 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:36:39.630966    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0cf10bb-9d19-4779-9e55-e7c0cae84429-tls-certs\") pod \"sub-test-model-5f50cb-kserve-dd956c885-gcxpr\" (UID: \"d0cf10bb-9d19-4779-9e55-e7c0cae84429\") " pod="ai-tenant-e2e-shared-b-5f50cb/sub-test-model-5f50cb-kserve-dd956c885-gcxpr"
Jul 27 21:36:39.631118 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:36:39.631047    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0cf10bb-9d19-4779-9e55-e7c0cae84429-dshm\") pod \"sub-test-model-5f50cb-kserve-dd956c885-gcxpr\" (UID: \"d0cf10bb-9d19-4779-9e55-e7c0cae84429\") " pod="ai-tenant-e2e-shared-b-5f50cb/sub-test-model-5f50cb-kserve-dd956c885-gcxpr"
Jul 27 21:36:39.631118 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:36:39.631107    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0cf10bb-9d19-4779-9e55-e7c0cae84429-home\") pod \"sub-test-model-5f50cb-kserve-dd956c885-gcxpr\" (UID: \"d0cf10bb-9d19-4779-9e55-e7c0cae84429\") " pod="ai-tenant-e2e-shared-b-5f50cb/sub-test-model-5f50cb-kserve-dd956c885-gcxpr"
Jul 27 21:36:39.732291 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:36:39.732247    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0cf10bb-9d19-4779-9e55-e7c0cae84429-model-cache\") pod \"sub-test-model-5f50cb-kserve-dd956c885-gcxpr\" (UID: \"d0cf10bb-9d19-4779-9e55-e7c0cae84429\") " pod="ai-tenant-e2e-shared-b-5f50cb/sub-test-model-5f50cb-kserve-dd956c885-gcxpr"
Jul 27 21:36:39.732489 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:36:39.732310    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0cf10bb-9d19-4779-9e55-e7c0cae84429-tls-certs\") pod \"sub-test-model-5f50cb-kserve-dd956c885-gcxpr\" (UID: \"d0cf10bb-9d19-4779-9e55-e7c0cae84429\") " pod="ai-tenant-e2e-shared-b-5f50cb/sub-test-model-5f50cb-kserve-dd956c885-gcxpr"
Jul 27 21:36:39.732489 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:36:39.732366    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0cf10bb-9d19-4779-9e55-e7c0cae84429-dshm\") pod \"sub-test-model-5f50cb-kserve-dd956c885-gcxpr\" (UID: \"d0cf10bb-9d19-4779-9e55-e7c0cae84429\") " pod="ai-tenant-e2e-shared-b-5f50cb/sub-test-model-5f50cb-kserve-dd956c885-gcxpr"
Jul 27 21:36:39.732489 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:36:39.732396    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0cf10bb-9d19-4779-9e55-e7c0cae84429-home\") pod \"sub-test-model-5f50cb-kserve-dd956c885-gcxpr\" (UID: \"d0cf10bb-9d19-4779-9e55-e7c0cae84429\") " pod="ai-tenant-e2e-shared-b-5f50cb/sub-test-model-5f50cb-kserve-dd956c885-gcxpr"
Jul 27 21:36:39.732489 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:36:39.732425    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0cf10bb-9d19-4779-9e55-e7c0cae84429-tmp-dir\") pod \"sub-test-model-5f50cb-kserve-dd956c885-gcxpr\" (UID: \"d0cf10bb-9d19-4779-9e55-e7c0cae84429\") " pod="ai-tenant-e2e-shared-b-5f50cb/sub-test-model-5f50cb-kserve-dd956c885-gcxpr"
Jul 27 21:36:39.732489 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:36:39.732454    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbzh4\" (UniqueName: \"kubernetes.io/projected/d0cf10bb-9d19-4779-9e55-e7c0cae84429-kube-api-access-zbzh4\") pod \"sub-test-model-5f50cb-kserve-dd956c885-gcxpr\" (UID: \"d0cf10bb-9d19-4779-9e55-e7c0cae84429\") " pod="ai-tenant-e2e-shared-b-5f50cb/sub-test-model-5f50cb-kserve-dd956c885-gcxpr"
Jul 27 21:36:39.732749 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:36:39.732656    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0cf10bb-9d19-4779-9e55-e7c0cae84429-model-cache\") pod \"sub-test-model-5f50cb-kserve-dd956c885-gcxpr\" (UID: \"d0cf10bb-9d19-4779-9e55-e7c0cae84429\") " pod="ai-tenant-e2e-shared-b-5f50cb/sub-test-model-5f50cb-kserve-dd956c885-gcxpr"
Jul 27 21:36:39.732829 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:36:39.732806    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0cf10bb-9d19-4779-9e55-e7c0cae84429-tmp-dir\") pod \"sub-test-model-5f50cb-kserve-dd956c885-gcxpr\" (UID: \"d0cf10bb-9d19-4779-9e55-e7c0cae84429\") " pod="ai-tenant-e2e-shared-b-5f50cb/sub-test-model-5f50cb-kserve-dd956c885-gcxpr"
Jul 27 21:36:39.732883 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:36:39.732833    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0cf10bb-9d19-4779-9e55-e7c0cae84429-home\") pod \"sub-test-model-5f50cb-kserve-dd956c885-gcxpr\" (UID: \"d0cf10bb-9d19-4779-9e55-e7c0cae84429\") " pod="ai-tenant-e2e-shared-b-5f50cb/sub-test-model-5f50cb-kserve-dd956c885-gcxpr"
Jul 27 21:36:39.734819 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:36:39.734790    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0cf10bb-9d19-4779-9e55-e7c0cae84429-dshm\") pod \"sub-test-model-5f50cb-kserve-dd956c885-gcxpr\" (UID: \"d0cf10bb-9d19-4779-9e55-e7c0cae84429\") " pod="ai-tenant-e2e-shared-b-5f50cb/sub-test-model-5f50cb-kserve-dd956c885-gcxpr"
Jul 27 21:36:39.735053 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:36:39.735037    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0cf10bb-9d19-4779-9e55-e7c0cae84429-tls-certs\") pod \"sub-test-model-5f50cb-kserve-dd956c885-gcxpr\" (UID: \"d0cf10bb-9d19-4779-9e55-e7c0cae84429\") " pod="ai-tenant-e2e-shared-b-5f50cb/sub-test-model-5f50cb-kserve-dd956c885-gcxpr"
Jul 27 21:36:39.742281 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:36:39.742252    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbzh4\" (UniqueName: \"kubernetes.io/projected/d0cf10bb-9d19-4779-9e55-e7c0cae84429-kube-api-access-zbzh4\") pod \"sub-test-model-5f50cb-kserve-dd956c885-gcxpr\" (UID: \"d0cf10bb-9d19-4779-9e55-e7c0cae84429\") " pod="ai-tenant-e2e-shared-b-5f50cb/sub-test-model-5f50cb-kserve-dd956c885-gcxpr"
Jul 27 21:36:39.816410 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:36:39.816380    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-5f50cb/sub-test-model-5f50cb-kserve-dd956c885-gcxpr"
Jul 27 21:36:39.954069 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:36:39.954043    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-5f50cb/sub-test-model-5f50cb-kserve-dd956c885-gcxpr"]
Jul 27 21:36:39.956383 ip-10-0-141-64 kubenswrapper[2560]: W0727 21:36:39.956354    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd0cf10bb_9d19_4779_9e55_e7c0cae84429.slice/crio-4698acdaf77d47dbe86f4b54e3843c82fb601149ec3f08abe776578825281031 WatchSource:0}: Error finding container 4698acdaf77d47dbe86f4b54e3843c82fb601149ec3f08abe776578825281031: Status 404 returned error can't find the container with id 4698acdaf77d47dbe86f4b54e3843c82fb601149ec3f08abe776578825281031
Jul 27 21:36:40.286282 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:36:40.286246    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-5f50cb/sub-test-model-5f50cb-kserve-dd956c885-gcxpr" event={"ID":"d0cf10bb-9d19-4779-9e55-e7c0cae84429","Type":"ContainerStarted","Data":"6183998204fe89aeaf4f5919bedd3c7dae26c31189db4bd17df232f79da2a225"}
Jul 27 21:36:40.286282 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:36:40.286287    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-5f50cb/sub-test-model-5f50cb-kserve-dd956c885-gcxpr" event={"ID":"d0cf10bb-9d19-4779-9e55-e7c0cae84429","Type":"ContainerStarted","Data":"4698acdaf77d47dbe86f4b54e3843c82fb601149ec3f08abe776578825281031"}
Jul 27 21:36:40.305628 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:36:40.305572    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-5f50cb/sub-test-model-5f50cb-kserve-dd956c885-gcxpr" podStartSLOduration=1.3055557389999999 podStartE2EDuration="1.305555739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:36:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:36:40.304600342 +0000 UTC m=+2717.056462136" watchObservedRunningTime="2026-07-27 21:36:40.305555739 +0000 UTC m=+2717.057417549"
Jul 27 21:36:43.093976 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:36:43.093945    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:36:49.817349 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:36:49.817310    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-5f50cb/sub-test-model-5f50cb-kserve-dd956c885-gcxpr"
Jul 27 21:36:49.817830 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:36:49.817391    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-5f50cb/sub-test-model-5f50cb-kserve-dd956c885-gcxpr"
Jul 27 21:36:49.830075 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:36:49.830047    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-5f50cb/sub-test-model-5f50cb-kserve-dd956c885-gcxpr"
Jul 27 21:36:50.345951 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:36:50.345922    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-5f50cb/sub-test-model-5f50cb-kserve-dd956c885-gcxpr"
Jul 27 21:37:30.898328 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:37:30.898287    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:37:49.794249 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:37:49.794211    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:37:56.988009 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:37:56.987975    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-5f50cb/sub-test-model-5f50cb-kserve-dd956c885-gcxpr"]
Jul 27 21:38:08.208035 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:38:08.207998    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:38:12.008529 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:38:12.008473    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-5f50cb/sub-test-model-5f50cb-kserve-dd956c885-gcxpr" podUID="d0cf10bb-9d19-4779-9e55-e7c0cae84429" containerName="main" containerID="cri-o://6183998204fe89aeaf4f5919bedd3c7dae26c31189db4bd17df232f79da2a225" gracePeriod=45
Jul 27 21:38:12.262611 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:38:12.262551    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-5f50cb/sub-test-model-5f50cb-kserve-dd956c885-gcxpr"
Jul 27 21:38:12.365552 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:38:12.365514    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0cf10bb-9d19-4779-9e55-e7c0cae84429-model-cache\") pod \"d0cf10bb-9d19-4779-9e55-e7c0cae84429\" (UID: \"d0cf10bb-9d19-4779-9e55-e7c0cae84429\") "
Jul 27 21:38:12.365738 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:38:12.365580    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0cf10bb-9d19-4779-9e55-e7c0cae84429-dshm\") pod \"d0cf10bb-9d19-4779-9e55-e7c0cae84429\" (UID: \"d0cf10bb-9d19-4779-9e55-e7c0cae84429\") "
Jul 27 21:38:12.365738 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:38:12.365645    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0cf10bb-9d19-4779-9e55-e7c0cae84429-tmp-dir\") pod \"d0cf10bb-9d19-4779-9e55-e7c0cae84429\" (UID: \"d0cf10bb-9d19-4779-9e55-e7c0cae84429\") "
Jul 27 21:38:12.365738 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:38:12.365668    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0cf10bb-9d19-4779-9e55-e7c0cae84429-home\") pod \"d0cf10bb-9d19-4779-9e55-e7c0cae84429\" (UID: \"d0cf10bb-9d19-4779-9e55-e7c0cae84429\") "
Jul 27 21:38:12.365907 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:38:12.365763    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zbzh4\" (UniqueName: \"kubernetes.io/projected/d0cf10bb-9d19-4779-9e55-e7c0cae84429-kube-api-access-zbzh4\") pod \"d0cf10bb-9d19-4779-9e55-e7c0cae84429\" (UID: \"d0cf10bb-9d19-4779-9e55-e7c0cae84429\") "
Jul 27 21:38:12.365907 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:38:12.365795    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0cf10bb-9d19-4779-9e55-e7c0cae84429-tls-certs\") pod \"d0cf10bb-9d19-4779-9e55-e7c0cae84429\" (UID: \"d0cf10bb-9d19-4779-9e55-e7c0cae84429\") "
Jul 27 21:38:12.365907 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:38:12.365862    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0cf10bb-9d19-4779-9e55-e7c0cae84429-model-cache" (OuterVolumeSpecName: "model-cache") pod "d0cf10bb-9d19-4779-9e55-e7c0cae84429" (UID: "d0cf10bb-9d19-4779-9e55-e7c0cae84429"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:38:12.366037 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:38:12.365916    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0cf10bb-9d19-4779-9e55-e7c0cae84429-home" (OuterVolumeSpecName: "home") pod "d0cf10bb-9d19-4779-9e55-e7c0cae84429" (UID: "d0cf10bb-9d19-4779-9e55-e7c0cae84429"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:38:12.366037 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:38:12.365929    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0cf10bb-9d19-4779-9e55-e7c0cae84429-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d0cf10bb-9d19-4779-9e55-e7c0cae84429" (UID: "d0cf10bb-9d19-4779-9e55-e7c0cae84429"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:38:12.366189 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:38:12.366171    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0cf10bb-9d19-4779-9e55-e7c0cae84429-model-cache\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:38:12.366244 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:38:12.366194    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0cf10bb-9d19-4779-9e55-e7c0cae84429-tmp-dir\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:38:12.366244 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:38:12.366203    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0cf10bb-9d19-4779-9e55-e7c0cae84429-home\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:38:12.368081 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:38:12.368037    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0cf10bb-9d19-4779-9e55-e7c0cae84429-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d0cf10bb-9d19-4779-9e55-e7c0cae84429" (UID: "d0cf10bb-9d19-4779-9e55-e7c0cae84429"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:38:12.368258 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:38:12.368141    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0cf10bb-9d19-4779-9e55-e7c0cae84429-dshm" (OuterVolumeSpecName: "dshm") pod "d0cf10bb-9d19-4779-9e55-e7c0cae84429" (UID: "d0cf10bb-9d19-4779-9e55-e7c0cae84429"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:38:12.368258 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:38:12.368219    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0cf10bb-9d19-4779-9e55-e7c0cae84429-kube-api-access-zbzh4" (OuterVolumeSpecName: "kube-api-access-zbzh4") pod "d0cf10bb-9d19-4779-9e55-e7c0cae84429" (UID: "d0cf10bb-9d19-4779-9e55-e7c0cae84429"). InnerVolumeSpecName "kube-api-access-zbzh4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:38:12.467531 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:38:12.467498    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0cf10bb-9d19-4779-9e55-e7c0cae84429-dshm\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:38:12.467531 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:38:12.467526    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbzh4\" (UniqueName: \"kubernetes.io/projected/d0cf10bb-9d19-4779-9e55-e7c0cae84429-kube-api-access-zbzh4\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:38:12.467531 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:38:12.467536    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0cf10bb-9d19-4779-9e55-e7c0cae84429-tls-certs\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:38:12.744367 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:38:12.744329    2560 generic.go:356] "Generic (PLEG): container finished" podID="d0cf10bb-9d19-4779-9e55-e7c0cae84429" containerID="6183998204fe89aeaf4f5919bedd3c7dae26c31189db4bd17df232f79da2a225" exitCode=0
Jul 27 21:38:12.744570 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:38:12.744371    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-5f50cb/sub-test-model-5f50cb-kserve-dd956c885-gcxpr"
Jul 27 21:38:12.744570 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:38:12.744411    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-5f50cb/sub-test-model-5f50cb-kserve-dd956c885-gcxpr" event={"ID":"d0cf10bb-9d19-4779-9e55-e7c0cae84429","Type":"ContainerDied","Data":"6183998204fe89aeaf4f5919bedd3c7dae26c31189db4bd17df232f79da2a225"}
Jul 27 21:38:12.744570 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:38:12.744462    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-5f50cb/sub-test-model-5f50cb-kserve-dd956c885-gcxpr" event={"ID":"d0cf10bb-9d19-4779-9e55-e7c0cae84429","Type":"ContainerDied","Data":"4698acdaf77d47dbe86f4b54e3843c82fb601149ec3f08abe776578825281031"}
Jul 27 21:38:12.744570 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:38:12.744485    2560 scope.go:117] "RemoveContainer" containerID="6183998204fe89aeaf4f5919bedd3c7dae26c31189db4bd17df232f79da2a225"
Jul 27 21:38:12.753569 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:38:12.753552    2560 scope.go:117] "RemoveContainer" containerID="6183998204fe89aeaf4f5919bedd3c7dae26c31189db4bd17df232f79da2a225"
Jul 27 21:38:12.753839 ip-10-0-141-64 kubenswrapper[2560]: E0727 21:38:12.753821    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6183998204fe89aeaf4f5919bedd3c7dae26c31189db4bd17df232f79da2a225\": container with ID starting with 6183998204fe89aeaf4f5919bedd3c7dae26c31189db4bd17df232f79da2a225 not found: ID does not exist" containerID="6183998204fe89aeaf4f5919bedd3c7dae26c31189db4bd17df232f79da2a225"
Jul 27 21:38:12.753891 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:38:12.753848    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6183998204fe89aeaf4f5919bedd3c7dae26c31189db4bd17df232f79da2a225"} err="failed to get container status \"6183998204fe89aeaf4f5919bedd3c7dae26c31189db4bd17df232f79da2a225\": rpc error: code = NotFound desc = could not find container \"6183998204fe89aeaf4f5919bedd3c7dae26c31189db4bd17df232f79da2a225\": container with ID starting with 6183998204fe89aeaf4f5919bedd3c7dae26c31189db4bd17df232f79da2a225 not found: ID does not exist"
Jul 27 21:38:12.765286 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:38:12.765265    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-5f50cb/sub-test-model-5f50cb-kserve-dd956c885-gcxpr"]
Jul 27 21:38:12.776458 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:38:12.772613    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-5f50cb/sub-test-model-5f50cb-kserve-dd956c885-gcxpr"]
Jul 27 21:38:13.847405 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:38:13.847370    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0cf10bb-9d19-4779-9e55-e7c0cae84429" path="/var/lib/kubelet/pods/d0cf10bb-9d19-4779-9e55-e7c0cae84429/volumes"
Jul 27 21:38:17.906674 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:38:17.906633    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:38:34.805000 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:38:34.804968    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-5f50cb/rate-test-model-5f50cb-kserve-7c69d7c95f-jwj2d"]
Jul 27 21:38:34.805578 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:38:34.805332    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0cf10bb-9d19-4779-9e55-e7c0cae84429" containerName="main"
Jul 27 21:38:34.805578 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:38:34.805345    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0cf10bb-9d19-4779-9e55-e7c0cae84429" containerName="main"
Jul 27 21:38:34.805578 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:38:34.805449    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0cf10bb-9d19-4779-9e55-e7c0cae84429" containerName="main"
Jul 27 21:38:34.808577 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:38:34.808556    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-5f50cb/rate-test-model-5f50cb-kserve-7c69d7c95f-jwj2d"
Jul 27 21:38:34.811453 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:38:34.811418    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-5f50cb\"/\"openshift-service-ca.crt\""
Jul 27 21:38:34.811638 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:38:34.811615    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-5f50cb\"/\"default-dockercfg-54kfm\""
Jul 27 21:38:34.812542 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:38:34.812520    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-5f50cb\"/\"kube-root-ca.crt\""
Jul 27 21:38:34.812662 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:38:34.812561    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-5f50cb\"/\"rate-test-model-5f50cb-kserve-self-signed-certs\""
Jul 27 21:38:34.819133 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:38:34.819069    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-5f50cb/rate-test-model-5f50cb-kserve-7c69d7c95f-jwj2d"]
Jul 27 21:38:34.876383 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:38:34.876350    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7a39b1e-74a3-443f-b5be-f40fbfd6af68-dshm\") pod \"rate-test-model-5f50cb-kserve-7c69d7c95f-jwj2d\" (UID: \"a7a39b1e-74a3-443f-b5be-f40fbfd6af68\") " pod="ai-tenant-e2e-shared-b-5f50cb/rate-test-model-5f50cb-kserve-7c69d7c95f-jwj2d"
Jul 27 21:38:34.876529 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:38:34.876384    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7a39b1e-74a3-443f-b5be-f40fbfd6af68-model-cache\") pod \"rate-test-model-5f50cb-kserve-7c69d7c95f-jwj2d\" (UID: \"a7a39b1e-74a3-443f-b5be-f40fbfd6af68\") " pod="ai-tenant-e2e-shared-b-5f50cb/rate-test-model-5f50cb-kserve-7c69d7c95f-jwj2d"
Jul 27 21:38:34.876529 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:38:34.876430    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddmlx\" (UniqueName: \"kubernetes.io/projected/a7a39b1e-74a3-443f-b5be-f40fbfd6af68-kube-api-access-ddmlx\") pod \"rate-test-model-5f50cb-kserve-7c69d7c95f-jwj2d\" (UID: \"a7a39b1e-74a3-443f-b5be-f40fbfd6af68\") " pod="ai-tenant-e2e-shared-b-5f50cb/rate-test-model-5f50cb-kserve-7c69d7c95f-jwj2d"
Jul 27 21:38:34.876529 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:38:34.876460    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7a39b1e-74a3-443f-b5be-f40fbfd6af68-tmp-dir\") pod \"rate-test-model-5f50cb-kserve-7c69d7c95f-jwj2d\" (UID: \"a7a39b1e-74a3-443f-b5be-f40fbfd6af68\") " pod="ai-tenant-e2e-shared-b-5f50cb/rate-test-model-5f50cb-kserve-7c69d7c95f-jwj2d"
Jul 27 21:38:34.876529 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:38:34.876484    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7a39b1e-74a3-443f-b5be-f40fbfd6af68-home\") pod \"rate-test-model-5f50cb-kserve-7c69d7c95f-jwj2d\" (UID: \"a7a39b1e-74a3-443f-b5be-f40fbfd6af68\") " pod="ai-tenant-e2e-shared-b-5f50cb/rate-test-model-5f50cb-kserve-7c69d7c95f-jwj2d"
Jul 27 21:38:34.876684 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:38:34.876604    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7a39b1e-74a3-443f-b5be-f40fbfd6af68-tls-certs\") pod \"rate-test-model-5f50cb-kserve-7c69d7c95f-jwj2d\" (UID: \"a7a39b1e-74a3-443f-b5be-f40fbfd6af68\") " pod="ai-tenant-e2e-shared-b-5f50cb/rate-test-model-5f50cb-kserve-7c69d7c95f-jwj2d"
Jul 27 21:38:34.977477 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:38:34.977446    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7a39b1e-74a3-443f-b5be-f40fbfd6af68-tls-certs\") pod \"rate-test-model-5f50cb-kserve-7c69d7c95f-jwj2d\" (UID: \"a7a39b1e-74a3-443f-b5be-f40fbfd6af68\") " pod="ai-tenant-e2e-shared-b-5f50cb/rate-test-model-5f50cb-kserve-7c69d7c95f-jwj2d"
Jul 27 21:38:34.977638 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:38:34.977500    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7a39b1e-74a3-443f-b5be-f40fbfd6af68-dshm\") pod \"rate-test-model-5f50cb-kserve-7c69d7c95f-jwj2d\" (UID: \"a7a39b1e-74a3-443f-b5be-f40fbfd6af68\") " pod="ai-tenant-e2e-shared-b-5f50cb/rate-test-model-5f50cb-kserve-7c69d7c95f-jwj2d"
Jul 27 21:38:34.977638 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:38:34.977516    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7a39b1e-74a3-443f-b5be-f40fbfd6af68-model-cache\") pod \"rate-test-model-5f50cb-kserve-7c69d7c95f-jwj2d\" (UID: \"a7a39b1e-74a3-443f-b5be-f40fbfd6af68\") " pod="ai-tenant-e2e-shared-b-5f50cb/rate-test-model-5f50cb-kserve-7c69d7c95f-jwj2d"
Jul 27 21:38:34.977638 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:38:34.977557    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddmlx\" (UniqueName: \"kubernetes.io/projected/a7a39b1e-74a3-443f-b5be-f40fbfd6af68-kube-api-access-ddmlx\") pod \"rate-test-model-5f50cb-kserve-7c69d7c95f-jwj2d\" (UID: \"a7a39b1e-74a3-443f-b5be-f40fbfd6af68\") " pod="ai-tenant-e2e-shared-b-5f50cb/rate-test-model-5f50cb-kserve-7c69d7c95f-jwj2d"
Jul 27 21:38:34.977927 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:38:34.977673    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7a39b1e-74a3-443f-b5be-f40fbfd6af68-tmp-dir\") pod \"rate-test-model-5f50cb-kserve-7c69d7c95f-jwj2d\" (UID: \"a7a39b1e-74a3-443f-b5be-f40fbfd6af68\") " pod="ai-tenant-e2e-shared-b-5f50cb/rate-test-model-5f50cb-kserve-7c69d7c95f-jwj2d"
Jul 27 21:38:34.978000 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:38:34.977968    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7a39b1e-74a3-443f-b5be-f40fbfd6af68-home\") pod \"rate-test-model-5f50cb-kserve-7c69d7c95f-jwj2d\" (UID: \"a7a39b1e-74a3-443f-b5be-f40fbfd6af68\") " pod="ai-tenant-e2e-shared-b-5f50cb/rate-test-model-5f50cb-kserve-7c69d7c95f-jwj2d"
Jul 27 21:38:34.978054 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:38:34.978024    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7a39b1e-74a3-443f-b5be-f40fbfd6af68-model-cache\") pod \"rate-test-model-5f50cb-kserve-7c69d7c95f-jwj2d\" (UID: \"a7a39b1e-74a3-443f-b5be-f40fbfd6af68\") " pod="ai-tenant-e2e-shared-b-5f50cb/rate-test-model-5f50cb-kserve-7c69d7c95f-jwj2d"
Jul 27 21:38:34.978217 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:38:34.978179    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7a39b1e-74a3-443f-b5be-f40fbfd6af68-tmp-dir\") pod \"rate-test-model-5f50cb-kserve-7c69d7c95f-jwj2d\" (UID: \"a7a39b1e-74a3-443f-b5be-f40fbfd6af68\") " pod="ai-tenant-e2e-shared-b-5f50cb/rate-test-model-5f50cb-kserve-7c69d7c95f-jwj2d"
Jul 27 21:38:34.978378 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:38:34.978357    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7a39b1e-74a3-443f-b5be-f40fbfd6af68-home\") pod \"rate-test-model-5f50cb-kserve-7c69d7c95f-jwj2d\" (UID: \"a7a39b1e-74a3-443f-b5be-f40fbfd6af68\") " pod="ai-tenant-e2e-shared-b-5f50cb/rate-test-model-5f50cb-kserve-7c69d7c95f-jwj2d"
Jul 27 21:38:34.980330 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:38:34.980310    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7a39b1e-74a3-443f-b5be-f40fbfd6af68-dshm\") pod \"rate-test-model-5f50cb-kserve-7c69d7c95f-jwj2d\" (UID: \"a7a39b1e-74a3-443f-b5be-f40fbfd6af68\") " pod="ai-tenant-e2e-shared-b-5f50cb/rate-test-model-5f50cb-kserve-7c69d7c95f-jwj2d"
Jul 27 21:38:34.980517 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:38:34.980499    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7a39b1e-74a3-443f-b5be-f40fbfd6af68-tls-certs\") pod \"rate-test-model-5f50cb-kserve-7c69d7c95f-jwj2d\" (UID: \"a7a39b1e-74a3-443f-b5be-f40fbfd6af68\") " pod="ai-tenant-e2e-shared-b-5f50cb/rate-test-model-5f50cb-kserve-7c69d7c95f-jwj2d"
Jul 27 21:38:34.987156 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:38:34.987128    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddmlx\" (UniqueName: \"kubernetes.io/projected/a7a39b1e-74a3-443f-b5be-f40fbfd6af68-kube-api-access-ddmlx\") pod \"rate-test-model-5f50cb-kserve-7c69d7c95f-jwj2d\" (UID: \"a7a39b1e-74a3-443f-b5be-f40fbfd6af68\") " pod="ai-tenant-e2e-shared-b-5f50cb/rate-test-model-5f50cb-kserve-7c69d7c95f-jwj2d"
Jul 27 21:38:35.120192 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:38:35.120110    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-5f50cb/rate-test-model-5f50cb-kserve-7c69d7c95f-jwj2d"
Jul 27 21:38:35.259927 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:38:35.259898    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-5f50cb/rate-test-model-5f50cb-kserve-7c69d7c95f-jwj2d"]
Jul 27 21:38:35.262158 ip-10-0-141-64 kubenswrapper[2560]: W0727 21:38:35.262125    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda7a39b1e_74a3_443f_b5be_f40fbfd6af68.slice/crio-1370eea8c3c22b691c89258068f52c719d5df319231c35712e570fac0de07826 WatchSource:0}: Error finding container 1370eea8c3c22b691c89258068f52c719d5df319231c35712e570fac0de07826: Status 404 returned error can't find the container with id 1370eea8c3c22b691c89258068f52c719d5df319231c35712e570fac0de07826
Jul 27 21:38:35.853607 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:38:35.853562    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-5f50cb/rate-test-model-5f50cb-kserve-7c69d7c95f-jwj2d" event={"ID":"a7a39b1e-74a3-443f-b5be-f40fbfd6af68","Type":"ContainerStarted","Data":"b6bae84686af76efba0d88fc9c96490dde7be2053faf22ce52e4404bf554af49"}
Jul 27 21:38:35.853607 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:38:35.853609    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-5f50cb/rate-test-model-5f50cb-kserve-7c69d7c95f-jwj2d" event={"ID":"a7a39b1e-74a3-443f-b5be-f40fbfd6af68","Type":"ContainerStarted","Data":"1370eea8c3c22b691c89258068f52c719d5df319231c35712e570fac0de07826"}
Jul 27 21:38:35.873065 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:38:35.873013    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-5f50cb/rate-test-model-5f50cb-kserve-7c69d7c95f-jwj2d" podStartSLOduration=1.872995798 podStartE2EDuration="1.872995798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:38:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:38:35.870934897 +0000 UTC m=+2832.622796707" watchObservedRunningTime="2026-07-27 21:38:35.872995798 +0000 UTC m=+2832.624857599"
Jul 27 21:38:38.195005 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:38:38.194973    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:38:45.120629 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:38:45.120592    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-5f50cb/rate-test-model-5f50cb-kserve-7c69d7c95f-jwj2d"
Jul 27 21:38:45.121035 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:38:45.120642    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-5f50cb/rate-test-model-5f50cb-kserve-7c69d7c95f-jwj2d"
Jul 27 21:38:45.133083 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:38:45.133056    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-5f50cb/rate-test-model-5f50cb-kserve-7c69d7c95f-jwj2d"
Jul 27 21:38:45.923539 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:38:45.923509    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-5f50cb/rate-test-model-5f50cb-kserve-7c69d7c95f-jwj2d"
Jul 27 21:39:21.395766 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:39:21.395677    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:39:35.917789 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:39:35.917747    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:39:53.701827 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:39:53.701789    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:39:59.580978 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:39:59.580944    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-5f50cb/rate-test-model-5f50cb-kserve-7c69d7c95f-jwj2d"]
Jul 27 21:40:09.286594 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:09.286555    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:40:14.600639 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:14.600591    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-5f50cb/rate-test-model-5f50cb-kserve-7c69d7c95f-jwj2d" podUID="a7a39b1e-74a3-443f-b5be-f40fbfd6af68" containerName="main" containerID="cri-o://b6bae84686af76efba0d88fc9c96490dde7be2053faf22ce52e4404bf554af49" gracePeriod=45
Jul 27 21:40:14.853176 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:14.853117    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-5f50cb/rate-test-model-5f50cb-kserve-7c69d7c95f-jwj2d"
Jul 27 21:40:14.969232 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:14.969189    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ddmlx\" (UniqueName: \"kubernetes.io/projected/a7a39b1e-74a3-443f-b5be-f40fbfd6af68-kube-api-access-ddmlx\") pod \"a7a39b1e-74a3-443f-b5be-f40fbfd6af68\" (UID: \"a7a39b1e-74a3-443f-b5be-f40fbfd6af68\") "
Jul 27 21:40:14.969427 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:14.969283    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7a39b1e-74a3-443f-b5be-f40fbfd6af68-tmp-dir\") pod \"a7a39b1e-74a3-443f-b5be-f40fbfd6af68\" (UID: \"a7a39b1e-74a3-443f-b5be-f40fbfd6af68\") "
Jul 27 21:40:14.969427 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:14.969318    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7a39b1e-74a3-443f-b5be-f40fbfd6af68-home\") pod \"a7a39b1e-74a3-443f-b5be-f40fbfd6af68\" (UID: \"a7a39b1e-74a3-443f-b5be-f40fbfd6af68\") "
Jul 27 21:40:14.969427 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:14.969336    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7a39b1e-74a3-443f-b5be-f40fbfd6af68-model-cache\") pod \"a7a39b1e-74a3-443f-b5be-f40fbfd6af68\" (UID: \"a7a39b1e-74a3-443f-b5be-f40fbfd6af68\") "
Jul 27 21:40:14.969427 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:14.969368    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7a39b1e-74a3-443f-b5be-f40fbfd6af68-dshm\") pod \"a7a39b1e-74a3-443f-b5be-f40fbfd6af68\" (UID: \"a7a39b1e-74a3-443f-b5be-f40fbfd6af68\") "
Jul 27 21:40:14.969427 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:14.969391    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7a39b1e-74a3-443f-b5be-f40fbfd6af68-tls-certs\") pod \"a7a39b1e-74a3-443f-b5be-f40fbfd6af68\" (UID: \"a7a39b1e-74a3-443f-b5be-f40fbfd6af68\") "
Jul 27 21:40:14.969651 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:14.969591    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7a39b1e-74a3-443f-b5be-f40fbfd6af68-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a7a39b1e-74a3-443f-b5be-f40fbfd6af68" (UID: "a7a39b1e-74a3-443f-b5be-f40fbfd6af68"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:40:14.969651 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:14.969604    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7a39b1e-74a3-443f-b5be-f40fbfd6af68-home" (OuterVolumeSpecName: "home") pod "a7a39b1e-74a3-443f-b5be-f40fbfd6af68" (UID: "a7a39b1e-74a3-443f-b5be-f40fbfd6af68"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:40:14.969651 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:14.969627    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7a39b1e-74a3-443f-b5be-f40fbfd6af68-model-cache" (OuterVolumeSpecName: "model-cache") pod "a7a39b1e-74a3-443f-b5be-f40fbfd6af68" (UID: "a7a39b1e-74a3-443f-b5be-f40fbfd6af68"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:40:14.969805 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:14.969711    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7a39b1e-74a3-443f-b5be-f40fbfd6af68-tmp-dir\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:40:14.969805 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:14.969723    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7a39b1e-74a3-443f-b5be-f40fbfd6af68-home\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:40:14.969805 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:14.969731    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7a39b1e-74a3-443f-b5be-f40fbfd6af68-model-cache\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:40:14.971602 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:14.971577    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7a39b1e-74a3-443f-b5be-f40fbfd6af68-kube-api-access-ddmlx" (OuterVolumeSpecName: "kube-api-access-ddmlx") pod "a7a39b1e-74a3-443f-b5be-f40fbfd6af68" (UID: "a7a39b1e-74a3-443f-b5be-f40fbfd6af68"). InnerVolumeSpecName "kube-api-access-ddmlx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:40:14.971602 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:14.971585    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7a39b1e-74a3-443f-b5be-f40fbfd6af68-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a7a39b1e-74a3-443f-b5be-f40fbfd6af68" (UID: "a7a39b1e-74a3-443f-b5be-f40fbfd6af68"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:40:14.971602 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:14.971577    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7a39b1e-74a3-443f-b5be-f40fbfd6af68-dshm" (OuterVolumeSpecName: "dshm") pod "a7a39b1e-74a3-443f-b5be-f40fbfd6af68" (UID: "a7a39b1e-74a3-443f-b5be-f40fbfd6af68"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:40:15.070987 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:15.070953    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddmlx\" (UniqueName: \"kubernetes.io/projected/a7a39b1e-74a3-443f-b5be-f40fbfd6af68-kube-api-access-ddmlx\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:40:15.070987 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:15.070982    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7a39b1e-74a3-443f-b5be-f40fbfd6af68-dshm\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:40:15.070987 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:15.070993    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7a39b1e-74a3-443f-b5be-f40fbfd6af68-tls-certs\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:40:15.338169 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:15.338128    2560 generic.go:356] "Generic (PLEG): container finished" podID="a7a39b1e-74a3-443f-b5be-f40fbfd6af68" containerID="b6bae84686af76efba0d88fc9c96490dde7be2053faf22ce52e4404bf554af49" exitCode=0
Jul 27 21:40:15.338169 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:15.338168    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-5f50cb/rate-test-model-5f50cb-kserve-7c69d7c95f-jwj2d"
Jul 27 21:40:15.338423 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:15.338206    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-5f50cb/rate-test-model-5f50cb-kserve-7c69d7c95f-jwj2d" event={"ID":"a7a39b1e-74a3-443f-b5be-f40fbfd6af68","Type":"ContainerDied","Data":"b6bae84686af76efba0d88fc9c96490dde7be2053faf22ce52e4404bf554af49"}
Jul 27 21:40:15.338423 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:15.338245    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-5f50cb/rate-test-model-5f50cb-kserve-7c69d7c95f-jwj2d" event={"ID":"a7a39b1e-74a3-443f-b5be-f40fbfd6af68","Type":"ContainerDied","Data":"1370eea8c3c22b691c89258068f52c719d5df319231c35712e570fac0de07826"}
Jul 27 21:40:15.338423 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:15.338262    2560 scope.go:117] "RemoveContainer" containerID="b6bae84686af76efba0d88fc9c96490dde7be2053faf22ce52e4404bf554af49"
Jul 27 21:40:15.347611 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:15.347596    2560 scope.go:117] "RemoveContainer" containerID="b6bae84686af76efba0d88fc9c96490dde7be2053faf22ce52e4404bf554af49"
Jul 27 21:40:15.347865 ip-10-0-141-64 kubenswrapper[2560]: E0727 21:40:15.347846    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6bae84686af76efba0d88fc9c96490dde7be2053faf22ce52e4404bf554af49\": container with ID starting with b6bae84686af76efba0d88fc9c96490dde7be2053faf22ce52e4404bf554af49 not found: ID does not exist" containerID="b6bae84686af76efba0d88fc9c96490dde7be2053faf22ce52e4404bf554af49"
Jul 27 21:40:15.347929 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:15.347877    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6bae84686af76efba0d88fc9c96490dde7be2053faf22ce52e4404bf554af49"} err="failed to get container status \"b6bae84686af76efba0d88fc9c96490dde7be2053faf22ce52e4404bf554af49\": rpc error: code = NotFound desc = could not find container \"b6bae84686af76efba0d88fc9c96490dde7be2053faf22ce52e4404bf554af49\": container with ID starting with b6bae84686af76efba0d88fc9c96490dde7be2053faf22ce52e4404bf554af49 not found: ID does not exist"
Jul 27 21:40:15.359537 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:15.359512    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-5f50cb/rate-test-model-5f50cb-kserve-7c69d7c95f-jwj2d"]
Jul 27 21:40:15.361239 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:15.361218    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-5f50cb/rate-test-model-5f50cb-kserve-7c69d7c95f-jwj2d"]
Jul 27 21:40:15.848170 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:15.848134    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7a39b1e-74a3-443f-b5be-f40fbfd6af68" path="/var/lib/kubelet/pods/a7a39b1e-74a3-443f-b5be-f40fbfd6af68/volumes"
Jul 27 21:40:19.239044 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:19.239015    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-b4bf5c-openshift-default-84869c884b-sxxsq"]
Jul 27 21:40:19.239583 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:19.239561    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7a39b1e-74a3-443f-b5be-f40fbfd6af68" containerName="main"
Jul 27 21:40:19.239629 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:19.239588    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7a39b1e-74a3-443f-b5be-f40fbfd6af68" containerName="main"
Jul 27 21:40:19.239699 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:19.239688    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7a39b1e-74a3-443f-b5be-f40fbfd6af68" containerName="main"
Jul 27 21:40:19.244346 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:19.244329    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-b4bf5c-openshift-default-84869c884b-sxxsq"
Jul 27 21:40:19.246761 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:19.246742    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-b4bf5c-openshift-default-dockercfg-bf79g\""
Jul 27 21:40:19.257789 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:19.257766    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-b4bf5c-openshift-default-84869c884b-sxxsq"]
Jul 27 21:40:19.309035 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:19.309003    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/911cf451-bf1d-4c8b-a240-288b2cd55fc1-istio-podinfo\") pod \"e2e-ipp-a-b4bf5c-openshift-default-84869c884b-sxxsq\" (UID: \"911cf451-bf1d-4c8b-a240-288b2cd55fc1\") " pod="openshift-ingress/e2e-ipp-a-b4bf5c-openshift-default-84869c884b-sxxsq"
Jul 27 21:40:19.309202 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:19.309062    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/911cf451-bf1d-4c8b-a240-288b2cd55fc1-credential-socket\") pod \"e2e-ipp-a-b4bf5c-openshift-default-84869c884b-sxxsq\" (UID: \"911cf451-bf1d-4c8b-a240-288b2cd55fc1\") " pod="openshift-ingress/e2e-ipp-a-b4bf5c-openshift-default-84869c884b-sxxsq"
Jul 27 21:40:19.309202 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:19.309101    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/911cf451-bf1d-4c8b-a240-288b2cd55fc1-istio-data\") pod \"e2e-ipp-a-b4bf5c-openshift-default-84869c884b-sxxsq\" (UID: \"911cf451-bf1d-4c8b-a240-288b2cd55fc1\") " pod="openshift-ingress/e2e-ipp-a-b4bf5c-openshift-default-84869c884b-sxxsq"
Jul 27 21:40:19.309202 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:19.309130    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/911cf451-bf1d-4c8b-a240-288b2cd55fc1-workload-certs\") pod \"e2e-ipp-a-b4bf5c-openshift-default-84869c884b-sxxsq\" (UID: \"911cf451-bf1d-4c8b-a240-288b2cd55fc1\") " pod="openshift-ingress/e2e-ipp-a-b4bf5c-openshift-default-84869c884b-sxxsq"
Jul 27 21:40:19.309202 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:19.309180    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/911cf451-bf1d-4c8b-a240-288b2cd55fc1-workload-socket\") pod \"e2e-ipp-a-b4bf5c-openshift-default-84869c884b-sxxsq\" (UID: \"911cf451-bf1d-4c8b-a240-288b2cd55fc1\") " pod="openshift-ingress/e2e-ipp-a-b4bf5c-openshift-default-84869c884b-sxxsq"
Jul 27 21:40:19.309342 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:19.309212    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/911cf451-bf1d-4c8b-a240-288b2cd55fc1-istio-token\") pod \"e2e-ipp-a-b4bf5c-openshift-default-84869c884b-sxxsq\" (UID: \"911cf451-bf1d-4c8b-a240-288b2cd55fc1\") " pod="openshift-ingress/e2e-ipp-a-b4bf5c-openshift-default-84869c884b-sxxsq"
Jul 27 21:40:19.309342 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:19.309242    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/911cf451-bf1d-4c8b-a240-288b2cd55fc1-istiod-ca-cert\") pod \"e2e-ipp-a-b4bf5c-openshift-default-84869c884b-sxxsq\" (UID: \"911cf451-bf1d-4c8b-a240-288b2cd55fc1\") " pod="openshift-ingress/e2e-ipp-a-b4bf5c-openshift-default-84869c884b-sxxsq"
Jul 27 21:40:19.309342 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:19.309260    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/911cf451-bf1d-4c8b-a240-288b2cd55fc1-istio-envoy\") pod \"e2e-ipp-a-b4bf5c-openshift-default-84869c884b-sxxsq\" (UID: \"911cf451-bf1d-4c8b-a240-288b2cd55fc1\") " pod="openshift-ingress/e2e-ipp-a-b4bf5c-openshift-default-84869c884b-sxxsq"
Jul 27 21:40:19.309342 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:19.309285    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffzgc\" (UniqueName: \"kubernetes.io/projected/911cf451-bf1d-4c8b-a240-288b2cd55fc1-kube-api-access-ffzgc\") pod \"e2e-ipp-a-b4bf5c-openshift-default-84869c884b-sxxsq\" (UID: \"911cf451-bf1d-4c8b-a240-288b2cd55fc1\") " pod="openshift-ingress/e2e-ipp-a-b4bf5c-openshift-default-84869c884b-sxxsq"
Jul 27 21:40:19.409767 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:19.409728    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/911cf451-bf1d-4c8b-a240-288b2cd55fc1-credential-socket\") pod \"e2e-ipp-a-b4bf5c-openshift-default-84869c884b-sxxsq\" (UID: \"911cf451-bf1d-4c8b-a240-288b2cd55fc1\") " pod="openshift-ingress/e2e-ipp-a-b4bf5c-openshift-default-84869c884b-sxxsq"
Jul 27 21:40:19.409767 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:19.409766    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/911cf451-bf1d-4c8b-a240-288b2cd55fc1-istio-data\") pod \"e2e-ipp-a-b4bf5c-openshift-default-84869c884b-sxxsq\" (UID: \"911cf451-bf1d-4c8b-a240-288b2cd55fc1\") " pod="openshift-ingress/e2e-ipp-a-b4bf5c-openshift-default-84869c884b-sxxsq"
Jul 27 21:40:19.410025 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:19.409802    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/911cf451-bf1d-4c8b-a240-288b2cd55fc1-workload-certs\") pod \"e2e-ipp-a-b4bf5c-openshift-default-84869c884b-sxxsq\" (UID: \"911cf451-bf1d-4c8b-a240-288b2cd55fc1\") " pod="openshift-ingress/e2e-ipp-a-b4bf5c-openshift-default-84869c884b-sxxsq"
Jul 27 21:40:19.410025 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:19.409837    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/911cf451-bf1d-4c8b-a240-288b2cd55fc1-workload-socket\") pod \"e2e-ipp-a-b4bf5c-openshift-default-84869c884b-sxxsq\" (UID: \"911cf451-bf1d-4c8b-a240-288b2cd55fc1\") " pod="openshift-ingress/e2e-ipp-a-b4bf5c-openshift-default-84869c884b-sxxsq"
Jul 27 21:40:19.410025 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:19.409858    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/911cf451-bf1d-4c8b-a240-288b2cd55fc1-istio-token\") pod \"e2e-ipp-a-b4bf5c-openshift-default-84869c884b-sxxsq\" (UID: \"911cf451-bf1d-4c8b-a240-288b2cd55fc1\") " pod="openshift-ingress/e2e-ipp-a-b4bf5c-openshift-default-84869c884b-sxxsq"
Jul 27 21:40:19.410025 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:19.409884    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/911cf451-bf1d-4c8b-a240-288b2cd55fc1-istiod-ca-cert\") pod \"e2e-ipp-a-b4bf5c-openshift-default-84869c884b-sxxsq\" (UID: \"911cf451-bf1d-4c8b-a240-288b2cd55fc1\") " pod="openshift-ingress/e2e-ipp-a-b4bf5c-openshift-default-84869c884b-sxxsq"
Jul 27 21:40:19.410025 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:19.409930    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/911cf451-bf1d-4c8b-a240-288b2cd55fc1-istio-envoy\") pod \"e2e-ipp-a-b4bf5c-openshift-default-84869c884b-sxxsq\" (UID: \"911cf451-bf1d-4c8b-a240-288b2cd55fc1\") " pod="openshift-ingress/e2e-ipp-a-b4bf5c-openshift-default-84869c884b-sxxsq"
Jul 27 21:40:19.410025 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:19.409990    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffzgc\" (UniqueName: \"kubernetes.io/projected/911cf451-bf1d-4c8b-a240-288b2cd55fc1-kube-api-access-ffzgc\") pod \"e2e-ipp-a-b4bf5c-openshift-default-84869c884b-sxxsq\" (UID: \"911cf451-bf1d-4c8b-a240-288b2cd55fc1\") " pod="openshift-ingress/e2e-ipp-a-b4bf5c-openshift-default-84869c884b-sxxsq"
Jul 27 21:40:19.410351 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:19.410106    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/911cf451-bf1d-4c8b-a240-288b2cd55fc1-istio-podinfo\") pod \"e2e-ipp-a-b4bf5c-openshift-default-84869c884b-sxxsq\" (UID: \"911cf451-bf1d-4c8b-a240-288b2cd55fc1\") " pod="openshift-ingress/e2e-ipp-a-b4bf5c-openshift-default-84869c884b-sxxsq"
Jul 27 21:40:19.410351 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:19.410205    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/911cf451-bf1d-4c8b-a240-288b2cd55fc1-istio-data\") pod \"e2e-ipp-a-b4bf5c-openshift-default-84869c884b-sxxsq\" (UID: \"911cf451-bf1d-4c8b-a240-288b2cd55fc1\") " pod="openshift-ingress/e2e-ipp-a-b4bf5c-openshift-default-84869c884b-sxxsq"
Jul 27 21:40:19.410351 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:19.410300    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/911cf451-bf1d-4c8b-a240-288b2cd55fc1-workload-certs\") pod \"e2e-ipp-a-b4bf5c-openshift-default-84869c884b-sxxsq\" (UID: \"911cf451-bf1d-4c8b-a240-288b2cd55fc1\") " pod="openshift-ingress/e2e-ipp-a-b4bf5c-openshift-default-84869c884b-sxxsq"
Jul 27 21:40:19.410513 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:19.410407    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/911cf451-bf1d-4c8b-a240-288b2cd55fc1-credential-socket\") pod \"e2e-ipp-a-b4bf5c-openshift-default-84869c884b-sxxsq\" (UID: \"911cf451-bf1d-4c8b-a240-288b2cd55fc1\") " pod="openshift-ingress/e2e-ipp-a-b4bf5c-openshift-default-84869c884b-sxxsq"
Jul 27 21:40:19.410513 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:19.410408    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/911cf451-bf1d-4c8b-a240-288b2cd55fc1-workload-socket\") pod \"e2e-ipp-a-b4bf5c-openshift-default-84869c884b-sxxsq\" (UID: \"911cf451-bf1d-4c8b-a240-288b2cd55fc1\") " pod="openshift-ingress/e2e-ipp-a-b4bf5c-openshift-default-84869c884b-sxxsq"
Jul 27 21:40:19.410727 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:19.410707    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/911cf451-bf1d-4c8b-a240-288b2cd55fc1-istiod-ca-cert\") pod \"e2e-ipp-a-b4bf5c-openshift-default-84869c884b-sxxsq\" (UID: \"911cf451-bf1d-4c8b-a240-288b2cd55fc1\") " pod="openshift-ingress/e2e-ipp-a-b4bf5c-openshift-default-84869c884b-sxxsq"
Jul 27 21:40:19.412399 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:19.412376    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/911cf451-bf1d-4c8b-a240-288b2cd55fc1-istio-envoy\") pod \"e2e-ipp-a-b4bf5c-openshift-default-84869c884b-sxxsq\" (UID: \"911cf451-bf1d-4c8b-a240-288b2cd55fc1\") " pod="openshift-ingress/e2e-ipp-a-b4bf5c-openshift-default-84869c884b-sxxsq"
Jul 27 21:40:19.412718 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:19.412699    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/911cf451-bf1d-4c8b-a240-288b2cd55fc1-istio-podinfo\") pod \"e2e-ipp-a-b4bf5c-openshift-default-84869c884b-sxxsq\" (UID: \"911cf451-bf1d-4c8b-a240-288b2cd55fc1\") " pod="openshift-ingress/e2e-ipp-a-b4bf5c-openshift-default-84869c884b-sxxsq"
Jul 27 21:40:19.417734 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:19.417711    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/911cf451-bf1d-4c8b-a240-288b2cd55fc1-istio-token\") pod \"e2e-ipp-a-b4bf5c-openshift-default-84869c884b-sxxsq\" (UID: \"911cf451-bf1d-4c8b-a240-288b2cd55fc1\") " pod="openshift-ingress/e2e-ipp-a-b4bf5c-openshift-default-84869c884b-sxxsq"
Jul 27 21:40:19.417965 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:19.417946    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffzgc\" (UniqueName: \"kubernetes.io/projected/911cf451-bf1d-4c8b-a240-288b2cd55fc1-kube-api-access-ffzgc\") pod \"e2e-ipp-a-b4bf5c-openshift-default-84869c884b-sxxsq\" (UID: \"911cf451-bf1d-4c8b-a240-288b2cd55fc1\") " pod="openshift-ingress/e2e-ipp-a-b4bf5c-openshift-default-84869c884b-sxxsq"
Jul 27 21:40:19.554946 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:19.554912    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-b4bf5c-openshift-default-84869c884b-sxxsq"
Jul 27 21:40:19.706684 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:19.706657    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-b4bf5c-openshift-default-84869c884b-sxxsq"]
Jul 27 21:40:19.709282 ip-10-0-141-64 kubenswrapper[2560]: W0727 21:40:19.709255    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod911cf451_bf1d_4c8b_a240_288b2cd55fc1.slice/crio-9a3302c43dd10974cb467a465d7ec7bce015f648378432527c03b86f883d5d2f WatchSource:0}: Error finding container 9a3302c43dd10974cb467a465d7ec7bce015f648378432527c03b86f883d5d2f: Status 404 returned error can't find the container with id 9a3302c43dd10974cb467a465d7ec7bce015f648378432527c03b86f883d5d2f
Jul 27 21:40:19.711376 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:19.711344    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 27 21:40:19.711471 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:19.711410    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 27 21:40:19.711471 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:19.711438    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 27 21:40:20.366048 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:20.366010    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-b4bf5c-openshift-default-84869c884b-sxxsq" event={"ID":"911cf451-bf1d-4c8b-a240-288b2cd55fc1","Type":"ContainerStarted","Data":"004ef90f58495d2ada9a0e1736ac957ab099af7db4d7299166eb28c12d733728"}
Jul 27 21:40:20.366447 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:20.366062    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-b4bf5c-openshift-default-84869c884b-sxxsq" event={"ID":"911cf451-bf1d-4c8b-a240-288b2cd55fc1","Type":"ContainerStarted","Data":"9a3302c43dd10974cb467a465d7ec7bce015f648378432527c03b86f883d5d2f"}
Jul 27 21:40:20.392008 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:20.391953    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-b4bf5c-openshift-default-84869c884b-sxxsq" podStartSLOduration=1.391938051 podStartE2EDuration="1.391938051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:40:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:40:20.39134789 +0000 UTC m=+2937.143209714" watchObservedRunningTime="2026-07-27 21:40:20.391938051 +0000 UTC m=+2937.143799846"
Jul 27 21:40:20.556033 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:20.555997    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-b4bf5c-openshift-default-84869c884b-sxxsq"
Jul 27 21:40:20.560746 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:20.560723    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-b4bf5c-openshift-default-84869c884b-sxxsq"
Jul 27 21:40:21.371016 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:21.370983    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-b4bf5c-openshift-default-84869c884b-sxxsq"
Jul 27 21:40:21.372149 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:21.372130    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-b4bf5c-openshift-default-84869c884b-sxxsq"
Jul 27 21:40:31.298437 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:31.298393    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:40:37.412327 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:37.412265    2560 patch_prober.go:28] interesting pod/e2e-ipp-a-b4bf5c-openshift-default-84869c884b-sxxsq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 27 21:40:37.412848 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:37.412369    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-b4bf5c-openshift-default-84869c884b-sxxsq" podUID="911cf451-bf1d-4c8b-a240-288b2cd55fc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 27 21:40:38.214296 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:38.214262    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-9e51a0-openshift-default-85b958c9f4-ntd42"]
Jul 27 21:40:38.218162 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:38.218144    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-9e51a0-openshift-default-85b958c9f4-ntd42"
Jul 27 21:40:38.220756 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:38.220733    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-9e51a0-openshift-default-dockercfg-8frjn\""
Jul 27 21:40:38.229919 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:38.229893    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-9e51a0-openshift-default-85b958c9f4-ntd42"]
Jul 27 21:40:38.296151 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:38.296111    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fc339fb-bbea-4776-b172-fc631a1dce80-workload-certs\") pod \"e2e-ipp-b-9e51a0-openshift-default-85b958c9f4-ntd42\" (UID: \"0fc339fb-bbea-4776-b172-fc631a1dce80\") " pod="openshift-ingress/e2e-ipp-b-9e51a0-openshift-default-85b958c9f4-ntd42"
Jul 27 21:40:38.296313 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:38.296218    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0fc339fb-bbea-4776-b172-fc631a1dce80-istio-data\") pod \"e2e-ipp-b-9e51a0-openshift-default-85b958c9f4-ntd42\" (UID: \"0fc339fb-bbea-4776-b172-fc631a1dce80\") " pod="openshift-ingress/e2e-ipp-b-9e51a0-openshift-default-85b958c9f4-ntd42"
Jul 27 21:40:38.296313 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:38.296250    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8b9jt\" (UniqueName: \"kubernetes.io/projected/0fc339fb-bbea-4776-b172-fc631a1dce80-kube-api-access-8b9jt\") pod \"e2e-ipp-b-9e51a0-openshift-default-85b958c9f4-ntd42\" (UID: \"0fc339fb-bbea-4776-b172-fc631a1dce80\") " pod="openshift-ingress/e2e-ipp-b-9e51a0-openshift-default-85b958c9f4-ntd42"
Jul 27 21:40:38.296898 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:38.296314    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0fc339fb-bbea-4776-b172-fc631a1dce80-istio-token\") pod \"e2e-ipp-b-9e51a0-openshift-default-85b958c9f4-ntd42\" (UID: \"0fc339fb-bbea-4776-b172-fc631a1dce80\") " pod="openshift-ingress/e2e-ipp-b-9e51a0-openshift-default-85b958c9f4-ntd42"
Jul 27 21:40:38.296898 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:38.296344    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fc339fb-bbea-4776-b172-fc631a1dce80-workload-socket\") pod \"e2e-ipp-b-9e51a0-openshift-default-85b958c9f4-ntd42\" (UID: \"0fc339fb-bbea-4776-b172-fc631a1dce80\") " pod="openshift-ingress/e2e-ipp-b-9e51a0-openshift-default-85b958c9f4-ntd42"
Jul 27 21:40:38.296898 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:38.296404    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0fc339fb-bbea-4776-b172-fc631a1dce80-istio-podinfo\") pod \"e2e-ipp-b-9e51a0-openshift-default-85b958c9f4-ntd42\" (UID: \"0fc339fb-bbea-4776-b172-fc631a1dce80\") " pod="openshift-ingress/e2e-ipp-b-9e51a0-openshift-default-85b958c9f4-ntd42"
Jul 27 21:40:38.296898 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:38.296487    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0fc339fb-bbea-4776-b172-fc631a1dce80-istiod-ca-cert\") pod \"e2e-ipp-b-9e51a0-openshift-default-85b958c9f4-ntd42\" (UID: \"0fc339fb-bbea-4776-b172-fc631a1dce80\") " pod="openshift-ingress/e2e-ipp-b-9e51a0-openshift-default-85b958c9f4-ntd42"
Jul 27 21:40:38.296898 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:38.296525    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fc339fb-bbea-4776-b172-fc631a1dce80-credential-socket\") pod \"e2e-ipp-b-9e51a0-openshift-default-85b958c9f4-ntd42\" (UID: \"0fc339fb-bbea-4776-b172-fc631a1dce80\") " pod="openshift-ingress/e2e-ipp-b-9e51a0-openshift-default-85b958c9f4-ntd42"
Jul 27 21:40:38.296898 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:38.296620    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0fc339fb-bbea-4776-b172-fc631a1dce80-istio-envoy\") pod \"e2e-ipp-b-9e51a0-openshift-default-85b958c9f4-ntd42\" (UID: \"0fc339fb-bbea-4776-b172-fc631a1dce80\") " pod="openshift-ingress/e2e-ipp-b-9e51a0-openshift-default-85b958c9f4-ntd42"
Jul 27 21:40:38.397723 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:38.397677    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0fc339fb-bbea-4776-b172-fc631a1dce80-istio-envoy\") pod \"e2e-ipp-b-9e51a0-openshift-default-85b958c9f4-ntd42\" (UID: \"0fc339fb-bbea-4776-b172-fc631a1dce80\") " pod="openshift-ingress/e2e-ipp-b-9e51a0-openshift-default-85b958c9f4-ntd42"
Jul 27 21:40:38.397942 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:38.397750    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fc339fb-bbea-4776-b172-fc631a1dce80-workload-certs\") pod \"e2e-ipp-b-9e51a0-openshift-default-85b958c9f4-ntd42\" (UID: \"0fc339fb-bbea-4776-b172-fc631a1dce80\") " pod="openshift-ingress/e2e-ipp-b-9e51a0-openshift-default-85b958c9f4-ntd42"
Jul 27 21:40:38.397942 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:38.397898    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0fc339fb-bbea-4776-b172-fc631a1dce80-istio-data\") pod \"e2e-ipp-b-9e51a0-openshift-default-85b958c9f4-ntd42\" (UID: \"0fc339fb-bbea-4776-b172-fc631a1dce80\") " pod="openshift-ingress/e2e-ipp-b-9e51a0-openshift-default-85b958c9f4-ntd42"
Jul 27 21:40:38.398082 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:38.397946    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8b9jt\" (UniqueName: \"kubernetes.io/projected/0fc339fb-bbea-4776-b172-fc631a1dce80-kube-api-access-8b9jt\") pod \"e2e-ipp-b-9e51a0-openshift-default-85b958c9f4-ntd42\" (UID: \"0fc339fb-bbea-4776-b172-fc631a1dce80\") " pod="openshift-ingress/e2e-ipp-b-9e51a0-openshift-default-85b958c9f4-ntd42"
Jul 27 21:40:38.398082 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:38.397987    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0fc339fb-bbea-4776-b172-fc631a1dce80-istio-token\") pod \"e2e-ipp-b-9e51a0-openshift-default-85b958c9f4-ntd42\" (UID: \"0fc339fb-bbea-4776-b172-fc631a1dce80\") " pod="openshift-ingress/e2e-ipp-b-9e51a0-openshift-default-85b958c9f4-ntd42"
Jul 27 21:40:38.398082 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:38.398014    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fc339fb-bbea-4776-b172-fc631a1dce80-workload-socket\") pod \"e2e-ipp-b-9e51a0-openshift-default-85b958c9f4-ntd42\" (UID: \"0fc339fb-bbea-4776-b172-fc631a1dce80\") " pod="openshift-ingress/e2e-ipp-b-9e51a0-openshift-default-85b958c9f4-ntd42"
Jul 27 21:40:38.398082 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:38.398057    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0fc339fb-bbea-4776-b172-fc631a1dce80-istio-podinfo\") pod \"e2e-ipp-b-9e51a0-openshift-default-85b958c9f4-ntd42\" (UID: \"0fc339fb-bbea-4776-b172-fc631a1dce80\") " pod="openshift-ingress/e2e-ipp-b-9e51a0-openshift-default-85b958c9f4-ntd42"
Jul 27 21:40:38.398082 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:38.398105    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0fc339fb-bbea-4776-b172-fc631a1dce80-istiod-ca-cert\") pod \"e2e-ipp-b-9e51a0-openshift-default-85b958c9f4-ntd42\" (UID: \"0fc339fb-bbea-4776-b172-fc631a1dce80\") " pod="openshift-ingress/e2e-ipp-b-9e51a0-openshift-default-85b958c9f4-ntd42"
Jul 27 21:40:38.398621 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:38.398146    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fc339fb-bbea-4776-b172-fc631a1dce80-credential-socket\") pod \"e2e-ipp-b-9e51a0-openshift-default-85b958c9f4-ntd42\" (UID: \"0fc339fb-bbea-4776-b172-fc631a1dce80\") " pod="openshift-ingress/e2e-ipp-b-9e51a0-openshift-default-85b958c9f4-ntd42"
Jul 27 21:40:38.398621 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:38.398253    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fc339fb-bbea-4776-b172-fc631a1dce80-workload-certs\") pod \"e2e-ipp-b-9e51a0-openshift-default-85b958c9f4-ntd42\" (UID: \"0fc339fb-bbea-4776-b172-fc631a1dce80\") " pod="openshift-ingress/e2e-ipp-b-9e51a0-openshift-default-85b958c9f4-ntd42"
Jul 27 21:40:38.398621 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:38.398492    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0fc339fb-bbea-4776-b172-fc631a1dce80-istio-data\") pod \"e2e-ipp-b-9e51a0-openshift-default-85b958c9f4-ntd42\" (UID: \"0fc339fb-bbea-4776-b172-fc631a1dce80\") " pod="openshift-ingress/e2e-ipp-b-9e51a0-openshift-default-85b958c9f4-ntd42"
Jul 27 21:40:38.398805 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:38.398631    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fc339fb-bbea-4776-b172-fc631a1dce80-workload-socket\") pod \"e2e-ipp-b-9e51a0-openshift-default-85b958c9f4-ntd42\" (UID: \"0fc339fb-bbea-4776-b172-fc631a1dce80\") " pod="openshift-ingress/e2e-ipp-b-9e51a0-openshift-default-85b958c9f4-ntd42"
Jul 27 21:40:38.398805 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:38.398709    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fc339fb-bbea-4776-b172-fc631a1dce80-credential-socket\") pod \"e2e-ipp-b-9e51a0-openshift-default-85b958c9f4-ntd42\" (UID: \"0fc339fb-bbea-4776-b172-fc631a1dce80\") " pod="openshift-ingress/e2e-ipp-b-9e51a0-openshift-default-85b958c9f4-ntd42"
Jul 27 21:40:38.399190 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:38.399169    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0fc339fb-bbea-4776-b172-fc631a1dce80-istiod-ca-cert\") pod \"e2e-ipp-b-9e51a0-openshift-default-85b958c9f4-ntd42\" (UID: \"0fc339fb-bbea-4776-b172-fc631a1dce80\") " pod="openshift-ingress/e2e-ipp-b-9e51a0-openshift-default-85b958c9f4-ntd42"
Jul 27 21:40:38.400459 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:38.400438    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0fc339fb-bbea-4776-b172-fc631a1dce80-istio-envoy\") pod \"e2e-ipp-b-9e51a0-openshift-default-85b958c9f4-ntd42\" (UID: \"0fc339fb-bbea-4776-b172-fc631a1dce80\") " pod="openshift-ingress/e2e-ipp-b-9e51a0-openshift-default-85b958c9f4-ntd42"
Jul 27 21:40:38.401010 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:38.400994    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0fc339fb-bbea-4776-b172-fc631a1dce80-istio-podinfo\") pod \"e2e-ipp-b-9e51a0-openshift-default-85b958c9f4-ntd42\" (UID: \"0fc339fb-bbea-4776-b172-fc631a1dce80\") " pod="openshift-ingress/e2e-ipp-b-9e51a0-openshift-default-85b958c9f4-ntd42"
Jul 27 21:40:38.417872 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:38.417849    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0fc339fb-bbea-4776-b172-fc631a1dce80-istio-token\") pod \"e2e-ipp-b-9e51a0-openshift-default-85b958c9f4-ntd42\" (UID: \"0fc339fb-bbea-4776-b172-fc631a1dce80\") " pod="openshift-ingress/e2e-ipp-b-9e51a0-openshift-default-85b958c9f4-ntd42"
Jul 27 21:40:38.419688 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:38.419664    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8b9jt\" (UniqueName: \"kubernetes.io/projected/0fc339fb-bbea-4776-b172-fc631a1dce80-kube-api-access-8b9jt\") pod \"e2e-ipp-b-9e51a0-openshift-default-85b958c9f4-ntd42\" (UID: \"0fc339fb-bbea-4776-b172-fc631a1dce80\") " pod="openshift-ingress/e2e-ipp-b-9e51a0-openshift-default-85b958c9f4-ntd42"
Jul 27 21:40:38.530279 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:38.530243    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-9e51a0-openshift-default-85b958c9f4-ntd42"
Jul 27 21:40:38.677433 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:38.677255    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-9e51a0-openshift-default-85b958c9f4-ntd42"]
Jul 27 21:40:38.679494 ip-10-0-141-64 kubenswrapper[2560]: W0727 21:40:38.679466    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fc339fb_bbea_4776_b172_fc631a1dce80.slice/crio-cf36341632f381e784896262d7641e132b2651b90df48d2f46555a5addb566b2 WatchSource:0}: Error finding container cf36341632f381e784896262d7641e132b2651b90df48d2f46555a5addb566b2: Status 404 returned error can't find the container with id cf36341632f381e784896262d7641e132b2651b90df48d2f46555a5addb566b2
Jul 27 21:40:38.681689 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:38.681654    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 27 21:40:38.681773 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:38.681723    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 27 21:40:38.681773 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:38.681758    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 27 21:40:39.460370 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:39.460339    2560 generic.go:356] "Generic (PLEG): container finished" podID="911cf451-bf1d-4c8b-a240-288b2cd55fc1" containerID="004ef90f58495d2ada9a0e1736ac957ab099af7db4d7299166eb28c12d733728" exitCode=0
Jul 27 21:40:39.460825 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:39.460419    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-b4bf5c-openshift-default-84869c884b-sxxsq" event={"ID":"911cf451-bf1d-4c8b-a240-288b2cd55fc1","Type":"ContainerDied","Data":"004ef90f58495d2ada9a0e1736ac957ab099af7db4d7299166eb28c12d733728"}
Jul 27 21:40:39.460928 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:39.460911    2560 scope.go:117] "RemoveContainer" containerID="004ef90f58495d2ada9a0e1736ac957ab099af7db4d7299166eb28c12d733728"
Jul 27 21:40:39.462053 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:39.462028    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-9e51a0-openshift-default-85b958c9f4-ntd42" event={"ID":"0fc339fb-bbea-4776-b172-fc631a1dce80","Type":"ContainerStarted","Data":"7ea5fc862caae62dfb73027c0b73290f5d7667bc76e2ea34e94eca9f7cc10232"}
Jul 27 21:40:39.462168 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:39.462077    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-9e51a0-openshift-default-85b958c9f4-ntd42" event={"ID":"0fc339fb-bbea-4776-b172-fc631a1dce80","Type":"ContainerStarted","Data":"cf36341632f381e784896262d7641e132b2651b90df48d2f46555a5addb566b2"}
Jul 27 21:40:39.462217 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:39.462181    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 27 21:40:39.462280 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:39.462263    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 27 21:40:39.462325 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:39.462311    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 27 21:40:39.503058 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:39.503003    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-9e51a0-openshift-default-85b958c9f4-ntd42" podStartSLOduration=1.5029829829999999 podStartE2EDuration="1.502982983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:40:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:40:39.500206075 +0000 UTC m=+2956.252067905" watchObservedRunningTime="2026-07-27 21:40:39.502982983 +0000 UTC m=+2956.254844779"
Jul 27 21:40:39.531146 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:39.531122    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-9e51a0-openshift-default-85b958c9f4-ntd42"
Jul 27 21:40:39.532525 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:39.532501    2560 patch_prober.go:28] interesting pod/e2e-ipp-b-9e51a0-openshift-default-85b958c9f4-ntd42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Jul 27 21:40:39.532634 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:39.532541    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-9e51a0-openshift-default-85b958c9f4-ntd42" podUID="0fc339fb-bbea-4776-b172-fc631a1dce80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Jul 27 21:40:40.470832 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:40.470792    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-b4bf5c-openshift-default-84869c884b-sxxsq" event={"ID":"911cf451-bf1d-4c8b-a240-288b2cd55fc1","Type":"ContainerStarted","Data":"fe1fcc4e681a6057902ac0c8b3aa955861f9ce096b782d363dc4d0610f04b700"}
Jul 27 21:40:40.534851 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:40.534818    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-9e51a0-openshift-default-85b958c9f4-ntd42"
Jul 27 21:40:40.555784 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:40.555745    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-b4bf5c-openshift-default-84869c884b-sxxsq"
Jul 27 21:40:40.556057 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:40.556026    2560 patch_prober.go:28] interesting pod/e2e-ipp-a-b4bf5c-openshift-default-84869c884b-sxxsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Jul 27 21:40:40.556184 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:40.556123    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-b4bf5c-openshift-default-84869c884b-sxxsq" podUID="911cf451-bf1d-4c8b-a240-288b2cd55fc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Jul 27 21:40:41.476217 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:41.476174    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-9e51a0-openshift-default-85b958c9f4-ntd42"
Jul 27 21:40:41.476954 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:41.476934    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-9e51a0-openshift-default-85b958c9f4-ntd42"
Jul 27 21:40:41.564115 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:41.560438    2560 patch_prober.go:28] interesting pod/e2e-ipp-a-b4bf5c-openshift-default-84869c884b-sxxsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Jul 27 21:40:41.564115 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:41.560501    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-b4bf5c-openshift-default-84869c884b-sxxsq" podUID="911cf451-bf1d-4c8b-a240-288b2cd55fc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Jul 27 21:40:42.556107 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:42.556047    2560 patch_prober.go:28] interesting pod/e2e-ipp-a-b4bf5c-openshift-default-84869c884b-sxxsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Jul 27 21:40:42.557159 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:42.557121    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-b4bf5c-openshift-default-84869c884b-sxxsq" podUID="911cf451-bf1d-4c8b-a240-288b2cd55fc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Jul 27 21:40:43.556152 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:43.555690    2560 patch_prober.go:28] interesting pod/e2e-ipp-a-b4bf5c-openshift-default-84869c884b-sxxsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Jul 27 21:40:43.556152 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:43.555766    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-b4bf5c-openshift-default-84869c884b-sxxsq" podUID="911cf451-bf1d-4c8b-a240-288b2cd55fc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Jul 27 21:40:44.559138 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:44.559080    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-b4bf5c-openshift-default-84869c884b-sxxsq"
Jul 27 21:40:44.559623 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:44.559459    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-b4bf5c-openshift-default-84869c884b-sxxsq"
Jul 27 21:40:44.560151 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:44.560129    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-b4bf5c-openshift-default-84869c884b-sxxsq"
Jul 27 21:40:49.301928 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:49.301890    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:40:57.336549 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:57.335715    2560 patch_prober.go:28] interesting pod/e2e-ipp-b-9e51a0-openshift-default-85b958c9f4-ntd42 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": read tcp 10.133.0.2:41802->10.133.0.55:15021: read: connection reset by peer" start-of-body=
Jul 27 21:40:57.336549 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:57.335797    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-9e51a0-openshift-default-85b958c9f4-ntd42" podUID="0fc339fb-bbea-4776-b172-fc631a1dce80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": read tcp 10.133.0.2:41802->10.133.0.55:15021: read: connection reset by peer"
Jul 27 21:40:57.556915 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:57.556881    2560 generic.go:356] "Generic (PLEG): container finished" podID="0fc339fb-bbea-4776-b172-fc631a1dce80" containerID="7ea5fc862caae62dfb73027c0b73290f5d7667bc76e2ea34e94eca9f7cc10232" exitCode=0
Jul 27 21:40:57.556915 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:57.556918    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-9e51a0-openshift-default-85b958c9f4-ntd42" event={"ID":"0fc339fb-bbea-4776-b172-fc631a1dce80","Type":"ContainerDied","Data":"7ea5fc862caae62dfb73027c0b73290f5d7667bc76e2ea34e94eca9f7cc10232"}
Jul 27 21:40:57.557306 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:57.557292    2560 scope.go:117] "RemoveContainer" containerID="7ea5fc862caae62dfb73027c0b73290f5d7667bc76e2ea34e94eca9f7cc10232"
Jul 27 21:40:57.558379 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:57.558348    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 27 21:40:57.558508 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:57.558433    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 27 21:40:57.558508 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:57.558479    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 27 21:40:58.567814 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:58.567783    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-9e51a0-openshift-default-85b958c9f4-ntd42" event={"ID":"0fc339fb-bbea-4776-b172-fc631a1dce80","Type":"ContainerStarted","Data":"3025c3228bf39a3eac6c13de1fea4186293220072e913d29f786bd7a045ae1a1"}
Jul 27 21:40:59.530850 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:59.530806    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-9e51a0-openshift-default-85b958c9f4-ntd42"
Jul 27 21:40:59.531101 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:59.530951    2560 patch_prober.go:28] interesting pod/e2e-ipp-b-9e51a0-openshift-default-85b958c9f4-ntd42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Jul 27 21:40:59.531101 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:40:59.530999    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-9e51a0-openshift-default-85b958c9f4-ntd42" podUID="0fc339fb-bbea-4776-b172-fc631a1dce80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Jul 27 21:41:00.530671 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:41:00.530626    2560 patch_prober.go:28] interesting pod/e2e-ipp-b-9e51a0-openshift-default-85b958c9f4-ntd42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Jul 27 21:41:00.531080 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:41:00.530694    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-9e51a0-openshift-default-85b958c9f4-ntd42" podUID="0fc339fb-bbea-4776-b172-fc631a1dce80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Jul 27 21:41:01.531443 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:41:01.531401    2560 patch_prober.go:28] interesting pod/e2e-ipp-b-9e51a0-openshift-default-85b958c9f4-ntd42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Jul 27 21:41:01.531927 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:41:01.531468    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-9e51a0-openshift-default-85b958c9f4-ntd42" podUID="0fc339fb-bbea-4776-b172-fc631a1dce80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Jul 27 21:41:02.534029 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:41:02.534001    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-9e51a0-openshift-default-85b958c9f4-ntd42"
Jul 27 21:41:02.534444 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:41:02.534306    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-9e51a0-openshift-default-85b958c9f4-ntd42"
Jul 27 21:41:02.535022 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:41:02.535004    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-9e51a0-openshift-default-85b958c9f4-ntd42"
Jul 27 21:41:18.303321 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:41:18.303282    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:41:23.943438 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:41:23.943409    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-z6zxv_8a6b7ae6-6c21-453d-99b9-303066d19915/console-operator/1.log"
Jul 27 21:41:23.947473 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:41:23.947449    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-z6zxv_8a6b7ae6-6c21-453d-99b9-303066d19915/console-operator/1.log"
Jul 27 21:41:23.950357 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:41:23.950333    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-95hg7_433c431a-b5bf-4734-a03f-6f0a16895ded/ovn-acl-logging/0.log"
Jul 27 21:41:23.953831 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:41:23.953811    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-95hg7_433c431a-b5bf-4734-a03f-6f0a16895ded/ovn-acl-logging/0.log"
Jul 27 21:41:44.703114 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:41:44.703064    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:42:06.597563 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:42:06.597529    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:42:15.449310 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:42:15.449203    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-400a5b-gvtzk"]
Jul 27 21:42:15.454517 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:42:15.454492    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-400a5b-gvtzk"
Jul 27 21:42:15.457871 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:42:15.457845    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 27 21:42:15.458015 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:42:15.457844    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j5rjm\""
Jul 27 21:42:15.461590 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:42:15.461566    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-400a5b-gvtzk"]
Jul 27 21:42:15.523429 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:42:15.523387    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ed4782e-5f6a-4b17-8db3-34cf40061089-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-400a5b-gvtzk\" (UID: \"2ed4782e-5f6a-4b17-8db3-34cf40061089\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-400a5b-gvtzk"
Jul 27 21:42:15.624870 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:42:15.624830    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ed4782e-5f6a-4b17-8db3-34cf40061089-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-400a5b-gvtzk\" (UID: \"2ed4782e-5f6a-4b17-8db3-34cf40061089\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-400a5b-gvtzk"
Jul 27 21:42:15.625464 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:42:15.625446    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ed4782e-5f6a-4b17-8db3-34cf40061089-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-400a5b-gvtzk\" (UID: \"2ed4782e-5f6a-4b17-8db3-34cf40061089\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-400a5b-gvtzk"
Jul 27 21:42:15.767980 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:42:15.767949    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-400a5b-gvtzk"
Jul 27 21:42:15.902369 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:42:15.902266    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-400a5b-gvtzk"]
Jul 27 21:42:15.905214 ip-10-0-141-64 kubenswrapper[2560]: W0727 21:42:15.905187    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ed4782e_5f6a_4b17_8db3_34cf40061089.slice/crio-dfaeb04f0737ac71088f164c7676e07e8c0adda368a248619db7944910f67e06 WatchSource:0}: Error finding container dfaeb04f0737ac71088f164c7676e07e8c0adda368a248619db7944910f67e06: Status 404 returned error can't find the container with id dfaeb04f0737ac71088f164c7676e07e8c0adda368a248619db7944910f67e06
Jul 27 21:42:15.954608 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:42:15.954574    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-400a5b-gvtzk" event={"ID":"2ed4782e-5f6a-4b17-8db3-34cf40061089","Type":"ContainerStarted","Data":"dfaeb04f0737ac71088f164c7676e07e8c0adda368a248619db7944910f67e06"}
Jul 27 21:42:16.960868 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:42:16.960775    2560 generic.go:356] "Generic (PLEG): container finished" podID="2ed4782e-5f6a-4b17-8db3-34cf40061089" containerID="09df62337662743e0e9c0b05aa319f27331766ab808c873fbb1b2b9c2a2cb452" exitCode=0
Jul 27 21:42:16.960868 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:42:16.960821    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-400a5b-gvtzk" event={"ID":"2ed4782e-5f6a-4b17-8db3-34cf40061089","Type":"ContainerDied","Data":"09df62337662743e0e9c0b05aa319f27331766ab808c873fbb1b2b9c2a2cb452"}
Jul 27 21:42:18.103398 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:42:18.103374    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-400a5b-gvtzk"
Jul 27 21:42:18.248710 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:42:18.248624    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ed4782e-5f6a-4b17-8db3-34cf40061089-maas-api-service-ca\") pod \"2ed4782e-5f6a-4b17-8db3-34cf40061089\" (UID: \"2ed4782e-5f6a-4b17-8db3-34cf40061089\") "
Jul 27 21:42:18.248989 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:42:18.248962    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ed4782e-5f6a-4b17-8db3-34cf40061089-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2ed4782e-5f6a-4b17-8db3-34cf40061089" (UID: "2ed4782e-5f6a-4b17-8db3-34cf40061089"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:42:18.350289 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:42:18.350256    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ed4782e-5f6a-4b17-8db3-34cf40061089-maas-api-service-ca\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:42:18.971407 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:42:18.971375    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-400a5b-gvtzk" event={"ID":"2ed4782e-5f6a-4b17-8db3-34cf40061089","Type":"ContainerDied","Data":"dfaeb04f0737ac71088f164c7676e07e8c0adda368a248619db7944910f67e06"}
Jul 27 21:42:18.971560 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:42:18.971419    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-400a5b-gvtzk"
Jul 27 21:42:18.971560 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:42:18.971423    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dfaeb04f0737ac71088f164c7676e07e8c0adda368a248619db7944910f67e06"
Jul 27 21:42:30.530043 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:42:30.529996    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-400a5b-gvtzk"]
Jul 27 21:42:30.534237 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:42:30.534205    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-400a5b-gvtzk"]
Jul 27 21:42:31.848319 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:42:31.848280    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ed4782e-5f6a-4b17-8db3-34cf40061089" path="/var/lib/kubelet/pods/2ed4782e-5f6a-4b17-8db3-34cf40061089/volumes"
Jul 27 21:42:34.191572 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:42:34.191536    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:42:57.382663 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:42:57.382628    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:43:07.395589 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:43:07.395549    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:43:50.078676 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:43:50.078636    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-b4bf5c-openshift-default-84869c884b-sxxsq"]
Jul 27 21:43:50.079269 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:43:50.078839    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-b4bf5c-openshift-default-84869c884b-sxxsq" podUID="911cf451-bf1d-4c8b-a240-288b2cd55fc1" containerName="istio-proxy" containerID="cri-o://fe1fcc4e681a6057902ac0c8b3aa955861f9ce096b782d363dc4d0610f04b700" gracePeriod=30
Jul 27 21:43:55.335515 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:43:55.335489    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-b4bf5c-openshift-default-84869c884b-sxxsq"
Jul 27 21:43:55.341794 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:43:55.341773    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/911cf451-bf1d-4c8b-a240-288b2cd55fc1-workload-socket\") pod \"911cf451-bf1d-4c8b-a240-288b2cd55fc1\" (UID: \"911cf451-bf1d-4c8b-a240-288b2cd55fc1\") "
Jul 27 21:43:55.341910 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:43:55.341809    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/911cf451-bf1d-4c8b-a240-288b2cd55fc1-workload-certs\") pod \"911cf451-bf1d-4c8b-a240-288b2cd55fc1\" (UID: \"911cf451-bf1d-4c8b-a240-288b2cd55fc1\") "
Jul 27 21:43:55.341910 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:43:55.341845    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/911cf451-bf1d-4c8b-a240-288b2cd55fc1-istiod-ca-cert\") pod \"911cf451-bf1d-4c8b-a240-288b2cd55fc1\" (UID: \"911cf451-bf1d-4c8b-a240-288b2cd55fc1\") "
Jul 27 21:43:55.341910 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:43:55.341870    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/911cf451-bf1d-4c8b-a240-288b2cd55fc1-istio-podinfo\") pod \"911cf451-bf1d-4c8b-a240-288b2cd55fc1\" (UID: \"911cf451-bf1d-4c8b-a240-288b2cd55fc1\") "
Jul 27 21:43:55.341910 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:43:55.341885    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/911cf451-bf1d-4c8b-a240-288b2cd55fc1-istio-data\") pod \"911cf451-bf1d-4c8b-a240-288b2cd55fc1\" (UID: \"911cf451-bf1d-4c8b-a240-288b2cd55fc1\") "
Jul 27 21:43:55.341910 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:43:55.341903    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ffzgc\" (UniqueName: \"kubernetes.io/projected/911cf451-bf1d-4c8b-a240-288b2cd55fc1-kube-api-access-ffzgc\") pod \"911cf451-bf1d-4c8b-a240-288b2cd55fc1\" (UID: \"911cf451-bf1d-4c8b-a240-288b2cd55fc1\") "
Jul 27 21:43:55.342201 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:43:55.341930    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/911cf451-bf1d-4c8b-a240-288b2cd55fc1-istio-envoy\") pod \"911cf451-bf1d-4c8b-a240-288b2cd55fc1\" (UID: \"911cf451-bf1d-4c8b-a240-288b2cd55fc1\") "
Jul 27 21:43:55.342201 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:43:55.341957    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/911cf451-bf1d-4c8b-a240-288b2cd55fc1-istio-token\") pod \"911cf451-bf1d-4c8b-a240-288b2cd55fc1\" (UID: \"911cf451-bf1d-4c8b-a240-288b2cd55fc1\") "
Jul 27 21:43:55.342201 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:43:55.341981    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/911cf451-bf1d-4c8b-a240-288b2cd55fc1-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "911cf451-bf1d-4c8b-a240-288b2cd55fc1" (UID: "911cf451-bf1d-4c8b-a240-288b2cd55fc1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:43:55.342201 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:43:55.342020    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/911cf451-bf1d-4c8b-a240-288b2cd55fc1-credential-socket\") pod \"911cf451-bf1d-4c8b-a240-288b2cd55fc1\" (UID: \"911cf451-bf1d-4c8b-a240-288b2cd55fc1\") "
Jul 27 21:43:55.342414 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:43:55.342195    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/911cf451-bf1d-4c8b-a240-288b2cd55fc1-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "911cf451-bf1d-4c8b-a240-288b2cd55fc1" (UID: "911cf451-bf1d-4c8b-a240-288b2cd55fc1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:43:55.342414 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:43:55.342282    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/911cf451-bf1d-4c8b-a240-288b2cd55fc1-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "911cf451-bf1d-4c8b-a240-288b2cd55fc1" (UID: "911cf451-bf1d-4c8b-a240-288b2cd55fc1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:43:55.342414 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:43:55.342309    2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/911cf451-bf1d-4c8b-a240-288b2cd55fc1-workload-socket\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:43:55.342414 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:43:55.342327    2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/911cf451-bf1d-4c8b-a240-288b2cd55fc1-workload-certs\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:43:55.342716 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:43:55.342666    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/911cf451-bf1d-4c8b-a240-288b2cd55fc1-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "911cf451-bf1d-4c8b-a240-288b2cd55fc1" (UID: "911cf451-bf1d-4c8b-a240-288b2cd55fc1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:43:55.343033 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:43:55.343008    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/911cf451-bf1d-4c8b-a240-288b2cd55fc1-istio-data" (OuterVolumeSpecName: "istio-data") pod "911cf451-bf1d-4c8b-a240-288b2cd55fc1" (UID: "911cf451-bf1d-4c8b-a240-288b2cd55fc1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:43:55.344357 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:43:55.344335    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/911cf451-bf1d-4c8b-a240-288b2cd55fc1-kube-api-access-ffzgc" (OuterVolumeSpecName: "kube-api-access-ffzgc") pod "911cf451-bf1d-4c8b-a240-288b2cd55fc1" (UID: "911cf451-bf1d-4c8b-a240-288b2cd55fc1"). InnerVolumeSpecName "kube-api-access-ffzgc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:43:55.344629 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:43:55.344580    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/911cf451-bf1d-4c8b-a240-288b2cd55fc1-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "911cf451-bf1d-4c8b-a240-288b2cd55fc1" (UID: "911cf451-bf1d-4c8b-a240-288b2cd55fc1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 27 21:43:55.344966 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:43:55.344943    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/911cf451-bf1d-4c8b-a240-288b2cd55fc1-istio-token" (OuterVolumeSpecName: "istio-token") pod "911cf451-bf1d-4c8b-a240-288b2cd55fc1" (UID: "911cf451-bf1d-4c8b-a240-288b2cd55fc1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:43:55.344966 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:43:55.344941    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/911cf451-bf1d-4c8b-a240-288b2cd55fc1-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "911cf451-bf1d-4c8b-a240-288b2cd55fc1" (UID: "911cf451-bf1d-4c8b-a240-288b2cd55fc1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:43:55.438643 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:43:55.438611    2560 generic.go:356] "Generic (PLEG): container finished" podID="911cf451-bf1d-4c8b-a240-288b2cd55fc1" containerID="fe1fcc4e681a6057902ac0c8b3aa955861f9ce096b782d363dc4d0610f04b700" exitCode=0
Jul 27 21:43:55.438828 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:43:55.438656    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-b4bf5c-openshift-default-84869c884b-sxxsq" event={"ID":"911cf451-bf1d-4c8b-a240-288b2cd55fc1","Type":"ContainerDied","Data":"fe1fcc4e681a6057902ac0c8b3aa955861f9ce096b782d363dc4d0610f04b700"}
Jul 27 21:43:55.438828 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:43:55.438696    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-b4bf5c-openshift-default-84869c884b-sxxsq" event={"ID":"911cf451-bf1d-4c8b-a240-288b2cd55fc1","Type":"ContainerDied","Data":"9a3302c43dd10974cb467a465d7ec7bce015f648378432527c03b86f883d5d2f"}
Jul 27 21:43:55.438828 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:43:55.438717    2560 scope.go:117] "RemoveContainer" containerID="fe1fcc4e681a6057902ac0c8b3aa955861f9ce096b782d363dc4d0610f04b700"
Jul 27 21:43:55.438828 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:43:55.438657    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-b4bf5c-openshift-default-84869c884b-sxxsq"
Jul 27 21:43:55.442958 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:43:55.442922    2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/911cf451-bf1d-4c8b-a240-288b2cd55fc1-istiod-ca-cert\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:43:55.442958 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:43:55.442953    2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/911cf451-bf1d-4c8b-a240-288b2cd55fc1-istio-podinfo\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:43:55.443115 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:43:55.442969    2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/911cf451-bf1d-4c8b-a240-288b2cd55fc1-istio-data\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:43:55.443115 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:43:55.442983    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffzgc\" (UniqueName: \"kubernetes.io/projected/911cf451-bf1d-4c8b-a240-288b2cd55fc1-kube-api-access-ffzgc\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:43:55.443115 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:43:55.442997    2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/911cf451-bf1d-4c8b-a240-288b2cd55fc1-istio-envoy\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:43:55.443115 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:43:55.443010    2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/911cf451-bf1d-4c8b-a240-288b2cd55fc1-istio-token\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:43:55.443115 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:43:55.443023    2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/911cf451-bf1d-4c8b-a240-288b2cd55fc1-credential-socket\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:43:55.449959 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:43:55.449917    2560 scope.go:117] "RemoveContainer" containerID="004ef90f58495d2ada9a0e1736ac957ab099af7db4d7299166eb28c12d733728"
Jul 27 21:43:55.458207 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:43:55.458191    2560 scope.go:117] "RemoveContainer" containerID="fe1fcc4e681a6057902ac0c8b3aa955861f9ce096b782d363dc4d0610f04b700"
Jul 27 21:43:55.458459 ip-10-0-141-64 kubenswrapper[2560]: E0727 21:43:55.458440    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe1fcc4e681a6057902ac0c8b3aa955861f9ce096b782d363dc4d0610f04b700\": container with ID starting with fe1fcc4e681a6057902ac0c8b3aa955861f9ce096b782d363dc4d0610f04b700 not found: ID does not exist" containerID="fe1fcc4e681a6057902ac0c8b3aa955861f9ce096b782d363dc4d0610f04b700"
Jul 27 21:43:55.458529 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:43:55.458466    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe1fcc4e681a6057902ac0c8b3aa955861f9ce096b782d363dc4d0610f04b700"} err="failed to get container status \"fe1fcc4e681a6057902ac0c8b3aa955861f9ce096b782d363dc4d0610f04b700\": rpc error: code = NotFound desc = could not find container \"fe1fcc4e681a6057902ac0c8b3aa955861f9ce096b782d363dc4d0610f04b700\": container with ID starting with fe1fcc4e681a6057902ac0c8b3aa955861f9ce096b782d363dc4d0610f04b700 not found: ID does not exist"
Jul 27 21:43:55.458529 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:43:55.458481    2560 scope.go:117] "RemoveContainer" containerID="004ef90f58495d2ada9a0e1736ac957ab099af7db4d7299166eb28c12d733728"
Jul 27 21:43:55.458685 ip-10-0-141-64 kubenswrapper[2560]: E0727 21:43:55.458670    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"004ef90f58495d2ada9a0e1736ac957ab099af7db4d7299166eb28c12d733728\": container with ID starting with 004ef90f58495d2ada9a0e1736ac957ab099af7db4d7299166eb28c12d733728 not found: ID does not exist" containerID="004ef90f58495d2ada9a0e1736ac957ab099af7db4d7299166eb28c12d733728"
Jul 27 21:43:55.458729 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:43:55.458688    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"004ef90f58495d2ada9a0e1736ac957ab099af7db4d7299166eb28c12d733728"} err="failed to get container status \"004ef90f58495d2ada9a0e1736ac957ab099af7db4d7299166eb28c12d733728\": rpc error: code = NotFound desc = could not find container \"004ef90f58495d2ada9a0e1736ac957ab099af7db4d7299166eb28c12d733728\": container with ID starting with 004ef90f58495d2ada9a0e1736ac957ab099af7db4d7299166eb28c12d733728 not found: ID does not exist"
Jul 27 21:43:55.465511 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:43:55.465487    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-b4bf5c-openshift-default-84869c884b-sxxsq"]
Jul 27 21:43:55.469680 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:43:55.469660    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-b4bf5c-openshift-default-84869c884b-sxxsq"]
Jul 27 21:43:55.847870 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:43:55.847834    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="911cf451-bf1d-4c8b-a240-288b2cd55fc1" path="/var/lib/kubelet/pods/911cf451-bf1d-4c8b-a240-288b2cd55fc1/volumes"
Jul 27 21:44:04.196518 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:44:04.196474    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:44:12.130066 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:44:12.130031    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-9e51a0-openshift-default-85b958c9f4-ntd42"]
Jul 27 21:44:12.130529 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:44:12.130254    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-9e51a0-openshift-default-85b958c9f4-ntd42" podUID="0fc339fb-bbea-4776-b172-fc631a1dce80" containerName="istio-proxy" containerID="cri-o://3025c3228bf39a3eac6c13de1fea4186293220072e913d29f786bd7a045ae1a1" gracePeriod=30
Jul 27 21:44:17.390244 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:44:17.390216    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-9e51a0-openshift-default-85b958c9f4-ntd42"
Jul 27 21:44:17.452857 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:44:17.452781    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0fc339fb-bbea-4776-b172-fc631a1dce80-istio-data\") pod \"0fc339fb-bbea-4776-b172-fc631a1dce80\" (UID: \"0fc339fb-bbea-4776-b172-fc631a1dce80\") "
Jul 27 21:44:17.452857 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:44:17.452815    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0fc339fb-bbea-4776-b172-fc631a1dce80-istio-envoy\") pod \"0fc339fb-bbea-4776-b172-fc631a1dce80\" (UID: \"0fc339fb-bbea-4776-b172-fc631a1dce80\") "
Jul 27 21:44:17.452857 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:44:17.452837    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fc339fb-bbea-4776-b172-fc631a1dce80-credential-socket\") pod \"0fc339fb-bbea-4776-b172-fc631a1dce80\" (UID: \"0fc339fb-bbea-4776-b172-fc631a1dce80\") "
Jul 27 21:44:17.453179 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:44:17.452878    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8b9jt\" (UniqueName: \"kubernetes.io/projected/0fc339fb-bbea-4776-b172-fc631a1dce80-kube-api-access-8b9jt\") pod \"0fc339fb-bbea-4776-b172-fc631a1dce80\" (UID: \"0fc339fb-bbea-4776-b172-fc631a1dce80\") "
Jul 27 21:44:17.453179 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:44:17.452913    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0fc339fb-bbea-4776-b172-fc631a1dce80-istiod-ca-cert\") pod \"0fc339fb-bbea-4776-b172-fc631a1dce80\" (UID: \"0fc339fb-bbea-4776-b172-fc631a1dce80\") "
Jul 27 21:44:17.453179 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:44:17.452966    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fc339fb-bbea-4776-b172-fc631a1dce80-workload-socket\") pod \"0fc339fb-bbea-4776-b172-fc631a1dce80\" (UID: \"0fc339fb-bbea-4776-b172-fc631a1dce80\") "
Jul 27 21:44:17.453179 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:44:17.452996    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0fc339fb-bbea-4776-b172-fc631a1dce80-istio-token\") pod \"0fc339fb-bbea-4776-b172-fc631a1dce80\" (UID: \"0fc339fb-bbea-4776-b172-fc631a1dce80\") "
Jul 27 21:44:17.453179 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:44:17.453017    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0fc339fb-bbea-4776-b172-fc631a1dce80-istio-podinfo\") pod \"0fc339fb-bbea-4776-b172-fc631a1dce80\" (UID: \"0fc339fb-bbea-4776-b172-fc631a1dce80\") "
Jul 27 21:44:17.453179 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:44:17.453050    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fc339fb-bbea-4776-b172-fc631a1dce80-workload-certs\") pod \"0fc339fb-bbea-4776-b172-fc631a1dce80\" (UID: \"0fc339fb-bbea-4776-b172-fc631a1dce80\") "
Jul 27 21:44:17.453471 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:44:17.453231    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fc339fb-bbea-4776-b172-fc631a1dce80-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0fc339fb-bbea-4776-b172-fc631a1dce80" (UID: "0fc339fb-bbea-4776-b172-fc631a1dce80"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:44:17.453471 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:44:17.453412    2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fc339fb-bbea-4776-b172-fc631a1dce80-credential-socket\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:44:17.453566 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:44:17.453463    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fc339fb-bbea-4776-b172-fc631a1dce80-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0fc339fb-bbea-4776-b172-fc631a1dce80" (UID: "0fc339fb-bbea-4776-b172-fc631a1dce80"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:44:17.453566 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:44:17.453389    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fc339fb-bbea-4776-b172-fc631a1dce80-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0fc339fb-bbea-4776-b172-fc631a1dce80" (UID: "0fc339fb-bbea-4776-b172-fc631a1dce80"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:44:17.453679 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:44:17.453642    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fc339fb-bbea-4776-b172-fc631a1dce80-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0fc339fb-bbea-4776-b172-fc631a1dce80" (UID: "0fc339fb-bbea-4776-b172-fc631a1dce80"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:44:17.453679 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:44:17.453665    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fc339fb-bbea-4776-b172-fc631a1dce80-istio-data" (OuterVolumeSpecName: "istio-data") pod "0fc339fb-bbea-4776-b172-fc631a1dce80" (UID: "0fc339fb-bbea-4776-b172-fc631a1dce80"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:44:17.455534 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:44:17.455502    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0fc339fb-bbea-4776-b172-fc631a1dce80-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0fc339fb-bbea-4776-b172-fc631a1dce80" (UID: "0fc339fb-bbea-4776-b172-fc631a1dce80"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 27 21:44:17.455649 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:44:17.455590    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fc339fb-bbea-4776-b172-fc631a1dce80-istio-token" (OuterVolumeSpecName: "istio-token") pod "0fc339fb-bbea-4776-b172-fc631a1dce80" (UID: "0fc339fb-bbea-4776-b172-fc631a1dce80"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:44:17.455697 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:44:17.455643    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fc339fb-bbea-4776-b172-fc631a1dce80-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0fc339fb-bbea-4776-b172-fc631a1dce80" (UID: "0fc339fb-bbea-4776-b172-fc631a1dce80"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:44:17.455697 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:44:17.455662    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fc339fb-bbea-4776-b172-fc631a1dce80-kube-api-access-8b9jt" (OuterVolumeSpecName: "kube-api-access-8b9jt") pod "0fc339fb-bbea-4776-b172-fc631a1dce80" (UID: "0fc339fb-bbea-4776-b172-fc631a1dce80"). InnerVolumeSpecName "kube-api-access-8b9jt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:44:17.550047 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:44:17.550016    2560 generic.go:356] "Generic (PLEG): container finished" podID="0fc339fb-bbea-4776-b172-fc631a1dce80" containerID="3025c3228bf39a3eac6c13de1fea4186293220072e913d29f786bd7a045ae1a1" exitCode=0
Jul 27 21:44:17.550196 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:44:17.550062    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-9e51a0-openshift-default-85b958c9f4-ntd42"
Jul 27 21:44:17.550196 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:44:17.550117    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-9e51a0-openshift-default-85b958c9f4-ntd42" event={"ID":"0fc339fb-bbea-4776-b172-fc631a1dce80","Type":"ContainerDied","Data":"3025c3228bf39a3eac6c13de1fea4186293220072e913d29f786bd7a045ae1a1"}
Jul 27 21:44:17.550196 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:44:17.550175    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-9e51a0-openshift-default-85b958c9f4-ntd42" event={"ID":"0fc339fb-bbea-4776-b172-fc631a1dce80","Type":"ContainerDied","Data":"cf36341632f381e784896262d7641e132b2651b90df48d2f46555a5addb566b2"}
Jul 27 21:44:17.550305 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:44:17.550201    2560 scope.go:117] "RemoveContainer" containerID="3025c3228bf39a3eac6c13de1fea4186293220072e913d29f786bd7a045ae1a1"
Jul 27 21:44:17.553963 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:44:17.553934    2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fc339fb-bbea-4776-b172-fc631a1dce80-workload-certs\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:44:17.553963 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:44:17.553952    2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0fc339fb-bbea-4776-b172-fc631a1dce80-istio-data\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:44:17.553963 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:44:17.553963    2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0fc339fb-bbea-4776-b172-fc631a1dce80-istio-envoy\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:44:17.554220 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:44:17.553975    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8b9jt\" (UniqueName: \"kubernetes.io/projected/0fc339fb-bbea-4776-b172-fc631a1dce80-kube-api-access-8b9jt\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:44:17.554220 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:44:17.553985    2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0fc339fb-bbea-4776-b172-fc631a1dce80-istiod-ca-cert\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:44:17.554220 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:44:17.553994    2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fc339fb-bbea-4776-b172-fc631a1dce80-workload-socket\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:44:17.554220 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:44:17.554002    2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0fc339fb-bbea-4776-b172-fc631a1dce80-istio-token\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:44:17.554220 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:44:17.554010    2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0fc339fb-bbea-4776-b172-fc631a1dce80-istio-podinfo\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:44:17.559224 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:44:17.559207    2560 scope.go:117] "RemoveContainer" containerID="7ea5fc862caae62dfb73027c0b73290f5d7667bc76e2ea34e94eca9f7cc10232"
Jul 27 21:44:17.566954 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:44:17.566937    2560 scope.go:117] "RemoveContainer" containerID="3025c3228bf39a3eac6c13de1fea4186293220072e913d29f786bd7a045ae1a1"
Jul 27 21:44:17.567262 ip-10-0-141-64 kubenswrapper[2560]: E0727 21:44:17.567243    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3025c3228bf39a3eac6c13de1fea4186293220072e913d29f786bd7a045ae1a1\": container with ID starting with 3025c3228bf39a3eac6c13de1fea4186293220072e913d29f786bd7a045ae1a1 not found: ID does not exist" containerID="3025c3228bf39a3eac6c13de1fea4186293220072e913d29f786bd7a045ae1a1"
Jul 27 21:44:17.567310 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:44:17.567269    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3025c3228bf39a3eac6c13de1fea4186293220072e913d29f786bd7a045ae1a1"} err="failed to get container status \"3025c3228bf39a3eac6c13de1fea4186293220072e913d29f786bd7a045ae1a1\": rpc error: code = NotFound desc = could not find container \"3025c3228bf39a3eac6c13de1fea4186293220072e913d29f786bd7a045ae1a1\": container with ID starting with 3025c3228bf39a3eac6c13de1fea4186293220072e913d29f786bd7a045ae1a1 not found: ID does not exist"
Jul 27 21:44:17.567310 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:44:17.567284    2560 scope.go:117] "RemoveContainer" containerID="7ea5fc862caae62dfb73027c0b73290f5d7667bc76e2ea34e94eca9f7cc10232"
Jul 27 21:44:17.567508 ip-10-0-141-64 kubenswrapper[2560]: E0727 21:44:17.567488    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ea5fc862caae62dfb73027c0b73290f5d7667bc76e2ea34e94eca9f7cc10232\": container with ID starting with 7ea5fc862caae62dfb73027c0b73290f5d7667bc76e2ea34e94eca9f7cc10232 not found: ID does not exist" containerID="7ea5fc862caae62dfb73027c0b73290f5d7667bc76e2ea34e94eca9f7cc10232"
Jul 27 21:44:17.567564 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:44:17.567516    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ea5fc862caae62dfb73027c0b73290f5d7667bc76e2ea34e94eca9f7cc10232"} err="failed to get container status \"7ea5fc862caae62dfb73027c0b73290f5d7667bc76e2ea34e94eca9f7cc10232\": rpc error: code = NotFound desc = could not find container \"7ea5fc862caae62dfb73027c0b73290f5d7667bc76e2ea34e94eca9f7cc10232\": container with ID starting with 7ea5fc862caae62dfb73027c0b73290f5d7667bc76e2ea34e94eca9f7cc10232 not found: ID does not exist"
Jul 27 21:44:17.575723 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:44:17.575705    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-9e51a0-openshift-default-85b958c9f4-ntd42"]
Jul 27 21:44:17.582019 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:44:17.581993    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-9e51a0-openshift-default-85b958c9f4-ntd42"]
Jul 27 21:44:17.847656 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:44:17.847624    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0fc339fb-bbea-4776-b172-fc631a1dce80" path="/var/lib/kubelet/pods/0fc339fb-bbea-4776-b172-fc631a1dce80/volumes"
Jul 27 21:44:26.290460 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:44:26.290419    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:44:34.160801 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:44:34.160763    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-6dd721-openshift-default-55b7fb86b4-622lp"]
Jul 27 21:44:34.161218 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:44:34.161026    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-6dd721-openshift-default-55b7fb86b4-622lp" podUID="8b059bf6-3c81-41ed-b200-965fa935795f" containerName="istio-proxy" containerID="cri-o://4b347a473a46fbab2678a965d81ca9ad462f7faa395fae2387db9e9db434978d" gracePeriod=30
Jul 27 21:44:39.418204 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:44:39.418173    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-6dd721-openshift-default-55b7fb86b4-622lp"
Jul 27 21:44:39.449181 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:44:39.449075    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b059bf6-3c81-41ed-b200-965fa935795f-istio-token\") pod \"8b059bf6-3c81-41ed-b200-965fa935795f\" (UID: \"8b059bf6-3c81-41ed-b200-965fa935795f\") "
Jul 27 21:44:39.449181 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:44:39.449163    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-45t4t\" (UniqueName: \"kubernetes.io/projected/8b059bf6-3c81-41ed-b200-965fa935795f-kube-api-access-45t4t\") pod \"8b059bf6-3c81-41ed-b200-965fa935795f\" (UID: \"8b059bf6-3c81-41ed-b200-965fa935795f\") "
Jul 27 21:44:39.449389 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:44:39.449201    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b059bf6-3c81-41ed-b200-965fa935795f-istio-podinfo\") pod \"8b059bf6-3c81-41ed-b200-965fa935795f\" (UID: \"8b059bf6-3c81-41ed-b200-965fa935795f\") "
Jul 27 21:44:39.449389 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:44:39.449227    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b059bf6-3c81-41ed-b200-965fa935795f-workload-certs\") pod \"8b059bf6-3c81-41ed-b200-965fa935795f\" (UID: \"8b059bf6-3c81-41ed-b200-965fa935795f\") "
Jul 27 21:44:39.449389 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:44:39.449281    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b059bf6-3c81-41ed-b200-965fa935795f-credential-socket\") pod \"8b059bf6-3c81-41ed-b200-965fa935795f\" (UID: \"8b059bf6-3c81-41ed-b200-965fa935795f\") "
Jul 27 21:44:39.449389 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:44:39.449303    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b059bf6-3c81-41ed-b200-965fa935795f-istio-data\") pod \"8b059bf6-3c81-41ed-b200-965fa935795f\" (UID: \"8b059bf6-3c81-41ed-b200-965fa935795f\") "
Jul 27 21:44:39.449389 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:44:39.449376    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b059bf6-3c81-41ed-b200-965fa935795f-istiod-ca-cert\") pod \"8b059bf6-3c81-41ed-b200-965fa935795f\" (UID: \"8b059bf6-3c81-41ed-b200-965fa935795f\") "
Jul 27 21:44:39.449655 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:44:39.449415    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b059bf6-3c81-41ed-b200-965fa935795f-workload-socket\") pod \"8b059bf6-3c81-41ed-b200-965fa935795f\" (UID: \"8b059bf6-3c81-41ed-b200-965fa935795f\") "
Jul 27 21:44:39.449655 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:44:39.449458    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b059bf6-3c81-41ed-b200-965fa935795f-istio-envoy\") pod \"8b059bf6-3c81-41ed-b200-965fa935795f\" (UID: \"8b059bf6-3c81-41ed-b200-965fa935795f\") "
Jul 27 21:44:39.449655 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:44:39.449578    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b059bf6-3c81-41ed-b200-965fa935795f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8b059bf6-3c81-41ed-b200-965fa935795f" (UID: "8b059bf6-3c81-41ed-b200-965fa935795f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:44:39.449655 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:44:39.449596    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b059bf6-3c81-41ed-b200-965fa935795f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8b059bf6-3c81-41ed-b200-965fa935795f" (UID: "8b059bf6-3c81-41ed-b200-965fa935795f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:44:39.449975 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:44:39.449789    2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b059bf6-3c81-41ed-b200-965fa935795f-workload-certs\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:44:39.449975 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:44:39.449810    2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b059bf6-3c81-41ed-b200-965fa935795f-credential-socket\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:44:39.449975 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:44:39.449819    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b059bf6-3c81-41ed-b200-965fa935795f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8b059bf6-3c81-41ed-b200-965fa935795f" (UID: "8b059bf6-3c81-41ed-b200-965fa935795f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:44:39.449975 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:44:39.449820    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b059bf6-3c81-41ed-b200-965fa935795f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8b059bf6-3c81-41ed-b200-965fa935795f" (UID: "8b059bf6-3c81-41ed-b200-965fa935795f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:44:39.451011 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:44:39.450970    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b059bf6-3c81-41ed-b200-965fa935795f-istio-data" (OuterVolumeSpecName: "istio-data") pod "8b059bf6-3c81-41ed-b200-965fa935795f" (UID: "8b059bf6-3c81-41ed-b200-965fa935795f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:44:39.451701 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:44:39.451670    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8b059bf6-3c81-41ed-b200-965fa935795f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8b059bf6-3c81-41ed-b200-965fa935795f" (UID: "8b059bf6-3c81-41ed-b200-965fa935795f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 27 21:44:39.451806 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:44:39.451763    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b059bf6-3c81-41ed-b200-965fa935795f-kube-api-access-45t4t" (OuterVolumeSpecName: "kube-api-access-45t4t") pod "8b059bf6-3c81-41ed-b200-965fa935795f" (UID: "8b059bf6-3c81-41ed-b200-965fa935795f"). InnerVolumeSpecName "kube-api-access-45t4t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:44:39.451948 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:44:39.451927    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b059bf6-3c81-41ed-b200-965fa935795f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8b059bf6-3c81-41ed-b200-965fa935795f" (UID: "8b059bf6-3c81-41ed-b200-965fa935795f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:44:39.452251 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:44:39.452234    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b059bf6-3c81-41ed-b200-965fa935795f-istio-token" (OuterVolumeSpecName: "istio-token") pod "8b059bf6-3c81-41ed-b200-965fa935795f" (UID: "8b059bf6-3c81-41ed-b200-965fa935795f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:44:39.550264 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:44:39.550228    2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b059bf6-3c81-41ed-b200-965fa935795f-istio-data\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:44:39.550264 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:44:39.550257    2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b059bf6-3c81-41ed-b200-965fa935795f-istiod-ca-cert\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:44:39.550264 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:44:39.550266    2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b059bf6-3c81-41ed-b200-965fa935795f-workload-socket\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:44:39.550264 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:44:39.550274    2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b059bf6-3c81-41ed-b200-965fa935795f-istio-envoy\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:44:39.550529 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:44:39.550282    2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b059bf6-3c81-41ed-b200-965fa935795f-istio-token\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:44:39.550529 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:44:39.550292    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45t4t\" (UniqueName: \"kubernetes.io/projected/8b059bf6-3c81-41ed-b200-965fa935795f-kube-api-access-45t4t\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:44:39.550529 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:44:39.550299    2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b059bf6-3c81-41ed-b200-965fa935795f-istio-podinfo\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:44:39.657750 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:44:39.657714    2560 generic.go:356] "Generic (PLEG): container finished" podID="8b059bf6-3c81-41ed-b200-965fa935795f" containerID="4b347a473a46fbab2678a965d81ca9ad462f7faa395fae2387db9e9db434978d" exitCode=0
Jul 27 21:44:39.657930 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:44:39.657756    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-6dd721-openshift-default-55b7fb86b4-622lp"
Jul 27 21:44:39.657930 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:44:39.657783    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-6dd721-openshift-default-55b7fb86b4-622lp" event={"ID":"8b059bf6-3c81-41ed-b200-965fa935795f","Type":"ContainerDied","Data":"4b347a473a46fbab2678a965d81ca9ad462f7faa395fae2387db9e9db434978d"}
Jul 27 21:44:39.657930 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:44:39.657831    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-6dd721-openshift-default-55b7fb86b4-622lp" event={"ID":"8b059bf6-3c81-41ed-b200-965fa935795f","Type":"ContainerDied","Data":"5e9697572127cfbd98aee9cd738fc895be189361b8d6daaa57002008c710add2"}
Jul 27 21:44:39.657930 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:44:39.657852    2560 scope.go:117] "RemoveContainer" containerID="4b347a473a46fbab2678a965d81ca9ad462f7faa395fae2387db9e9db434978d"
Jul 27 21:44:39.667077 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:44:39.667062    2560 scope.go:117] "RemoveContainer" containerID="c563cf2cde488be91a55e5dc2f30e67410d582f01d8577329c37fdddd2ce622f"
Jul 27 21:44:39.674778 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:44:39.674759    2560 scope.go:117] "RemoveContainer" containerID="4b347a473a46fbab2678a965d81ca9ad462f7faa395fae2387db9e9db434978d"
Jul 27 21:44:39.675024 ip-10-0-141-64 kubenswrapper[2560]: E0727 21:44:39.675006    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b347a473a46fbab2678a965d81ca9ad462f7faa395fae2387db9e9db434978d\": container with ID starting with 4b347a473a46fbab2678a965d81ca9ad462f7faa395fae2387db9e9db434978d not found: ID does not exist" containerID="4b347a473a46fbab2678a965d81ca9ad462f7faa395fae2387db9e9db434978d"
Jul 27 21:44:39.675073 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:44:39.675032    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b347a473a46fbab2678a965d81ca9ad462f7faa395fae2387db9e9db434978d"} err="failed to get container status \"4b347a473a46fbab2678a965d81ca9ad462f7faa395fae2387db9e9db434978d\": rpc error: code = NotFound desc = could not find container \"4b347a473a46fbab2678a965d81ca9ad462f7faa395fae2387db9e9db434978d\": container with ID starting with 4b347a473a46fbab2678a965d81ca9ad462f7faa395fae2387db9e9db434978d not found: ID does not exist"
Jul 27 21:44:39.675073 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:44:39.675049    2560 scope.go:117] "RemoveContainer" containerID="c563cf2cde488be91a55e5dc2f30e67410d582f01d8577329c37fdddd2ce622f"
Jul 27 21:44:39.675311 ip-10-0-141-64 kubenswrapper[2560]: E0727 21:44:39.675289    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c563cf2cde488be91a55e5dc2f30e67410d582f01d8577329c37fdddd2ce622f\": container with ID starting with c563cf2cde488be91a55e5dc2f30e67410d582f01d8577329c37fdddd2ce622f not found: ID does not exist" containerID="c563cf2cde488be91a55e5dc2f30e67410d582f01d8577329c37fdddd2ce622f"
Jul 27 21:44:39.675374 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:44:39.675321    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c563cf2cde488be91a55e5dc2f30e67410d582f01d8577329c37fdddd2ce622f"} err="failed to get container status \"c563cf2cde488be91a55e5dc2f30e67410d582f01d8577329c37fdddd2ce622f\": rpc error: code = NotFound desc = could not find container \"c563cf2cde488be91a55e5dc2f30e67410d582f01d8577329c37fdddd2ce622f\": container with ID starting with c563cf2cde488be91a55e5dc2f30e67410d582f01d8577329c37fdddd2ce622f not found: ID does not exist"
Jul 27 21:44:39.679580 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:44:39.679556    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-6dd721-openshift-default-55b7fb86b4-622lp"]
Jul 27 21:44:39.685116 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:44:39.685076    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-6dd721-openshift-default-55b7fb86b4-622lp"]
Jul 27 21:44:39.847297 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:44:39.847264    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b059bf6-3c81-41ed-b200-965fa935795f" path="/var/lib/kubelet/pods/8b059bf6-3c81-41ed-b200-965fa935795f/volumes"
Jul 27 21:44:48.104683 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:44:48.104648    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7znrn"]
Jul 27 21:44:56.203948 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:44:56.203908    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-5f50cb-openshift-default-5fcbf9d69b-tts4q"]
Jul 27 21:44:56.204386 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:44:56.204136    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-5f50cb-openshift-default-5fcbf9d69b-tts4q" podUID="00e24a4b-9175-46f4-b80e-42d8694750cb" containerName="istio-proxy" containerID="cri-o://550028813a7471e91b053b2c8a0c0e111e47b11bd8ffb7a7990f01b91634e8cd" gracePeriod=30
Jul 27 21:45:01.489451 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:01.489425    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-5f50cb-openshift-default-5fcbf9d69b-tts4q"
Jul 27 21:45:01.541914 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:01.541883    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00e24a4b-9175-46f4-b80e-42d8694750cb-istio-data\") pod \"00e24a4b-9175-46f4-b80e-42d8694750cb\" (UID: \"00e24a4b-9175-46f4-b80e-42d8694750cb\") "
Jul 27 21:45:01.541914 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:01.541919    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qvjlb\" (UniqueName: \"kubernetes.io/projected/00e24a4b-9175-46f4-b80e-42d8694750cb-kube-api-access-qvjlb\") pod \"00e24a4b-9175-46f4-b80e-42d8694750cb\" (UID: \"00e24a4b-9175-46f4-b80e-42d8694750cb\") "
Jul 27 21:45:01.542176 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:01.541955    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00e24a4b-9175-46f4-b80e-42d8694750cb-istiod-ca-cert\") pod \"00e24a4b-9175-46f4-b80e-42d8694750cb\" (UID: \"00e24a4b-9175-46f4-b80e-42d8694750cb\") "
Jul 27 21:45:01.542176 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:01.541996    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00e24a4b-9175-46f4-b80e-42d8694750cb-workload-socket\") pod \"00e24a4b-9175-46f4-b80e-42d8694750cb\" (UID: \"00e24a4b-9175-46f4-b80e-42d8694750cb\") "
Jul 27 21:45:01.542176 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:01.542019    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00e24a4b-9175-46f4-b80e-42d8694750cb-istio-podinfo\") pod \"00e24a4b-9175-46f4-b80e-42d8694750cb\" (UID: \"00e24a4b-9175-46f4-b80e-42d8694750cb\") "
Jul 27 21:45:01.542176 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:01.542033    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00e24a4b-9175-46f4-b80e-42d8694750cb-istio-token\") pod \"00e24a4b-9175-46f4-b80e-42d8694750cb\" (UID: \"00e24a4b-9175-46f4-b80e-42d8694750cb\") "
Jul 27 21:45:01.542176 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:01.542051    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00e24a4b-9175-46f4-b80e-42d8694750cb-workload-certs\") pod \"00e24a4b-9175-46f4-b80e-42d8694750cb\" (UID: \"00e24a4b-9175-46f4-b80e-42d8694750cb\") "
Jul 27 21:45:01.542176 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:01.542079    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00e24a4b-9175-46f4-b80e-42d8694750cb-credential-socket\") pod \"00e24a4b-9175-46f4-b80e-42d8694750cb\" (UID: \"00e24a4b-9175-46f4-b80e-42d8694750cb\") "
Jul 27 21:45:01.542176 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:01.542160    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00e24a4b-9175-46f4-b80e-42d8694750cb-istio-envoy\") pod \"00e24a4b-9175-46f4-b80e-42d8694750cb\" (UID: \"00e24a4b-9175-46f4-b80e-42d8694750cb\") "
Jul 27 21:45:01.542530 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:01.542342    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00e24a4b-9175-46f4-b80e-42d8694750cb-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "00e24a4b-9175-46f4-b80e-42d8694750cb" (UID: "00e24a4b-9175-46f4-b80e-42d8694750cb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:45:01.542530 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:01.542426    2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00e24a4b-9175-46f4-b80e-42d8694750cb-workload-socket\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:45:01.542630 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:01.542540    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00e24a4b-9175-46f4-b80e-42d8694750cb-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "00e24a4b-9175-46f4-b80e-42d8694750cb" (UID: "00e24a4b-9175-46f4-b80e-42d8694750cb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:45:01.542630 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:01.542471    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00e24a4b-9175-46f4-b80e-42d8694750cb-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "00e24a4b-9175-46f4-b80e-42d8694750cb" (UID: "00e24a4b-9175-46f4-b80e-42d8694750cb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:45:01.542928 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:01.542900    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00e24a4b-9175-46f4-b80e-42d8694750cb-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "00e24a4b-9175-46f4-b80e-42d8694750cb" (UID: "00e24a4b-9175-46f4-b80e-42d8694750cb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:45:01.542928 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:01.542912    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00e24a4b-9175-46f4-b80e-42d8694750cb-istio-data" (OuterVolumeSpecName: "istio-data") pod "00e24a4b-9175-46f4-b80e-42d8694750cb" (UID: "00e24a4b-9175-46f4-b80e-42d8694750cb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:45:01.544445 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:01.544340    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/00e24a4b-9175-46f4-b80e-42d8694750cb-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "00e24a4b-9175-46f4-b80e-42d8694750cb" (UID: "00e24a4b-9175-46f4-b80e-42d8694750cb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 27 21:45:01.544445 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:01.544380    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00e24a4b-9175-46f4-b80e-42d8694750cb-kube-api-access-qvjlb" (OuterVolumeSpecName: "kube-api-access-qvjlb") pod "00e24a4b-9175-46f4-b80e-42d8694750cb" (UID: "00e24a4b-9175-46f4-b80e-42d8694750cb"). InnerVolumeSpecName "kube-api-access-qvjlb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:45:01.544445 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:01.544419    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00e24a4b-9175-46f4-b80e-42d8694750cb-istio-token" (OuterVolumeSpecName: "istio-token") pod "00e24a4b-9175-46f4-b80e-42d8694750cb" (UID: "00e24a4b-9175-46f4-b80e-42d8694750cb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:45:01.544626 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:01.544578    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00e24a4b-9175-46f4-b80e-42d8694750cb-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "00e24a4b-9175-46f4-b80e-42d8694750cb" (UID: "00e24a4b-9175-46f4-b80e-42d8694750cb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:45:01.643594 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:01.643555    2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00e24a4b-9175-46f4-b80e-42d8694750cb-istio-podinfo\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:45:01.643594 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:01.643592    2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00e24a4b-9175-46f4-b80e-42d8694750cb-istio-token\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:45:01.643800 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:01.643604    2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00e24a4b-9175-46f4-b80e-42d8694750cb-workload-certs\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:45:01.643800 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:01.643618    2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00e24a4b-9175-46f4-b80e-42d8694750cb-credential-socket\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:45:01.643800 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:01.643630    2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00e24a4b-9175-46f4-b80e-42d8694750cb-istio-envoy\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:45:01.643800 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:01.643642    2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00e24a4b-9175-46f4-b80e-42d8694750cb-istio-data\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:45:01.643800 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:01.643652    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvjlb\" (UniqueName: \"kubernetes.io/projected/00e24a4b-9175-46f4-b80e-42d8694750cb-kube-api-access-qvjlb\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:45:01.643800 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:01.643663    2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00e24a4b-9175-46f4-b80e-42d8694750cb-istiod-ca-cert\") on node \"ip-10-0-141-64.ec2.internal\" DevicePath \"\""
Jul 27 21:45:01.763379 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:01.763285    2560 generic.go:356] "Generic (PLEG): container finished" podID="00e24a4b-9175-46f4-b80e-42d8694750cb" containerID="550028813a7471e91b053b2c8a0c0e111e47b11bd8ffb7a7990f01b91634e8cd" exitCode=0
Jul 27 21:45:01.763379 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:01.763332    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-5f50cb-openshift-default-5fcbf9d69b-tts4q"
Jul 27 21:45:01.763379 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:01.763361    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-5f50cb-openshift-default-5fcbf9d69b-tts4q" event={"ID":"00e24a4b-9175-46f4-b80e-42d8694750cb","Type":"ContainerDied","Data":"550028813a7471e91b053b2c8a0c0e111e47b11bd8ffb7a7990f01b91634e8cd"}
Jul 27 21:45:01.763640 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:01.763404    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-5f50cb-openshift-default-5fcbf9d69b-tts4q" event={"ID":"00e24a4b-9175-46f4-b80e-42d8694750cb","Type":"ContainerDied","Data":"78439aaaf2f610b9999d1e29184b5dad0603c658e2554ecfd6b431a3e438b4e0"}
Jul 27 21:45:01.763640 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:01.763426    2560 scope.go:117] "RemoveContainer" containerID="550028813a7471e91b053b2c8a0c0e111e47b11bd8ffb7a7990f01b91634e8cd"
Jul 27 21:45:01.772743 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:01.772725    2560 scope.go:117] "RemoveContainer" containerID="f34160be023c9b3a7260d6fefb9db064a552ef2d13b01e86d44aca5de303d56d"
Jul 27 21:45:01.780548 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:01.780531    2560 scope.go:117] "RemoveContainer" containerID="550028813a7471e91b053b2c8a0c0e111e47b11bd8ffb7a7990f01b91634e8cd"
Jul 27 21:45:01.780808 ip-10-0-141-64 kubenswrapper[2560]: E0727 21:45:01.780789    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"550028813a7471e91b053b2c8a0c0e111e47b11bd8ffb7a7990f01b91634e8cd\": container with ID starting with 550028813a7471e91b053b2c8a0c0e111e47b11bd8ffb7a7990f01b91634e8cd not found: ID does not exist" containerID="550028813a7471e91b053b2c8a0c0e111e47b11bd8ffb7a7990f01b91634e8cd"
Jul 27 21:45:01.780885 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:01.780820    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"550028813a7471e91b053b2c8a0c0e111e47b11bd8ffb7a7990f01b91634e8cd"} err="failed to get container status \"550028813a7471e91b053b2c8a0c0e111e47b11bd8ffb7a7990f01b91634e8cd\": rpc error: code = NotFound desc = could not find container \"550028813a7471e91b053b2c8a0c0e111e47b11bd8ffb7a7990f01b91634e8cd\": container with ID starting with 550028813a7471e91b053b2c8a0c0e111e47b11bd8ffb7a7990f01b91634e8cd not found: ID does not exist"
Jul 27 21:45:01.780885 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:01.780843    2560 scope.go:117] "RemoveContainer" containerID="f34160be023c9b3a7260d6fefb9db064a552ef2d13b01e86d44aca5de303d56d"
Jul 27 21:45:01.781101 ip-10-0-141-64 kubenswrapper[2560]: E0727 21:45:01.781068    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f34160be023c9b3a7260d6fefb9db064a552ef2d13b01e86d44aca5de303d56d\": container with ID starting with f34160be023c9b3a7260d6fefb9db064a552ef2d13b01e86d44aca5de303d56d not found: ID does not exist" containerID="f34160be023c9b3a7260d6fefb9db064a552ef2d13b01e86d44aca5de303d56d"
Jul 27 21:45:01.781150 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:01.781106    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f34160be023c9b3a7260d6fefb9db064a552ef2d13b01e86d44aca5de303d56d"} err="failed to get container status \"f34160be023c9b3a7260d6fefb9db064a552ef2d13b01e86d44aca5de303d56d\": rpc error: code = NotFound desc = could not find container \"f34160be023c9b3a7260d6fefb9db064a552ef2d13b01e86d44aca5de303d56d\": container with ID starting with f34160be023c9b3a7260d6fefb9db064a552ef2d13b01e86d44aca5de303d56d not found: ID does not exist"
Jul 27 21:45:01.786565 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:01.786540    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-5f50cb-openshift-default-5fcbf9d69b-tts4q"]
Jul 27 21:45:01.790978 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:01.790956    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-5f50cb-openshift-default-5fcbf9d69b-tts4q"]
Jul 27 21:45:01.847906 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:01.847871    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="00e24a4b-9175-46f4-b80e-42d8694750cb" path="/var/lib/kubelet/pods/00e24a4b-9175-46f4-b80e-42d8694750cb/volumes"
Jul 27 21:45:02.843704 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:02.843668    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-cd8kf_b88c26a8-d71c-460d-a2db-386767155bba/manager/0.log"
Jul 27 21:45:02.975905 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:02.975874    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-bgxv4_83957a08-9031-4434-a11d-7dc22b82220c/manager/0.log"
Jul 27 21:45:03.396299 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:03.396270    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-8f6xg_a5bf166f-b209-416f-ae7e-98f18db1c2e5/manager/0.log"
Jul 27 21:45:03.528698 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:03.528671    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5c6d5848fb-b8zrz_164f878d-db0a-4fa2-aed1-0ae818bee556/copy-manifests/0.log"
Jul 27 21:45:03.542211 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:03.542172    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5c6d5848fb-b8zrz_164f878d-db0a-4fa2-aed1-0ae818bee556/manager/0.log"
Jul 27 21:45:04.346922 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:04.346892    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29753115-t8g6v_c8e679a9-15f2-4f34-81f6-399734e7a830/cleanup/0.log"
Jul 27 21:45:04.470744 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:04.470715    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29753130-54ln4_a88707a8-fd98-4349-b1e6-d3fabc7c5c78/cleanup/0.log"
Jul 27 21:45:06.392791 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:06.392755    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-nm6pr_7df103ae-114b-4595-b00c-e1d2b44d882d/manager/0.log"
Jul 27 21:45:06.905774 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:06.905745    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-7znrn_bde6ee24-561a-417d-b2f8-1f24d1c55733/limitador/0.log"
Jul 27 21:45:07.035737 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:07.035703    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-zk927_8c076ad1-452f-4f41-9c2c-7684990f388f/manager/0.log"
Jul 27 21:45:07.534419 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:07.534387    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-bch22_235c0fce-11cf-42aa-93c8-74a39b3348b4/discovery/0.log"
Jul 27 21:45:07.786376 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:07.786272    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6598f7b856-nsqwk_40340708-0cf9-4b9c-9331-6155dbee39b3/kube-auth-proxy/0.log"
Jul 27 21:45:08.674753 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:08.674720    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-m5tt9_3552283d-6dcb-4cd2-a850-621c4b698567/main/0.log"
Jul 27 21:45:08.807128 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:08.807097    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-jszvk_759ce03b-f36f-48a3-a497-2b2dd4ab5f4e/main/0.log"
Jul 27 21:45:18.245522 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:18.245438    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-dtrjg_4b924e42-83bc-459f-b1a2-f115f9bce63f/global-pull-secret-syncer/0.log"
Jul 27 21:45:18.476870 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:18.476831    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-sg57x_9a766ad8-ce17-462e-b45f-867a6c6c44e4/konnectivity-agent/0.log"
Jul 27 21:45:18.548576 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:18.548533    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-64.ec2.internal_5fa776c99382db40a7e9c25fe5c95038/haproxy/0.log"
Jul 27 21:45:24.129987 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:24.129955    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-nm6pr_7df103ae-114b-4595-b00c-e1d2b44d882d/manager/0.log"
Jul 27 21:45:24.414935 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:24.414847    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-7znrn_bde6ee24-561a-417d-b2f8-1f24d1c55733/limitador/0.log"
Jul 27 21:45:24.489563 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:24.489531    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-zk927_8c076ad1-452f-4f41-9c2c-7684990f388f/manager/0.log"
Jul 27 21:45:26.574265 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:26.574233    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mj6h5_3e1e5615-3312-4080-bbfa-2a92c24bb888/node-exporter/0.log"
Jul 27 21:45:26.601586 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:26.601557    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mj6h5_3e1e5615-3312-4080-bbfa-2a92c24bb888/kube-rbac-proxy/0.log"
Jul 27 21:45:26.622096 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:26.622071    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mj6h5_3e1e5615-3312-4080-bbfa-2a92c24bb888/init-textfile/0.log"
Jul 27 21:45:27.074835 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:27.074808    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7444f7bc55-qltlp_39155db8-f2d3-4a9d-8ef8-ed19f16b62d8/thanos-query/0.log"
Jul 27 21:45:27.097494 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:27.097473    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7444f7bc55-qltlp_39155db8-f2d3-4a9d-8ef8-ed19f16b62d8/kube-rbac-proxy-web/0.log"
Jul 27 21:45:27.121276 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:27.121249    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7444f7bc55-qltlp_39155db8-f2d3-4a9d-8ef8-ed19f16b62d8/kube-rbac-proxy/0.log"
Jul 27 21:45:27.143411 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:27.143384    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7444f7bc55-qltlp_39155db8-f2d3-4a9d-8ef8-ed19f16b62d8/prom-label-proxy/0.log"
Jul 27 21:45:27.167470 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:27.167446    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7444f7bc55-qltlp_39155db8-f2d3-4a9d-8ef8-ed19f16b62d8/kube-rbac-proxy-rules/0.log"
Jul 27 21:45:27.188812 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:27.188795    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7444f7bc55-qltlp_39155db8-f2d3-4a9d-8ef8-ed19f16b62d8/kube-rbac-proxy-metrics/0.log"
Jul 27 21:45:27.985170 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:27.985138    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-wd6n7/perf-node-gather-daemonset-sw68b"]
Jul 27 21:45:27.985551 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:27.985461    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ed4782e-5f6a-4b17-8db3-34cf40061089" containerName="revoke-keys"
Jul 27 21:45:27.985551 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:27.985471    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ed4782e-5f6a-4b17-8db3-34cf40061089" containerName="revoke-keys"
Jul 27 21:45:27.985551 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:27.985483    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00e24a4b-9175-46f4-b80e-42d8694750cb" containerName="istio-proxy"
Jul 27 21:45:27.985551 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:27.985489    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="00e24a4b-9175-46f4-b80e-42d8694750cb" containerName="istio-proxy"
Jul 27 21:45:27.985551 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:27.985500    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b059bf6-3c81-41ed-b200-965fa935795f" containerName="istio-proxy"
Jul 27 21:45:27.985551 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:27.985505    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b059bf6-3c81-41ed-b200-965fa935795f" containerName="istio-proxy"
Jul 27 21:45:27.985551 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:27.985512    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="911cf451-bf1d-4c8b-a240-288b2cd55fc1" containerName="istio-proxy"
Jul 27 21:45:27.985551 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:27.985518    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="911cf451-bf1d-4c8b-a240-288b2cd55fc1" containerName="istio-proxy"
Jul 27 21:45:27.985551 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:27.985524    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00e24a4b-9175-46f4-b80e-42d8694750cb" containerName="istio-proxy"
Jul 27 21:45:27.985551 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:27.985530    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="00e24a4b-9175-46f4-b80e-42d8694750cb" containerName="istio-proxy"
Jul 27 21:45:27.985551 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:27.985536    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="911cf451-bf1d-4c8b-a240-288b2cd55fc1" containerName="istio-proxy"
Jul 27 21:45:27.985551 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:27.985541    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="911cf451-bf1d-4c8b-a240-288b2cd55fc1" containerName="istio-proxy"
Jul 27 21:45:27.985551 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:27.985553    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b059bf6-3c81-41ed-b200-965fa935795f" containerName="istio-proxy"
Jul 27 21:45:27.985551 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:27.985558    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b059bf6-3c81-41ed-b200-965fa935795f" containerName="istio-proxy"
Jul 27 21:45:27.985979 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:27.985566    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0fc339fb-bbea-4776-b172-fc631a1dce80" containerName="istio-proxy"
Jul 27 21:45:27.985979 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:27.985571    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="0fc339fb-bbea-4776-b172-fc631a1dce80" containerName="istio-proxy"
Jul 27 21:45:27.985979 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:27.985576    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0fc339fb-bbea-4776-b172-fc631a1dce80" containerName="istio-proxy"
Jul 27 21:45:27.985979 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:27.985582    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="0fc339fb-bbea-4776-b172-fc631a1dce80" containerName="istio-proxy"
Jul 27 21:45:27.985979 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:27.985631    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="0fc339fb-bbea-4776-b172-fc631a1dce80" containerName="istio-proxy"
Jul 27 21:45:27.985979 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:27.985638    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="911cf451-bf1d-4c8b-a240-288b2cd55fc1" containerName="istio-proxy"
Jul 27 21:45:27.985979 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:27.985645    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="00e24a4b-9175-46f4-b80e-42d8694750cb" containerName="istio-proxy"
Jul 27 21:45:27.985979 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:27.985649    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b059bf6-3c81-41ed-b200-965fa935795f" containerName="istio-proxy"
Jul 27 21:45:27.985979 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:27.985654    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ed4782e-5f6a-4b17-8db3-34cf40061089" containerName="revoke-keys"
Jul 27 21:45:27.985979 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:27.985659    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="0fc339fb-bbea-4776-b172-fc631a1dce80" containerName="istio-proxy"
Jul 27 21:45:27.985979 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:27.985663    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="911cf451-bf1d-4c8b-a240-288b2cd55fc1" containerName="istio-proxy"
Jul 27 21:45:27.985979 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:27.985669    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="00e24a4b-9175-46f4-b80e-42d8694750cb" containerName="istio-proxy"
Jul 27 21:45:27.989761 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:27.989745    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wd6n7/perf-node-gather-daemonset-sw68b"
Jul 27 21:45:27.992395 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:27.992377    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wd6n7\"/\"kube-root-ca.crt\""
Jul 27 21:45:27.993372 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:27.993341    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wd6n7\"/\"openshift-service-ca.crt\""
Jul 27 21:45:27.993514 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:27.993390    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-wd6n7\"/\"default-dockercfg-pmb4h\""
Jul 27 21:45:27.995952 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:27.995930    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wd6n7/perf-node-gather-daemonset-sw68b"]
Jul 27 21:45:28.059286 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:28.059244    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/36f42b0d-db8d-40b7-9d4a-d44435a846e8-proc\") pod \"perf-node-gather-daemonset-sw68b\" (UID: \"36f42b0d-db8d-40b7-9d4a-d44435a846e8\") " pod="openshift-must-gather-wd6n7/perf-node-gather-daemonset-sw68b"
Jul 27 21:45:28.059459 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:28.059304    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/36f42b0d-db8d-40b7-9d4a-d44435a846e8-podres\") pod \"perf-node-gather-daemonset-sw68b\" (UID: \"36f42b0d-db8d-40b7-9d4a-d44435a846e8\") " pod="openshift-must-gather-wd6n7/perf-node-gather-daemonset-sw68b"
Jul 27 21:45:28.059459 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:28.059361    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/36f42b0d-db8d-40b7-9d4a-d44435a846e8-lib-modules\") pod \"perf-node-gather-daemonset-sw68b\" (UID: \"36f42b0d-db8d-40b7-9d4a-d44435a846e8\") " pod="openshift-must-gather-wd6n7/perf-node-gather-daemonset-sw68b"
Jul 27 21:45:28.059459 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:28.059414    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/36f42b0d-db8d-40b7-9d4a-d44435a846e8-sys\") pod \"perf-node-gather-daemonset-sw68b\" (UID: \"36f42b0d-db8d-40b7-9d4a-d44435a846e8\") " pod="openshift-must-gather-wd6n7/perf-node-gather-daemonset-sw68b"
Jul 27 21:45:28.059459 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:28.059430    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5fkw\" (UniqueName: \"kubernetes.io/projected/36f42b0d-db8d-40b7-9d4a-d44435a846e8-kube-api-access-g5fkw\") pod \"perf-node-gather-daemonset-sw68b\" (UID: \"36f42b0d-db8d-40b7-9d4a-d44435a846e8\") " pod="openshift-must-gather-wd6n7/perf-node-gather-daemonset-sw68b"
Jul 27 21:45:28.159882 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:28.159839    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/36f42b0d-db8d-40b7-9d4a-d44435a846e8-podres\") pod \"perf-node-gather-daemonset-sw68b\" (UID: \"36f42b0d-db8d-40b7-9d4a-d44435a846e8\") " pod="openshift-must-gather-wd6n7/perf-node-gather-daemonset-sw68b"
Jul 27 21:45:28.159882 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:28.159885    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/36f42b0d-db8d-40b7-9d4a-d44435a846e8-lib-modules\") pod \"perf-node-gather-daemonset-sw68b\" (UID: \"36f42b0d-db8d-40b7-9d4a-d44435a846e8\") " pod="openshift-must-gather-wd6n7/perf-node-gather-daemonset-sw68b"
Jul 27 21:45:28.160155 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:28.159921    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/36f42b0d-db8d-40b7-9d4a-d44435a846e8-sys\") pod \"perf-node-gather-daemonset-sw68b\" (UID: \"36f42b0d-db8d-40b7-9d4a-d44435a846e8\") " pod="openshift-must-gather-wd6n7/perf-node-gather-daemonset-sw68b"
Jul 27 21:45:28.160155 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:28.159938    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5fkw\" (UniqueName: \"kubernetes.io/projected/36f42b0d-db8d-40b7-9d4a-d44435a846e8-kube-api-access-g5fkw\") pod \"perf-node-gather-daemonset-sw68b\" (UID: \"36f42b0d-db8d-40b7-9d4a-d44435a846e8\") " pod="openshift-must-gather-wd6n7/perf-node-gather-daemonset-sw68b"
Jul 27 21:45:28.160155 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:28.159956    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/36f42b0d-db8d-40b7-9d4a-d44435a846e8-proc\") pod \"perf-node-gather-daemonset-sw68b\" (UID: \"36f42b0d-db8d-40b7-9d4a-d44435a846e8\") " pod="openshift-must-gather-wd6n7/perf-node-gather-daemonset-sw68b"
Jul 27 21:45:28.160155 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:28.160025    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/36f42b0d-db8d-40b7-9d4a-d44435a846e8-sys\") pod \"perf-node-gather-daemonset-sw68b\" (UID: \"36f42b0d-db8d-40b7-9d4a-d44435a846e8\") " pod="openshift-must-gather-wd6n7/perf-node-gather-daemonset-sw68b"
Jul 27 21:45:28.160155 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:28.160026    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/36f42b0d-db8d-40b7-9d4a-d44435a846e8-podres\") pod \"perf-node-gather-daemonset-sw68b\" (UID: \"36f42b0d-db8d-40b7-9d4a-d44435a846e8\") " pod="openshift-must-gather-wd6n7/perf-node-gather-daemonset-sw68b"
Jul 27 21:45:28.160155 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:28.160082    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/36f42b0d-db8d-40b7-9d4a-d44435a846e8-lib-modules\") pod \"perf-node-gather-daemonset-sw68b\" (UID: \"36f42b0d-db8d-40b7-9d4a-d44435a846e8\") " pod="openshift-must-gather-wd6n7/perf-node-gather-daemonset-sw68b"
Jul 27 21:45:28.160363 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:28.160157    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/36f42b0d-db8d-40b7-9d4a-d44435a846e8-proc\") pod \"perf-node-gather-daemonset-sw68b\" (UID: \"36f42b0d-db8d-40b7-9d4a-d44435a846e8\") " pod="openshift-must-gather-wd6n7/perf-node-gather-daemonset-sw68b"
Jul 27 21:45:28.168536 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:28.168509    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5fkw\" (UniqueName: \"kubernetes.io/projected/36f42b0d-db8d-40b7-9d4a-d44435a846e8-kube-api-access-g5fkw\") pod \"perf-node-gather-daemonset-sw68b\" (UID: \"36f42b0d-db8d-40b7-9d4a-d44435a846e8\") " pod="openshift-must-gather-wd6n7/perf-node-gather-daemonset-sw68b"
Jul 27 21:45:28.300733 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:28.300710    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wd6n7/perf-node-gather-daemonset-sw68b"
Jul 27 21:45:28.436304 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:28.436268    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wd6n7/perf-node-gather-daemonset-sw68b"]
Jul 27 21:45:28.439477 ip-10-0-141-64 kubenswrapper[2560]: W0727 21:45:28.439443    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod36f42b0d_db8d_40b7_9d4a_d44435a846e8.slice/crio-d2f0766daf72596e1be44403b9e6c342f8d4430d00d5bf7653e2e0d6acc1e8d3 WatchSource:0}: Error finding container d2f0766daf72596e1be44403b9e6c342f8d4430d00d5bf7653e2e0d6acc1e8d3: Status 404 returned error can't find the container with id d2f0766daf72596e1be44403b9e6c342f8d4430d00d5bf7653e2e0d6acc1e8d3
Jul 27 21:45:28.896195 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:28.896153    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wd6n7/perf-node-gather-daemonset-sw68b" event={"ID":"36f42b0d-db8d-40b7-9d4a-d44435a846e8","Type":"ContainerStarted","Data":"8ef8a027bcaccb23023c15ab2fd5f77195e6c9bf1decb1ff174c2e05c7e6855f"}
Jul 27 21:45:28.896374 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:28.896218    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wd6n7/perf-node-gather-daemonset-sw68b" event={"ID":"36f42b0d-db8d-40b7-9d4a-d44435a846e8","Type":"ContainerStarted","Data":"d2f0766daf72596e1be44403b9e6c342f8d4430d00d5bf7653e2e0d6acc1e8d3"}
Jul 27 21:45:28.896374 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:28.896245    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-wd6n7/perf-node-gather-daemonset-sw68b"
Jul 27 21:45:28.921500 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:28.921450    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-wd6n7/perf-node-gather-daemonset-sw68b" podStartSLOduration=1.921435443 podStartE2EDuration="1.921435443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:45:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:45:28.920112699 +0000 UTC m=+3245.671974516" watchObservedRunningTime="2026-07-27 21:45:28.921435443 +0000 UTC m=+3245.673297279"
Jul 27 21:45:29.027442 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:29.027411    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-z6zxv_8a6b7ae6-6c21-453d-99b9-303066d19915/console-operator/1.log"
Jul 27 21:45:29.032436 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:29.032400    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-z6zxv_8a6b7ae6-6c21-453d-99b9-303066d19915/console-operator/2.log"
Jul 27 21:45:29.546481 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:29.546454    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-95d667bb8-sxpj4_d62a4442-c805-40e3-919f-4bf0f1f4dd47/console/0.log"
Jul 27 21:45:31.308369 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:31.308336    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-ds6vw_a9187e8b-eb38-4984-b284-39419b5de51b/dns/0.log"
Jul 27 21:45:31.330417 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:31.330387    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-ds6vw_a9187e8b-eb38-4984-b284-39419b5de51b/kube-rbac-proxy/0.log"
Jul 27 21:45:31.404530 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:31.404497    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-g5tm9_95f7e24a-fe2f-451e-ae08-fa9342e04cef/dns-node-resolver/0.log"
Jul 27 21:45:31.960200 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:31.960173    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-jm574_412ea27a-182d-40cc-917e-4686a4ff65cb/node-ca/0.log"
Jul 27 21:45:33.072543 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:33.072504    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-bch22_235c0fce-11cf-42aa-93c8-74a39b3348b4/discovery/0.log"
Jul 27 21:45:33.125643 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:33.125617    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6598f7b856-nsqwk_40340708-0cf9-4b9c-9331-6155dbee39b3/kube-auth-proxy/0.log"
Jul 27 21:45:34.071057 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:34.071029    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-blb74_ef72a491-3d46-4039-9c8b-a6a9eb853223/serve-healthcheck-canary/0.log"
Jul 27 21:45:34.805658 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:34.805627    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mb7xd_1c5f5fd5-7bf8-4d82-9c39-ca9e923ec6b4/kube-rbac-proxy/0.log"
Jul 27 21:45:34.831450 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:34.831420    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mb7xd_1c5f5fd5-7bf8-4d82-9c39-ca9e923ec6b4/exporter/0.log"
Jul 27 21:45:34.854097 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:34.854064    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mb7xd_1c5f5fd5-7bf8-4d82-9c39-ca9e923ec6b4/extractor/0.log"
Jul 27 21:45:34.911124 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:34.911071    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-wd6n7/perf-node-gather-daemonset-sw68b"
Jul 27 21:45:37.006105 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:37.006063    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-cd8kf_b88c26a8-d71c-460d-a2db-386767155bba/manager/0.log"
Jul 27 21:45:37.196955 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:37.196923    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-bgxv4_83957a08-9031-4434-a11d-7dc22b82220c/manager/0.log"
Jul 27 21:45:37.428108 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:37.428069    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-8f6xg_a5bf166f-b209-416f-ae7e-98f18db1c2e5/manager/0.log"
Jul 27 21:45:37.502030 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:37.502003    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5c6d5848fb-b8zrz_164f878d-db0a-4fa2-aed1-0ae818bee556/manager/0.log"
Jul 27 21:45:37.519440 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:37.519417    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5c6d5848fb-b8zrz_164f878d-db0a-4fa2-aed1-0ae818bee556/copy-manifests/0.log"
Jul 27 21:45:38.887667 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:38.887632    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-bch22_235c0fce-11cf-42aa-93c8-74a39b3348b4/discovery/0.log"
Jul 27 21:45:38.921924 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:38.921892    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6598f7b856-nsqwk_40340708-0cf9-4b9c-9331-6155dbee39b3/kube-auth-proxy/0.log"
Jul 27 21:45:40.567985 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:40.567952    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-bch22_235c0fce-11cf-42aa-93c8-74a39b3348b4/discovery/0.log"
Jul 27 21:45:40.596771 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:40.596734    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6598f7b856-nsqwk_40340708-0cf9-4b9c-9331-6155dbee39b3/kube-auth-proxy/0.log"
Jul 27 21:45:44.548255 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:44.548222    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-7vsbq_1da96bd6-098f-41d7-9be9-904d53068b43/kube-storage-version-migrator-operator/1.log"
Jul 27 21:45:44.549452 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:44.549422    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-7vsbq_1da96bd6-098f-41d7-9be9-904d53068b43/kube-storage-version-migrator-operator/0.log"
Jul 27 21:45:45.607930 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:45.607898    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-djcnj_eae6d6c0-9844-45a8-a5ef-9c6f1ecc2810/kube-multus-additional-cni-plugins/0.log"
Jul 27 21:45:45.630260 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:45.630231    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-djcnj_eae6d6c0-9844-45a8-a5ef-9c6f1ecc2810/egress-router-binary-copy/0.log"
Jul 27 21:45:45.651051 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:45.651026    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-djcnj_eae6d6c0-9844-45a8-a5ef-9c6f1ecc2810/cni-plugins/0.log"
Jul 27 21:45:45.671850 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:45.671827    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-djcnj_eae6d6c0-9844-45a8-a5ef-9c6f1ecc2810/bond-cni-plugin/0.log"
Jul 27 21:45:45.692721 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:45.692700    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-djcnj_eae6d6c0-9844-45a8-a5ef-9c6f1ecc2810/routeoverride-cni/0.log"
Jul 27 21:45:45.715111 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:45.715066    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-djcnj_eae6d6c0-9844-45a8-a5ef-9c6f1ecc2810/whereabouts-cni-bincopy/0.log"
Jul 27 21:45:45.737638 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:45.737605    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-djcnj_eae6d6c0-9844-45a8-a5ef-9c6f1ecc2810/whereabouts-cni/0.log"
Jul 27 21:45:46.168865 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:46.168830    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-td78h_dbae1ee6-3616-432d-8ef9-255ad4cd444e/kube-multus/0.log"
Jul 27 21:45:46.243821 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:46.243786    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-r6g8d_fac6ee6e-225a-4402-8834-69ff744b07f9/network-metrics-daemon/0.log"
Jul 27 21:45:46.266155 ip-10-0-141-64 kubenswrapper[2560]: I0727 21:45:46.266125    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-r6g8d_fac6ee6e-225a-4402-8834-69ff744b07f9/kube-rbac-proxy/0.log"