Jul 09 20:01:16.611499 ip-10-0-140-63 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 09 20:01:16.611508 ip-10-0-140-63 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 09 20:01:16.611516 ip-10-0-140-63 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 09 20:01:16.611839 ip-10-0-140-63 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 09 20:01:26.707445 ip-10-0-140-63 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 09 20:01:26.707463 ip-10-0-140-63 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 3fb440e76cb5479c88659cb0c11edf33 --
Jul 09 20:03:29.446659 ip-10-0-140-63 systemd[1]: Starting Kubernetes Kubelet...
Jul 09 20:03:29.954116 ip-10-0-140-63 kubenswrapper[2562]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 09 20:03:29.954116 ip-10-0-140-63 kubenswrapper[2562]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 09 20:03:29.954116 ip-10-0-140-63 kubenswrapper[2562]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 09 20:03:29.954116 ip-10-0-140-63 kubenswrapper[2562]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 09 20:03:29.954116 ip-10-0-140-63 kubenswrapper[2562]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 09 20:03:29.954116 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.908851    2562 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 09 20:03:29.954116 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.912725    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 20:03:29.954116 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.912738    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 20:03:29.954116 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.912742    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 20:03:29.954116 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.912747    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 20:03:29.954116 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.912750    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 20:03:29.954116 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.912753    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 20:03:29.954116 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.912756    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 20:03:29.954116 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.912759    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 20:03:29.955661 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.912765    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 20:03:29.955661 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.912769    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 20:03:29.955661 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.912771    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 20:03:29.955661 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.912774    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 20:03:29.955661 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.912777    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 20:03:29.955661 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.912780    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 20:03:29.955661 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.912783    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 20:03:29.955661 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.912786    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 20:03:29.955661 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.912789    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 20:03:29.955661 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.912791    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 20:03:29.955661 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.912795    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 20:03:29.955661 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.912798    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 20:03:29.955661 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.912801    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 20:03:29.955661 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.912803    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 20:03:29.955661 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.912806    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 20:03:29.955661 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.912810    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 20:03:29.955661 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.912813    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 20:03:29.955661 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.912816    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 20:03:29.955661 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.912819    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 20:03:29.955661 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.912822    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 20:03:29.956524 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.912825    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 20:03:29.956524 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.912828    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 20:03:29.956524 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.912831    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 20:03:29.956524 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.912834    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 20:03:29.956524 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.912838    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 20:03:29.956524 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.912841    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 20:03:29.956524 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.912844    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 20:03:29.956524 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.912848    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 20:03:29.956524 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.912851    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 20:03:29.956524 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.912854    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 20:03:29.956524 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.912857    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 20:03:29.956524 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.912860    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 20:03:29.956524 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.912864    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 20:03:29.956524 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.912867    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 20:03:29.956524 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.912870    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 20:03:29.956524 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.912873    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 20:03:29.956524 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.912876    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 20:03:29.956524 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.912879    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 20:03:29.956524 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.912881    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 20:03:29.956524 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.912884    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 20:03:29.957146 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.912887    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 20:03:29.957146 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.912890    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 20:03:29.957146 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.912892    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 20:03:29.957146 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.912928    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 20:03:29.957146 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.912933    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 20:03:29.957146 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.912937    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 20:03:29.957146 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.912940    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 20:03:29.957146 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.912944    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 20:03:29.957146 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.912947    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 20:03:29.957146 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.912950    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 20:03:29.957146 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.912953    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 20:03:29.957146 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.912957    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 20:03:29.957146 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.912960    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 20:03:29.957146 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.912963    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 20:03:29.957146 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.912966    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 20:03:29.957146 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.912969    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 20:03:29.957146 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.912972    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 20:03:29.957146 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.912975    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 20:03:29.957146 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.912978    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 20:03:29.957146 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.912981    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 20:03:29.957826 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.912984    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 20:03:29.957826 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.912987    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 20:03:29.957826 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.912991    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 20:03:29.957826 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.912994    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 20:03:29.957826 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.912997    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 20:03:29.957826 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.913000    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 20:03:29.957826 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.913003    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 20:03:29.957826 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.913006    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 20:03:29.957826 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.913009    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 20:03:29.957826 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.913014    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 20:03:29.957826 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.913018    2562 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 20:03:29.957826 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.913022    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 20:03:29.957826 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.913025    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 20:03:29.957826 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.913028    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 20:03:29.957826 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.913030    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 20:03:29.957826 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.913033    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 20:03:29.957826 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.913036    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 20:03:29.957826 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.913039    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 20:03:29.957826 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.913042    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 20:03:29.958494 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.913045    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 20:03:29.958494 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.913047    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 20:03:29.958494 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.913051    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 20:03:29.958494 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.913065    2562 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 20:03:29.958494 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.913071    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 20:03:29.958494 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.913075    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 20:03:29.958494 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.913078    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 20:03:29.958494 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.913082    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 20:03:29.958494 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.913085    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 20:03:29.958494 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.913088    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 20:03:29.958494 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.913091    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 20:03:29.958494 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.913094    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 20:03:29.958494 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.913903    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 20:03:29.958494 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.913912    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 20:03:29.958494 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.913915    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 20:03:29.958494 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.913918    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 20:03:29.958494 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.913923    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 20:03:29.958494 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.913927    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 20:03:29.958494 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.913930    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 20:03:29.959210 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.913934    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 20:03:29.959210 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.913936    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 20:03:29.959210 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.913940    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 20:03:29.959210 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.913943    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 20:03:29.959210 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.913945    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 20:03:29.959210 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.913948    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 20:03:29.959210 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.913951    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 20:03:29.959210 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.913954    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 20:03:29.959210 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.913957    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 20:03:29.959210 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.913960    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 20:03:29.959210 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.913962    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 20:03:29.959210 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.913966    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 20:03:29.959210 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.913968    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 20:03:29.959210 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.913971    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 20:03:29.959210 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.913974    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 20:03:29.959210 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.913977    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 20:03:29.959210 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.913980    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 20:03:29.959210 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.913982    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 20:03:29.959210 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.913985    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 20:03:29.959210 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.913988    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 20:03:29.959937 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.913991    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 20:03:29.959937 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.913993    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 20:03:29.959937 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.913996    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 20:03:29.959937 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.913999    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 20:03:29.959937 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914002    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 20:03:29.959937 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914005    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 20:03:29.959937 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914009    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 20:03:29.959937 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914013    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 20:03:29.959937 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914016    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 20:03:29.959937 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914024    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 20:03:29.959937 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914027    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 20:03:29.959937 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914030    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 20:03:29.959937 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914033    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 20:03:29.959937 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914041    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 20:03:29.959937 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914044    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 20:03:29.959937 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914048    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 20:03:29.959937 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914052    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 20:03:29.959937 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914057    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 20:03:29.959937 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914060    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 20:03:29.959937 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914062    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 20:03:29.960587 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914065    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 20:03:29.960587 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914068    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 20:03:29.960587 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914071    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 20:03:29.960587 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914074    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 20:03:29.960587 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914076    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 20:03:29.960587 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914080    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 20:03:29.960587 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914084    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 20:03:29.960587 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914087    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 20:03:29.960587 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914090    2562 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 20:03:29.960587 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914093    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 20:03:29.960587 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914096    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 20:03:29.960587 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914099    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 20:03:29.960587 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914102    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 20:03:29.960587 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914105    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 20:03:29.960587 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914107    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 20:03:29.960587 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914110    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 20:03:29.960587 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914113    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 20:03:29.960587 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914116    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 20:03:29.960587 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914119    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 20:03:29.961166 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914121    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 20:03:29.961166 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914124    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 20:03:29.961166 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914129    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 20:03:29.961166 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914132    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 20:03:29.961166 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914135    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 20:03:29.961166 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914138    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 20:03:29.961166 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914141    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 20:03:29.961166 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914144    2562 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 20:03:29.961166 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914147    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 20:03:29.961166 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914150    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 20:03:29.961166 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914152    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 20:03:29.961166 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914158    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 20:03:29.961166 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914162    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 20:03:29.961166 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914165    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 20:03:29.961166 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914168    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 20:03:29.961166 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914172    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 20:03:29.961166 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914176    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 20:03:29.961166 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914179    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 20:03:29.961166 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914182    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 20:03:29.961800 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914185    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 20:03:29.961800 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914187    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 20:03:29.961800 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914190    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 20:03:29.961800 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914193    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 20:03:29.961800 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914196    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 20:03:29.961800 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914199    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 20:03:29.961800 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914202    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 20:03:29.961800 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914205    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 20:03:29.961800 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914208    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 20:03:29.961800 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914211    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 20:03:29.961800 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914213    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 20:03:29.961800 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914216    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 20:03:29.961800 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914219    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 20:03:29.961800 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914222    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 20:03:29.961800 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914633    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 20:03:29.961800 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914640    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 20:03:29.961800 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914644    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 20:03:29.961800 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914647    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 20:03:29.961800 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914650    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 20:03:29.961800 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914653    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 20:03:29.962369 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914657    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 20:03:29.962369 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914661    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 20:03:29.962369 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914665    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 20:03:29.962369 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914668    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 20:03:29.962369 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914672    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 20:03:29.962369 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914675    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 20:03:29.962369 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914678    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 20:03:29.962369 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914681    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 20:03:29.962369 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914684    2562 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 20:03:29.962369 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914688    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 20:03:29.962369 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914691    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 20:03:29.962369 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914694    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 20:03:29.962369 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914697    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 20:03:29.962369 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914700    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 20:03:29.962369 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914704    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 20:03:29.962369 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914706    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 20:03:29.962369 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914709    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 20:03:29.962369 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914712    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 20:03:29.962369 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914716    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 20:03:29.962369 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914718    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 20:03:29.962997 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914721    2562 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 20:03:29.962997 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914724    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 20:03:29.962997 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914727    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 20:03:29.962997 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914729    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 20:03:29.962997 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914732    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 20:03:29.962997 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914735    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 20:03:29.962997 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914737    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 20:03:29.962997 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914742    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 20:03:29.962997 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914745    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 20:03:29.962997 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914748    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 20:03:29.962997 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914750    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 20:03:29.962997 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914755    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 20:03:29.962997 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914758    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 20:03:29.962997 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914764    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 20:03:29.962997 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914766    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 20:03:29.962997 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914770    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 20:03:29.962997 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914773    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 20:03:29.962997 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914775    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 20:03:29.962997 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914778    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 20:03:29.963540 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914781    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 20:03:29.963540 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914783    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 20:03:29.963540 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914786    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 20:03:29.963540 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914788    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 20:03:29.963540 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914791    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 20:03:29.963540 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914794    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 20:03:29.963540 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914796    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 20:03:29.963540 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914799    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 20:03:29.963540 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914802    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 20:03:29.963540 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914805    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 20:03:29.963540 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914807    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 20:03:29.963540 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914810    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 20:03:29.963540 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914813    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 20:03:29.963540 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914816    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 20:03:29.963540 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914819    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 20:03:29.963540 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914821    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 20:03:29.963540 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914824    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 20:03:29.963540 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914827    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 20:03:29.963540 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914829    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 20:03:29.963540 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914832    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 20:03:29.964130 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914836    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 20:03:29.964130 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914839    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 20:03:29.964130 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914842    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 20:03:29.964130 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914845    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 20:03:29.964130 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914847    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 20:03:29.964130 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914850    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 20:03:29.964130 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914854    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 20:03:29.964130 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914857    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 20:03:29.964130 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914860    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 20:03:29.964130 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914864    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 20:03:29.964130 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914868    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 20:03:29.964130 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914871    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 20:03:29.964130 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914875    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 20:03:29.964130 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914878    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 20:03:29.964130 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914881    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 20:03:29.964130 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914884    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 20:03:29.964130 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914886    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 20:03:29.964130 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914889    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 20:03:29.964130 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914892    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 20:03:29.964682 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914895    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 20:03:29.964682 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914898    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 20:03:29.964682 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914901    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 20:03:29.964682 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914903    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 20:03:29.964682 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914906    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 20:03:29.964682 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914909    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 20:03:29.964682 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914911    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 20:03:29.964682 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914914    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 20:03:29.964682 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914917    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 20:03:29.964682 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914919    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 20:03:29.964682 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914922    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 20:03:29.964682 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914925    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 20:03:29.964682 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914927    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 20:03:29.964682 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914932    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 20:03:29.964682 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.914935    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 20:03:29.964682 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915012    2562 flags.go:64] FLAG: --address="0.0.0.0"
Jul 09 20:03:29.964682 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915019    2562 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 09 20:03:29.964682 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915027    2562 flags.go:64] FLAG: --anonymous-auth="true"
Jul 09 20:03:29.964682 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915032    2562 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 09 20:03:29.964682 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915039    2562 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 09 20:03:29.964682 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915043    2562 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 09 20:03:29.965308 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915048    2562 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 09 20:03:29.965308 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915053    2562 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 09 20:03:29.965308 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915057    2562 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 09 20:03:29.965308 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915060    2562 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 09 20:03:29.965308 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915064    2562 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 09 20:03:29.965308 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915068    2562 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 09 20:03:29.965308 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915071    2562 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 09 20:03:29.965308 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915074    2562 flags.go:64] FLAG: --cgroup-root=""
Jul 09 20:03:29.965308 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915078    2562 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 09 20:03:29.965308 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915081    2562 flags.go:64] FLAG: --client-ca-file=""
Jul 09 20:03:29.965308 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915084    2562 flags.go:64] FLAG: --cloud-provider="external"
Jul 09 20:03:29.965308 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915088    2562 flags.go:64] FLAG: --cluster-dns="[]"
Jul 09 20:03:29.965308 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915093    2562 flags.go:64] FLAG: --cluster-domain=""
Jul 09 20:03:29.965308 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915096    2562 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 09 20:03:29.965308 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915099    2562 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 09 20:03:29.965308 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915102    2562 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 09 20:03:29.965308 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915106    2562 flags.go:64] FLAG: --container-log-max-files="5"
Jul 09 20:03:29.965308 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915110    2562 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 09 20:03:29.965308 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915114    2562 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 09 20:03:29.965308 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915117    2562 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 09 20:03:29.965308 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915121    2562 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 09 20:03:29.965308 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915124    2562 flags.go:64] FLAG: --contention-profiling="false"
Jul 09 20:03:29.965308 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915127    2562 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 09 20:03:29.965308 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915131    2562 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 09 20:03:29.966138 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915135    2562 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 09 20:03:29.966138 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915139    2562 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 09 20:03:29.966138 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915143    2562 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 09 20:03:29.966138 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915146    2562 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 09 20:03:29.966138 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915150    2562 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 09 20:03:29.966138 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915153    2562 flags.go:64] FLAG: --enable-load-reader="false"
Jul 09 20:03:29.966138 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915158    2562 flags.go:64] FLAG: --enable-server="true"
Jul 09 20:03:29.966138 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915161    2562 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 09 20:03:29.966138 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915166    2562 flags.go:64] FLAG: --event-burst="100"
Jul 09 20:03:29.966138 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915170    2562 flags.go:64] FLAG: --event-qps="50"
Jul 09 20:03:29.966138 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915173    2562 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 09 20:03:29.966138 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915177    2562 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 09 20:03:29.966138 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915181    2562 flags.go:64] FLAG: --eviction-hard=""
Jul 09 20:03:29.966138 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915185    2562 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 09 20:03:29.966138 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915188    2562 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 09 20:03:29.966138 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915192    2562 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 09 20:03:29.966138 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915195    2562 flags.go:64] FLAG: --eviction-soft=""
Jul 09 20:03:29.966138 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915198    2562 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 09 20:03:29.966138 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915202    2562 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 09 20:03:29.966138 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915205    2562 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 09 20:03:29.966138 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915208    2562 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 09 20:03:29.966138 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915211    2562 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 09 20:03:29.966138 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915214    2562 flags.go:64] FLAG: --fail-swap-on="true"
Jul 09 20:03:29.966138 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915218    2562 flags.go:64] FLAG: --feature-gates=""
Jul 09 20:03:29.966138 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915222    2562 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 09 20:03:29.967332 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915225    2562 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 09 20:03:29.967332 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915229    2562 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 09 20:03:29.967332 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915240    2562 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 09 20:03:29.967332 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915244    2562 flags.go:64] FLAG: --healthz-port="10248"
Jul 09 20:03:29.967332 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915247    2562 flags.go:64] FLAG: --help="false"
Jul 09 20:03:29.967332 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915250    2562 flags.go:64] FLAG: --hostname-override="ip-10-0-140-63.ec2.internal"
Jul 09 20:03:29.967332 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915254    2562 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 09 20:03:29.967332 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915260    2562 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 09 20:03:29.967332 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915263    2562 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 09 20:03:29.967332 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915267    2562 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 09 20:03:29.967332 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915271    2562 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 09 20:03:29.967332 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915274    2562 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 09 20:03:29.967332 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915277    2562 flags.go:64] FLAG: --image-service-endpoint=""
Jul 09 20:03:29.967332 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915283    2562 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 09 20:03:29.967332 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915286    2562 flags.go:64] FLAG: --kube-api-burst="100"
Jul 09 20:03:29.967332 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915289    2562 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 09 20:03:29.967332 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915292    2562 flags.go:64] FLAG: --kube-api-qps="50"
Jul 09 20:03:29.967332 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915297    2562 flags.go:64] FLAG: --kube-reserved=""
Jul 09 20:03:29.967332 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915300    2562 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 09 20:03:29.967332 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915303    2562 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 09 20:03:29.967332 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915307    2562 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 09 20:03:29.967332 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915310    2562 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 09 20:03:29.967332 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915313    2562 flags.go:64] FLAG: --lock-file=""
Jul 09 20:03:29.967332 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915316    2562 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 09 20:03:29.970734 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915319    2562 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 09 20:03:29.970734 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915322    2562 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 09 20:03:29.970734 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915328    2562 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 09 20:03:29.970734 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915331    2562 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 09 20:03:29.970734 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915334    2562 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 09 20:03:29.970734 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915338    2562 flags.go:64] FLAG: --logging-format="text"
Jul 09 20:03:29.970734 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915341    2562 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 09 20:03:29.970734 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915344    2562 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 09 20:03:29.970734 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915347    2562 flags.go:64] FLAG: --manifest-url=""
Jul 09 20:03:29.970734 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915350    2562 flags.go:64] FLAG: --manifest-url-header=""
Jul 09 20:03:29.970734 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915355    2562 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 09 20:03:29.970734 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915359    2562 flags.go:64] FLAG: --max-open-files="1000000"
Jul 09 20:03:29.970734 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915369    2562 flags.go:64] FLAG: --max-pods="110"
Jul 09 20:03:29.970734 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915372    2562 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 09 20:03:29.970734 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915389    2562 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 09 20:03:29.970734 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915395    2562 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 09 20:03:29.970734 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915398    2562 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 09 20:03:29.970734 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915401    2562 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 09 20:03:29.970734 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915404    2562 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 09 20:03:29.970734 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915408    2562 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 09 20:03:29.970734 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915416    2562 flags.go:64] FLAG: --node-status-max-images="50"
Jul 09 20:03:29.970734 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915421    2562 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 09 20:03:29.970734 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915424    2562 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 09 20:03:29.970734 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915427    2562 flags.go:64] FLAG: --pod-cidr=""
Jul 09 20:03:29.971640 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915430    2562 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926"
Jul 09 20:03:29.971640 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915436    2562 flags.go:64] FLAG: --pod-manifest-path=""
Jul 09 20:03:29.971640 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915441    2562 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 09 20:03:29.971640 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915444    2562 flags.go:64] FLAG: --pods-per-core="0"
Jul 09 20:03:29.971640 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915448    2562 flags.go:64] FLAG: --port="10250"
Jul 09 20:03:29.971640 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915451    2562 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 09 20:03:29.971640 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915454    2562 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-071cc4f83bc9aea2d"
Jul 09 20:03:29.971640 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915458    2562 flags.go:64] FLAG: --qos-reserved=""
Jul 09 20:03:29.971640 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915461    2562 flags.go:64] FLAG: --read-only-port="10255"
Jul 09 20:03:29.971640 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915464    2562 flags.go:64] FLAG: --register-node="true"
Jul 09 20:03:29.971640 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915468    2562 flags.go:64] FLAG: --register-with-taints=""
Jul 09 20:03:29.971640 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915472    2562 flags.go:64] FLAG: --registry-burst="10"
Jul 09 20:03:29.971640 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915475    2562 flags.go:64] FLAG: --registry-qps="5"
Jul 09 20:03:29.971640 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915478    2562 flags.go:64] FLAG: --reserved-cpus=""
Jul 09 20:03:29.971640 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915481    2562 flags.go:64] FLAG: --reserved-memory=""
Jul 09 20:03:29.971640 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915485    2562 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 09 20:03:29.971640 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915489    2562 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 09 20:03:29.971640 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915492    2562 flags.go:64] FLAG: --rotate-certificates="false"
Jul 09 20:03:29.971640 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915495    2562 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 09 20:03:29.971640 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915498    2562 flags.go:64] FLAG: --runonce="false"
Jul 09 20:03:29.971640 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915501    2562 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 09 20:03:29.971640 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915504    2562 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 09 20:03:29.971640 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915508    2562 flags.go:64] FLAG: --seccomp-default="false"
Jul 09 20:03:29.971640 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915511    2562 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 09 20:03:29.971640 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915516    2562 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 09 20:03:30.060537 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915519    2562 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 09 20:03:30.060537 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915523    2562 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 09 20:03:30.060537 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915526    2562 flags.go:64] FLAG: --storage-driver-password="root"
Jul 09 20:03:30.060537 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915529    2562 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 09 20:03:30.060537 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915534    2562 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 09 20:03:30.060537 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915537    2562 flags.go:64] FLAG: --storage-driver-user="root"
Jul 09 20:03:30.060537 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915540    2562 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 09 20:03:30.060537 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915544    2562 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 09 20:03:30.060537 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915547    2562 flags.go:64] FLAG: --system-cgroups=""
Jul 09 20:03:30.060537 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915550    2562 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 09 20:03:30.060537 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915556    2562 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 09 20:03:30.060537 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915559    2562 flags.go:64] FLAG: --tls-cert-file=""
Jul 09 20:03:30.060537 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915563    2562 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 09 20:03:30.060537 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915567    2562 flags.go:64] FLAG: --tls-min-version=""
Jul 09 20:03:30.060537 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915570    2562 flags.go:64] FLAG: --tls-private-key-file=""
Jul 09 20:03:30.060537 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915573    2562 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 09 20:03:30.060537 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915576    2562 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 09 20:03:30.060537 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915579    2562 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 09 20:03:30.060537 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915583    2562 flags.go:64] FLAG: --v="2"
Jul 09 20:03:30.060537 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915587    2562 flags.go:64] FLAG: --version="false"
Jul 09 20:03:30.060537 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915591    2562 flags.go:64] FLAG: --vmodule=""
Jul 09 20:03:30.060537 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915595    2562 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 09 20:03:30.060537 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.915598    2562 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 09 20:03:30.060537 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.915747    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 20:03:30.060537 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.915753    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 20:03:30.004050 ip-10-0-140-63 systemd[1]: Started Kubernetes Kubelet.
Jul 09 20:03:30.062060 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.915757    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 20:03:30.062060 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.915761    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 20:03:30.062060 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.915765    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 20:03:30.062060 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.915768    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 20:03:30.062060 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.915771    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 20:03:30.062060 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.915774    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 20:03:30.062060 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.915779    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 20:03:30.062060 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.915782    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 20:03:30.062060 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.915785    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 20:03:30.062060 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.915788    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 20:03:30.062060 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.915790    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 20:03:30.062060 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.915795    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 20:03:30.062060 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.915797    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 20:03:30.062060 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.915800    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 20:03:30.062060 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.915803    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 20:03:30.062060 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.915806    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 20:03:30.062060 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.915809    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 20:03:30.062060 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.915812    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 20:03:30.062060 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.915816    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 20:03:30.062060 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.915819    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 20:03:30.063323 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.915822    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 20:03:30.063323 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.915825    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 20:03:30.063323 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.915828    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 20:03:30.063323 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.915831    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 20:03:30.063323 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.915833    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 20:03:30.063323 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.915836    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 20:03:30.063323 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.915839    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 20:03:30.063323 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.915841    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 20:03:30.063323 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.915844    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 20:03:30.063323 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.915847    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 20:03:30.063323 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.915850    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 20:03:30.063323 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.915853    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 20:03:30.063323 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.915855    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 20:03:30.063323 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.915858    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 20:03:30.063323 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.915861    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 20:03:30.063323 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.915864    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 20:03:30.063323 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.915867    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 20:03:30.063323 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.915870    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 20:03:30.063323 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.915874    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 20:03:30.063323 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.915877    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 20:03:30.064165 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.915880    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 20:03:30.064165 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.915883    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 20:03:30.064165 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.915886    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 20:03:30.064165 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.915890    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 20:03:30.064165 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.915893    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 20:03:30.064165 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.915896    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 20:03:30.064165 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.915899    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 20:03:30.064165 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.915902    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 20:03:30.064165 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.915905    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 20:03:30.064165 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.915908    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 20:03:30.064165 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.915911    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 20:03:30.064165 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.915914    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 20:03:30.064165 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.915917    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 20:03:30.064165 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.915920    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 20:03:30.064165 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.915923    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 20:03:30.064165 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.915926    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 20:03:30.064165 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.915928    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 20:03:30.064165 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.915931    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 20:03:30.064165 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.915934    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 20:03:30.064791 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.915938    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 20:03:30.064791 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.915942    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 20:03:30.064791 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.915945    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 20:03:30.064791 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.915949    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 20:03:30.064791 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.915952    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 20:03:30.064791 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.915954    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 20:03:30.064791 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.915958    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 20:03:30.064791 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.915961    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 20:03:30.064791 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.915964    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 20:03:30.064791 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.915968    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 20:03:30.064791 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.915972    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 20:03:30.064791 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.915977    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 20:03:30.064791 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.915980    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 20:03:30.064791 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.915983    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 20:03:30.064791 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.915986    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 20:03:30.064791 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.915989    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 20:03:30.064791 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.915994    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 20:03:30.064791 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.915997    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 20:03:30.064791 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.915999    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 20:03:30.065458 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.916002    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 20:03:30.065458 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.916005    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 20:03:30.065458 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.916007    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 20:03:30.065458 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.916010    2562 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 20:03:30.065458 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.916013    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 20:03:30.065458 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.916016    2562 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 20:03:30.065458 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.916019    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 20:03:30.065458 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.916022    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 20:03:30.065458 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.916025    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 20:03:30.065458 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.916028    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 20:03:30.065458 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.916031    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 20:03:30.065458 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.916034    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 20:03:30.065458 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.916037    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 20:03:30.065458 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.916039    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 20:03:30.065458 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.916042    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 20:03:30.065458 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.916045    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 20:03:30.065458 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.916047    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 20:03:30.065458 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.916050    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 20:03:30.065458 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.916053    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 20:03:30.066073 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.916061    2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 09 20:03:30.066073 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.925595    2562 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 09 20:03:30.066073 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.925614    2562 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 09 20:03:30.066073 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.926562    2562 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 09 20:03:30.066073 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.926572    2562 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 09 20:03:30.066073 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.926636    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 20:03:30.066073 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.926642    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 20:03:30.066073 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.926645    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 20:03:30.066073 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.926650    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 20:03:30.066073 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.926655    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 20:03:30.066073 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.926658    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 20:03:30.066073 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.926662    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 20:03:30.066073 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.926665    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 20:03:30.066073 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.926668    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 20:03:30.066656 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.926671    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 20:03:30.066656 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.926674    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 20:03:30.066656 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.926677    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 20:03:30.066656 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.926680    2562 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 20:03:30.066656 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.926683    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 20:03:30.066656 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.926686    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 20:03:30.066656 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.926689    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 20:03:30.066656 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.926692    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 20:03:30.066656 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.926696    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 20:03:30.066656 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.926699    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 20:03:30.066656 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.926702    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 20:03:30.066656 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.926705    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 20:03:30.066656 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.926708    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 20:03:30.066656 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.926711    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 20:03:30.066656 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.926715    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 20:03:30.066656 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.926718    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 20:03:30.066656 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.926721    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 20:03:30.066656 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.926723    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 20:03:30.066656 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.926726    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 20:03:30.066656 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.926729    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 20:03:30.067317 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.926733    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 20:03:30.067317 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.926736    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 20:03:30.067317 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.926739    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 20:03:30.067317 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.926742    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 20:03:30.067317 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.926745    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 20:03:30.067317 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.926748    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 20:03:30.067317 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.926751    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 20:03:30.067317 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.926753    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 20:03:30.067317 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.926758    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 20:03:30.067317 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.926762    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 20:03:30.067317 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.926766    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 20:03:30.067317 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.926769    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 20:03:30.067317 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.926773    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 20:03:30.067317 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.926776    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 20:03:30.067317 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.926783    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 20:03:30.067317 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.926786    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 20:03:30.067317 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.926789    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 20:03:30.067317 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.926793    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 20:03:30.067317 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.926796    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 20:03:30.068064 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.926799    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 20:03:30.068064 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.926802    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 20:03:30.068064 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.926807    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 20:03:30.068064 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.926811    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 20:03:30.068064 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.926814    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 20:03:30.068064 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.926817    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 20:03:30.068064 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.926821    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 20:03:30.068064 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.926824    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 20:03:30.068064 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.926827    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 20:03:30.068064 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.926829    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 20:03:30.068064 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.926832    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 20:03:30.068064 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.926835    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 20:03:30.068064 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.926838    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 20:03:30.068064 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.926841    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 20:03:30.068064 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.926844    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 20:03:30.068064 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.926847    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 20:03:30.068064 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.926850    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 20:03:30.068064 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.926853    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 20:03:30.068064 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.926855    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 20:03:30.068064 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.926858    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 20:03:30.068755 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.926861    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 20:03:30.068755 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.926864    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 20:03:30.068755 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.926867    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 20:03:30.068755 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.926869    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 20:03:30.068755 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.926875    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 20:03:30.068755 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.926877    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 20:03:30.068755 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.926880    2562 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 20:03:30.068755 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.926887    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 20:03:30.068755 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.926890    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 20:03:30.068755 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.926893    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 20:03:30.068755 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.926896    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 20:03:30.068755 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.926900    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 20:03:30.068755 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.926903    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 20:03:30.068755 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.926906    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 20:03:30.068755 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.926909    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 20:03:30.068755 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.926912    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 20:03:30.068755 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.926915    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 20:03:30.068755 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.926918    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 20:03:30.068755 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.926921    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 20:03:30.068755 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.926924    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 20:03:30.069355 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.926926    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 20:03:30.069355 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.926929    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 20:03:30.069355 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.926932    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 20:03:30.069355 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.926935    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 20:03:30.069355 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.926938    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 20:03:30.069355 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.926940    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 20:03:30.069355 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.926943    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 20:03:30.069355 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.926946    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 20:03:30.069355 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.926949    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 20:03:30.069355 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.926952    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 20:03:30.069355 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.926954    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 20:03:30.069355 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.926960    2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 09 20:03:30.069355 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.927110    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 20:03:30.069355 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.927116    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 20:03:30.069797 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.927120    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 20:03:30.069797 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.927123    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 20:03:30.069797 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.927127    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 20:03:30.069797 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.927131    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 20:03:30.069797 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.927133    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 20:03:30.069797 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.927136    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 20:03:30.069797 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.927141    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 20:03:30.069797 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.927144    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 20:03:30.069797 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.927147    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 20:03:30.069797 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.927149    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 20:03:30.069797 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.927155    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 20:03:30.069797 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.927160    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 20:03:30.069797 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.927163    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 20:03:30.069797 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.927167    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 20:03:30.069797 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.927170    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 20:03:30.069797 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.927174    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 20:03:30.069797 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.927176    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 20:03:30.069797 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.927179    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 20:03:30.069797 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.927182    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 20:03:30.070487 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.927185    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 20:03:30.070487 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.927189    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 20:03:30.070487 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.927191    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 20:03:30.070487 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.927194    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 20:03:30.070487 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.927197    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 20:03:30.070487 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.927199    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 20:03:30.070487 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.927202    2562 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 20:03:30.070487 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.927205    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 20:03:30.070487 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.927208    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 20:03:30.070487 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.927211    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 20:03:30.070487 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.927214    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 20:03:30.070487 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.927216    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 20:03:30.070487 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.927219    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 20:03:30.070487 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.927222    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 20:03:30.070487 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.927225    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 20:03:30.070487 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.927229    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 20:03:30.070487 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.927232    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 20:03:30.070487 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.927235    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 20:03:30.070487 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.927238    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 20:03:30.070487 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.927242    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 20:03:30.071165 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.927245    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 20:03:30.071165 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.927248    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 20:03:30.071165 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.927250    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 20:03:30.071165 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.927253    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 20:03:30.071165 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.927257    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 20:03:30.071165 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.927260    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 20:03:30.071165 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.927263    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 20:03:30.071165 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.927266    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 20:03:30.071165 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.927271    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 20:03:30.071165 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.927274    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 20:03:30.071165 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.927277    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 20:03:30.071165 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.927280    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 20:03:30.071165 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.927283    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 20:03:30.071165 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.927287    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 20:03:30.071165 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.927290    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 20:03:30.071165 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.927293    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 20:03:30.071165 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.927296    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 20:03:30.071165 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.927299    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 20:03:30.071165 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.927302    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 20:03:30.071838 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.927305    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 20:03:30.071838 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.927308    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 20:03:30.071838 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.927311    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 20:03:30.071838 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.927314    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 20:03:30.071838 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.927317    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 20:03:30.071838 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.927319    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 20:03:30.071838 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.927323    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 20:03:30.071838 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.927325    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 20:03:30.071838 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.927332    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 20:03:30.071838 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.927335    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 20:03:30.071838 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.927338    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 20:03:30.071838 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.927341    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 20:03:30.071838 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.927345    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 20:03:30.071838 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.927347    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 20:03:30.071838 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.927350    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 20:03:30.071838 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.927353    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 20:03:30.071838 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.927356    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 20:03:30.071838 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.927359    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 20:03:30.071838 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.927379    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 20:03:30.071838 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.927382    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 20:03:30.072570 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.927385    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 20:03:30.072570 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.927388    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 20:03:30.072570 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.927391    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 20:03:30.072570 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.927393    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 20:03:30.072570 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.927396    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 20:03:30.072570 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.927399    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 20:03:30.072570 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.927402    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 20:03:30.072570 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.927405    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 20:03:30.072570 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.927408    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 20:03:30.072570 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.927410    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 20:03:30.072570 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.927414    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 20:03:30.072570 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.927416    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 20:03:30.072570 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.927419    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 20:03:30.072570 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.927422    2562 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 20:03:30.072570 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.927426    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 20:03:30.072570 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.927430    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 20:03:30.072570 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.927434    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 20:03:30.072570 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.927436    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 20:03:30.072570 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:29.927439    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 20:03:30.073289 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.927444    2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 09 20:03:30.073289 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.927562    2562 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 09 20:03:30.073289 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.929753    2562 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 09 20:03:30.073289 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.931283    2562 server.go:1019] "Starting client certificate rotation"
Jul 09 20:03:30.073289 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.931412    2562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 09 20:03:30.073289 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.931454    2562 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 09 20:03:30.073289 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.935771    2562 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 09 20:03:30.073289 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.937443    2562 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 09 20:03:30.073289 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.955695    2562 log.go:25] "Validated CRI v1 runtime API"
Jul 09 20:03:30.073289 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.958499    2562 log.go:25] "Validated CRI v1 image API"
Jul 09 20:03:30.073289 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.959576    2562 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 09 20:03:30.073289 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.966322    2562 fs.go:135] Filesystem UUIDs: map[1c01c24e-fa88-41cb-b79d-41b71fd4dc4e:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 b04d89c6-b5d2-4ea3-bd1d-1ca4d55bc2d4:/dev/nvme0n1p4]
Jul 09 20:03:30.073766 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.966397    2562 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 09 20:03:30.074042 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.979334    2562 manager.go:216] Machine: {Timestamp:2026-07-09 20:03:29.977590112 +0000 UTC m=+0.395259921 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099702 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2d4ef16c7b65ff108d5b25064ed918 SystemUUID:ec2d4ef1-6c7b-65ff-108d-5b25064ed918 BootID:3fb440e7-6cb5-479c-8865-9cb0c11edf33 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:78:9f:dc:e9:45 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:78:9f:dc:e9:45 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:6a:9c:8f:99:92:44 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 09 20:03:30.074042 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.979461    2562 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 09 20:03:30.074042 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.979557    2562 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 09 20:03:30.074042 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.981223    2562 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 09 20:03:30.074042 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.983225    2562 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 09 20:03:30.074042 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.983252    2562 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-63.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 09 20:03:30.074042 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.984023    2562 topology_manager.go:138] "Creating topology manager with none policy"
Jul 09 20:03:30.246645 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.984033    2562 container_manager_linux.go:309] "Creating device plugin manager"
Jul 09 20:03:30.246645 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.984041    2562 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 09 20:03:30.246645 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.984071    2562 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 09 20:03:30.246645 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.985160    2562 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 09 20:03:30.246645 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.988361    2562 state_mem.go:36] "Initialized new in-memory state store"
Jul 09 20:03:30.246645 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.988507    2562 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 09 20:03:30.246645 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.988532    2562 kubelet.go:486] "Attempting to sync node with API server"
Jul 09 20:03:30.246645 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.988545    2562 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 09 20:03:30.246645 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.988561    2562 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 09 20:03:30.246645 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.988571    2562 kubelet.go:398] "Adding apiserver pod source"
Jul 09 20:03:30.246645 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.988581    2562 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 09 20:03:30.246645 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.988933    2562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 09 20:03:30.246645 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.988951    2562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 09 20:03:30.246645 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.993847    2562 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 09 20:03:30.246645 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.995727    2562 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 09 20:03:30.246645 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.995762    2562 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 09 20:03:30.246645 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.996420    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 09 20:03:30.246645 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.996435    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 09 20:03:30.246645 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.996441    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 09 20:03:30.246645 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.996447    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 09 20:03:30.246645 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.996455    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 09 20:03:30.246645 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.996461    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 09 20:03:30.246645 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.996813    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 09 20:03:30.246645 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.996821    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 09 20:03:30.246645 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.996828    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 09 20:03:30.246645 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.996836    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 09 20:03:30.246645 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.996846    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 09 20:03:30.246645 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.996854    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 09 20:03:30.246645 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.998493    2562 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-brhsc"
Jul 09 20:03:30.246645 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.999014    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 09 20:03:30.246645 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:29.999025    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 09 20:03:30.246645 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:30.002990    2562 server.go:1268] "Started kubelet"
Jul 09 20:03:30.246645 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:30.007129    2562 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-63.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 09 20:03:30.246645 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:03:30.007253    2562 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 09 20:03:30.246645 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:03:30.007323    2562 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-63.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 09 20:03:30.246645 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:30.007481    2562 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 09 20:03:30.246645 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:30.007520    2562 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 09 20:03:30.246645 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:30.007556    2562 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 09 20:03:30.246645 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:30.009036    2562 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 09 20:03:30.246645 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:30.009240    2562 server.go:310] "Adding debug handlers to kubelet server"
Jul 09 20:03:30.266730 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:30.012553    2562 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 09 20:03:30.266730 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:30.012957    2562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 09 20:03:30.266730 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:03:30.014374    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-63.ec2.internal\" not found"
Jul 09 20:03:30.266730 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:30.014522    2562 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-brhsc"
Jul 09 20:03:30.266730 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:30.014957    2562 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 09 20:03:30.266730 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:30.014978    2562 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 09 20:03:30.266730 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:30.015133    2562 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 09 20:03:30.266730 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:30.015201    2562 reconstruct.go:97] "Volume reconstruction finished"
Jul 09 20:03:30.266730 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:30.015210    2562 reconciler.go:29] "Reconciler: start to sync state"
Jul 09 20:03:30.266730 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:30.015655    2562 factory.go:55] Registering systemd factory
Jul 09 20:03:30.266730 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:30.015668    2562 factory.go:223] Registration of the systemd container factory successfully
Jul 09 20:03:30.266730 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:30.015905    2562 factory.go:153] Registering CRI-O factory
Jul 09 20:03:30.266730 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:30.015921    2562 factory.go:223] Registration of the crio container factory successfully
Jul 09 20:03:30.266730 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:30.015999    2562 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 09 20:03:30.266730 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:30.016033    2562 factory.go:103] Registering Raw factory
Jul 09 20:03:30.266730 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:30.016052    2562 manager.go:1196] Started watching for new ooms in manager
Jul 09 20:03:30.266730 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:03:30.016281    2562 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 09 20:03:30.266730 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:30.017292    2562 manager.go:349] Starting recovery of all containers
Jul 09 20:03:30.266730 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:30.024429    2562 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 20:03:30.266730 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:03:30.026826    2562 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-140-63.ec2.internal\" not found" node="ip-10-0-140-63.ec2.internal"
Jul 09 20:03:30.266730 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:30.041883    2562 manager.go:354] Recovery completed
Jul 09 20:03:30.266730 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:30.047828    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 20:03:30.266730 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:30.050028    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-63.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 20:03:30.266730 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:30.050054    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-63.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 20:03:30.266730 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:30.050065    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-63.ec2.internal" event="NodeHasSufficientPID"
Jul 09 20:03:30.266730 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:30.050756    2562 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 09 20:03:30.266730 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:30.050764    2562 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 09 20:03:30.266730 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:30.050783    2562 state_mem.go:36] "Initialized new in-memory state store"
Jul 09 20:03:30.266730 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:30.052039    2562 policy_none.go:49] "None policy: Start"
Jul 09 20:03:30.266730 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:30.052052    2562 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 09 20:03:30.266730 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:30.052063    2562 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 09 20:03:30.266730 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:30.053338    2562 policy_none.go:47] "Start"
Jul 09 20:03:30.266730 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:30.053549    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 20:03:30.266730 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:30.060196    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-63.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 20:03:30.266730 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:30.060364    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-63.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 20:03:30.266730 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:30.060380    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-63.ec2.internal" event="NodeHasSufficientPID"
Jul 09 20:03:30.266730 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:30.083969    2562 manager.go:337] "Starting Device Plugin manager"
Jul 09 20:03:30.266730 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:03:30.084004    2562 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 09 20:03:30.266730 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:30.084015    2562 server.go:87] "Starting device plugin registration server"
Jul 09 20:03:30.266730 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:30.084263    2562 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 09 20:03:30.266730 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:30.084274    2562 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 09 20:03:30.266730 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:30.084400    2562 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 09 20:03:30.266730 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:30.084482    2562 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 09 20:03:30.266730 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:30.084493    2562 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 09 20:03:30.268926 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:03:30.085207    2562 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 09 20:03:30.268926 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:03:30.085260    2562 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-63.ec2.internal\" not found"
Jul 09 20:03:30.268926 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:30.186706    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 20:03:30.268926 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:30.187857    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-63.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 20:03:30.268926 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:30.187894    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-63.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 20:03:30.268926 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:30.187905    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-63.ec2.internal" event="NodeHasSufficientPID"
Jul 09 20:03:30.268926 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:30.187933    2562 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-140-63.ec2.internal"
Jul 09 20:03:30.268926 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:30.197194    2562 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-140-63.ec2.internal"
Jul 09 20:03:30.268926 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:03:30.197219    2562 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-63.ec2.internal\": node \"ip-10-0-140-63.ec2.internal\" not found"
Jul 09 20:03:30.268926 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:03:30.211087    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-63.ec2.internal\" not found"
Jul 09 20:03:30.268926 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:30.233881    2562 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 09 20:03:30.268926 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:30.235210    2562 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 09 20:03:30.268926 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:30.235246    2562 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 09 20:03:30.268926 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:30.235269    2562 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 09 20:03:30.268926 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:03:30.235301    2562 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 09 20:03:30.268926 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:30.237401    2562 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 20:03:30.482201 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:03:30.311447    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-63.ec2.internal\" not found"
Jul 09 20:03:30.482201 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:30.336642    2562 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-140-63.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-63.ec2.internal"]
Jul 09 20:03:30.482201 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:30.336739    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 20:03:30.482201 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:30.338629    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-63.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 20:03:30.482201 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:30.338665    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-63.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 20:03:30.482201 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:30.338676    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-63.ec2.internal" event="NodeHasSufficientPID"
Jul 09 20:03:30.482201 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:30.339931    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 20:03:30.482201 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:30.340041    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-63.ec2.internal"
Jul 09 20:03:30.482201 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:30.340076    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 20:03:30.482201 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:30.341265    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-63.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 20:03:30.482201 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:30.341292    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-63.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 20:03:30.482201 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:30.341323    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-63.ec2.internal" event="NodeHasSufficientPID"
Jul 09 20:03:30.482201 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:30.341486    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-63.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 20:03:30.482201 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:30.341509    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-63.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 20:03:30.482201 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:30.341519    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-63.ec2.internal" event="NodeHasSufficientPID"
Jul 09 20:03:30.482201 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:30.343365    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-63.ec2.internal"
Jul 09 20:03:30.482201 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:30.343411    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 20:03:30.482201 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:30.344023    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-63.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 20:03:30.482201 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:30.344041    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-63.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 20:03:30.482201 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:30.344051    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-63.ec2.internal" event="NodeHasSufficientPID"
Jul 09 20:03:30.482201 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:03:30.365983    2562 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-63.ec2.internal\" not found" node="ip-10-0-140-63.ec2.internal"
Jul 09 20:03:30.482201 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:03:30.370315    2562 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-63.ec2.internal\" not found" node="ip-10-0-140-63.ec2.internal"
Jul 09 20:03:30.482201 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:03:30.413539    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-63.ec2.internal\" not found"
Jul 09 20:03:30.482201 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:30.416058    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/61c3c5dbc96e6f9e2e03d9211834c2ca-config\") pod \"kube-apiserver-proxy-ip-10-0-140-63.ec2.internal\" (UID: \"61c3c5dbc96e6f9e2e03d9211834c2ca\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-63.ec2.internal"
Jul 09 20:03:30.482201 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:30.416081    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8af726f313e368ccaa2fb3f2a4a8af61-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-63.ec2.internal\" (UID: \"8af726f313e368ccaa2fb3f2a4a8af61\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-63.ec2.internal"
Jul 09 20:03:30.482201 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:30.416107    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8af726f313e368ccaa2fb3f2a4a8af61-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-63.ec2.internal\" (UID: \"8af726f313e368ccaa2fb3f2a4a8af61\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-63.ec2.internal"
Jul 09 20:03:30.685364 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:03:30.515252    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-63.ec2.internal\" not found"
Jul 09 20:03:30.685364 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:30.516967    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/61c3c5dbc96e6f9e2e03d9211834c2ca-config\") pod \"kube-apiserver-proxy-ip-10-0-140-63.ec2.internal\" (UID: \"61c3c5dbc96e6f9e2e03d9211834c2ca\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-63.ec2.internal"
Jul 09 20:03:30.685364 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:30.517002    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8af726f313e368ccaa2fb3f2a4a8af61-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-63.ec2.internal\" (UID: \"8af726f313e368ccaa2fb3f2a4a8af61\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-63.ec2.internal"
Jul 09 20:03:30.685364 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:30.517019    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8af726f313e368ccaa2fb3f2a4a8af61-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-63.ec2.internal\" (UID: \"8af726f313e368ccaa2fb3f2a4a8af61\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-63.ec2.internal"
Jul 09 20:03:30.685364 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:30.517035    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/61c3c5dbc96e6f9e2e03d9211834c2ca-config\") pod \"kube-apiserver-proxy-ip-10-0-140-63.ec2.internal\" (UID: \"61c3c5dbc96e6f9e2e03d9211834c2ca\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-63.ec2.internal"
Jul 09 20:03:30.685364 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:30.517049    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8af726f313e368ccaa2fb3f2a4a8af61-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-63.ec2.internal\" (UID: \"8af726f313e368ccaa2fb3f2a4a8af61\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-63.ec2.internal"
Jul 09 20:03:30.685364 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:30.517063    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8af726f313e368ccaa2fb3f2a4a8af61-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-63.ec2.internal\" (UID: \"8af726f313e368ccaa2fb3f2a4a8af61\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-63.ec2.internal"
Jul 09 20:03:30.685364 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:03:30.615623    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-63.ec2.internal\" not found"
Jul 09 20:03:30.685364 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:30.670891    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-63.ec2.internal"
Jul 09 20:03:30.685364 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:30.673170    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-63.ec2.internal"
Jul 09 20:03:30.871785 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:03:30.716062    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-63.ec2.internal\" not found"
Jul 09 20:03:30.871785 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:03:30.820399    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-63.ec2.internal\" not found"
Jul 09 20:03:30.871785 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:30.828223    2562 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 20:03:31.058921 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:30.914808    2562 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-63.ec2.internal"
Jul 09 20:03:31.058921 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:30.927097    2562 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 09 20:03:31.058921 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:30.929179    2562 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-63.ec2.internal"
Jul 09 20:03:31.058921 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:30.930407    2562 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 09 20:03:31.058921 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:30.930519    2562 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 09 20:03:31.058921 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:30.930571    2562 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 09 20:03:31.058921 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:03:30.930583    2562 kubelet.go:3250] "Failed creating a mirror pod" err="Post \"https://aad1b3a813b2b4dd79cd4cb2e9a35288-e7840672136983b1.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/kube-system/pods\": read tcp 10.0.140.63:47206->44.218.80.32:6443: use of closed network connection" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-63.ec2.internal"
Jul 09 20:03:31.058921 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:30.990047    2562 apiserver.go:52] "Watching apiserver"
Jul 09 20:03:31.058921 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.003029    2562 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 09 20:03:31.058921 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.003423    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-vwttv","openshift-image-registry/node-ca-n2tvb","openshift-network-diagnostics/network-check-target-6b5z9","openshift-network-operator/iptables-alerter-j6mws","openshift-ovn-kubernetes/ovnkube-node-hds4g","kube-system/global-pull-secret-syncer-dccxz","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nl8h","openshift-cluster-node-tuning-operator/tuned-87zd2","openshift-dns/node-resolver-8r78k","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-63.ec2.internal","openshift-multus/multus-additional-cni-plugins-dz4cq","openshift-multus/multus-rvxnk","openshift-multus/network-metrics-daemon-ksd8v"]
Jul 09 20:03:31.058921 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.005142    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nl8h"
Jul 09 20:03:31.058921 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.006655    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-n2tvb"
Jul 09 20:03:31.058921 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.007901    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6b5z9"
Jul 09 20:03:31.058921 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:03:31.007950    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6b5z9" podUID="4fd1b369-9b81-495d-9608-9fc4d81295e2"
Jul 09 20:03:31.058921 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.008049    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 09 20:03:31.058921 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.008354    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 09 20:03:31.058921 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.008396    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-5k92c\""
Jul 09 20:03:31.058921 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.008488    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 09 20:03:31.058921 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.009260    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-j6mws"
Jul 09 20:03:31.058921 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.009703    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 09 20:03:31.058921 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.009959    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 09 20:03:31.058921 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.010595    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-755w5\""
Jul 09 20:03:31.058921 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.010598    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 09 20:03:31.058921 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.010742    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-hds4g"
Jul 09 20:03:31.060289 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.013623    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dccxz"
Jul 09 20:03:31.060289 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:03:31.013708    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dccxz" podUID="13454dc6-77a2-4dc9-adc2-81791d1cd463"
Jul 09 20:03:31.060289 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.015427    2562 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 09 20:03:31.060289 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.015451    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vwttv"
Jul 09 20:03:31.060289 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.016581    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-87zd2"
Jul 09 20:03:31.060289 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.017292    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 09 20:03:31.060289 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.017294    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 09 20:03:31.060289 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.017311    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 09 20:03:31.060289 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.017476    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 09 20:03:31.060289 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.017522    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 09 20:03:31.060289 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.017537    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 09 20:03:31.060289 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.017563    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 09 20:03:31.060289 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.017590    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 09 20:03:31.060289 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.017795    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-4kk25\""
Jul 09 20:03:31.060289 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.017886    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 09 20:03:31.060289 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.017936    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-5f9xc\""
Jul 09 20:03:31.060289 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.018042    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-8r78k"
Jul 09 20:03:31.060289 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.018427    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-w2jfv\""
Jul 09 20:03:31.060289 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.018748    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 09 20:03:31.060289 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.018765    2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-08 19:58:30 +0000 UTC" deadline="2028-01-28 05:11:17.791784374 +0000 UTC"
Jul 09 20:03:31.060289 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.018799    2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13617h7m46.772987308s"
Jul 09 20:03:31.060289 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.019295    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-dz4cq"
Jul 09 20:03:31.060289 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.019787    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 09 20:03:31.060289 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.019965    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 09 20:03:31.060289 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.019993    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 09 20:03:31.060289 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.020116    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-vvvz5\""
Jul 09 20:03:31.060289 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.020486    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a304eb99-5afb-4710-9e69-51ca0c329ba5-etc-modprobe-d\") pod \"tuned-87zd2\" (UID: \"a304eb99-5afb-4710-9e69-51ca0c329ba5\") " pod="openshift-cluster-node-tuning-operator/tuned-87zd2"
Jul 09 20:03:31.061290 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.020507    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a304eb99-5afb-4710-9e69-51ca0c329ba5-etc-sysctl-conf\") pod \"tuned-87zd2\" (UID: \"a304eb99-5afb-4710-9e69-51ca0c329ba5\") " pod="openshift-cluster-node-tuning-operator/tuned-87zd2"
Jul 09 20:03:31.061290 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.020524    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a304eb99-5afb-4710-9e69-51ca0c329ba5-etc-systemd\") pod \"tuned-87zd2\" (UID: \"a304eb99-5afb-4710-9e69-51ca0c329ba5\") " pod="openshift-cluster-node-tuning-operator/tuned-87zd2"
Jul 09 20:03:31.061290 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.020539    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a304eb99-5afb-4710-9e69-51ca0c329ba5-etc-tuned\") pod \"tuned-87zd2\" (UID: \"a304eb99-5afb-4710-9e69-51ca0c329ba5\") " pod="openshift-cluster-node-tuning-operator/tuned-87zd2"
Jul 09 20:03:31.061290 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.020556    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rst6\" (UniqueName: \"kubernetes.io/projected/785ecd20-a92f-4924-b2f3-663697381982-kube-api-access-2rst6\") pod \"node-ca-n2tvb\" (UID: \"785ecd20-a92f-4924-b2f3-663697381982\") " pod="openshift-image-registry/node-ca-n2tvb"
Jul 09 20:03:31.061290 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.020572    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/62b8139e-cc18-4428-9373-a64195b6cbdd-device-dir\") pod \"aws-ebs-csi-driver-node-9nl8h\" (UID: \"62b8139e-cc18-4428-9373-a64195b6cbdd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nl8h"
Jul 09 20:03:31.061290 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.020617    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8zbm\" (UniqueName: \"kubernetes.io/projected/0a94e16d-ed38-41a6-a71d-49e96e5ae7c7-kube-api-access-v8zbm\") pod \"iptables-alerter-j6mws\" (UID: \"0a94e16d-ed38-41a6-a71d-49e96e5ae7c7\") " pod="openshift-network-operator/iptables-alerter-j6mws"
Jul 09 20:03:31.061290 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.020643    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/62b8139e-cc18-4428-9373-a64195b6cbdd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9nl8h\" (UID: \"62b8139e-cc18-4428-9373-a64195b6cbdd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nl8h"
Jul 09 20:03:31.061290 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.020659    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/62b8139e-cc18-4428-9373-a64195b6cbdd-socket-dir\") pod \"aws-ebs-csi-driver-node-9nl8h\" (UID: \"62b8139e-cc18-4428-9373-a64195b6cbdd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nl8h"
Jul 09 20:03:31.061290 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.020673    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a304eb99-5afb-4710-9e69-51ca0c329ba5-etc-sysconfig\") pod \"tuned-87zd2\" (UID: \"a304eb99-5afb-4710-9e69-51ca0c329ba5\") " pod="openshift-cluster-node-tuning-operator/tuned-87zd2"
Jul 09 20:03:31.061290 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.020688    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rk6gw\" (UniqueName: \"kubernetes.io/projected/a304eb99-5afb-4710-9e69-51ca0c329ba5-kube-api-access-rk6gw\") pod \"tuned-87zd2\" (UID: \"a304eb99-5afb-4710-9e69-51ca0c329ba5\") " pod="openshift-cluster-node-tuning-operator/tuned-87zd2"
Jul 09 20:03:31.061290 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.020703    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/785ecd20-a92f-4924-b2f3-663697381982-host\") pod \"node-ca-n2tvb\" (UID: \"785ecd20-a92f-4924-b2f3-663697381982\") " pod="openshift-image-registry/node-ca-n2tvb"
Jul 09 20:03:31.061290 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.020717    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/785ecd20-a92f-4924-b2f3-663697381982-serviceca\") pod \"node-ca-n2tvb\" (UID: \"785ecd20-a92f-4924-b2f3-663697381982\") " pod="openshift-image-registry/node-ca-n2tvb"
Jul 09 20:03:31.061290 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.020733    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/00761abf-a23d-4f99-ba0f-bcb623060f98-agent-certs\") pod \"konnectivity-agent-vwttv\" (UID: \"00761abf-a23d-4f99-ba0f-bcb623060f98\") " pod="kube-system/konnectivity-agent-vwttv"
Jul 09 20:03:31.061786 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.020765    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gdm2\" (UniqueName: \"kubernetes.io/projected/62b8139e-cc18-4428-9373-a64195b6cbdd-kube-api-access-7gdm2\") pod \"aws-ebs-csi-driver-node-9nl8h\" (UID: \"62b8139e-cc18-4428-9373-a64195b6cbdd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nl8h"
Jul 09 20:03:31.061786 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.020822    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a304eb99-5afb-4710-9e69-51ca0c329ba5-etc-sysctl-d\") pod \"tuned-87zd2\" (UID: \"a304eb99-5afb-4710-9e69-51ca0c329ba5\") " pod="openshift-cluster-node-tuning-operator/tuned-87zd2"
Jul 09 20:03:31.061786 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.020859    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khzxc\" (UniqueName: \"kubernetes.io/projected/4fd1b369-9b81-495d-9608-9fc4d81295e2-kube-api-access-khzxc\") pod \"network-check-target-6b5z9\" (UID: \"4fd1b369-9b81-495d-9608-9fc4d81295e2\") " pod="openshift-network-diagnostics/network-check-target-6b5z9"
Jul 09 20:03:31.061786 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.020885    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0a94e16d-ed38-41a6-a71d-49e96e5ae7c7-iptables-alerter-script\") pod \"iptables-alerter-j6mws\" (UID: \"0a94e16d-ed38-41a6-a71d-49e96e5ae7c7\") " pod="openshift-network-operator/iptables-alerter-j6mws"
Jul 09 20:03:31.061786 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.020909    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0a94e16d-ed38-41a6-a71d-49e96e5ae7c7-host-slash\") pod \"iptables-alerter-j6mws\" (UID: \"0a94e16d-ed38-41a6-a71d-49e96e5ae7c7\") " pod="openshift-network-operator/iptables-alerter-j6mws"
Jul 09 20:03:31.061786 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.020932    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4b4468fd-1708-487b-8d69-bb7d817e25de-host-slash\") pod \"ovnkube-node-hds4g\" (UID: \"4b4468fd-1708-487b-8d69-bb7d817e25de\") " pod="openshift-ovn-kubernetes/ovnkube-node-hds4g"
Jul 09 20:03:31.061786 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.020955    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4b4468fd-1708-487b-8d69-bb7d817e25de-var-lib-openvswitch\") pod \"ovnkube-node-hds4g\" (UID: \"4b4468fd-1708-487b-8d69-bb7d817e25de\") " pod="openshift-ovn-kubernetes/ovnkube-node-hds4g"
Jul 09 20:03:31.061786 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.020984    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4b4468fd-1708-487b-8d69-bb7d817e25de-host-run-ovn-kubernetes\") pod \"ovnkube-node-hds4g\" (UID: \"4b4468fd-1708-487b-8d69-bb7d817e25de\") " pod="openshift-ovn-kubernetes/ovnkube-node-hds4g"
Jul 09 20:03:31.061786 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.021001    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4b4468fd-1708-487b-8d69-bb7d817e25de-host-cni-netd\") pod \"ovnkube-node-hds4g\" (UID: \"4b4468fd-1708-487b-8d69-bb7d817e25de\") " pod="openshift-ovn-kubernetes/ovnkube-node-hds4g"
Jul 09 20:03:31.061786 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.021027    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a304eb99-5afb-4710-9e69-51ca0c329ba5-lib-modules\") pod \"tuned-87zd2\" (UID: \"a304eb99-5afb-4710-9e69-51ca0c329ba5\") " pod="openshift-cluster-node-tuning-operator/tuned-87zd2"
Jul 09 20:03:31.061786 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.021052    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a304eb99-5afb-4710-9e69-51ca0c329ba5-tmp\") pod \"tuned-87zd2\" (UID: \"a304eb99-5afb-4710-9e69-51ca0c329ba5\") " pod="openshift-cluster-node-tuning-operator/tuned-87zd2"
Jul 09 20:03:31.061786 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.021076    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/62b8139e-cc18-4428-9373-a64195b6cbdd-registration-dir\") pod \"aws-ebs-csi-driver-node-9nl8h\" (UID: \"62b8139e-cc18-4428-9373-a64195b6cbdd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nl8h"
Jul 09 20:03:31.061786 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.021093    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-rvxnk"
Jul 09 20:03:31.061786 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.021101    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/62b8139e-cc18-4428-9373-a64195b6cbdd-sys-fs\") pod \"aws-ebs-csi-driver-node-9nl8h\" (UID: \"62b8139e-cc18-4428-9373-a64195b6cbdd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nl8h"
Jul 09 20:03:31.062398 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.021125    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4b4468fd-1708-487b-8d69-bb7d817e25de-host-run-netns\") pod \"ovnkube-node-hds4g\" (UID: \"4b4468fd-1708-487b-8d69-bb7d817e25de\") " pod="openshift-ovn-kubernetes/ovnkube-node-hds4g"
Jul 09 20:03:31.062398 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.021149    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4b4468fd-1708-487b-8d69-bb7d817e25de-run-systemd\") pod \"ovnkube-node-hds4g\" (UID: \"4b4468fd-1708-487b-8d69-bb7d817e25de\") " pod="openshift-ovn-kubernetes/ovnkube-node-hds4g"
Jul 09 20:03:31.062398 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.021175    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4b4468fd-1708-487b-8d69-bb7d817e25de-ovnkube-config\") pod \"ovnkube-node-hds4g\" (UID: \"4b4468fd-1708-487b-8d69-bb7d817e25de\") " pod="openshift-ovn-kubernetes/ovnkube-node-hds4g"
Jul 09 20:03:31.062398 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.021198    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/62b8139e-cc18-4428-9373-a64195b6cbdd-etc-selinux\") pod \"aws-ebs-csi-driver-node-9nl8h\" (UID: \"62b8139e-cc18-4428-9373-a64195b6cbdd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nl8h"
Jul 09 20:03:31.062398 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.021223    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a304eb99-5afb-4710-9e69-51ca0c329ba5-var-lib-kubelet\") pod \"tuned-87zd2\" (UID: \"a304eb99-5afb-4710-9e69-51ca0c329ba5\") " pod="openshift-cluster-node-tuning-operator/tuned-87zd2"
Jul 09 20:03:31.062398 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.021247    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4b4468fd-1708-487b-8d69-bb7d817e25de-log-socket\") pod \"ovnkube-node-hds4g\" (UID: \"4b4468fd-1708-487b-8d69-bb7d817e25de\") " pod="openshift-ovn-kubernetes/ovnkube-node-hds4g"
Jul 09 20:03:31.062398 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.021293    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4b4468fd-1708-487b-8d69-bb7d817e25de-host-cni-bin\") pod \"ovnkube-node-hds4g\" (UID: \"4b4468fd-1708-487b-8d69-bb7d817e25de\") " pod="openshift-ovn-kubernetes/ovnkube-node-hds4g"
Jul 09 20:03:31.062398 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.021320    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4b4468fd-1708-487b-8d69-bb7d817e25de-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hds4g\" (UID: \"4b4468fd-1708-487b-8d69-bb7d817e25de\") " pod="openshift-ovn-kubernetes/ovnkube-node-hds4g"
Jul 09 20:03:31.062398 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.021414    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/13454dc6-77a2-4dc9-adc2-81791d1cd463-original-pull-secret\") pod \"global-pull-secret-syncer-dccxz\" (UID: \"13454dc6-77a2-4dc9-adc2-81791d1cd463\") " pod="kube-system/global-pull-secret-syncer-dccxz"
Jul 09 20:03:31.062398 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.021491    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a304eb99-5afb-4710-9e69-51ca0c329ba5-host\") pod \"tuned-87zd2\" (UID: \"a304eb99-5afb-4710-9e69-51ca0c329ba5\") " pod="openshift-cluster-node-tuning-operator/tuned-87zd2"
Jul 09 20:03:31.062398 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.021520    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/00761abf-a23d-4f99-ba0f-bcb623060f98-konnectivity-ca\") pod \"konnectivity-agent-vwttv\" (UID: \"00761abf-a23d-4f99-ba0f-bcb623060f98\") " pod="kube-system/konnectivity-agent-vwttv"
Jul 09 20:03:31.062398 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.021834    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-8d7mv\""
Jul 09 20:03:31.062398 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.021875    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4b4468fd-1708-487b-8d69-bb7d817e25de-host-kubelet\") pod \"ovnkube-node-hds4g\" (UID: \"4b4468fd-1708-487b-8d69-bb7d817e25de\") " pod="openshift-ovn-kubernetes/ovnkube-node-hds4g"
Jul 09 20:03:31.062398 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.021511    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 09 20:03:31.062398 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.021918    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4b4468fd-1708-487b-8d69-bb7d817e25de-run-ovn\") pod \"ovnkube-node-hds4g\" (UID: \"4b4468fd-1708-487b-8d69-bb7d817e25de\") " pod="openshift-ovn-kubernetes/ovnkube-node-hds4g"
Jul 09 20:03:31.062918 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.021978    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4b4468fd-1708-487b-8d69-bb7d817e25de-env-overrides\") pod \"ovnkube-node-hds4g\" (UID: \"4b4468fd-1708-487b-8d69-bb7d817e25de\") " pod="openshift-ovn-kubernetes/ovnkube-node-hds4g"
Jul 09 20:03:31.062918 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.022001    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 09 20:03:31.062918 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.022011    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4b4468fd-1708-487b-8d69-bb7d817e25de-ovnkube-script-lib\") pod \"ovnkube-node-hds4g\" (UID: \"4b4468fd-1708-487b-8d69-bb7d817e25de\") " pod="openshift-ovn-kubernetes/ovnkube-node-hds4g"
Jul 09 20:03:31.062918 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.022072    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/13454dc6-77a2-4dc9-adc2-81791d1cd463-dbus\") pod \"global-pull-secret-syncer-dccxz\" (UID: \"13454dc6-77a2-4dc9-adc2-81791d1cd463\") " pod="kube-system/global-pull-secret-syncer-dccxz"
Jul 09 20:03:31.062918 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.022107    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a304eb99-5afb-4710-9e69-51ca0c329ba5-etc-kubernetes\") pod \"tuned-87zd2\" (UID: \"a304eb99-5afb-4710-9e69-51ca0c329ba5\") " pod="openshift-cluster-node-tuning-operator/tuned-87zd2"
Jul 09 20:03:31.062918 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.022137    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a304eb99-5afb-4710-9e69-51ca0c329ba5-sys\") pod \"tuned-87zd2\" (UID: \"a304eb99-5afb-4710-9e69-51ca0c329ba5\") " pod="openshift-cluster-node-tuning-operator/tuned-87zd2"
Jul 09 20:03:31.062918 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.022170    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4b4468fd-1708-487b-8d69-bb7d817e25de-etc-openvswitch\") pod \"ovnkube-node-hds4g\" (UID: \"4b4468fd-1708-487b-8d69-bb7d817e25de\") " pod="openshift-ovn-kubernetes/ovnkube-node-hds4g"
Jul 09 20:03:31.062918 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.022216    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4b4468fd-1708-487b-8d69-bb7d817e25de-run-openvswitch\") pod \"ovnkube-node-hds4g\" (UID: \"4b4468fd-1708-487b-8d69-bb7d817e25de\") " pod="openshift-ovn-kubernetes/ovnkube-node-hds4g"
Jul 09 20:03:31.062918 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.022248    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 09 20:03:31.062918 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.022249    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4b4468fd-1708-487b-8d69-bb7d817e25de-ovn-node-metrics-cert\") pod \"ovnkube-node-hds4g\" (UID: \"4b4468fd-1708-487b-8d69-bb7d817e25de\") " pod="openshift-ovn-kubernetes/ovnkube-node-hds4g"
Jul 09 20:03:31.062918 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.022378    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gdr2\" (UniqueName: \"kubernetes.io/projected/4b4468fd-1708-487b-8d69-bb7d817e25de-kube-api-access-8gdr2\") pod \"ovnkube-node-hds4g\" (UID: \"4b4468fd-1708-487b-8d69-bb7d817e25de\") " pod="openshift-ovn-kubernetes/ovnkube-node-hds4g"
Jul 09 20:03:31.062918 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.022683    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/13454dc6-77a2-4dc9-adc2-81791d1cd463-kubelet-config\") pod \"global-pull-secret-syncer-dccxz\" (UID: \"13454dc6-77a2-4dc9-adc2-81791d1cd463\") " pod="kube-system/global-pull-secret-syncer-dccxz"
Jul 09 20:03:31.062918 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.022715    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a304eb99-5afb-4710-9e69-51ca0c329ba5-run\") pod \"tuned-87zd2\" (UID: \"a304eb99-5afb-4710-9e69-51ca0c329ba5\") " pod="openshift-cluster-node-tuning-operator/tuned-87zd2"
Jul 09 20:03:31.062918 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.022757    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4b4468fd-1708-487b-8d69-bb7d817e25de-systemd-units\") pod \"ovnkube-node-hds4g\" (UID: \"4b4468fd-1708-487b-8d69-bb7d817e25de\") " pod="openshift-ovn-kubernetes/ovnkube-node-hds4g"
Jul 09 20:03:31.062918 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.022723    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 09 20:03:31.062918 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.023046    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 09 20:03:31.062918 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.023267    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 09 20:03:31.063478 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.022791    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 09 20:03:31.063478 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.022788    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4b4468fd-1708-487b-8d69-bb7d817e25de-node-log\") pod \"ovnkube-node-hds4g\" (UID: \"4b4468fd-1708-487b-8d69-bb7d817e25de\") " pod="openshift-ovn-kubernetes/ovnkube-node-hds4g"
Jul 09 20:03:31.063478 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.023829    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-bgxxg\""
Jul 09 20:03:31.063478 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.023906    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 09 20:03:31.063478 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.023987    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-xpxsp\""
Jul 09 20:03:31.063478 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.026351    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ksd8v"
Jul 09 20:03:31.063478 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:03:31.026444    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ksd8v" podUID="14a1d22f-43ff-4eb8-a89d-920551b2ef1f"
Jul 09 20:03:31.063478 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.031186    2562 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 09 20:03:31.063478 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.057038    2562 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-fg6r4"
Jul 09 20:03:31.238357 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.068513    2562 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-fg6r4"
Jul 09 20:03:31.238357 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.115744    2562 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 09 20:03:31.238357 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.123876    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d592535b-5457-48d0-ae48-caed8efc04fc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dz4cq\" (UID: \"d592535b-5457-48d0-ae48-caed8efc04fc\") " pod="openshift-multus/multus-additional-cni-plugins-dz4cq"
Jul 09 20:03:31.238357 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.123905    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4b4468fd-1708-487b-8d69-bb7d817e25de-host-run-netns\") pod \"ovnkube-node-hds4g\" (UID: \"4b4468fd-1708-487b-8d69-bb7d817e25de\") " pod="openshift-ovn-kubernetes/ovnkube-node-hds4g"
Jul 09 20:03:31.238357 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.123924    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gdr2\" (UniqueName: \"kubernetes.io/projected/4b4468fd-1708-487b-8d69-bb7d817e25de-kube-api-access-8gdr2\") pod \"ovnkube-node-hds4g\" (UID: \"4b4468fd-1708-487b-8d69-bb7d817e25de\") " pod="openshift-ovn-kubernetes/ovnkube-node-hds4g"
Jul 09 20:03:31.238357 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.123941    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a304eb99-5afb-4710-9e69-51ca0c329ba5-etc-modprobe-d\") pod \"tuned-87zd2\" (UID: \"a304eb99-5afb-4710-9e69-51ca0c329ba5\") " pod="openshift-cluster-node-tuning-operator/tuned-87zd2"
Jul 09 20:03:31.238357 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.123958    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a304eb99-5afb-4710-9e69-51ca0c329ba5-etc-systemd\") pod \"tuned-87zd2\" (UID: \"a304eb99-5afb-4710-9e69-51ca0c329ba5\") " pod="openshift-cluster-node-tuning-operator/tuned-87zd2"
Jul 09 20:03:31.238357 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.123999    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4b4468fd-1708-487b-8d69-bb7d817e25de-host-run-netns\") pod \"ovnkube-node-hds4g\" (UID: \"4b4468fd-1708-487b-8d69-bb7d817e25de\") " pod="openshift-ovn-kubernetes/ovnkube-node-hds4g"
Jul 09 20:03:31.238357 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.124047    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a304eb99-5afb-4710-9e69-51ca0c329ba5-etc-modprobe-d\") pod \"tuned-87zd2\" (UID: \"a304eb99-5afb-4710-9e69-51ca0c329ba5\") " pod="openshift-cluster-node-tuning-operator/tuned-87zd2"
Jul 09 20:03:31.238357 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.124076    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a304eb99-5afb-4710-9e69-51ca0c329ba5-etc-tuned\") pod \"tuned-87zd2\" (UID: \"a304eb99-5afb-4710-9e69-51ca0c329ba5\") " pod="openshift-cluster-node-tuning-operator/tuned-87zd2"
Jul 09 20:03:31.238357 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.124117    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5204f9ff-c575-4889-91ff-3cf3ffcd032b-system-cni-dir\") pod \"multus-rvxnk\" (UID: \"5204f9ff-c575-4889-91ff-3cf3ffcd032b\") " pod="openshift-multus/multus-rvxnk"
Jul 09 20:03:31.238357 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.124085    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a304eb99-5afb-4710-9e69-51ca0c329ba5-etc-systemd\") pod \"tuned-87zd2\" (UID: \"a304eb99-5afb-4710-9e69-51ca0c329ba5\") " pod="openshift-cluster-node-tuning-operator/tuned-87zd2"
Jul 09 20:03:31.238357 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.124146    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5204f9ff-c575-4889-91ff-3cf3ffcd032b-multus-daemon-config\") pod \"multus-rvxnk\" (UID: \"5204f9ff-c575-4889-91ff-3cf3ffcd032b\") " pod="openshift-multus/multus-rvxnk"
Jul 09 20:03:31.238357 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.124175    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0a94e16d-ed38-41a6-a71d-49e96e5ae7c7-host-slash\") pod \"iptables-alerter-j6mws\" (UID: \"0a94e16d-ed38-41a6-a71d-49e96e5ae7c7\") " pod="openshift-network-operator/iptables-alerter-j6mws"
Jul 09 20:03:31.238357 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.124203    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/62b8139e-cc18-4428-9373-a64195b6cbdd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9nl8h\" (UID: \"62b8139e-cc18-4428-9373-a64195b6cbdd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nl8h"
Jul 09 20:03:31.238357 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.124228    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rk6gw\" (UniqueName: \"kubernetes.io/projected/a304eb99-5afb-4710-9e69-51ca0c329ba5-kube-api-access-rk6gw\") pod \"tuned-87zd2\" (UID: \"a304eb99-5afb-4710-9e69-51ca0c329ba5\") " pod="openshift-cluster-node-tuning-operator/tuned-87zd2"
Jul 09 20:03:31.239524 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.124268    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0a94e16d-ed38-41a6-a71d-49e96e5ae7c7-host-slash\") pod \"iptables-alerter-j6mws\" (UID: \"0a94e16d-ed38-41a6-a71d-49e96e5ae7c7\") " pod="openshift-network-operator/iptables-alerter-j6mws"
Jul 09 20:03:31.239524 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.124275    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5204f9ff-c575-4889-91ff-3cf3ffcd032b-host-run-netns\") pod \"multus-rvxnk\" (UID: \"5204f9ff-c575-4889-91ff-3cf3ffcd032b\") " pod="openshift-multus/multus-rvxnk"
Jul 09 20:03:31.239524 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.124288    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/62b8139e-cc18-4428-9373-a64195b6cbdd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9nl8h\" (UID: \"62b8139e-cc18-4428-9373-a64195b6cbdd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nl8h"
Jul 09 20:03:31.239524 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.124300    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d592535b-5457-48d0-ae48-caed8efc04fc-os-release\") pod \"multus-additional-cni-plugins-dz4cq\" (UID: \"d592535b-5457-48d0-ae48-caed8efc04fc\") " pod="openshift-multus/multus-additional-cni-plugins-dz4cq"
Jul 09 20:03:31.239524 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.124344    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0a94e16d-ed38-41a6-a71d-49e96e5ae7c7-iptables-alerter-script\") pod \"iptables-alerter-j6mws\" (UID: \"0a94e16d-ed38-41a6-a71d-49e96e5ae7c7\") " pod="openshift-network-operator/iptables-alerter-j6mws"
Jul 09 20:03:31.239524 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.124362    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4b4468fd-1708-487b-8d69-bb7d817e25de-host-slash\") pod \"ovnkube-node-hds4g\" (UID: \"4b4468fd-1708-487b-8d69-bb7d817e25de\") " pod="openshift-ovn-kubernetes/ovnkube-node-hds4g"
Jul 09 20:03:31.239524 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.124379    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4b4468fd-1708-487b-8d69-bb7d817e25de-host-run-ovn-kubernetes\") pod \"ovnkube-node-hds4g\" (UID: \"4b4468fd-1708-487b-8d69-bb7d817e25de\") " pod="openshift-ovn-kubernetes/ovnkube-node-hds4g"
Jul 09 20:03:31.239524 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.124402    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5204f9ff-c575-4889-91ff-3cf3ffcd032b-multus-cni-dir\") pod \"multus-rvxnk\" (UID: \"5204f9ff-c575-4889-91ff-3cf3ffcd032b\") " pod="openshift-multus/multus-rvxnk"
Jul 09 20:03:31.239524 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.124437    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5204f9ff-c575-4889-91ff-3cf3ffcd032b-hostroot\") pod \"multus-rvxnk\" (UID: \"5204f9ff-c575-4889-91ff-3cf3ffcd032b\") " pod="openshift-multus/multus-rvxnk"
Jul 09 20:03:31.239524 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.124454    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4b4468fd-1708-487b-8d69-bb7d817e25de-host-run-ovn-kubernetes\") pod \"ovnkube-node-hds4g\" (UID: \"4b4468fd-1708-487b-8d69-bb7d817e25de\") " pod="openshift-ovn-kubernetes/ovnkube-node-hds4g"
Jul 09 20:03:31.239524 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.124462    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a304eb99-5afb-4710-9e69-51ca0c329ba5-lib-modules\") pod \"tuned-87zd2\" (UID: \"a304eb99-5afb-4710-9e69-51ca0c329ba5\") " pod="openshift-cluster-node-tuning-operator/tuned-87zd2"
Jul 09 20:03:31.239524 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.124514    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/62b8139e-cc18-4428-9373-a64195b6cbdd-registration-dir\") pod \"aws-ebs-csi-driver-node-9nl8h\" (UID: \"62b8139e-cc18-4428-9373-a64195b6cbdd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nl8h"
Jul 09 20:03:31.239524 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.124555    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4b4468fd-1708-487b-8d69-bb7d817e25de-run-systemd\") pod \"ovnkube-node-hds4g\" (UID: \"4b4468fd-1708-487b-8d69-bb7d817e25de\") " pod="openshift-ovn-kubernetes/ovnkube-node-hds4g"
Jul 09 20:03:31.239524 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.124584    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4b4468fd-1708-487b-8d69-bb7d817e25de-host-slash\") pod \"ovnkube-node-hds4g\" (UID: \"4b4468fd-1708-487b-8d69-bb7d817e25de\") " pod="openshift-ovn-kubernetes/ovnkube-node-hds4g"
Jul 09 20:03:31.240451 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.124590    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a304eb99-5afb-4710-9e69-51ca0c329ba5-lib-modules\") pod \"tuned-87zd2\" (UID: \"a304eb99-5afb-4710-9e69-51ca0c329ba5\") " pod="openshift-cluster-node-tuning-operator/tuned-87zd2"
Jul 09 20:03:31.240451 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.124602    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/62b8139e-cc18-4428-9373-a64195b6cbdd-registration-dir\") pod \"aws-ebs-csi-driver-node-9nl8h\" (UID: \"62b8139e-cc18-4428-9373-a64195b6cbdd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nl8h"
Jul 09 20:03:31.240451 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.124610    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4b4468fd-1708-487b-8d69-bb7d817e25de-run-systemd\") pod \"ovnkube-node-hds4g\" (UID: \"4b4468fd-1708-487b-8d69-bb7d817e25de\") " pod="openshift-ovn-kubernetes/ovnkube-node-hds4g"
Jul 09 20:03:31.240451 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.124601    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5204f9ff-c575-4889-91ff-3cf3ffcd032b-os-release\") pod \"multus-rvxnk\" (UID: \"5204f9ff-c575-4889-91ff-3cf3ffcd032b\") " pod="openshift-multus/multus-rvxnk"
Jul 09 20:03:31.240451 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.124645    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5204f9ff-c575-4889-91ff-3cf3ffcd032b-multus-socket-dir-parent\") pod \"multus-rvxnk\" (UID: \"5204f9ff-c575-4889-91ff-3cf3ffcd032b\") " pod="openshift-multus/multus-rvxnk"
Jul 09 20:03:31.240451 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.124674    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/dddbfeb6-339f-4aa6-b74f-26dfe9438a2c-hosts-file\") pod \"node-resolver-8r78k\" (UID: \"dddbfeb6-339f-4aa6-b74f-26dfe9438a2c\") " pod="openshift-dns/node-resolver-8r78k"
Jul 09 20:03:31.240451 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.124694    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dddbfeb6-339f-4aa6-b74f-26dfe9438a2c-tmp-dir\") pod \"node-resolver-8r78k\" (UID: \"dddbfeb6-339f-4aa6-b74f-26dfe9438a2c\") " pod="openshift-dns/node-resolver-8r78k"
Jul 09 20:03:31.240451 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.124713    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4b4468fd-1708-487b-8d69-bb7d817e25de-run-ovn\") pod \"ovnkube-node-hds4g\" (UID: \"4b4468fd-1708-487b-8d69-bb7d817e25de\") " pod="openshift-ovn-kubernetes/ovnkube-node-hds4g"
Jul 09 20:03:31.240451 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.124730    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4b4468fd-1708-487b-8d69-bb7d817e25de-ovnkube-script-lib\") pod \"ovnkube-node-hds4g\" (UID: \"4b4468fd-1708-487b-8d69-bb7d817e25de\") " pod="openshift-ovn-kubernetes/ovnkube-node-hds4g"
Jul 09 20:03:31.240451 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.124745    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5204f9ff-c575-4889-91ff-3cf3ffcd032b-cnibin\") pod \"multus-rvxnk\" (UID: \"5204f9ff-c575-4889-91ff-3cf3ffcd032b\") " pod="openshift-multus/multus-rvxnk"
Jul 09 20:03:31.240451 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.124761    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a304eb99-5afb-4710-9e69-51ca0c329ba5-sys\") pod \"tuned-87zd2\" (UID: \"a304eb99-5afb-4710-9e69-51ca0c329ba5\") " pod="openshift-cluster-node-tuning-operator/tuned-87zd2"
Jul 09 20:03:31.240451 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.124789    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4b4468fd-1708-487b-8d69-bb7d817e25de-run-openvswitch\") pod \"ovnkube-node-hds4g\" (UID: \"4b4468fd-1708-487b-8d69-bb7d817e25de\") " pod="openshift-ovn-kubernetes/ovnkube-node-hds4g"
Jul 09 20:03:31.240451 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.124812    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4b4468fd-1708-487b-8d69-bb7d817e25de-run-ovn\") pod \"ovnkube-node-hds4g\" (UID: \"4b4468fd-1708-487b-8d69-bb7d817e25de\") " pod="openshift-ovn-kubernetes/ovnkube-node-hds4g"
Jul 09 20:03:31.240451 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.124827    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4b4468fd-1708-487b-8d69-bb7d817e25de-run-openvswitch\") pod \"ovnkube-node-hds4g\" (UID: \"4b4468fd-1708-487b-8d69-bb7d817e25de\") " pod="openshift-ovn-kubernetes/ovnkube-node-hds4g"
Jul 09 20:03:31.240451 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.124857    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/13454dc6-77a2-4dc9-adc2-81791d1cd463-kubelet-config\") pod \"global-pull-secret-syncer-dccxz\" (UID: \"13454dc6-77a2-4dc9-adc2-81791d1cd463\") " pod="kube-system/global-pull-secret-syncer-dccxz"
Jul 09 20:03:31.241455 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.124869    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a304eb99-5afb-4710-9e69-51ca0c329ba5-sys\") pod \"tuned-87zd2\" (UID: \"a304eb99-5afb-4710-9e69-51ca0c329ba5\") " pod="openshift-cluster-node-tuning-operator/tuned-87zd2"
Jul 09 20:03:31.241455 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.124883    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d592535b-5457-48d0-ae48-caed8efc04fc-system-cni-dir\") pod \"multus-additional-cni-plugins-dz4cq\" (UID: \"d592535b-5457-48d0-ae48-caed8efc04fc\") " pod="openshift-multus/multus-additional-cni-plugins-dz4cq"
Jul 09 20:03:31.241455 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.124907    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d592535b-5457-48d0-ae48-caed8efc04fc-cni-binary-copy\") pod \"multus-additional-cni-plugins-dz4cq\" (UID: \"d592535b-5457-48d0-ae48-caed8efc04fc\") " pod="openshift-multus/multus-additional-cni-plugins-dz4cq"
Jul 09 20:03:31.241455 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.124920    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/13454dc6-77a2-4dc9-adc2-81791d1cd463-kubelet-config\") pod \"global-pull-secret-syncer-dccxz\" (UID: \"13454dc6-77a2-4dc9-adc2-81791d1cd463\") " pod="kube-system/global-pull-secret-syncer-dccxz"
Jul 09 20:03:31.241455 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.124933    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d592535b-5457-48d0-ae48-caed8efc04fc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dz4cq\" (UID: \"d592535b-5457-48d0-ae48-caed8efc04fc\") " pod="openshift-multus/multus-additional-cni-plugins-dz4cq"
Jul 09 20:03:31.241455 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.124954    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4b4468fd-1708-487b-8d69-bb7d817e25de-systemd-units\") pod \"ovnkube-node-hds4g\" (UID: \"4b4468fd-1708-487b-8d69-bb7d817e25de\") " pod="openshift-ovn-kubernetes/ovnkube-node-hds4g"
Jul 09 20:03:31.241455 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.124976    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5204f9ff-c575-4889-91ff-3cf3ffcd032b-host-run-k8s-cni-cncf-io\") pod \"multus-rvxnk\" (UID: \"5204f9ff-c575-4889-91ff-3cf3ffcd032b\") " pod="openshift-multus/multus-rvxnk"
Jul 09 20:03:31.241455 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.125001    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5204f9ff-c575-4889-91ff-3cf3ffcd032b-etc-kubernetes\") pod \"multus-rvxnk\" (UID: \"5204f9ff-c575-4889-91ff-3cf3ffcd032b\") " pod="openshift-multus/multus-rvxnk"
Jul 09 20:03:31.241455 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.125015    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0a94e16d-ed38-41a6-a71d-49e96e5ae7c7-iptables-alerter-script\") pod \"iptables-alerter-j6mws\" (UID: \"0a94e16d-ed38-41a6-a71d-49e96e5ae7c7\") " pod="openshift-network-operator/iptables-alerter-j6mws"
Jul 09 20:03:31.241455 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.125020    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4b4468fd-1708-487b-8d69-bb7d817e25de-systemd-units\") pod \"ovnkube-node-hds4g\" (UID: \"4b4468fd-1708-487b-8d69-bb7d817e25de\") " pod="openshift-ovn-kubernetes/ovnkube-node-hds4g"
Jul 09 20:03:31.241455 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.125049    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a304eb99-5afb-4710-9e69-51ca0c329ba5-etc-sysctl-conf\") pod \"tuned-87zd2\" (UID: \"a304eb99-5afb-4710-9e69-51ca0c329ba5\") " pod="openshift-cluster-node-tuning-operator/tuned-87zd2"
Jul 09 20:03:31.241455 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.125065    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rst6\" (UniqueName: \"kubernetes.io/projected/785ecd20-a92f-4924-b2f3-663697381982-kube-api-access-2rst6\") pod \"node-ca-n2tvb\" (UID: \"785ecd20-a92f-4924-b2f3-663697381982\") " pod="openshift-image-registry/node-ca-n2tvb"
Jul 09 20:03:31.241455 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.125088    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/62b8139e-cc18-4428-9373-a64195b6cbdd-device-dir\") pod \"aws-ebs-csi-driver-node-9nl8h\" (UID: \"62b8139e-cc18-4428-9373-a64195b6cbdd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nl8h"
Jul 09 20:03:31.241455 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.125104    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8zbm\" (UniqueName: \"kubernetes.io/projected/0a94e16d-ed38-41a6-a71d-49e96e5ae7c7-kube-api-access-v8zbm\") pod \"iptables-alerter-j6mws\" (UID: \"0a94e16d-ed38-41a6-a71d-49e96e5ae7c7\") " pod="openshift-network-operator/iptables-alerter-j6mws"
Jul 09 20:03:31.242417 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.125121    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/62b8139e-cc18-4428-9373-a64195b6cbdd-socket-dir\") pod \"aws-ebs-csi-driver-node-9nl8h\" (UID: \"62b8139e-cc18-4428-9373-a64195b6cbdd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nl8h"
Jul 09 20:03:31.242417 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.125145    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a304eb99-5afb-4710-9e69-51ca0c329ba5-etc-sysconfig\") pod \"tuned-87zd2\" (UID: \"a304eb99-5afb-4710-9e69-51ca0c329ba5\") " pod="openshift-cluster-node-tuning-operator/tuned-87zd2"
Jul 09 20:03:31.242417 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.125164    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a304eb99-5afb-4710-9e69-51ca0c329ba5-etc-sysctl-conf\") pod \"tuned-87zd2\" (UID: \"a304eb99-5afb-4710-9e69-51ca0c329ba5\") " pod="openshift-cluster-node-tuning-operator/tuned-87zd2"
Jul 09 20:03:31.242417 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.125179    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/62b8139e-cc18-4428-9373-a64195b6cbdd-device-dir\") pod \"aws-ebs-csi-driver-node-9nl8h\" (UID: \"62b8139e-cc18-4428-9373-a64195b6cbdd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nl8h"
Jul 09 20:03:31.242417 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.125190    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/785ecd20-a92f-4924-b2f3-663697381982-host\") pod \"node-ca-n2tvb\" (UID: \"785ecd20-a92f-4924-b2f3-663697381982\") " pod="openshift-image-registry/node-ca-n2tvb"
Jul 09 20:03:31.242417 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.125154    2562 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 09 20:03:31.242417 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.125235    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/62b8139e-cc18-4428-9373-a64195b6cbdd-socket-dir\") pod \"aws-ebs-csi-driver-node-9nl8h\" (UID: \"62b8139e-cc18-4428-9373-a64195b6cbdd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nl8h"
Jul 09 20:03:31.242417 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.125237    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/785ecd20-a92f-4924-b2f3-663697381982-host\") pod \"node-ca-n2tvb\" (UID: \"785ecd20-a92f-4924-b2f3-663697381982\") " pod="openshift-image-registry/node-ca-n2tvb"
Jul 09 20:03:31.242417 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.125250    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a304eb99-5afb-4710-9e69-51ca0c329ba5-etc-sysconfig\") pod \"tuned-87zd2\" (UID: \"a304eb99-5afb-4710-9e69-51ca0c329ba5\") " pod="openshift-cluster-node-tuning-operator/tuned-87zd2"
Jul 09 20:03:31.242417 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.125291    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/785ecd20-a92f-4924-b2f3-663697381982-serviceca\") pod \"node-ca-n2tvb\" (UID: \"785ecd20-a92f-4924-b2f3-663697381982\") " pod="openshift-image-registry/node-ca-n2tvb"
Jul 09 20:03:31.242417 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.125263    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4b4468fd-1708-487b-8d69-bb7d817e25de-ovnkube-script-lib\") pod \"ovnkube-node-hds4g\" (UID: \"4b4468fd-1708-487b-8d69-bb7d817e25de\") " pod="openshift-ovn-kubernetes/ovnkube-node-hds4g"
Jul 09 20:03:31.242417 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.125311    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/00761abf-a23d-4f99-ba0f-bcb623060f98-agent-certs\") pod \"konnectivity-agent-vwttv\" (UID: \"00761abf-a23d-4f99-ba0f-bcb623060f98\") " pod="kube-system/konnectivity-agent-vwttv"
Jul 09 20:03:31.242417 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.125342    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gdm2\" (UniqueName: \"kubernetes.io/projected/62b8139e-cc18-4428-9373-a64195b6cbdd-kube-api-access-7gdm2\") pod \"aws-ebs-csi-driver-node-9nl8h\" (UID: \"62b8139e-cc18-4428-9373-a64195b6cbdd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nl8h"
Jul 09 20:03:31.242417 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.125360    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5204f9ff-c575-4889-91ff-3cf3ffcd032b-cni-binary-copy\") pod \"multus-rvxnk\" (UID: \"5204f9ff-c575-4889-91ff-3cf3ffcd032b\") " pod="openshift-multus/multus-rvxnk"
Jul 09 20:03:31.242417 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.125381    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xrrb\" (UniqueName: \"kubernetes.io/projected/5204f9ff-c575-4889-91ff-3cf3ffcd032b-kube-api-access-9xrrb\") pod \"multus-rvxnk\" (UID: \"5204f9ff-c575-4889-91ff-3cf3ffcd032b\") " pod="openshift-multus/multus-rvxnk"
Jul 09 20:03:31.243092 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.125518    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a304eb99-5afb-4710-9e69-51ca0c329ba5-etc-sysctl-d\") pod \"tuned-87zd2\" (UID: \"a304eb99-5afb-4710-9e69-51ca0c329ba5\") " pod="openshift-cluster-node-tuning-operator/tuned-87zd2"
Jul 09 20:03:31.243092 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.125557    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khzxc\" (UniqueName: \"kubernetes.io/projected/4fd1b369-9b81-495d-9608-9fc4d81295e2-kube-api-access-khzxc\") pod \"network-check-target-6b5z9\" (UID: \"4fd1b369-9b81-495d-9608-9fc4d81295e2\") " pod="openshift-network-diagnostics/network-check-target-6b5z9"
Jul 09 20:03:31.243092 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.125588    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4b4468fd-1708-487b-8d69-bb7d817e25de-var-lib-openvswitch\") pod \"ovnkube-node-hds4g\" (UID: \"4b4468fd-1708-487b-8d69-bb7d817e25de\") " pod="openshift-ovn-kubernetes/ovnkube-node-hds4g"
Jul 09 20:03:31.243092 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.125617    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4b4468fd-1708-487b-8d69-bb7d817e25de-host-cni-netd\") pod \"ovnkube-node-hds4g\" (UID: \"4b4468fd-1708-487b-8d69-bb7d817e25de\") " pod="openshift-ovn-kubernetes/ovnkube-node-hds4g"
Jul 09 20:03:31.243092 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.125646    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5204f9ff-c575-4889-91ff-3cf3ffcd032b-host-var-lib-cni-multus\") pod \"multus-rvxnk\" (UID: \"5204f9ff-c575-4889-91ff-3cf3ffcd032b\") " pod="openshift-multus/multus-rvxnk"
Jul 09 20:03:31.243092 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.125704    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a304eb99-5afb-4710-9e69-51ca0c329ba5-etc-sysctl-d\") pod \"tuned-87zd2\" (UID: \"a304eb99-5afb-4710-9e69-51ca0c329ba5\") " pod="openshift-cluster-node-tuning-operator/tuned-87zd2"
Jul 09 20:03:31.243092 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.125714    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4b4468fd-1708-487b-8d69-bb7d817e25de-host-cni-netd\") pod \"ovnkube-node-hds4g\" (UID: \"4b4468fd-1708-487b-8d69-bb7d817e25de\") " pod="openshift-ovn-kubernetes/ovnkube-node-hds4g"
Jul 09 20:03:31.243092 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.125704    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/785ecd20-a92f-4924-b2f3-663697381982-serviceca\") pod \"node-ca-n2tvb\" (UID: \"785ecd20-a92f-4924-b2f3-663697381982\") " pod="openshift-image-registry/node-ca-n2tvb"
Jul 09 20:03:31.243092 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.125696    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4b4468fd-1708-487b-8d69-bb7d817e25de-var-lib-openvswitch\") pod \"ovnkube-node-hds4g\" (UID: \"4b4468fd-1708-487b-8d69-bb7d817e25de\") " pod="openshift-ovn-kubernetes/ovnkube-node-hds4g"
Jul 09 20:03:31.243092 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.125741    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a304eb99-5afb-4710-9e69-51ca0c329ba5-tmp\") pod \"tuned-87zd2\" (UID: \"a304eb99-5afb-4710-9e69-51ca0c329ba5\") " pod="openshift-cluster-node-tuning-operator/tuned-87zd2"
Jul 09 20:03:31.243092 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.125765    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/62b8139e-cc18-4428-9373-a64195b6cbdd-sys-fs\") pod \"aws-ebs-csi-driver-node-9nl8h\" (UID: \"62b8139e-cc18-4428-9373-a64195b6cbdd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nl8h"
Jul 09 20:03:31.243092 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.125800    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4b4468fd-1708-487b-8d69-bb7d817e25de-ovnkube-config\") pod \"ovnkube-node-hds4g\" (UID: \"4b4468fd-1708-487b-8d69-bb7d817e25de\") " pod="openshift-ovn-kubernetes/ovnkube-node-hds4g"
Jul 09 20:03:31.243092 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.125828    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/13454dc6-77a2-4dc9-adc2-81791d1cd463-original-pull-secret\") pod \"global-pull-secret-syncer-dccxz\" (UID: \"13454dc6-77a2-4dc9-adc2-81791d1cd463\") " pod="kube-system/global-pull-secret-syncer-dccxz"
Jul 09 20:03:31.243092 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.125844    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/62b8139e-cc18-4428-9373-a64195b6cbdd-sys-fs\") pod \"aws-ebs-csi-driver-node-9nl8h\" (UID: \"62b8139e-cc18-4428-9373-a64195b6cbdd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nl8h"
Jul 09 20:03:31.243934 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.125859    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txq7t\" (UniqueName: \"kubernetes.io/projected/d592535b-5457-48d0-ae48-caed8efc04fc-kube-api-access-txq7t\") pod \"multus-additional-cni-plugins-dz4cq\" (UID: \"d592535b-5457-48d0-ae48-caed8efc04fc\") " pod="openshift-multus/multus-additional-cni-plugins-dz4cq"
Jul 09 20:03:31.243934 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.125887    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/62b8139e-cc18-4428-9373-a64195b6cbdd-etc-selinux\") pod \"aws-ebs-csi-driver-node-9nl8h\" (UID: \"62b8139e-cc18-4428-9373-a64195b6cbdd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nl8h"
Jul 09 20:03:31.243934 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:03:31.125976    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 20:03:31.243934 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.126004    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a304eb99-5afb-4710-9e69-51ca0c329ba5-var-lib-kubelet\") pod \"tuned-87zd2\" (UID: \"a304eb99-5afb-4710-9e69-51ca0c329ba5\") " pod="openshift-cluster-node-tuning-operator/tuned-87zd2"
Jul 09 20:03:31.243934 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.126027    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/62b8139e-cc18-4428-9373-a64195b6cbdd-etc-selinux\") pod \"aws-ebs-csi-driver-node-9nl8h\" (UID: \"62b8139e-cc18-4428-9373-a64195b6cbdd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nl8h"
Jul 09 20:03:31.243934 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:03:31.126046    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13454dc6-77a2-4dc9-adc2-81791d1cd463-original-pull-secret podName:13454dc6-77a2-4dc9-adc2-81791d1cd463 nodeName:}" failed. No retries permitted until 2026-07-09 20:03:31.626023716 +0000 UTC m=+2.043693525 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/13454dc6-77a2-4dc9-adc2-81791d1cd463-original-pull-secret") pod "global-pull-secret-syncer-dccxz" (UID: "13454dc6-77a2-4dc9-adc2-81791d1cd463") : object "kube-system"/"original-pull-secret" not registered
Jul 09 20:03:31.243934 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.126064    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4b4468fd-1708-487b-8d69-bb7d817e25de-log-socket\") pod \"ovnkube-node-hds4g\" (UID: \"4b4468fd-1708-487b-8d69-bb7d817e25de\") " pod="openshift-ovn-kubernetes/ovnkube-node-hds4g"
Jul 09 20:03:31.243934 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.126091    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4b4468fd-1708-487b-8d69-bb7d817e25de-host-cni-bin\") pod \"ovnkube-node-hds4g\" (UID: \"4b4468fd-1708-487b-8d69-bb7d817e25de\") " pod="openshift-ovn-kubernetes/ovnkube-node-hds4g"
Jul 09 20:03:31.243934 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.126071    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a304eb99-5afb-4710-9e69-51ca0c329ba5-var-lib-kubelet\") pod \"tuned-87zd2\" (UID: \"a304eb99-5afb-4710-9e69-51ca0c329ba5\") " pod="openshift-cluster-node-tuning-operator/tuned-87zd2"
Jul 09 20:03:31.243934 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.126125    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4b4468fd-1708-487b-8d69-bb7d817e25de-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hds4g\" (UID: \"4b4468fd-1708-487b-8d69-bb7d817e25de\") " pod="openshift-ovn-kubernetes/ovnkube-node-hds4g"
Jul 09 20:03:31.243934 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.126159    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d592535b-5457-48d0-ae48-caed8efc04fc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dz4cq\" (UID: \"d592535b-5457-48d0-ae48-caed8efc04fc\") " pod="openshift-multus/multus-additional-cni-plugins-dz4cq"
Jul 09 20:03:31.243934 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.126184    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a304eb99-5afb-4710-9e69-51ca0c329ba5-host\") pod \"tuned-87zd2\" (UID: \"a304eb99-5afb-4710-9e69-51ca0c329ba5\") " pod="openshift-cluster-node-tuning-operator/tuned-87zd2"
Jul 09 20:03:31.243934 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.126209    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/00761abf-a23d-4f99-ba0f-bcb623060f98-konnectivity-ca\") pod \"konnectivity-agent-vwttv\" (UID: \"00761abf-a23d-4f99-ba0f-bcb623060f98\") " pod="kube-system/konnectivity-agent-vwttv"
Jul 09 20:03:31.243934 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.126230    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4b4468fd-1708-487b-8d69-bb7d817e25de-host-kubelet\") pod \"ovnkube-node-hds4g\" (UID: \"4b4468fd-1708-487b-8d69-bb7d817e25de\") " pod="openshift-ovn-kubernetes/ovnkube-node-hds4g"
Jul 09 20:03:31.244814 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.126251    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4b4468fd-1708-487b-8d69-bb7d817e25de-env-overrides\") pod \"ovnkube-node-hds4g\" (UID: \"4b4468fd-1708-487b-8d69-bb7d817e25de\") " pod="openshift-ovn-kubernetes/ovnkube-node-hds4g"
Jul 09 20:03:31.244814 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.126270    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4b4468fd-1708-487b-8d69-bb7d817e25de-ovnkube-config\") pod \"ovnkube-node-hds4g\" (UID: \"4b4468fd-1708-487b-8d69-bb7d817e25de\") " pod="openshift-ovn-kubernetes/ovnkube-node-hds4g"
Jul 09 20:03:31.244814 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.126281    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4b4468fd-1708-487b-8d69-bb7d817e25de-log-socket\") pod \"ovnkube-node-hds4g\" (UID: \"4b4468fd-1708-487b-8d69-bb7d817e25de\") " pod="openshift-ovn-kubernetes/ovnkube-node-hds4g"
Jul 09 20:03:31.244814 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.126287    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/13454dc6-77a2-4dc9-adc2-81791d1cd463-dbus\") pod \"global-pull-secret-syncer-dccxz\" (UID: \"13454dc6-77a2-4dc9-adc2-81791d1cd463\") " pod="kube-system/global-pull-secret-syncer-dccxz"
Jul 09 20:03:31.244814 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.126314    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5204f9ff-c575-4889-91ff-3cf3ffcd032b-host-var-lib-kubelet\") pod \"multus-rvxnk\" (UID: \"5204f9ff-c575-4889-91ff-3cf3ffcd032b\") " pod="openshift-multus/multus-rvxnk"
Jul 09 20:03:31.244814 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.126358    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4b4468fd-1708-487b-8d69-bb7d817e25de-host-kubelet\") pod \"ovnkube-node-hds4g\" (UID: \"4b4468fd-1708-487b-8d69-bb7d817e25de\") " pod="openshift-ovn-kubernetes/ovnkube-node-hds4g"
Jul 09 20:03:31.244814 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.126360    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d592535b-5457-48d0-ae48-caed8efc04fc-cnibin\") pod \"multus-additional-cni-plugins-dz4cq\" (UID: \"d592535b-5457-48d0-ae48-caed8efc04fc\") " pod="openshift-multus/multus-additional-cni-plugins-dz4cq"
Jul 09 20:03:31.244814 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.126412    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a304eb99-5afb-4710-9e69-51ca0c329ba5-etc-kubernetes\") pod \"tuned-87zd2\" (UID: \"a304eb99-5afb-4710-9e69-51ca0c329ba5\") " pod="openshift-cluster-node-tuning-operator/tuned-87zd2"
Jul 09 20:03:31.244814 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.126399    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a304eb99-5afb-4710-9e69-51ca0c329ba5-host\") pod \"tuned-87zd2\" (UID: \"a304eb99-5afb-4710-9e69-51ca0c329ba5\") " pod="openshift-cluster-node-tuning-operator/tuned-87zd2"
Jul 09 20:03:31.244814 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.126440    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4b4468fd-1708-487b-8d69-bb7d817e25de-etc-openvswitch\") pod \"ovnkube-node-hds4g\" (UID: \"4b4468fd-1708-487b-8d69-bb7d817e25de\") " pod="openshift-ovn-kubernetes/ovnkube-node-hds4g"
Jul 09 20:03:31.244814 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.126489    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4b4468fd-1708-487b-8d69-bb7d817e25de-etc-openvswitch\") pod \"ovnkube-node-hds4g\" (UID: \"4b4468fd-1708-487b-8d69-bb7d817e25de\") " pod="openshift-ovn-kubernetes/ovnkube-node-hds4g"
Jul 09 20:03:31.244814 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.126532    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4b4468fd-1708-487b-8d69-bb7d817e25de-host-cni-bin\") pod \"ovnkube-node-hds4g\" (UID: \"4b4468fd-1708-487b-8d69-bb7d817e25de\") " pod="openshift-ovn-kubernetes/ovnkube-node-hds4g"
Jul 09 20:03:31.244814 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.126573    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4b4468fd-1708-487b-8d69-bb7d817e25de-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hds4g\" (UID: \"4b4468fd-1708-487b-8d69-bb7d817e25de\") " pod="openshift-ovn-kubernetes/ovnkube-node-hds4g"
Jul 09 20:03:31.244814 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.126736    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4b4468fd-1708-487b-8d69-bb7d817e25de-env-overrides\") pod \"ovnkube-node-hds4g\" (UID: \"4b4468fd-1708-487b-8d69-bb7d817e25de\") " pod="openshift-ovn-kubernetes/ovnkube-node-hds4g"
Jul 09 20:03:31.245918 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.126479    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4b4468fd-1708-487b-8d69-bb7d817e25de-ovn-node-metrics-cert\") pod \"ovnkube-node-hds4g\" (UID: \"4b4468fd-1708-487b-8d69-bb7d817e25de\") " pod="openshift-ovn-kubernetes/ovnkube-node-hds4g"
Jul 09 20:03:31.245918 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.126857    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5204f9ff-c575-4889-91ff-3cf3ffcd032b-multus-conf-dir\") pod \"multus-rvxnk\" (UID: \"5204f9ff-c575-4889-91ff-3cf3ffcd032b\") " pod="openshift-multus/multus-rvxnk"
Jul 09 20:03:31.245918 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.126890    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/00761abf-a23d-4f99-ba0f-bcb623060f98-konnectivity-ca\") pod \"konnectivity-agent-vwttv\" (UID: \"00761abf-a23d-4f99-ba0f-bcb623060f98\") " pod="kube-system/konnectivity-agent-vwttv"
Jul 09 20:03:31.245918 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.126884    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a304eb99-5afb-4710-9e69-51ca0c329ba5-etc-kubernetes\") pod \"tuned-87zd2\" (UID: \"a304eb99-5afb-4710-9e69-51ca0c329ba5\") " pod="openshift-cluster-node-tuning-operator/tuned-87zd2"
Jul 09 20:03:31.245918 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.126922    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14a1d22f-43ff-4eb8-a89d-920551b2ef1f-metrics-certs\") pod \"network-metrics-daemon-ksd8v\" (UID: \"14a1d22f-43ff-4eb8-a89d-920551b2ef1f\") " pod="openshift-multus/network-metrics-daemon-ksd8v"
Jul 09 20:03:31.245918 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.126942    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7qqq\" (UniqueName: \"kubernetes.io/projected/14a1d22f-43ff-4eb8-a89d-920551b2ef1f-kube-api-access-n7qqq\") pod \"network-metrics-daemon-ksd8v\" (UID: \"14a1d22f-43ff-4eb8-a89d-920551b2ef1f\") " pod="openshift-multus/network-metrics-daemon-ksd8v"
Jul 09 20:03:31.245918 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.126977    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nctvg\" (UniqueName: \"kubernetes.io/projected/dddbfeb6-339f-4aa6-b74f-26dfe9438a2c-kube-api-access-nctvg\") pod \"node-resolver-8r78k\" (UID: \"dddbfeb6-339f-4aa6-b74f-26dfe9438a2c\") " pod="openshift-dns/node-resolver-8r78k"
Jul 09 20:03:31.245918 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.127005    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a304eb99-5afb-4710-9e69-51ca0c329ba5-run\") pod \"tuned-87zd2\" (UID: \"a304eb99-5afb-4710-9e69-51ca0c329ba5\") " pod="openshift-cluster-node-tuning-operator/tuned-87zd2"
Jul 09 20:03:31.245918 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.127034    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4b4468fd-1708-487b-8d69-bb7d817e25de-node-log\") pod \"ovnkube-node-hds4g\" (UID: \"4b4468fd-1708-487b-8d69-bb7d817e25de\") " pod="openshift-ovn-kubernetes/ovnkube-node-hds4g"
Jul 09 20:03:31.245918 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.127053    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a304eb99-5afb-4710-9e69-51ca0c329ba5-run\") pod \"tuned-87zd2\" (UID: \"a304eb99-5afb-4710-9e69-51ca0c329ba5\") " pod="openshift-cluster-node-tuning-operator/tuned-87zd2"
Jul 09 20:03:31.245918 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.127033    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/13454dc6-77a2-4dc9-adc2-81791d1cd463-dbus\") pod \"global-pull-secret-syncer-dccxz\" (UID: \"13454dc6-77a2-4dc9-adc2-81791d1cd463\") " pod="kube-system/global-pull-secret-syncer-dccxz"
Jul 09 20:03:31.245918 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.127072    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5204f9ff-c575-4889-91ff-3cf3ffcd032b-host-var-lib-cni-bin\") pod \"multus-rvxnk\" (UID: \"5204f9ff-c575-4889-91ff-3cf3ffcd032b\") " pod="openshift-multus/multus-rvxnk"
Jul 09 20:03:31.245918 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.127077    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4b4468fd-1708-487b-8d69-bb7d817e25de-node-log\") pod \"ovnkube-node-hds4g\" (UID: \"4b4468fd-1708-487b-8d69-bb7d817e25de\") " pod="openshift-ovn-kubernetes/ovnkube-node-hds4g"
Jul 09 20:03:31.245918 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.127099    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5204f9ff-c575-4889-91ff-3cf3ffcd032b-host-run-multus-certs\") pod \"multus-rvxnk\" (UID: \"5204f9ff-c575-4889-91ff-3cf3ffcd032b\") " pod="openshift-multus/multus-rvxnk"
Jul 09 20:03:31.246822 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.128699    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a304eb99-5afb-4710-9e69-51ca0c329ba5-tmp\") pod \"tuned-87zd2\" (UID: \"a304eb99-5afb-4710-9e69-51ca0c329ba5\") " pod="openshift-cluster-node-tuning-operator/tuned-87zd2"
Jul 09 20:03:31.246822 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.128748    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a304eb99-5afb-4710-9e69-51ca0c329ba5-etc-tuned\") pod \"tuned-87zd2\" (UID: \"a304eb99-5afb-4710-9e69-51ca0c329ba5\") " pod="openshift-cluster-node-tuning-operator/tuned-87zd2"
Jul 09 20:03:31.246822 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.128917    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4b4468fd-1708-487b-8d69-bb7d817e25de-ovn-node-metrics-cert\") pod \"ovnkube-node-hds4g\" (UID: \"4b4468fd-1708-487b-8d69-bb7d817e25de\") " pod="openshift-ovn-kubernetes/ovnkube-node-hds4g"
Jul 09 20:03:31.246822 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.129030    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/00761abf-a23d-4f99-ba0f-bcb623060f98-agent-certs\") pod \"konnectivity-agent-vwttv\" (UID: \"00761abf-a23d-4f99-ba0f-bcb623060f98\") " pod="kube-system/konnectivity-agent-vwttv"
Jul 09 20:03:31.246822 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:03:31.132613    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 20:03:31.246822 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:03:31.132639    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 20:03:31.246822 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:03:31.132653    2562 projected.go:196] Error preparing data for projected volume kube-api-access-khzxc for pod openshift-network-diagnostics/network-check-target-6b5z9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 20:03:31.246822 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:03:31.132730    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4fd1b369-9b81-495d-9608-9fc4d81295e2-kube-api-access-khzxc podName:4fd1b369-9b81-495d-9608-9fc4d81295e2 nodeName:}" failed. No retries permitted until 2026-07-09 20:03:31.632711633 +0000 UTC m=+2.050381446 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-khzxc" (UniqueName: "kubernetes.io/projected/4fd1b369-9b81-495d-9608-9fc4d81295e2-kube-api-access-khzxc") pod "network-check-target-6b5z9" (UID: "4fd1b369-9b81-495d-9608-9fc4d81295e2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 20:03:31.246822 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.133599    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rk6gw\" (UniqueName: \"kubernetes.io/projected/a304eb99-5afb-4710-9e69-51ca0c329ba5-kube-api-access-rk6gw\") pod \"tuned-87zd2\" (UID: \"a304eb99-5afb-4710-9e69-51ca0c329ba5\") " pod="openshift-cluster-node-tuning-operator/tuned-87zd2"
Jul 09 20:03:31.246822 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.133869    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gdr2\" (UniqueName: \"kubernetes.io/projected/4b4468fd-1708-487b-8d69-bb7d817e25de-kube-api-access-8gdr2\") pod \"ovnkube-node-hds4g\" (UID: \"4b4468fd-1708-487b-8d69-bb7d817e25de\") " pod="openshift-ovn-kubernetes/ovnkube-node-hds4g"
Jul 09 20:03:31.246822 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.133953    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8zbm\" (UniqueName: \"kubernetes.io/projected/0a94e16d-ed38-41a6-a71d-49e96e5ae7c7-kube-api-access-v8zbm\") pod \"iptables-alerter-j6mws\" (UID: \"0a94e16d-ed38-41a6-a71d-49e96e5ae7c7\") " pod="openshift-network-operator/iptables-alerter-j6mws"
Jul 09 20:03:31.246822 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.134521    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rst6\" (UniqueName: \"kubernetes.io/projected/785ecd20-a92f-4924-b2f3-663697381982-kube-api-access-2rst6\") pod \"node-ca-n2tvb\" (UID: \"785ecd20-a92f-4924-b2f3-663697381982\") " pod="openshift-image-registry/node-ca-n2tvb"
Jul 09 20:03:31.246822 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.135557    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gdm2\" (UniqueName: \"kubernetes.io/projected/62b8139e-cc18-4428-9373-a64195b6cbdd-kube-api-access-7gdm2\") pod \"aws-ebs-csi-driver-node-9nl8h\" (UID: \"62b8139e-cc18-4428-9373-a64195b6cbdd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nl8h"
Jul 09 20:03:31.246822 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.227938    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5204f9ff-c575-4889-91ff-3cf3ffcd032b-cni-binary-copy\") pod \"multus-rvxnk\" (UID: \"5204f9ff-c575-4889-91ff-3cf3ffcd032b\") " pod="openshift-multus/multus-rvxnk"
Jul 09 20:03:31.247974 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.227974    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xrrb\" (UniqueName: \"kubernetes.io/projected/5204f9ff-c575-4889-91ff-3cf3ffcd032b-kube-api-access-9xrrb\") pod \"multus-rvxnk\" (UID: \"5204f9ff-c575-4889-91ff-3cf3ffcd032b\") " pod="openshift-multus/multus-rvxnk"
Jul 09 20:03:31.247974 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.228012    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5204f9ff-c575-4889-91ff-3cf3ffcd032b-host-var-lib-cni-multus\") pod \"multus-rvxnk\" (UID: \"5204f9ff-c575-4889-91ff-3cf3ffcd032b\") " pod="openshift-multus/multus-rvxnk"
Jul 09 20:03:31.247974 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.228040    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txq7t\" (UniqueName: \"kubernetes.io/projected/d592535b-5457-48d0-ae48-caed8efc04fc-kube-api-access-txq7t\") pod \"multus-additional-cni-plugins-dz4cq\" (UID: \"d592535b-5457-48d0-ae48-caed8efc04fc\") " pod="openshift-multus/multus-additional-cni-plugins-dz4cq"
Jul 09 20:03:31.247974 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.228077    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d592535b-5457-48d0-ae48-caed8efc04fc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dz4cq\" (UID: \"d592535b-5457-48d0-ae48-caed8efc04fc\") " pod="openshift-multus/multus-additional-cni-plugins-dz4cq"
Jul 09 20:03:31.247974 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.228108    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5204f9ff-c575-4889-91ff-3cf3ffcd032b-host-var-lib-kubelet\") pod \"multus-rvxnk\" (UID: \"5204f9ff-c575-4889-91ff-3cf3ffcd032b\") " pod="openshift-multus/multus-rvxnk"
Jul 09 20:03:31.247974 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.228507    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d592535b-5457-48d0-ae48-caed8efc04fc-cnibin\") pod \"multus-additional-cni-plugins-dz4cq\" (UID: \"d592535b-5457-48d0-ae48-caed8efc04fc\") " pod="openshift-multus/multus-additional-cni-plugins-dz4cq"
Jul 09 20:03:31.247974 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.228544    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5204f9ff-c575-4889-91ff-3cf3ffcd032b-multus-conf-dir\") pod \"multus-rvxnk\" (UID: \"5204f9ff-c575-4889-91ff-3cf3ffcd032b\") " pod="openshift-multus/multus-rvxnk"
Jul 09 20:03:31.247974 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.228269    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5204f9ff-c575-4889-91ff-3cf3ffcd032b-host-var-lib-kubelet\") pod \"multus-rvxnk\" (UID: \"5204f9ff-c575-4889-91ff-3cf3ffcd032b\") " pod="openshift-multus/multus-rvxnk"
Jul 09 20:03:31.247974 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.228144    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5204f9ff-c575-4889-91ff-3cf3ffcd032b-host-var-lib-cni-multus\") pod \"multus-rvxnk\" (UID: \"5204f9ff-c575-4889-91ff-3cf3ffcd032b\") " pod="openshift-multus/multus-rvxnk"
Jul 09 20:03:31.247974 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.228594    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d592535b-5457-48d0-ae48-caed8efc04fc-cnibin\") pod \"multus-additional-cni-plugins-dz4cq\" (UID: \"d592535b-5457-48d0-ae48-caed8efc04fc\") " pod="openshift-multus/multus-additional-cni-plugins-dz4cq"
Jul 09 20:03:31.247974 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.228599    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5204f9ff-c575-4889-91ff-3cf3ffcd032b-multus-conf-dir\") pod \"multus-rvxnk\" (UID: \"5204f9ff-c575-4889-91ff-3cf3ffcd032b\") " pod="openshift-multus/multus-rvxnk"
Jul 09 20:03:31.247974 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.228628    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14a1d22f-43ff-4eb8-a89d-920551b2ef1f-metrics-certs\") pod \"network-metrics-daemon-ksd8v\" (UID: \"14a1d22f-43ff-4eb8-a89d-920551b2ef1f\") " pod="openshift-multus/network-metrics-daemon-ksd8v"
Jul 09 20:03:31.247974 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.228656    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7qqq\" (UniqueName: \"kubernetes.io/projected/14a1d22f-43ff-4eb8-a89d-920551b2ef1f-kube-api-access-n7qqq\") pod \"network-metrics-daemon-ksd8v\" (UID: \"14a1d22f-43ff-4eb8-a89d-920551b2ef1f\") " pod="openshift-multus/network-metrics-daemon-ksd8v"
Jul 09 20:03:31.247974 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.228684    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nctvg\" (UniqueName: \"kubernetes.io/projected/dddbfeb6-339f-4aa6-b74f-26dfe9438a2c-kube-api-access-nctvg\") pod \"node-resolver-8r78k\" (UID: \"dddbfeb6-339f-4aa6-b74f-26dfe9438a2c\") " pod="openshift-dns/node-resolver-8r78k"
Jul 09 20:03:31.247974 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:03:31.228701    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 20:03:31.248621 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.228713    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5204f9ff-c575-4889-91ff-3cf3ffcd032b-host-var-lib-cni-bin\") pod \"multus-rvxnk\" (UID: \"5204f9ff-c575-4889-91ff-3cf3ffcd032b\") " pod="openshift-multus/multus-rvxnk"
Jul 09 20:03:31.248621 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.228743    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5204f9ff-c575-4889-91ff-3cf3ffcd032b-host-run-multus-certs\") pod \"multus-rvxnk\" (UID: \"5204f9ff-c575-4889-91ff-3cf3ffcd032b\") " pod="openshift-multus/multus-rvxnk"
Jul 09 20:03:31.248621 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:03:31.228775    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/14a1d22f-43ff-4eb8-a89d-920551b2ef1f-metrics-certs podName:14a1d22f-43ff-4eb8-a89d-920551b2ef1f nodeName:}" failed. No retries permitted until 2026-07-09 20:03:31.728752951 +0000 UTC m=+2.146422763 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/14a1d22f-43ff-4eb8-a89d-920551b2ef1f-metrics-certs") pod "network-metrics-daemon-ksd8v" (UID: "14a1d22f-43ff-4eb8-a89d-920551b2ef1f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 20:03:31.248621 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.228788    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5204f9ff-c575-4889-91ff-3cf3ffcd032b-host-run-multus-certs\") pod \"multus-rvxnk\" (UID: \"5204f9ff-c575-4889-91ff-3cf3ffcd032b\") " pod="openshift-multus/multus-rvxnk"
Jul 09 20:03:31.248621 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.228788    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5204f9ff-c575-4889-91ff-3cf3ffcd032b-host-var-lib-cni-bin\") pod \"multus-rvxnk\" (UID: \"5204f9ff-c575-4889-91ff-3cf3ffcd032b\") " pod="openshift-multus/multus-rvxnk"
Jul 09 20:03:31.248621 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.228811    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d592535b-5457-48d0-ae48-caed8efc04fc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dz4cq\" (UID: \"d592535b-5457-48d0-ae48-caed8efc04fc\") " pod="openshift-multus/multus-additional-cni-plugins-dz4cq"
Jul 09 20:03:31.248621 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.228787    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d592535b-5457-48d0-ae48-caed8efc04fc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dz4cq\" (UID: \"d592535b-5457-48d0-ae48-caed8efc04fc\") " pod="openshift-multus/multus-additional-cni-plugins-dz4cq"
Jul 09 20:03:31.248621 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.228857    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5204f9ff-c575-4889-91ff-3cf3ffcd032b-system-cni-dir\") pod \"multus-rvxnk\" (UID: \"5204f9ff-c575-4889-91ff-3cf3ffcd032b\") " pod="openshift-multus/multus-rvxnk"
Jul 09 20:03:31.248621 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.228884    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5204f9ff-c575-4889-91ff-3cf3ffcd032b-multus-daemon-config\") pod \"multus-rvxnk\" (UID: \"5204f9ff-c575-4889-91ff-3cf3ffcd032b\") " pod="openshift-multus/multus-rvxnk"
Jul 09 20:03:31.248621 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.228928    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5204f9ff-c575-4889-91ff-3cf3ffcd032b-host-run-netns\") pod \"multus-rvxnk\" (UID: \"5204f9ff-c575-4889-91ff-3cf3ffcd032b\") " pod="openshift-multus/multus-rvxnk"
Jul 09 20:03:31.248621 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.228939    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5204f9ff-c575-4889-91ff-3cf3ffcd032b-system-cni-dir\") pod \"multus-rvxnk\" (UID: \"5204f9ff-c575-4889-91ff-3cf3ffcd032b\") " pod="openshift-multus/multus-rvxnk"
Jul 09 20:03:31.248621 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.228952    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d592535b-5457-48d0-ae48-caed8efc04fc-os-release\") pod \"multus-additional-cni-plugins-dz4cq\" (UID: \"d592535b-5457-48d0-ae48-caed8efc04fc\") " pod="openshift-multus/multus-additional-cni-plugins-dz4cq"
Jul 09 20:03:31.248621 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.228978    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d592535b-5457-48d0-ae48-caed8efc04fc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dz4cq\" (UID: \"d592535b-5457-48d0-ae48-caed8efc04fc\") " pod="openshift-multus/multus-additional-cni-plugins-dz4cq"
Jul 09 20:03:31.248621 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.228983    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5204f9ff-c575-4889-91ff-3cf3ffcd032b-multus-cni-dir\") pod \"multus-rvxnk\" (UID: \"5204f9ff-c575-4889-91ff-3cf3ffcd032b\") " pod="openshift-multus/multus-rvxnk"
Jul 09 20:03:31.249956 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.228990    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5204f9ff-c575-4889-91ff-3cf3ffcd032b-host-run-netns\") pod \"multus-rvxnk\" (UID: \"5204f9ff-c575-4889-91ff-3cf3ffcd032b\") " pod="openshift-multus/multus-rvxnk"
Jul 09 20:03:31.249956 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.229008    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5204f9ff-c575-4889-91ff-3cf3ffcd032b-hostroot\") pod \"multus-rvxnk\" (UID: \"5204f9ff-c575-4889-91ff-3cf3ffcd032b\") " pod="openshift-multus/multus-rvxnk"
Jul 09 20:03:31.249956 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.229041    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5204f9ff-c575-4889-91ff-3cf3ffcd032b-os-release\") pod \"multus-rvxnk\" (UID: \"5204f9ff-c575-4889-91ff-3cf3ffcd032b\") " pod="openshift-multus/multus-rvxnk"
Jul 09 20:03:31.249956 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.229048    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d592535b-5457-48d0-ae48-caed8efc04fc-os-release\") pod \"multus-additional-cni-plugins-dz4cq\" (UID: \"d592535b-5457-48d0-ae48-caed8efc04fc\") " pod="openshift-multus/multus-additional-cni-plugins-dz4cq"
Jul 09 20:03:31.249956 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.229052    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5204f9ff-c575-4889-91ff-3cf3ffcd032b-multus-cni-dir\") pod \"multus-rvxnk\" (UID: \"5204f9ff-c575-4889-91ff-3cf3ffcd032b\") " pod="openshift-multus/multus-rvxnk"
Jul 09 20:03:31.249956 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.229067    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5204f9ff-c575-4889-91ff-3cf3ffcd032b-multus-socket-dir-parent\") pod \"multus-rvxnk\" (UID: \"5204f9ff-c575-4889-91ff-3cf3ffcd032b\") " pod="openshift-multus/multus-rvxnk"
Jul 09 20:03:31.249956 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.229083    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5204f9ff-c575-4889-91ff-3cf3ffcd032b-hostroot\") pod \"multus-rvxnk\" (UID: \"5204f9ff-c575-4889-91ff-3cf3ffcd032b\") " pod="openshift-multus/multus-rvxnk"
Jul 09 20:03:31.249956 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.229095    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/dddbfeb6-339f-4aa6-b74f-26dfe9438a2c-hosts-file\") pod \"node-resolver-8r78k\" (UID: \"dddbfeb6-339f-4aa6-b74f-26dfe9438a2c\") " pod="openshift-dns/node-resolver-8r78k"
Jul 09 20:03:31.249956 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.229107    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5204f9ff-c575-4889-91ff-3cf3ffcd032b-os-release\") pod \"multus-rvxnk\" (UID: \"5204f9ff-c575-4889-91ff-3cf3ffcd032b\") " pod="openshift-multus/multus-rvxnk"
Jul 09 20:03:31.249956 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.229116    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5204f9ff-c575-4889-91ff-3cf3ffcd032b-multus-socket-dir-parent\") pod \"multus-rvxnk\" (UID: \"5204f9ff-c575-4889-91ff-3cf3ffcd032b\") " pod="openshift-multus/multus-rvxnk"
Jul 09 20:03:31.249956 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.229133    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dddbfeb6-339f-4aa6-b74f-26dfe9438a2c-tmp-dir\") pod \"node-resolver-8r78k\" (UID: \"dddbfeb6-339f-4aa6-b74f-26dfe9438a2c\") " pod="openshift-dns/node-resolver-8r78k"
Jul 09 20:03:31.249956 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.229145    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/dddbfeb6-339f-4aa6-b74f-26dfe9438a2c-hosts-file\") pod \"node-resolver-8r78k\" (UID: \"dddbfeb6-339f-4aa6-b74f-26dfe9438a2c\") " pod="openshift-dns/node-resolver-8r78k"
Jul 09 20:03:31.249956 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.229164    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5204f9ff-c575-4889-91ff-3cf3ffcd032b-cnibin\") pod \"multus-rvxnk\" (UID: \"5204f9ff-c575-4889-91ff-3cf3ffcd032b\") " pod="openshift-multus/multus-rvxnk"
Jul 09 20:03:31.249956 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.229195    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d592535b-5457-48d0-ae48-caed8efc04fc-system-cni-dir\") pod \"multus-additional-cni-plugins-dz4cq\" (UID: \"d592535b-5457-48d0-ae48-caed8efc04fc\") " pod="openshift-multus/multus-additional-cni-plugins-dz4cq"
Jul 09 20:03:31.249956 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.229230    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5204f9ff-c575-4889-91ff-3cf3ffcd032b-cni-binary-copy\") pod \"multus-rvxnk\" (UID: \"5204f9ff-c575-4889-91ff-3cf3ffcd032b\") " pod="openshift-multus/multus-rvxnk"
Jul 09 20:03:31.250624 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.229244    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d592535b-5457-48d0-ae48-caed8efc04fc-cni-binary-copy\") pod \"multus-additional-cni-plugins-dz4cq\" (UID: \"d592535b-5457-48d0-ae48-caed8efc04fc\") " pod="openshift-multus/multus-additional-cni-plugins-dz4cq"
Jul 09 20:03:31.250624 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.229272    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d592535b-5457-48d0-ae48-caed8efc04fc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dz4cq\" (UID: \"d592535b-5457-48d0-ae48-caed8efc04fc\") " pod="openshift-multus/multus-additional-cni-plugins-dz4cq"
Jul 09 20:03:31.250624 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.229276    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5204f9ff-c575-4889-91ff-3cf3ffcd032b-cnibin\") pod \"multus-rvxnk\" (UID: \"5204f9ff-c575-4889-91ff-3cf3ffcd032b\") " pod="openshift-multus/multus-rvxnk"
Jul 09 20:03:31.250624 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.229297    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d592535b-5457-48d0-ae48-caed8efc04fc-system-cni-dir\") pod \"multus-additional-cni-plugins-dz4cq\" (UID: \"d592535b-5457-48d0-ae48-caed8efc04fc\") " pod="openshift-multus/multus-additional-cni-plugins-dz4cq"
Jul 09 20:03:31.250624 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.229325    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5204f9ff-c575-4889-91ff-3cf3ffcd032b-host-run-k8s-cni-cncf-io\") pod \"multus-rvxnk\" (UID: \"5204f9ff-c575-4889-91ff-3cf3ffcd032b\") " pod="openshift-multus/multus-rvxnk"
Jul 09 20:03:31.250624 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.229398    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5204f9ff-c575-4889-91ff-3cf3ffcd032b-host-run-k8s-cni-cncf-io\") pod \"multus-rvxnk\" (UID: \"5204f9ff-c575-4889-91ff-3cf3ffcd032b\") " pod="openshift-multus/multus-rvxnk"
Jul 09 20:03:31.250624 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.229428    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5204f9ff-c575-4889-91ff-3cf3ffcd032b-etc-kubernetes\") pod \"multus-rvxnk\" (UID: \"5204f9ff-c575-4889-91ff-3cf3ffcd032b\") " pod="openshift-multus/multus-rvxnk"
Jul 09 20:03:31.250624 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.229508    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5204f9ff-c575-4889-91ff-3cf3ffcd032b-etc-kubernetes\") pod \"multus-rvxnk\" (UID: \"5204f9ff-c575-4889-91ff-3cf3ffcd032b\") " pod="openshift-multus/multus-rvxnk"
Jul 09 20:03:31.250624 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.229511    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5204f9ff-c575-4889-91ff-3cf3ffcd032b-multus-daemon-config\") pod \"multus-rvxnk\" (UID: \"5204f9ff-c575-4889-91ff-3cf3ffcd032b\") " pod="openshift-multus/multus-rvxnk"
Jul 09 20:03:31.250624 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.229509    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dddbfeb6-339f-4aa6-b74f-26dfe9438a2c-tmp-dir\") pod \"node-resolver-8r78k\" (UID: \"dddbfeb6-339f-4aa6-b74f-26dfe9438a2c\") " pod="openshift-dns/node-resolver-8r78k"
Jul 09 20:03:31.250624 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.229774    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d592535b-5457-48d0-ae48-caed8efc04fc-cni-binary-copy\") pod \"multus-additional-cni-plugins-dz4cq\" (UID: \"d592535b-5457-48d0-ae48-caed8efc04fc\") " pod="openshift-multus/multus-additional-cni-plugins-dz4cq"
Jul 09 20:03:31.250624 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.229794    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d592535b-5457-48d0-ae48-caed8efc04fc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dz4cq\" (UID: \"d592535b-5457-48d0-ae48-caed8efc04fc\") " pod="openshift-multus/multus-additional-cni-plugins-dz4cq"
Jul 09 20:03:31.250624 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.240424    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-txq7t\" (UniqueName: \"kubernetes.io/projected/d592535b-5457-48d0-ae48-caed8efc04fc-kube-api-access-txq7t\") pod \"multus-additional-cni-plugins-dz4cq\" (UID: \"d592535b-5457-48d0-ae48-caed8efc04fc\") " pod="openshift-multus/multus-additional-cni-plugins-dz4cq"
Jul 09 20:03:31.250624 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.241104    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xrrb\" (UniqueName: \"kubernetes.io/projected/5204f9ff-c575-4889-91ff-3cf3ffcd032b-kube-api-access-9xrrb\") pod \"multus-rvxnk\" (UID: \"5204f9ff-c575-4889-91ff-3cf3ffcd032b\") " pod="openshift-multus/multus-rvxnk"
Jul 09 20:03:31.251165 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.242011    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nctvg\" (UniqueName: \"kubernetes.io/projected/dddbfeb6-339f-4aa6-b74f-26dfe9438a2c-kube-api-access-nctvg\") pod \"node-resolver-8r78k\" (UID: \"dddbfeb6-339f-4aa6-b74f-26dfe9438a2c\") " pod="openshift-dns/node-resolver-8r78k"
Jul 09 20:03:31.251165 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.242082    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7qqq\" (UniqueName: \"kubernetes.io/projected/14a1d22f-43ff-4eb8-a89d-920551b2ef1f-kube-api-access-n7qqq\") pod \"network-metrics-daemon-ksd8v\" (UID: \"14a1d22f-43ff-4eb8-a89d-920551b2ef1f\") " pod="openshift-multus/network-metrics-daemon-ksd8v"
Jul 09 20:03:31.370504 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:31.309473    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61c3c5dbc96e6f9e2e03d9211834c2ca.slice/crio-7fd2c3cd2fa883e6ac1ae16c5a7d981349fee5ec237a7c6b692f454eeddc3893 WatchSource:0}: Error finding container 7fd2c3cd2fa883e6ac1ae16c5a7d981349fee5ec237a7c6b692f454eeddc3893: Status 404 returned error can't find the container with id 7fd2c3cd2fa883e6ac1ae16c5a7d981349fee5ec237a7c6b692f454eeddc3893
Jul 09 20:03:31.370504 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:31.309912    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8af726f313e368ccaa2fb3f2a4a8af61.slice/crio-65654c3fd54f91ad798c439ef0e7cedb358724f6776f8b1589e685578c3db102 WatchSource:0}: Error finding container 65654c3fd54f91ad798c439ef0e7cedb358724f6776f8b1589e685578c3db102: Status 404 returned error can't find the container with id 65654c3fd54f91ad798c439ef0e7cedb358724f6776f8b1589e685578c3db102
Jul 09 20:03:31.370504 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.314592    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 09 20:03:31.370504 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.330545    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nl8h"
Jul 09 20:03:31.370504 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:31.344490    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62b8139e_cc18_4428_9373_a64195b6cbdd.slice/crio-a2894e87cc46fd4d1abf51faa0d82596380e6b2014593c475a2b72f6ce0ff5b9 WatchSource:0}: Error finding container a2894e87cc46fd4d1abf51faa0d82596380e6b2014593c475a2b72f6ce0ff5b9: Status 404 returned error can't find the container with id a2894e87cc46fd4d1abf51faa0d82596380e6b2014593c475a2b72f6ce0ff5b9
Jul 09 20:03:31.370504 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.346066    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-n2tvb"
Jul 09 20:03:31.370504 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:31.353853    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod785ecd20_a92f_4924_b2f3_663697381982.slice/crio-9995218189d95825f19e8ee39d5dad343bf8ec7784a744dea1d4660148f2e1d3 WatchSource:0}: Error finding container 9995218189d95825f19e8ee39d5dad343bf8ec7784a744dea1d4660148f2e1d3: Status 404 returned error can't find the container with id 9995218189d95825f19e8ee39d5dad343bf8ec7784a744dea1d4660148f2e1d3
Jul 09 20:03:31.370504 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.354502    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-j6mws"
Jul 09 20:03:31.370504 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:31.362465    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a94e16d_ed38_41a6_a71d_49e96e5ae7c7.slice/crio-36a5e04cb7a04017a1beb0466c7162568f58e06538d4767e3e1c2bd119466aa5 WatchSource:0}: Error finding container 36a5e04cb7a04017a1beb0466c7162568f58e06538d4767e3e1c2bd119466aa5: Status 404 returned error can't find the container with id 36a5e04cb7a04017a1beb0466c7162568f58e06538d4767e3e1c2bd119466aa5
Jul 09 20:03:31.370504 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.368966    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-hds4g"
Jul 09 20:03:31.590230 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:31.387107    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b4468fd_1708_487b_8d69_bb7d817e25de.slice/crio-f77ff8795c8991da319501a47c794b9e76b51b57bf629021536047d57614f392 WatchSource:0}: Error finding container f77ff8795c8991da319501a47c794b9e76b51b57bf629021536047d57614f392: Status 404 returned error can't find the container with id f77ff8795c8991da319501a47c794b9e76b51b57bf629021536047d57614f392
Jul 09 20:03:31.590230 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.394222    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vwttv"
Jul 09 20:03:31.590230 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:31.402022    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00761abf_a23d_4f99_ba0f_bcb623060f98.slice/crio-05f783129b848bacdc90c737c9f0ba5830d2deb9d6b96221da012dd189cf5dc7 WatchSource:0}: Error finding container 05f783129b848bacdc90c737c9f0ba5830d2deb9d6b96221da012dd189cf5dc7: Status 404 returned error can't find the container with id 05f783129b848bacdc90c737c9f0ba5830d2deb9d6b96221da012dd189cf5dc7
Jul 09 20:03:31.590230 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.417525    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-87zd2"
Jul 09 20:03:31.590230 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.420445    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-8r78k"
Jul 09 20:03:31.590230 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:31.425012    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda304eb99_5afb_4710_9e69_51ca0c329ba5.slice/crio-219e0d5689fcb335332efe7971edcd088c2616de7adb9ee0efa04dea736cb674 WatchSource:0}: Error finding container 219e0d5689fcb335332efe7971edcd088c2616de7adb9ee0efa04dea736cb674: Status 404 returned error can't find the container with id 219e0d5689fcb335332efe7971edcd088c2616de7adb9ee0efa04dea736cb674
Jul 09 20:03:31.590230 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:31.433555    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddddbfeb6_339f_4aa6_b74f_26dfe9438a2c.slice/crio-9923cef82f610f6311ecc2e6b98bfe008b9b873373f3de1b2f7c55433621ae8f WatchSource:0}: Error finding container 9923cef82f610f6311ecc2e6b98bfe008b9b873373f3de1b2f7c55433621ae8f: Status 404 returned error can't find the container with id 9923cef82f610f6311ecc2e6b98bfe008b9b873373f3de1b2f7c55433621ae8f
Jul 09 20:03:31.590230 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.450003    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-dz4cq"
Jul 09 20:03:31.590230 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.457047    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-rvxnk"
Jul 09 20:03:31.590230 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:31.458791    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd592535b_5457_48d0_ae48_caed8efc04fc.slice/crio-207d81b5f63839dc0a0f00bb5b89ae41dfa9504ff30ce1e2f6d2aff6a7028ca4 WatchSource:0}: Error finding container 207d81b5f63839dc0a0f00bb5b89ae41dfa9504ff30ce1e2f6d2aff6a7028ca4: Status 404 returned error can't find the container with id 207d81b5f63839dc0a0f00bb5b89ae41dfa9504ff30ce1e2f6d2aff6a7028ca4
Jul 09 20:03:31.590230 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:03:31.468074    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5204f9ff_c575_4889_91ff_3cf3ffcd032b.slice/crio-5c5e2d560d792763a87416ea5109888877b995da8cc436c4ac40cc0b72d73137 WatchSource:0}: Error finding container 5c5e2d560d792763a87416ea5109888877b995da8cc436c4ac40cc0b72d73137: Status 404 returned error can't find the container with id 5c5e2d560d792763a87416ea5109888877b995da8cc436c4ac40cc0b72d73137
Jul 09 20:03:31.590230 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.469452    2562 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 20:03:31.590230 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:03:31.469914    2562 kuberuntime_manager.go:1459] "Unhandled Error" err="container kube-multus start failed in pod multus-rvxnk_openshift-multus(5204f9ff-c575-4889-91ff-3cf3ffcd032b): ErrImagePull: pull QPS exceeded" logger="UnhandledError"
Jul 09 20:03:31.590230 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:03:31.469944    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"kube-multus\" with ErrImagePull: \"pull QPS exceeded\"" pod="openshift-multus/multus-rvxnk" podUID="5204f9ff-c575-4889-91ff-3cf3ffcd032b"
Jul 09 20:03:31.763456 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.635433    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khzxc\" (UniqueName: \"kubernetes.io/projected/4fd1b369-9b81-495d-9608-9fc4d81295e2-kube-api-access-khzxc\") pod \"network-check-target-6b5z9\" (UID: \"4fd1b369-9b81-495d-9608-9fc4d81295e2\") " pod="openshift-network-diagnostics/network-check-target-6b5z9"
Jul 09 20:03:31.763456 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.635473    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/13454dc6-77a2-4dc9-adc2-81791d1cd463-original-pull-secret\") pod \"global-pull-secret-syncer-dccxz\" (UID: \"13454dc6-77a2-4dc9-adc2-81791d1cd463\") " pod="kube-system/global-pull-secret-syncer-dccxz"
Jul 09 20:03:31.763456 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:03:31.635589    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 20:03:31.763456 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:03:31.635651    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 20:03:31.763456 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:03:31.635677    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 20:03:31.763456 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:03:31.635691    2562 projected.go:196] Error preparing data for projected volume kube-api-access-khzxc for pod openshift-network-diagnostics/network-check-target-6b5z9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 20:03:31.763456 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:03:31.635658    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13454dc6-77a2-4dc9-adc2-81791d1cd463-original-pull-secret podName:13454dc6-77a2-4dc9-adc2-81791d1cd463 nodeName:}" failed. No retries permitted until 2026-07-09 20:03:32.635632755 +0000 UTC m=+3.053302559 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/13454dc6-77a2-4dc9-adc2-81791d1cd463-original-pull-secret") pod "global-pull-secret-syncer-dccxz" (UID: "13454dc6-77a2-4dc9-adc2-81791d1cd463") : object "kube-system"/"original-pull-secret" not registered
Jul 09 20:03:31.763456 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:03:31.635767    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4fd1b369-9b81-495d-9608-9fc4d81295e2-kube-api-access-khzxc podName:4fd1b369-9b81-495d-9608-9fc4d81295e2 nodeName:}" failed. No retries permitted until 2026-07-09 20:03:32.635749316 +0000 UTC m=+3.053419123 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-khzxc" (UniqueName: "kubernetes.io/projected/4fd1b369-9b81-495d-9608-9fc4d81295e2-kube-api-access-khzxc") pod "network-check-target-6b5z9" (UID: "4fd1b369-9b81-495d-9608-9fc4d81295e2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 20:03:31.763456 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:31.736718    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14a1d22f-43ff-4eb8-a89d-920551b2ef1f-metrics-certs\") pod \"network-metrics-daemon-ksd8v\" (UID: \"14a1d22f-43ff-4eb8-a89d-920551b2ef1f\") " pod="openshift-multus/network-metrics-daemon-ksd8v"
Jul 09 20:03:31.763456 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:03:31.736846    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 20:03:31.763456 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:03:31.736932    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/14a1d22f-43ff-4eb8-a89d-920551b2ef1f-metrics-certs podName:14a1d22f-43ff-4eb8-a89d-920551b2ef1f nodeName:}" failed. No retries permitted until 2026-07-09 20:03:32.736912326 +0000 UTC m=+3.154582118 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/14a1d22f-43ff-4eb8-a89d-920551b2ef1f-metrics-certs") pod "network-metrics-daemon-ksd8v" (UID: "14a1d22f-43ff-4eb8-a89d-920551b2ef1f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 20:03:32.071225 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:32.071105    2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-08 19:58:31 +0000 UTC" deadline="2028-02-13 12:44:26.832728043 +0000 UTC"
Jul 09 20:03:32.071225 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:32.071169    2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14008h40m54.761562913s"
Jul 09 20:03:32.179122 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:32.179043    2562 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 20:03:32.255261 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:32.254635    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dccxz"
Jul 09 20:03:32.255261 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:03:32.254810    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dccxz" podUID="13454dc6-77a2-4dc9-adc2-81791d1cd463"
Jul 09 20:03:32.257686 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:32.257587    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dz4cq" event={"ID":"d592535b-5457-48d0-ae48-caed8efc04fc","Type":"ContainerStarted","Data":"207d81b5f63839dc0a0f00bb5b89ae41dfa9504ff30ce1e2f6d2aff6a7028ca4"}
Jul 09 20:03:32.264802 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:32.264765    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-87zd2" event={"ID":"a304eb99-5afb-4710-9e69-51ca0c329ba5","Type":"ContainerStarted","Data":"219e0d5689fcb335332efe7971edcd088c2616de7adb9ee0efa04dea736cb674"}
Jul 09 20:03:32.269705 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:32.269672    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vwttv" event={"ID":"00761abf-a23d-4f99-ba0f-bcb623060f98","Type":"ContainerStarted","Data":"05f783129b848bacdc90c737c9f0ba5830d2deb9d6b96221da012dd189cf5dc7"}
Jul 09 20:03:32.279790 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:32.279753    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hds4g" event={"ID":"4b4468fd-1708-487b-8d69-bb7d817e25de","Type":"ContainerStarted","Data":"f77ff8795c8991da319501a47c794b9e76b51b57bf629021536047d57614f392"}
Jul 09 20:03:32.282950 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:32.282917    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nl8h" event={"ID":"62b8139e-cc18-4428-9373-a64195b6cbdd","Type":"ContainerStarted","Data":"a2894e87cc46fd4d1abf51faa0d82596380e6b2014593c475a2b72f6ce0ff5b9"}
Jul 09 20:03:32.285296 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:32.285266    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-rvxnk" event={"ID":"5204f9ff-c575-4889-91ff-3cf3ffcd032b","Type":"ContainerStarted","Data":"5c5e2d560d792763a87416ea5109888877b995da8cc436c4ac40cc0b72d73137"}
Jul 09 20:03:32.286994 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:03:32.286961    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"kube-multus\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5228585713df6fe73e63d85efc7cb37f2a929792a1afb637a77bf56a4f2a40f9\\\": ErrImagePull: pull QPS exceeded\"" pod="openshift-multus/multus-rvxnk" podUID="5204f9ff-c575-4889-91ff-3cf3ffcd032b"
Jul 09 20:03:32.288714 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:32.288679    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-8r78k" event={"ID":"dddbfeb6-339f-4aa6-b74f-26dfe9438a2c","Type":"ContainerStarted","Data":"9923cef82f610f6311ecc2e6b98bfe008b9b873373f3de1b2f7c55433621ae8f"}
Jul 09 20:03:32.292444 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:32.292418    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-j6mws" event={"ID":"0a94e16d-ed38-41a6-a71d-49e96e5ae7c7","Type":"ContainerStarted","Data":"36a5e04cb7a04017a1beb0466c7162568f58e06538d4767e3e1c2bd119466aa5"}
Jul 09 20:03:32.295163 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:32.295128    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-n2tvb" event={"ID":"785ecd20-a92f-4924-b2f3-663697381982","Type":"ContainerStarted","Data":"9995218189d95825f19e8ee39d5dad343bf8ec7784a744dea1d4660148f2e1d3"}
Jul 09 20:03:32.306005 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:32.305927    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-63.ec2.internal" event={"ID":"8af726f313e368ccaa2fb3f2a4a8af61","Type":"ContainerStarted","Data":"65654c3fd54f91ad798c439ef0e7cedb358724f6776f8b1589e685578c3db102"}
Jul 09 20:03:32.310005 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:32.309919    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-63.ec2.internal" event={"ID":"61c3c5dbc96e6f9e2e03d9211834c2ca","Type":"ContainerStarted","Data":"7fd2c3cd2fa883e6ac1ae16c5a7d981349fee5ec237a7c6b692f454eeddc3893"}
Jul 09 20:03:32.474657 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:32.474405    2562 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 20:03:32.646917 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:32.646181    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khzxc\" (UniqueName: \"kubernetes.io/projected/4fd1b369-9b81-495d-9608-9fc4d81295e2-kube-api-access-khzxc\") pod \"network-check-target-6b5z9\" (UID: \"4fd1b369-9b81-495d-9608-9fc4d81295e2\") " pod="openshift-network-diagnostics/network-check-target-6b5z9"
Jul 09 20:03:32.646917 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:32.646262    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/13454dc6-77a2-4dc9-adc2-81791d1cd463-original-pull-secret\") pod \"global-pull-secret-syncer-dccxz\" (UID: \"13454dc6-77a2-4dc9-adc2-81791d1cd463\") " pod="kube-system/global-pull-secret-syncer-dccxz"
Jul 09 20:03:32.646917 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:03:32.646463    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 20:03:32.646917 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:03:32.646536    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13454dc6-77a2-4dc9-adc2-81791d1cd463-original-pull-secret podName:13454dc6-77a2-4dc9-adc2-81791d1cd463 nodeName:}" failed. No retries permitted until 2026-07-09 20:03:34.646515609 +0000 UTC m=+5.064185417 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/13454dc6-77a2-4dc9-adc2-81791d1cd463-original-pull-secret") pod "global-pull-secret-syncer-dccxz" (UID: "13454dc6-77a2-4dc9-adc2-81791d1cd463") : object "kube-system"/"original-pull-secret" not registered
Jul 09 20:03:32.646917 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:03:32.646780    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 20:03:32.646917 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:03:32.646810    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 20:03:32.646917 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:03:32.646826    2562 projected.go:196] Error preparing data for projected volume kube-api-access-khzxc for pod openshift-network-diagnostics/network-check-target-6b5z9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 20:03:32.646917 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:03:32.646892    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4fd1b369-9b81-495d-9608-9fc4d81295e2-kube-api-access-khzxc podName:4fd1b369-9b81-495d-9608-9fc4d81295e2 nodeName:}" failed. No retries permitted until 2026-07-09 20:03:34.646873917 +0000 UTC m=+5.064543708 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-khzxc" (UniqueName: "kubernetes.io/projected/4fd1b369-9b81-495d-9608-9fc4d81295e2-kube-api-access-khzxc") pod "network-check-target-6b5z9" (UID: "4fd1b369-9b81-495d-9608-9fc4d81295e2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 20:03:32.746980 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:32.746885    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14a1d22f-43ff-4eb8-a89d-920551b2ef1f-metrics-certs\") pod \"network-metrics-daemon-ksd8v\" (UID: \"14a1d22f-43ff-4eb8-a89d-920551b2ef1f\") " pod="openshift-multus/network-metrics-daemon-ksd8v"
Jul 09 20:03:32.747167 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:03:32.747057    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 20:03:32.747167 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:03:32.747127    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/14a1d22f-43ff-4eb8-a89d-920551b2ef1f-metrics-certs podName:14a1d22f-43ff-4eb8-a89d-920551b2ef1f nodeName:}" failed. No retries permitted until 2026-07-09 20:03:34.747106905 +0000 UTC m=+5.164776699 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/14a1d22f-43ff-4eb8-a89d-920551b2ef1f-metrics-certs") pod "network-metrics-daemon-ksd8v" (UID: "14a1d22f-43ff-4eb8-a89d-920551b2ef1f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 20:03:33.072061 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:33.071957    2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-08 19:58:31 +0000 UTC" deadline="2028-04-18 00:58:12.256490312 +0000 UTC"
Jul 09 20:03:33.072061 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:33.072003    2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15556h54m39.18449286s"
Jul 09 20:03:33.236946 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:33.236910    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ksd8v"
Jul 09 20:03:33.237198 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:33.236919    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6b5z9"
Jul 09 20:03:33.237198 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:03:33.237053    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ksd8v" podUID="14a1d22f-43ff-4eb8-a89d-920551b2ef1f"
Jul 09 20:03:33.237198 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:03:33.237126    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6b5z9" podUID="4fd1b369-9b81-495d-9608-9fc4d81295e2"
Jul 09 20:03:34.237133 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:34.236599    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dccxz"
Jul 09 20:03:34.237133 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:03:34.236733    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dccxz" podUID="13454dc6-77a2-4dc9-adc2-81791d1cd463"
Jul 09 20:03:34.663608 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:34.663567    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khzxc\" (UniqueName: \"kubernetes.io/projected/4fd1b369-9b81-495d-9608-9fc4d81295e2-kube-api-access-khzxc\") pod \"network-check-target-6b5z9\" (UID: \"4fd1b369-9b81-495d-9608-9fc4d81295e2\") " pod="openshift-network-diagnostics/network-check-target-6b5z9"
Jul 09 20:03:34.663793 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:34.663651    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/13454dc6-77a2-4dc9-adc2-81791d1cd463-original-pull-secret\") pod \"global-pull-secret-syncer-dccxz\" (UID: \"13454dc6-77a2-4dc9-adc2-81791d1cd463\") " pod="kube-system/global-pull-secret-syncer-dccxz"
Jul 09 20:03:34.663851 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:03:34.663791    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 20:03:34.663851 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:03:34.663828    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 20:03:34.663851 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:03:34.663844    2562 projected.go:196] Error preparing data for projected volume kube-api-access-khzxc for pod openshift-network-diagnostics/network-check-target-6b5z9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 20:03:34.663974 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:03:34.663789    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 20:03:34.663974 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:03:34.663911    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4fd1b369-9b81-495d-9608-9fc4d81295e2-kube-api-access-khzxc podName:4fd1b369-9b81-495d-9608-9fc4d81295e2 nodeName:}" failed. No retries permitted until 2026-07-09 20:03:38.663888867 +0000 UTC m=+9.081558680 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-khzxc" (UniqueName: "kubernetes.io/projected/4fd1b369-9b81-495d-9608-9fc4d81295e2-kube-api-access-khzxc") pod "network-check-target-6b5z9" (UID: "4fd1b369-9b81-495d-9608-9fc4d81295e2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 20:03:34.663974 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:03:34.663931    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13454dc6-77a2-4dc9-adc2-81791d1cd463-original-pull-secret podName:13454dc6-77a2-4dc9-adc2-81791d1cd463 nodeName:}" failed. No retries permitted until 2026-07-09 20:03:38.663919377 +0000 UTC m=+9.081589177 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/13454dc6-77a2-4dc9-adc2-81791d1cd463-original-pull-secret") pod "global-pull-secret-syncer-dccxz" (UID: "13454dc6-77a2-4dc9-adc2-81791d1cd463") : object "kube-system"/"original-pull-secret" not registered
Jul 09 20:03:34.764653 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:34.764614    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14a1d22f-43ff-4eb8-a89d-920551b2ef1f-metrics-certs\") pod \"network-metrics-daemon-ksd8v\" (UID: \"14a1d22f-43ff-4eb8-a89d-920551b2ef1f\") " pod="openshift-multus/network-metrics-daemon-ksd8v"
Jul 09 20:03:34.764846 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:03:34.764801    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 20:03:34.764914 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:03:34.764871    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/14a1d22f-43ff-4eb8-a89d-920551b2ef1f-metrics-certs podName:14a1d22f-43ff-4eb8-a89d-920551b2ef1f nodeName:}" failed. No retries permitted until 2026-07-09 20:03:38.764848131 +0000 UTC m=+9.182517925 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/14a1d22f-43ff-4eb8-a89d-920551b2ef1f-metrics-certs") pod "network-metrics-daemon-ksd8v" (UID: "14a1d22f-43ff-4eb8-a89d-920551b2ef1f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 20:03:35.235781 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:35.235741    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ksd8v"
Jul 09 20:03:35.235966 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:35.235763    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6b5z9"
Jul 09 20:03:35.235966 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:03:35.235914    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ksd8v" podUID="14a1d22f-43ff-4eb8-a89d-920551b2ef1f"
Jul 09 20:03:35.236416 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:03:35.236386    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6b5z9" podUID="4fd1b369-9b81-495d-9608-9fc4d81295e2"
Jul 09 20:03:36.236659 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:36.236623    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dccxz"
Jul 09 20:03:36.237099 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:03:36.236760    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dccxz" podUID="13454dc6-77a2-4dc9-adc2-81791d1cd463"
Jul 09 20:03:37.236082 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:37.236045    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6b5z9"
Jul 09 20:03:37.236239 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:37.236045    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ksd8v"
Jul 09 20:03:37.236239 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:03:37.236152    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6b5z9" podUID="4fd1b369-9b81-495d-9608-9fc4d81295e2"
Jul 09 20:03:37.236239 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:03:37.236231    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ksd8v" podUID="14a1d22f-43ff-4eb8-a89d-920551b2ef1f"
Jul 09 20:03:38.236013 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:38.235972    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dccxz"
Jul 09 20:03:38.236461 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:03:38.236123    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dccxz" podUID="13454dc6-77a2-4dc9-adc2-81791d1cd463"
Jul 09 20:03:38.693994 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:38.693895    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khzxc\" (UniqueName: \"kubernetes.io/projected/4fd1b369-9b81-495d-9608-9fc4d81295e2-kube-api-access-khzxc\") pod \"network-check-target-6b5z9\" (UID: \"4fd1b369-9b81-495d-9608-9fc4d81295e2\") " pod="openshift-network-diagnostics/network-check-target-6b5z9"
Jul 09 20:03:38.693994 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:38.693945    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/13454dc6-77a2-4dc9-adc2-81791d1cd463-original-pull-secret\") pod \"global-pull-secret-syncer-dccxz\" (UID: \"13454dc6-77a2-4dc9-adc2-81791d1cd463\") " pod="kube-system/global-pull-secret-syncer-dccxz"
Jul 09 20:03:38.694219 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:03:38.694114    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 20:03:38.694219 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:03:38.694139    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 20:03:38.694219 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:03:38.694163    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 20:03:38.694219 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:03:38.694175    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13454dc6-77a2-4dc9-adc2-81791d1cd463-original-pull-secret podName:13454dc6-77a2-4dc9-adc2-81791d1cd463 nodeName:}" failed. No retries permitted until 2026-07-09 20:03:46.694159117 +0000 UTC m=+17.111828923 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/13454dc6-77a2-4dc9-adc2-81791d1cd463-original-pull-secret") pod "global-pull-secret-syncer-dccxz" (UID: "13454dc6-77a2-4dc9-adc2-81791d1cd463") : object "kube-system"/"original-pull-secret" not registered
Jul 09 20:03:38.694219 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:03:38.694177    2562 projected.go:196] Error preparing data for projected volume kube-api-access-khzxc for pod openshift-network-diagnostics/network-check-target-6b5z9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 20:03:38.694461 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:03:38.694234    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4fd1b369-9b81-495d-9608-9fc4d81295e2-kube-api-access-khzxc podName:4fd1b369-9b81-495d-9608-9fc4d81295e2 nodeName:}" failed. No retries permitted until 2026-07-09 20:03:46.694218198 +0000 UTC m=+17.111887987 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-khzxc" (UniqueName: "kubernetes.io/projected/4fd1b369-9b81-495d-9608-9fc4d81295e2-kube-api-access-khzxc") pod "network-check-target-6b5z9" (UID: "4fd1b369-9b81-495d-9608-9fc4d81295e2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 20:03:38.794992 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:38.794949    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14a1d22f-43ff-4eb8-a89d-920551b2ef1f-metrics-certs\") pod \"network-metrics-daemon-ksd8v\" (UID: \"14a1d22f-43ff-4eb8-a89d-920551b2ef1f\") " pod="openshift-multus/network-metrics-daemon-ksd8v"
Jul 09 20:03:38.795283 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:03:38.795117    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 20:03:38.795283 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:03:38.795203    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/14a1d22f-43ff-4eb8-a89d-920551b2ef1f-metrics-certs podName:14a1d22f-43ff-4eb8-a89d-920551b2ef1f nodeName:}" failed. No retries permitted until 2026-07-09 20:03:46.795179309 +0000 UTC m=+17.212849110 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/14a1d22f-43ff-4eb8-a89d-920551b2ef1f-metrics-certs") pod "network-metrics-daemon-ksd8v" (UID: "14a1d22f-43ff-4eb8-a89d-920551b2ef1f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 20:03:39.235644 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:39.235608    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ksd8v"
Jul 09 20:03:39.235834 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:39.235649    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6b5z9"
Jul 09 20:03:39.235834 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:03:39.235745    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ksd8v" podUID="14a1d22f-43ff-4eb8-a89d-920551b2ef1f"
Jul 09 20:03:39.235920 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:03:39.235881    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6b5z9" podUID="4fd1b369-9b81-495d-9608-9fc4d81295e2"
Jul 09 20:03:40.236386 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:40.236315    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dccxz"
Jul 09 20:03:40.236957 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:03:40.236444    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dccxz" podUID="13454dc6-77a2-4dc9-adc2-81791d1cd463"
Jul 09 20:03:41.236505 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:41.236473    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ksd8v"
Jul 09 20:03:41.236972 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:03:41.236619    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ksd8v" podUID="14a1d22f-43ff-4eb8-a89d-920551b2ef1f"
Jul 09 20:03:41.237049 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:41.237014    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6b5z9"
Jul 09 20:03:41.237136 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:03:41.237114    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6b5z9" podUID="4fd1b369-9b81-495d-9608-9fc4d81295e2"
Jul 09 20:03:42.236329 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:42.236024    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dccxz"
Jul 09 20:03:42.236438 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:03:42.236366    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dccxz" podUID="13454dc6-77a2-4dc9-adc2-81791d1cd463"
Jul 09 20:03:42.341042 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:42.341005    2562 generic.go:356] "Generic (PLEG): container finished" podID="d592535b-5457-48d0-ae48-caed8efc04fc" containerID="89c783f5ceb094b8187a7766146a9a5ef2ebe4e076a5ac8fc5ef8593ec123c77" exitCode=0
Jul 09 20:03:42.341513 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:42.341102    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dz4cq" event={"ID":"d592535b-5457-48d0-ae48-caed8efc04fc","Type":"ContainerDied","Data":"89c783f5ceb094b8187a7766146a9a5ef2ebe4e076a5ac8fc5ef8593ec123c77"}
Jul 09 20:03:42.342758 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:42.342729    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-87zd2" event={"ID":"a304eb99-5afb-4710-9e69-51ca0c329ba5","Type":"ContainerStarted","Data":"a267f09b066b6a4028ae4d8442c41868f17ba28629f51b08b6b8f1097f9cd88d"}
Jul 09 20:03:42.344365 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:42.344328    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vwttv" event={"ID":"00761abf-a23d-4f99-ba0f-bcb623060f98","Type":"ContainerStarted","Data":"dad22a2c4135fe99bd093236d60944a3c3553d74a7c5d862ffbdb9984af74a1e"}
Jul 09 20:03:42.346055 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:42.346022    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nl8h" event={"ID":"62b8139e-cc18-4428-9373-a64195b6cbdd","Type":"ContainerStarted","Data":"2a94f7002d81781fbb36bda69aa0b94ab104916a6f2a411c85ba01593eb35faa"}
Jul 09 20:03:42.347620 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:42.347591    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-8r78k" event={"ID":"dddbfeb6-339f-4aa6-b74f-26dfe9438a2c","Type":"ContainerStarted","Data":"02e8fce15c9b2f34022f033f9ccb82e4082cf0584e49cb1b8b5d7dde44b7d570"}
Jul 09 20:03:42.349162 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:42.349140    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-n2tvb" event={"ID":"785ecd20-a92f-4924-b2f3-663697381982","Type":"ContainerStarted","Data":"e6068b7d9254d2f11a92df4a3bcb1309938d631ae4244b77fa1c020768c27631"}
Jul 09 20:03:42.350775 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:42.350754    2562 generic.go:356] "Generic (PLEG): container finished" podID="8af726f313e368ccaa2fb3f2a4a8af61" containerID="f42dec201c9a5c2268d8f7d417ebe44a3026b856d6ac71c29849c37e04df83c7" exitCode=0
Jul 09 20:03:42.350860 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:42.350799    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-63.ec2.internal" event={"ID":"8af726f313e368ccaa2fb3f2a4a8af61","Type":"ContainerDied","Data":"f42dec201c9a5c2268d8f7d417ebe44a3026b856d6ac71c29849c37e04df83c7"}
Jul 09 20:03:42.352275 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:42.352252    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-63.ec2.internal" event={"ID":"61c3c5dbc96e6f9e2e03d9211834c2ca","Type":"ContainerStarted","Data":"995d50c94a874aba955b984f2fc22e955ccdbfacbfbf7564c8b3df909955dbdf"}
Jul 09 20:03:42.352437 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:42.352415    2562 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-63.ec2.internal"
Jul 09 20:03:42.361188 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:42.361168    2562 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 09 20:03:42.362276 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:42.362258    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-140-63.ec2.internal"]
Jul 09 20:03:42.381988 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:42.381920    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-63.ec2.internal" podStartSLOduration=0.381899597 podStartE2EDuration="381.899597ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 20:03:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 20:03:42.381864092 +0000 UTC m=+12.799533902" watchObservedRunningTime="2026-07-09 20:03:42.381899597 +0000 UTC m=+12.799569618"
Jul 09 20:03:42.400646 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:42.397384    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-n2tvb" podStartSLOduration=1.869931937 podStartE2EDuration="12.397365363s" totalImagesPullingTime="10.527433426s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 20:03:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 20:03:42.397112608 +0000 UTC m=+12.814782418" watchObservedRunningTime="2026-07-09 20:03:42.397365363 +0000 UTC m=+12.815035173"
Jul 09 20:03:42.413672 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:42.413623    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-87zd2" podStartSLOduration=1.933280695 podStartE2EDuration="12.413607882s" totalImagesPullingTime="10.480327187s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 20:03:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 20:03:42.413053803 +0000 UTC m=+12.830723613" watchObservedRunningTime="2026-07-09 20:03:42.413607882 +0000 UTC m=+12.831277690"
Jul 09 20:03:42.428107 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:42.428063    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-8r78k" podStartSLOduration=1.980091762 podStartE2EDuration="12.428048933s" totalImagesPullingTime="10.447957171s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 20:03:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 20:03:42.427838392 +0000 UTC m=+12.845508201" watchObservedRunningTime="2026-07-09 20:03:42.428048933 +0000 UTC m=+12.845718743"
Jul 09 20:03:42.456767 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:42.456721    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-vwttv" podStartSLOduration=1.979380446 podStartE2EDuration="12.456705547s" totalImagesPullingTime="10.477325101s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 20:03:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 20:03:42.456516018 +0000 UTC m=+12.874185827" watchObservedRunningTime="2026-07-09 20:03:42.456705547 +0000 UTC m=+12.874375356"
Jul 09 20:03:42.868566 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:42.868531    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-vwttv"
Jul 09 20:03:42.869287 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:42.869263    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-vwttv"
Jul 09 20:03:43.236620 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:43.236489    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ksd8v"
Jul 09 20:03:43.236816 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:03:43.236645    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ksd8v" podUID="14a1d22f-43ff-4eb8-a89d-920551b2ef1f"
Jul 09 20:03:43.236816 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:43.236680    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6b5z9"
Jul 09 20:03:43.236816 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:03:43.236787    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6b5z9" podUID="4fd1b369-9b81-495d-9608-9fc4d81295e2"
Jul 09 20:03:43.356468 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:43.356427    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-63.ec2.internal" event={"ID":"8af726f313e368ccaa2fb3f2a4a8af61","Type":"ContainerStarted","Data":"df10012cff84cc37d48dbe929742fad214f04862d894398dc06e6e12f1d0c1cf"}
Jul 09 20:03:43.356890 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:43.356730    2562 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-63.ec2.internal"
Jul 09 20:03:43.363737 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:43.363717    2562 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 09 20:03:43.363864 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:03:43.363759    2562 kubelet.go:3250] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-140-63.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-63.ec2.internal"
Jul 09 20:03:43.371470 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:43.371428    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-63.ec2.internal" podStartSLOduration=12.371413659 podStartE2EDuration="12.371413659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 20:03:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 20:03:43.370962174 +0000 UTC m=+13.788631983" watchObservedRunningTime="2026-07-09 20:03:43.371413659 +0000 UTC m=+13.789083468"
Jul 09 20:03:44.235536 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:44.235502    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dccxz"
Jul 09 20:03:44.235739 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:03:44.235682    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dccxz" podUID="13454dc6-77a2-4dc9-adc2-81791d1cd463"
Jul 09 20:03:44.360597 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:44.360551    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-j6mws" event={"ID":"0a94e16d-ed38-41a6-a71d-49e96e5ae7c7","Type":"ContainerStarted","Data":"011bdd96e8d58c0765bd08761c312817610d47f4b129d958855787297d119c28"}
Jul 09 20:03:44.362134 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:44.362110    2562 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 09 20:03:44.376763 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:44.376712    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-j6mws" podStartSLOduration=3.858281234 podStartE2EDuration="14.37669357s" totalImagesPullingTime="10.518412336s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 20:03:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 20:03:44.376348925 +0000 UTC m=+14.794018994" watchObservedRunningTime="2026-07-09 20:03:44.37669357 +0000 UTC m=+14.794363380"
Jul 09 20:03:45.235885 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:45.235787    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ksd8v"
Jul 09 20:03:45.236075 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:03:45.235944    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ksd8v" podUID="14a1d22f-43ff-4eb8-a89d-920551b2ef1f"
Jul 09 20:03:45.236075 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:45.235999    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6b5z9"
Jul 09 20:03:45.236186 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:03:45.236113    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6b5z9" podUID="4fd1b369-9b81-495d-9608-9fc4d81295e2"
Jul 09 20:03:46.236471 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:46.236430    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dccxz"
Jul 09 20:03:46.237206 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:03:46.236591    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dccxz" podUID="13454dc6-77a2-4dc9-adc2-81791d1cd463"
Jul 09 20:03:46.758593 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:46.758549    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khzxc\" (UniqueName: \"kubernetes.io/projected/4fd1b369-9b81-495d-9608-9fc4d81295e2-kube-api-access-khzxc\") pod \"network-check-target-6b5z9\" (UID: \"4fd1b369-9b81-495d-9608-9fc4d81295e2\") " pod="openshift-network-diagnostics/network-check-target-6b5z9"
Jul 09 20:03:46.758593 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:46.758598    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/13454dc6-77a2-4dc9-adc2-81791d1cd463-original-pull-secret\") pod \"global-pull-secret-syncer-dccxz\" (UID: \"13454dc6-77a2-4dc9-adc2-81791d1cd463\") " pod="kube-system/global-pull-secret-syncer-dccxz"
Jul 09 20:03:46.758833 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:03:46.758741    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 20:03:46.758833 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:03:46.758745    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 20:03:46.758833 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:03:46.758771    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 20:03:46.758833 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:03:46.758781    2562 projected.go:196] Error preparing data for projected volume kube-api-access-khzxc for pod openshift-network-diagnostics/network-check-target-6b5z9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 20:03:46.758833 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:03:46.758809    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13454dc6-77a2-4dc9-adc2-81791d1cd463-original-pull-secret podName:13454dc6-77a2-4dc9-adc2-81791d1cd463 nodeName:}" failed. No retries permitted until 2026-07-09 20:04:02.758791305 +0000 UTC m=+33.176461109 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/13454dc6-77a2-4dc9-adc2-81791d1cd463-original-pull-secret") pod "global-pull-secret-syncer-dccxz" (UID: "13454dc6-77a2-4dc9-adc2-81791d1cd463") : object "kube-system"/"original-pull-secret" not registered
Jul 09 20:03:46.758833 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:03:46.758831    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4fd1b369-9b81-495d-9608-9fc4d81295e2-kube-api-access-khzxc podName:4fd1b369-9b81-495d-9608-9fc4d81295e2 nodeName:}" failed. No retries permitted until 2026-07-09 20:04:02.758818844 +0000 UTC m=+33.176488653 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-khzxc" (UniqueName: "kubernetes.io/projected/4fd1b369-9b81-495d-9608-9fc4d81295e2-kube-api-access-khzxc") pod "network-check-target-6b5z9" (UID: "4fd1b369-9b81-495d-9608-9fc4d81295e2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 20:03:46.859086 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:46.859045    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14a1d22f-43ff-4eb8-a89d-920551b2ef1f-metrics-certs\") pod \"network-metrics-daemon-ksd8v\" (UID: \"14a1d22f-43ff-4eb8-a89d-920551b2ef1f\") " pod="openshift-multus/network-metrics-daemon-ksd8v"
Jul 09 20:03:46.859334 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:03:46.859220    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 20:03:46.859334 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:03:46.859285    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/14a1d22f-43ff-4eb8-a89d-920551b2ef1f-metrics-certs podName:14a1d22f-43ff-4eb8-a89d-920551b2ef1f nodeName:}" failed. No retries permitted until 2026-07-09 20:04:02.859270283 +0000 UTC m=+33.276940070 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/14a1d22f-43ff-4eb8-a89d-920551b2ef1f-metrics-certs") pod "network-metrics-daemon-ksd8v" (UID: "14a1d22f-43ff-4eb8-a89d-920551b2ef1f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 20:03:47.235929 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:47.235892    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6b5z9"
Jul 09 20:03:47.236104 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:47.235907    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ksd8v"
Jul 09 20:03:47.236104 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:03:47.236019    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6b5z9" podUID="4fd1b369-9b81-495d-9608-9fc4d81295e2"
Jul 09 20:03:47.236204 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:03:47.236110    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ksd8v" podUID="14a1d22f-43ff-4eb8-a89d-920551b2ef1f"
Jul 09 20:03:47.604619 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:47.604531    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-vwttv"
Jul 09 20:03:47.605082 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:47.604683    2562 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 09 20:03:47.605324 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:47.605286    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-vwttv"
Jul 09 20:03:48.235700 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:48.235667    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dccxz"
Jul 09 20:03:48.235869 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:03:48.235806    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dccxz" podUID="13454dc6-77a2-4dc9-adc2-81791d1cd463"
Jul 09 20:03:49.236320 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:49.236272    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ksd8v"
Jul 09 20:03:49.236320 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:49.236318    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6b5z9"
Jul 09 20:03:49.236942 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:03:49.236441    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ksd8v" podUID="14a1d22f-43ff-4eb8-a89d-920551b2ef1f"
Jul 09 20:03:49.236942 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:03:49.236565    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6b5z9" podUID="4fd1b369-9b81-495d-9608-9fc4d81295e2"
Jul 09 20:03:49.649801 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:49.649598    2562 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 09 20:03:50.098035 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:50.097881    2562 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-09T20:03:49.649784724Z","UUID":"985a8e68-d209-4398-8399-cfe0d69d1413","Handler":null,"Name":"","Endpoint":""}
Jul 09 20:03:50.101314 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:50.101209    2562 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 09 20:03:50.101314 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:50.101256    2562 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 09 20:03:50.236158 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:50.236125    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dccxz"
Jul 09 20:03:50.236318 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:03:50.236222    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dccxz" podUID="13454dc6-77a2-4dc9-adc2-81791d1cd463"
Jul 09 20:03:50.376816 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:50.376731    2562 generic.go:356] "Generic (PLEG): container finished" podID="d592535b-5457-48d0-ae48-caed8efc04fc" containerID="cf32724b1ec024827fe21f4b483ade24628e995636f4d9a6bdc25c8f8082433f" exitCode=0
Jul 09 20:03:50.377273 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:50.376816    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dz4cq" event={"ID":"d592535b-5457-48d0-ae48-caed8efc04fc","Type":"ContainerDied","Data":"cf32724b1ec024827fe21f4b483ade24628e995636f4d9a6bdc25c8f8082433f"}
Jul 09 20:03:50.379884 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:50.379853    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hds4g_4b4468fd-1708-487b-8d69-bb7d817e25de/ovn-acl-logging/0.log"
Jul 09 20:03:50.380265 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:50.380230    2562 generic.go:356] "Generic (PLEG): container finished" podID="4b4468fd-1708-487b-8d69-bb7d817e25de" containerID="669aa4d97c9b88054f7eda2fd06b3e24f7c0aa60f2eaa587883086e964429591" exitCode=1
Jul 09 20:03:50.380364 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:50.380291    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hds4g" event={"ID":"4b4468fd-1708-487b-8d69-bb7d817e25de","Type":"ContainerStarted","Data":"5267a32fb6298563dded77cc099572f3381328ab992a93bd3649798aa3ed0228"}
Jul 09 20:03:50.380419 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:50.380396    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hds4g" event={"ID":"4b4468fd-1708-487b-8d69-bb7d817e25de","Type":"ContainerStarted","Data":"4572a0f5469d2e7ce88dd2d98ee0ade6a5ed8cc90c236e8867c620cd6cdae21a"}
Jul 09 20:03:50.380466 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:50.380447    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hds4g" event={"ID":"4b4468fd-1708-487b-8d69-bb7d817e25de","Type":"ContainerStarted","Data":"96094ec4208d589f2722c9af9f421e6107200096238a817dad0a9746cf924682"}
Jul 09 20:03:50.380510 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:50.380496    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hds4g" event={"ID":"4b4468fd-1708-487b-8d69-bb7d817e25de","Type":"ContainerStarted","Data":"e7fef51638482a615f331d5e6ba5ec4c33ac50157c955f803602594ba36db63a"}
Jul 09 20:03:50.380557 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:50.380539    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hds4g" event={"ID":"4b4468fd-1708-487b-8d69-bb7d817e25de","Type":"ContainerDied","Data":"669aa4d97c9b88054f7eda2fd06b3e24f7c0aa60f2eaa587883086e964429591"}
Jul 09 20:03:50.380598 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:50.380581    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hds4g" event={"ID":"4b4468fd-1708-487b-8d69-bb7d817e25de","Type":"ContainerStarted","Data":"2dfe2a293bc74b7d781b182657cba0f0e3873e5d8701e1810641ae92886ad2b2"}
Jul 09 20:03:50.382279 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:50.382255    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nl8h" event={"ID":"62b8139e-cc18-4428-9373-a64195b6cbdd","Type":"ContainerStarted","Data":"4d34582bd6e80f27c4a17dbd399cff1af77fe6b276d9eeb0cb73144760f1caec"}
Jul 09 20:03:51.238329 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:51.236347    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6b5z9"
Jul 09 20:03:51.238329 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:51.236351    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ksd8v"
Jul 09 20:03:51.238329 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:03:51.236460    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6b5z9" podUID="4fd1b369-9b81-495d-9608-9fc4d81295e2"
Jul 09 20:03:51.238329 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:03:51.236521    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ksd8v" podUID="14a1d22f-43ff-4eb8-a89d-920551b2ef1f"
Jul 09 20:03:51.386545 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:51.386459    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nl8h" event={"ID":"62b8139e-cc18-4428-9373-a64195b6cbdd","Type":"ContainerStarted","Data":"ca879b4e917ef9a0d63c5c92e0af7946f59f52847b20386f008bc0e832b4e0f9"}
Jul 09 20:03:51.388677 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:51.388652    2562 generic.go:356] "Generic (PLEG): container finished" podID="d592535b-5457-48d0-ae48-caed8efc04fc" containerID="cd3f44da8791b6437979d363f53399ce38bd903128895ee16df9717728edfd44" exitCode=0
Jul 09 20:03:51.388780 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:51.388687    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dz4cq" event={"ID":"d592535b-5457-48d0-ae48-caed8efc04fc","Type":"ContainerDied","Data":"cd3f44da8791b6437979d363f53399ce38bd903128895ee16df9717728edfd44"}
Jul 09 20:03:51.404347 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:51.404280    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9nl8h" podStartSLOduration=2.568599086 podStartE2EDuration="21.404267148s" totalImagesPullingTime="18.835668062s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 20:03:30 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 20:03:51.403707192 +0000 UTC m=+21.821377000" watchObservedRunningTime="2026-07-09 20:03:51.404267148 +0000 UTC m=+21.821937007"
Jul 09 20:03:52.236069 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:52.236038    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dccxz"
Jul 09 20:03:52.236229 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:03:52.236148    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dccxz" podUID="13454dc6-77a2-4dc9-adc2-81791d1cd463"
Jul 09 20:03:52.398248 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:52.398228    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hds4g_4b4468fd-1708-487b-8d69-bb7d817e25de/ovn-acl-logging/0.log"
Jul 09 20:03:52.398655 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:52.398627    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hds4g" event={"ID":"4b4468fd-1708-487b-8d69-bb7d817e25de","Type":"ContainerStarted","Data":"329756bbd4f04f764be189f5fae905c192b78e74a6cce0fc314edf95bf3bec0c"}
Jul 09 20:03:53.235790 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:53.235755    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ksd8v"
Jul 09 20:03:53.235790 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:53.235792    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6b5z9"
Jul 09 20:03:53.236076 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:03:53.235883    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ksd8v" podUID="14a1d22f-43ff-4eb8-a89d-920551b2ef1f"
Jul 09 20:03:53.236076 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:03:53.236010    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6b5z9" podUID="4fd1b369-9b81-495d-9608-9fc4d81295e2"
Jul 09 20:03:53.403002 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:53.402961    2562 generic.go:356] "Generic (PLEG): container finished" podID="d592535b-5457-48d0-ae48-caed8efc04fc" containerID="a0e262c11595ee0417737994861a11d1f5256642bbd18f25fbc808fa2c00a0dd" exitCode=0
Jul 09 20:03:53.403472 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:53.403024    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dz4cq" event={"ID":"d592535b-5457-48d0-ae48-caed8efc04fc","Type":"ContainerDied","Data":"a0e262c11595ee0417737994861a11d1f5256642bbd18f25fbc808fa2c00a0dd"}
Jul 09 20:03:54.236078 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:54.236054    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dccxz"
Jul 09 20:03:54.236229 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:03:54.236164    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dccxz" podUID="13454dc6-77a2-4dc9-adc2-81791d1cd463"
Jul 09 20:03:54.409769 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:54.409573    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hds4g_4b4468fd-1708-487b-8d69-bb7d817e25de/ovn-acl-logging/0.log"
Jul 09 20:03:54.410450 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:54.410155    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hds4g" event={"ID":"4b4468fd-1708-487b-8d69-bb7d817e25de","Type":"ContainerStarted","Data":"3bf895904f8771d6270c5716cd4f9216d36e913b5ffd82a53ba7a494501cf475"}
Jul 09 20:03:54.410450 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:54.410432    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hds4g"
Jul 09 20:03:54.410538 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:54.410511    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hds4g"
Jul 09 20:03:54.410587 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:54.410559    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hds4g"
Jul 09 20:03:54.410692 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:54.410678    2562 scope.go:117] "RemoveContainer" containerID="669aa4d97c9b88054f7eda2fd06b3e24f7c0aa60f2eaa587883086e964429591"
Jul 09 20:03:54.436063 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:54.436040    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hds4g"
Jul 09 20:03:54.436163 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:54.436153    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hds4g"
Jul 09 20:03:55.235584 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:55.235549    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ksd8v"
Jul 09 20:03:55.235584 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:55.235578    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6b5z9"
Jul 09 20:03:55.235803 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:03:55.235686    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ksd8v" podUID="14a1d22f-43ff-4eb8-a89d-920551b2ef1f"
Jul 09 20:03:55.235853 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:03:55.235820    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6b5z9" podUID="4fd1b369-9b81-495d-9608-9fc4d81295e2"
Jul 09 20:03:55.417466 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:55.417422    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hds4g_4b4468fd-1708-487b-8d69-bb7d817e25de/ovn-acl-logging/0.log"
Jul 09 20:03:55.418112 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:55.418078    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hds4g" event={"ID":"4b4468fd-1708-487b-8d69-bb7d817e25de","Type":"ContainerStarted","Data":"47bfd01714b4b6a6a77ede0d36f2d72e4ae75714fd74ef411031b3cda4a6114a"}
Jul 09 20:03:55.448570 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:55.448508    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-hds4g" podStartSLOduration=7.236685517 podStartE2EDuration="25.448489713s" totalImagesPullingTime="18.211804196s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 20:03:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 20:03:55.448075052 +0000 UTC m=+25.865744862" watchObservedRunningTime="2026-07-09 20:03:55.448489713 +0000 UTC m=+25.866159522"
Jul 09 20:03:56.218838 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:56.218799    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-6b5z9"]
Jul 09 20:03:56.219686 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:56.219618    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6b5z9"
Jul 09 20:03:56.219686 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:56.219654    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-dccxz"]
Jul 09 20:03:56.219885 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:56.219745    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dccxz"
Jul 09 20:03:56.219885 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:03:56.219835    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dccxz" podUID="13454dc6-77a2-4dc9-adc2-81791d1cd463"
Jul 09 20:03:56.220101 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:03:56.220059    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6b5z9" podUID="4fd1b369-9b81-495d-9608-9fc4d81295e2"
Jul 09 20:03:56.231241 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:56.231215    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-ksd8v"]
Jul 09 20:03:56.231371 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:56.231338    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ksd8v"
Jul 09 20:03:56.231464 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:03:56.231442    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ksd8v" podUID="14a1d22f-43ff-4eb8-a89d-920551b2ef1f"
Jul 09 20:03:58.236574 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:58.236536    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dccxz"
Jul 09 20:03:58.237020 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:03:58.236686    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dccxz" podUID="13454dc6-77a2-4dc9-adc2-81791d1cd463"
Jul 09 20:03:58.237146 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:58.237121    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6b5z9"
Jul 09 20:03:58.237243 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:03:58.237224    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6b5z9" podUID="4fd1b369-9b81-495d-9608-9fc4d81295e2"
Jul 09 20:03:58.237404 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:03:58.237381    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ksd8v"
Jul 09 20:03:58.237534 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:03:58.237506    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ksd8v" podUID="14a1d22f-43ff-4eb8-a89d-920551b2ef1f"
Jul 09 20:04:00.237292 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:04:00.237256    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dccxz"
Jul 09 20:04:00.237783 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:04:00.237400    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6b5z9"
Jul 09 20:04:00.237783 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:04:00.237463    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ksd8v"
Jul 09 20:04:00.237783 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:04:00.237506    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dccxz" podUID="13454dc6-77a2-4dc9-adc2-81791d1cd463"
Jul 09 20:04:00.237783 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:04:00.237557    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ksd8v" podUID="14a1d22f-43ff-4eb8-a89d-920551b2ef1f"
Jul 09 20:04:00.237783 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:04:00.237604    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6b5z9" podUID="4fd1b369-9b81-495d-9608-9fc4d81295e2"
Jul 09 20:04:02.236901 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:04:02.236664    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dccxz"
Jul 09 20:04:02.237389 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:04:02.236664    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ksd8v"
Jul 09 20:04:02.237389 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:04:02.236984    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dccxz" podUID="13454dc6-77a2-4dc9-adc2-81791d1cd463"
Jul 09 20:04:02.237389 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:04:02.236732    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6b5z9"
Jul 09 20:04:02.237389 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:04:02.237088    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ksd8v" podUID="14a1d22f-43ff-4eb8-a89d-920551b2ef1f"
Jul 09 20:04:02.237389 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:04:02.237173    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6b5z9" podUID="4fd1b369-9b81-495d-9608-9fc4d81295e2"
Jul 09 20:04:02.781774 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:04:02.781749    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khzxc\" (UniqueName: \"kubernetes.io/projected/4fd1b369-9b81-495d-9608-9fc4d81295e2-kube-api-access-khzxc\") pod \"network-check-target-6b5z9\" (UID: \"4fd1b369-9b81-495d-9608-9fc4d81295e2\") " pod="openshift-network-diagnostics/network-check-target-6b5z9"
Jul 09 20:04:02.781897 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:04:02.781792    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/13454dc6-77a2-4dc9-adc2-81791d1cd463-original-pull-secret\") pod \"global-pull-secret-syncer-dccxz\" (UID: \"13454dc6-77a2-4dc9-adc2-81791d1cd463\") " pod="kube-system/global-pull-secret-syncer-dccxz"
Jul 09 20:04:02.781897 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:04:02.781887    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 20:04:02.782008 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:04:02.781916    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 20:04:02.782008 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:04:02.781936    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13454dc6-77a2-4dc9-adc2-81791d1cd463-original-pull-secret podName:13454dc6-77a2-4dc9-adc2-81791d1cd463 nodeName:}" failed. No retries permitted until 2026-07-09 20:04:34.781924 +0000 UTC m=+65.199593787 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/13454dc6-77a2-4dc9-adc2-81791d1cd463-original-pull-secret") pod "global-pull-secret-syncer-dccxz" (UID: "13454dc6-77a2-4dc9-adc2-81791d1cd463") : object "kube-system"/"original-pull-secret" not registered
Jul 09 20:04:02.782008 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:04:02.781940    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 20:04:02.782008 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:04:02.781956    2562 projected.go:196] Error preparing data for projected volume kube-api-access-khzxc for pod openshift-network-diagnostics/network-check-target-6b5z9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 20:04:02.782008 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:04:02.782008    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4fd1b369-9b81-495d-9608-9fc4d81295e2-kube-api-access-khzxc podName:4fd1b369-9b81-495d-9608-9fc4d81295e2 nodeName:}" failed. No retries permitted until 2026-07-09 20:04:34.781992003 +0000 UTC m=+65.199661804 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-khzxc" (UniqueName: "kubernetes.io/projected/4fd1b369-9b81-495d-9608-9fc4d81295e2-kube-api-access-khzxc") pod "network-check-target-6b5z9" (UID: "4fd1b369-9b81-495d-9608-9fc4d81295e2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 20:04:02.883039 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:04:02.882993    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14a1d22f-43ff-4eb8-a89d-920551b2ef1f-metrics-certs\") pod \"network-metrics-daemon-ksd8v\" (UID: \"14a1d22f-43ff-4eb8-a89d-920551b2ef1f\") " pod="openshift-multus/network-metrics-daemon-ksd8v"
Jul 09 20:04:02.883281 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:04:02.883135    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 20:04:02.883281 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:04:02.883203    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/14a1d22f-43ff-4eb8-a89d-920551b2ef1f-metrics-certs podName:14a1d22f-43ff-4eb8-a89d-920551b2ef1f nodeName:}" failed. No retries permitted until 2026-07-09 20:04:34.883187516 +0000 UTC m=+65.300857326 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/14a1d22f-43ff-4eb8-a89d-920551b2ef1f-metrics-certs") pod "network-metrics-daemon-ksd8v" (UID: "14a1d22f-43ff-4eb8-a89d-920551b2ef1f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 20:04:03.438688 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:04:03.438588    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-rvxnk" event={"ID":"5204f9ff-c575-4889-91ff-3cf3ffcd032b","Type":"ContainerStarted","Data":"13c6398d2848c963a74dd8a6028088fb60648414008a26a28ace126cd6f50af0"}
Jul 09 20:04:03.441214 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:04:03.441187    2562 generic.go:356] "Generic (PLEG): container finished" podID="d592535b-5457-48d0-ae48-caed8efc04fc" containerID="d5a5bd2a038e812c24511b4dcde8e95c072363c54da69207c2b2a6c2c3de88b6" exitCode=0
Jul 09 20:04:03.441356 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:04:03.441223    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dz4cq" event={"ID":"d592535b-5457-48d0-ae48-caed8efc04fc","Type":"ContainerDied","Data":"d5a5bd2a038e812c24511b4dcde8e95c072363c54da69207c2b2a6c2c3de88b6"}
Jul 09 20:04:03.456982 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:04:03.456937    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-rvxnk" podStartSLOduration=2.208854752 podStartE2EDuration="33.456924788s" totalImagesPullingTime="31.248070036s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 20:03:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=1 observedRunningTime="2026-07-09 20:04:03.456517682 +0000 UTC m=+33.874187491" watchObservedRunningTime="2026-07-09 20:04:03.456924788 +0000 UTC m=+33.874594647"
Jul 09 20:04:04.235679 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:04:04.235647    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ksd8v"
Jul 09 20:04:04.235879 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:04:04.235648    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dccxz"
Jul 09 20:04:04.235879 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:04:04.235660    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6b5z9"
Jul 09 20:04:04.236285 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:04:04.236143    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ksd8v" podUID="14a1d22f-43ff-4eb8-a89d-920551b2ef1f"
Jul 09 20:04:04.236285 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:04:04.236223    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dccxz" podUID="13454dc6-77a2-4dc9-adc2-81791d1cd463"
Jul 09 20:04:04.238919 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:04:04.236505    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6b5z9" podUID="4fd1b369-9b81-495d-9608-9fc4d81295e2"
Jul 09 20:04:04.359067 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:04:04.359042    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-63.ec2.internal" event="NodeReady"
Jul 09 20:04:04.359243 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:04:04.359170    2562 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 09 20:04:04.397107 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:04:04.397037    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-vhphk"]
Jul 09 20:04:04.415758 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:04:04.415731    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-pwjnv"]
Jul 09 20:04:04.415921 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:04:04.415867    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vhphk"
Jul 09 20:04:04.419133 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:04:04.418557    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-9lrpv\""
Jul 09 20:04:04.419133 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:04:04.418557    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 09 20:04:04.419133 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:04:04.418789    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 09 20:04:04.432595 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:04:04.432573    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-vhphk"]
Jul 09 20:04:04.432704 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:04:04.432605    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-pwjnv"]
Jul 09 20:04:04.432704 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:04:04.432683    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-pwjnv"
Jul 09 20:04:04.435614 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:04:04.435590    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 09 20:04:04.435737 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:04:04.435713    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 09 20:04:04.435836 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:04:04.435814    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-57xht\""
Jul 09 20:04:04.435943 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:04:04.435849    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 09 20:04:04.446535 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:04:04.446512    2562 generic.go:356] "Generic (PLEG): container finished" podID="d592535b-5457-48d0-ae48-caed8efc04fc" containerID="e7faef8a5f10bd1fc3f1d80c602c8f0265a8e9cfb71a36232fe12addbd9dea02" exitCode=0
Jul 09 20:04:04.446940 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:04:04.446546    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dz4cq" event={"ID":"d592535b-5457-48d0-ae48-caed8efc04fc","Type":"ContainerDied","Data":"e7faef8a5f10bd1fc3f1d80c602c8f0265a8e9cfb71a36232fe12addbd9dea02"}
Jul 09 20:04:04.493394 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:04:04.493353    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f3c5da7-a9c8-4238-840b-df34177f1096-tmp-dir\") pod \"dns-default-vhphk\" (UID: \"1f3c5da7-a9c8-4238-840b-df34177f1096\") " pod="openshift-dns/dns-default-vhphk"
Jul 09 20:04:04.493531 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:04:04.493398    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwczq\" (UniqueName: \"kubernetes.io/projected/1f3c5da7-a9c8-4238-840b-df34177f1096-kube-api-access-rwczq\") pod \"dns-default-vhphk\" (UID: \"1f3c5da7-a9c8-4238-840b-df34177f1096\") " pod="openshift-dns/dns-default-vhphk"
Jul 09 20:04:04.493531 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:04:04.493431    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fd8d29db-3983-4f37-9476-6dca7b1b33b7-cert\") pod \"ingress-canary-pwjnv\" (UID: \"fd8d29db-3983-4f37-9476-6dca7b1b33b7\") " pod="openshift-ingress-canary/ingress-canary-pwjnv"
Jul 09 20:04:04.493531 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:04:04.493466    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrxfp\" (UniqueName: \"kubernetes.io/projected/fd8d29db-3983-4f37-9476-6dca7b1b33b7-kube-api-access-vrxfp\") pod \"ingress-canary-pwjnv\" (UID: \"fd8d29db-3983-4f37-9476-6dca7b1b33b7\") " pod="openshift-ingress-canary/ingress-canary-pwjnv"
Jul 09 20:04:04.493682 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:04:04.493529    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1f3c5da7-a9c8-4238-840b-df34177f1096-metrics-tls\") pod \"dns-default-vhphk\" (UID: \"1f3c5da7-a9c8-4238-840b-df34177f1096\") " pod="openshift-dns/dns-default-vhphk"
Jul 09 20:04:04.493682 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:04:04.493569    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1f3c5da7-a9c8-4238-840b-df34177f1096-config-volume\") pod \"dns-default-vhphk\" (UID: \"1f3c5da7-a9c8-4238-840b-df34177f1096\") " pod="openshift-dns/dns-default-vhphk"
Jul 09 20:04:04.594689 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:04:04.594663    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fd8d29db-3983-4f37-9476-6dca7b1b33b7-cert\") pod \"ingress-canary-pwjnv\" (UID: \"fd8d29db-3983-4f37-9476-6dca7b1b33b7\") " pod="openshift-ingress-canary/ingress-canary-pwjnv"
Jul 09 20:04:04.594835 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:04:04.594699    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrxfp\" (UniqueName: \"kubernetes.io/projected/fd8d29db-3983-4f37-9476-6dca7b1b33b7-kube-api-access-vrxfp\") pod \"ingress-canary-pwjnv\" (UID: \"fd8d29db-3983-4f37-9476-6dca7b1b33b7\") " pod="openshift-ingress-canary/ingress-canary-pwjnv"
Jul 09 20:04:04.594835 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:04:04.594727    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1f3c5da7-a9c8-4238-840b-df34177f1096-metrics-tls\") pod \"dns-default-vhphk\" (UID: \"1f3c5da7-a9c8-4238-840b-df34177f1096\") " pod="openshift-dns/dns-default-vhphk"
Jul 09 20:04:04.594835 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:04:04.594756    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1f3c5da7-a9c8-4238-840b-df34177f1096-config-volume\") pod \"dns-default-vhphk\" (UID: \"1f3c5da7-a9c8-4238-840b-df34177f1096\") " pod="openshift-dns/dns-default-vhphk"
Jul 09 20:04:04.595072 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:04:04.595049    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 20:04:04.595141 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:04:04.595111    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f3c5da7-a9c8-4238-840b-df34177f1096-metrics-tls podName:1f3c5da7-a9c8-4238-840b-df34177f1096 nodeName:}" failed. No retries permitted until 2026-07-09 20:04:05.095093106 +0000 UTC m=+35.512762895 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1f3c5da7-a9c8-4238-840b-df34177f1096-metrics-tls") pod "dns-default-vhphk" (UID: "1f3c5da7-a9c8-4238-840b-df34177f1096") : secret "dns-default-metrics-tls" not found
Jul 09 20:04:04.595209 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:04:04.595134    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f3c5da7-a9c8-4238-840b-df34177f1096-tmp-dir\") pod \"dns-default-vhphk\" (UID: \"1f3c5da7-a9c8-4238-840b-df34177f1096\") " pod="openshift-dns/dns-default-vhphk"
Jul 09 20:04:04.595209 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:04:04.595172    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwczq\" (UniqueName: \"kubernetes.io/projected/1f3c5da7-a9c8-4238-840b-df34177f1096-kube-api-access-rwczq\") pod \"dns-default-vhphk\" (UID: \"1f3c5da7-a9c8-4238-840b-df34177f1096\") " pod="openshift-dns/dns-default-vhphk"
Jul 09 20:04:04.595403 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:04:04.595387    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1f3c5da7-a9c8-4238-840b-df34177f1096-config-volume\") pod \"dns-default-vhphk\" (UID: \"1f3c5da7-a9c8-4238-840b-df34177f1096\") " pod="openshift-dns/dns-default-vhphk"
Jul 09 20:04:04.595488 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:04:04.595470    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f3c5da7-a9c8-4238-840b-df34177f1096-tmp-dir\") pod \"dns-default-vhphk\" (UID: \"1f3c5da7-a9c8-4238-840b-df34177f1096\") " pod="openshift-dns/dns-default-vhphk"
Jul 09 20:04:04.598782 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:04:04.598764    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fd8d29db-3983-4f37-9476-6dca7b1b33b7-cert\") pod \"ingress-canary-pwjnv\" (UID: \"fd8d29db-3983-4f37-9476-6dca7b1b33b7\") " pod="openshift-ingress-canary/ingress-canary-pwjnv"
Jul 09 20:04:04.603649 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:04:04.603624    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwczq\" (UniqueName: \"kubernetes.io/projected/1f3c5da7-a9c8-4238-840b-df34177f1096-kube-api-access-rwczq\") pod \"dns-default-vhphk\" (UID: \"1f3c5da7-a9c8-4238-840b-df34177f1096\") " pod="openshift-dns/dns-default-vhphk"
Jul 09 20:04:04.603790 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:04:04.603770    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrxfp\" (UniqueName: \"kubernetes.io/projected/fd8d29db-3983-4f37-9476-6dca7b1b33b7-kube-api-access-vrxfp\") pod \"ingress-canary-pwjnv\" (UID: \"fd8d29db-3983-4f37-9476-6dca7b1b33b7\") " pod="openshift-ingress-canary/ingress-canary-pwjnv"
Jul 09 20:04:04.741075 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:04:04.741043    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-pwjnv"
Jul 09 20:04:04.896701 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:04:04.896507    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-pwjnv"]
Jul 09 20:04:04.901022 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:04:04.900995    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd8d29db_3983_4f37_9476_6dca7b1b33b7.slice/crio-78a1dfe23918aa7cec22cc12af8ba1c700598f680ee5b4727ae47e2c8ad3561a WatchSource:0}: Error finding container 78a1dfe23918aa7cec22cc12af8ba1c700598f680ee5b4727ae47e2c8ad3561a: Status 404 returned error can't find the container with id 78a1dfe23918aa7cec22cc12af8ba1c700598f680ee5b4727ae47e2c8ad3561a
Jul 09 20:04:05.098546 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:04:05.098423    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1f3c5da7-a9c8-4238-840b-df34177f1096-metrics-tls\") pod \"dns-default-vhphk\" (UID: \"1f3c5da7-a9c8-4238-840b-df34177f1096\") " pod="openshift-dns/dns-default-vhphk"
Jul 09 20:04:05.098701 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:04:05.098560    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 20:04:05.098701 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:04:05.098618    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f3c5da7-a9c8-4238-840b-df34177f1096-metrics-tls podName:1f3c5da7-a9c8-4238-840b-df34177f1096 nodeName:}" failed. No retries permitted until 2026-07-09 20:04:06.098603061 +0000 UTC m=+36.516272847 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1f3c5da7-a9c8-4238-840b-df34177f1096-metrics-tls") pod "dns-default-vhphk" (UID: "1f3c5da7-a9c8-4238-840b-df34177f1096") : secret "dns-default-metrics-tls" not found
Jul 09 20:04:05.449625 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:04:05.449537    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-pwjnv" event={"ID":"fd8d29db-3983-4f37-9476-6dca7b1b33b7","Type":"ContainerStarted","Data":"78a1dfe23918aa7cec22cc12af8ba1c700598f680ee5b4727ae47e2c8ad3561a"}
Jul 09 20:04:05.452590 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:04:05.452565    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dz4cq" event={"ID":"d592535b-5457-48d0-ae48-caed8efc04fc","Type":"ContainerStarted","Data":"b0e355d9c245244f5d5468bc1b5515d018cfde475618a6697b1ce363ea5f5ab2"}
Jul 09 20:04:05.506262 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:04:05.506209    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-dz4cq" podStartSLOduration=6.876113344 podStartE2EDuration="35.506193098s" totalImagesPullingTime="28.630079754s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 20:03:30 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 20:04:05.501458516 +0000 UTC m=+35.919128307" watchObservedRunningTime="2026-07-09 20:04:05.506193098 +0000 UTC m=+35.923862906"
Jul 09 20:04:06.107918 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:04:06.107828    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1f3c5da7-a9c8-4238-840b-df34177f1096-metrics-tls\") pod \"dns-default-vhphk\" (UID: \"1f3c5da7-a9c8-4238-840b-df34177f1096\") " pod="openshift-dns/dns-default-vhphk"
Jul 09 20:04:06.108169 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:04:06.108001    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 20:04:06.108169 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:04:06.108076    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f3c5da7-a9c8-4238-840b-df34177f1096-metrics-tls podName:1f3c5da7-a9c8-4238-840b-df34177f1096 nodeName:}" failed. No retries permitted until 2026-07-09 20:04:08.108060471 +0000 UTC m=+38.525730262 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1f3c5da7-a9c8-4238-840b-df34177f1096-metrics-tls") pod "dns-default-vhphk" (UID: "1f3c5da7-a9c8-4238-840b-df34177f1096") : secret "dns-default-metrics-tls" not found
Jul 09 20:04:06.235615 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:04:06.235578    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6b5z9"
Jul 09 20:04:06.235885 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:04:06.235588    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ksd8v"
Jul 09 20:04:06.235885 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:04:06.235722    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dccxz"
Jul 09 20:04:06.238969 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:04:06.238940    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 09 20:04:06.239105 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:04:06.238950    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 09 20:04:06.240526 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:04:06.240506    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-s29qd\""
Jul 09 20:04:06.240526 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:04:06.240523    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 09 20:04:06.240711 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:04:06.240593    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-kv5k8\""
Jul 09 20:04:06.240711 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:04:06.240524    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 09 20:04:08.123539 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:04:08.123507    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1f3c5da7-a9c8-4238-840b-df34177f1096-metrics-tls\") pod \"dns-default-vhphk\" (UID: \"1f3c5da7-a9c8-4238-840b-df34177f1096\") " pod="openshift-dns/dns-default-vhphk"
Jul 09 20:04:08.123994 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:04:08.123616    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 20:04:08.123994 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:04:08.123671    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f3c5da7-a9c8-4238-840b-df34177f1096-metrics-tls podName:1f3c5da7-a9c8-4238-840b-df34177f1096 nodeName:}" failed. No retries permitted until 2026-07-09 20:04:12.123655746 +0000 UTC m=+42.541325534 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1f3c5da7-a9c8-4238-840b-df34177f1096-metrics-tls") pod "dns-default-vhphk" (UID: "1f3c5da7-a9c8-4238-840b-df34177f1096") : secret "dns-default-metrics-tls" not found
Jul 09 20:04:08.462740 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:04:08.462704    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-pwjnv" event={"ID":"fd8d29db-3983-4f37-9476-6dca7b1b33b7","Type":"ContainerStarted","Data":"fb074c7e3ece3c975e183ae4bc60c690e9c5e644a5b152a6b92cf7d9ccd08db6"}
Jul 09 20:04:08.478649 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:04:08.478605    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-pwjnv" podStartSLOduration=1.255469601 podStartE2EDuration="4.478589296s" totalImagesPullingTime="3.223119695s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 20:04:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 20:04:08.47779405 +0000 UTC m=+38.895463859" watchObservedRunningTime="2026-07-09 20:04:08.478589296 +0000 UTC m=+38.896259145"
Jul 09 20:04:12.149804 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:04:12.149628    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1f3c5da7-a9c8-4238-840b-df34177f1096-metrics-tls\") pod \"dns-default-vhphk\" (UID: \"1f3c5da7-a9c8-4238-840b-df34177f1096\") " pod="openshift-dns/dns-default-vhphk"
Jul 09 20:04:12.149804 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:04:12.149780    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 20:04:12.150223 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:04:12.149854    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f3c5da7-a9c8-4238-840b-df34177f1096-metrics-tls podName:1f3c5da7-a9c8-4238-840b-df34177f1096 nodeName:}" failed. No retries permitted until 2026-07-09 20:04:20.149837746 +0000 UTC m=+50.567507546 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1f3c5da7-a9c8-4238-840b-df34177f1096-metrics-tls") pod "dns-default-vhphk" (UID: "1f3c5da7-a9c8-4238-840b-df34177f1096") : secret "dns-default-metrics-tls" not found
Jul 09 20:04:20.207982 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:04:20.207942    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1f3c5da7-a9c8-4238-840b-df34177f1096-metrics-tls\") pod \"dns-default-vhphk\" (UID: \"1f3c5da7-a9c8-4238-840b-df34177f1096\") " pod="openshift-dns/dns-default-vhphk"
Jul 09 20:04:20.208566 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:04:20.208061    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 20:04:20.208566 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:04:20.208113    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f3c5da7-a9c8-4238-840b-df34177f1096-metrics-tls podName:1f3c5da7-a9c8-4238-840b-df34177f1096 nodeName:}" failed. No retries permitted until 2026-07-09 20:04:36.208099655 +0000 UTC m=+66.625769443 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1f3c5da7-a9c8-4238-840b-df34177f1096-metrics-tls") pod "dns-default-vhphk" (UID: "1f3c5da7-a9c8-4238-840b-df34177f1096") : secret "dns-default-metrics-tls" not found
Jul 09 20:04:26.432260 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:04:26.432232    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hds4g"
Jul 09 20:04:34.805638 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:04:34.805594    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khzxc\" (UniqueName: \"kubernetes.io/projected/4fd1b369-9b81-495d-9608-9fc4d81295e2-kube-api-access-khzxc\") pod \"network-check-target-6b5z9\" (UID: \"4fd1b369-9b81-495d-9608-9fc4d81295e2\") " pod="openshift-network-diagnostics/network-check-target-6b5z9"
Jul 09 20:04:34.805638 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:04:34.805643    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/13454dc6-77a2-4dc9-adc2-81791d1cd463-original-pull-secret\") pod \"global-pull-secret-syncer-dccxz\" (UID: \"13454dc6-77a2-4dc9-adc2-81791d1cd463\") " pod="kube-system/global-pull-secret-syncer-dccxz"
Jul 09 20:04:34.808996 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:04:34.808978    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 09 20:04:34.809055 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:04:34.809000    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 09 20:04:34.819132 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:04:34.819113    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 09 20:04:34.820070 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:04:34.820040    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/13454dc6-77a2-4dc9-adc2-81791d1cd463-original-pull-secret\") pod \"global-pull-secret-syncer-dccxz\" (UID: \"13454dc6-77a2-4dc9-adc2-81791d1cd463\") " pod="kube-system/global-pull-secret-syncer-dccxz"
Jul 09 20:04:34.829149 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:04:34.829130    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-khzxc\" (UniqueName: \"kubernetes.io/projected/4fd1b369-9b81-495d-9608-9fc4d81295e2-kube-api-access-khzxc\") pod \"network-check-target-6b5z9\" (UID: \"4fd1b369-9b81-495d-9608-9fc4d81295e2\") " pod="openshift-network-diagnostics/network-check-target-6b5z9"
Jul 09 20:04:34.906239 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:04:34.906204    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14a1d22f-43ff-4eb8-a89d-920551b2ef1f-metrics-certs\") pod \"network-metrics-daemon-ksd8v\" (UID: \"14a1d22f-43ff-4eb8-a89d-920551b2ef1f\") " pod="openshift-multus/network-metrics-daemon-ksd8v"
Jul 09 20:04:34.909312 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:04:34.909279    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 09 20:04:34.916851 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:04:34.916834    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 09 20:04:34.916906 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:04:34.916893    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/14a1d22f-43ff-4eb8-a89d-920551b2ef1f-metrics-certs podName:14a1d22f-43ff-4eb8-a89d-920551b2ef1f nodeName:}" failed. No retries permitted until 2026-07-09 20:05:38.916876513 +0000 UTC m=+129.334546317 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/14a1d22f-43ff-4eb8-a89d-920551b2ef1f-metrics-certs") pod "network-metrics-daemon-ksd8v" (UID: "14a1d22f-43ff-4eb8-a89d-920551b2ef1f") : secret "metrics-daemon-secret" not found
Jul 09 20:04:35.051688 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:04:35.051653    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-kv5k8\""
Jul 09 20:04:35.059293 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:04:35.059245    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6b5z9"
Jul 09 20:04:35.061554 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:04:35.061532    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dccxz"
Jul 09 20:04:35.236701 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:04:35.236673    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-6b5z9"]
Jul 09 20:04:35.239245 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:04:35.239213    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fd1b369_9b81_495d_9608_9fc4d81295e2.slice/crio-3eb863260775ba6d7e94ad5f608aeebdacdd532d5f24568f793a18be26fd0898 WatchSource:0}: Error finding container 3eb863260775ba6d7e94ad5f608aeebdacdd532d5f24568f793a18be26fd0898: Status 404 returned error can't find the container with id 3eb863260775ba6d7e94ad5f608aeebdacdd532d5f24568f793a18be26fd0898
Jul 09 20:04:35.270360 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:04:35.270331    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-dccxz"]
Jul 09 20:04:35.273158 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:04:35.273131    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13454dc6_77a2_4dc9_adc2_81791d1cd463.slice/crio-3ce61257b6ebd5eee0df6d982dc6c04efb7a5acbf315db9288c32a664e7d8201 WatchSource:0}: Error finding container 3ce61257b6ebd5eee0df6d982dc6c04efb7a5acbf315db9288c32a664e7d8201: Status 404 returned error can't find the container with id 3ce61257b6ebd5eee0df6d982dc6c04efb7a5acbf315db9288c32a664e7d8201
Jul 09 20:04:35.528772 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:04:35.528736    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-dccxz" event={"ID":"13454dc6-77a2-4dc9-adc2-81791d1cd463","Type":"ContainerStarted","Data":"3ce61257b6ebd5eee0df6d982dc6c04efb7a5acbf315db9288c32a664e7d8201"}
Jul 09 20:04:35.529727 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:04:35.529701    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-6b5z9" event={"ID":"4fd1b369-9b81-495d-9608-9fc4d81295e2","Type":"ContainerStarted","Data":"3eb863260775ba6d7e94ad5f608aeebdacdd532d5f24568f793a18be26fd0898"}
Jul 09 20:04:36.217005 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:04:36.216963    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1f3c5da7-a9c8-4238-840b-df34177f1096-metrics-tls\") pod \"dns-default-vhphk\" (UID: \"1f3c5da7-a9c8-4238-840b-df34177f1096\") " pod="openshift-dns/dns-default-vhphk"
Jul 09 20:04:36.217516 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:04:36.217135    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 20:04:36.217516 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:04:36.217218    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f3c5da7-a9c8-4238-840b-df34177f1096-metrics-tls podName:1f3c5da7-a9c8-4238-840b-df34177f1096 nodeName:}" failed. No retries permitted until 2026-07-09 20:05:08.217195053 +0000 UTC m=+98.634864845 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1f3c5da7-a9c8-4238-840b-df34177f1096-metrics-tls") pod "dns-default-vhphk" (UID: "1f3c5da7-a9c8-4238-840b-df34177f1096") : secret "dns-default-metrics-tls" not found
Jul 09 20:04:40.545259 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:04:40.545207    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-dccxz" event={"ID":"13454dc6-77a2-4dc9-adc2-81791d1cd463","Type":"ContainerStarted","Data":"2dada5f8855d00c5a178ad040ee186bd38d64be436fccb4018a62a1c6b3c8277"}
Jul 09 20:04:40.546649 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:04:40.546626    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-6b5z9" event={"ID":"4fd1b369-9b81-495d-9608-9fc4d81295e2","Type":"ContainerStarted","Data":"22067d84c079e7f65804e320d87da33021565c0212b1c8616e6de386bec5bd08"}
Jul 09 20:04:40.546787 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:04:40.546747    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-6b5z9"
Jul 09 20:04:40.577999 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:04:40.577946    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-dccxz" podStartSLOduration=65.398949035 podStartE2EDuration="1m10.577927295s" totalImagesPullingTime="5.17897826s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 20:03:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 20:04:40.562174617 +0000 UTC m=+70.979844426" watchObservedRunningTime="2026-07-09 20:04:40.577927295 +0000 UTC m=+70.995597103"
Jul 09 20:04:40.578577 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:04:40.578550    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-6b5z9" podStartSLOduration=65.369286683 podStartE2EDuration="1m10.578538859s" totalImagesPullingTime="5.209252176s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 20:03:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 20:04:40.577436038 +0000 UTC m=+70.995105860" watchObservedRunningTime="2026-07-09 20:04:40.578538859 +0000 UTC m=+70.996208712"
Jul 09 20:05:08.225220 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:05:08.225176    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1f3c5da7-a9c8-4238-840b-df34177f1096-metrics-tls\") pod \"dns-default-vhphk\" (UID: \"1f3c5da7-a9c8-4238-840b-df34177f1096\") " pod="openshift-dns/dns-default-vhphk"
Jul 09 20:05:08.225686 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:05:08.225332    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 20:05:08.225686 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:05:08.225396    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f3c5da7-a9c8-4238-840b-df34177f1096-metrics-tls podName:1f3c5da7-a9c8-4238-840b-df34177f1096 nodeName:}" failed. No retries permitted until 2026-07-09 20:06:12.225380653 +0000 UTC m=+162.643050441 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1f3c5da7-a9c8-4238-840b-df34177f1096-metrics-tls") pod "dns-default-vhphk" (UID: "1f3c5da7-a9c8-4238-840b-df34177f1096") : secret "dns-default-metrics-tls" not found
Jul 09 20:05:11.550865 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:05:11.550827    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-6b5z9"
Jul 09 20:05:38.933526 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:05:38.933482    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14a1d22f-43ff-4eb8-a89d-920551b2ef1f-metrics-certs\") pod \"network-metrics-daemon-ksd8v\" (UID: \"14a1d22f-43ff-4eb8-a89d-920551b2ef1f\") " pod="openshift-multus/network-metrics-daemon-ksd8v"
Jul 09 20:05:38.934059 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:05:38.933631    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 09 20:05:38.934059 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:05:38.933694    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/14a1d22f-43ff-4eb8-a89d-920551b2ef1f-metrics-certs podName:14a1d22f-43ff-4eb8-a89d-920551b2ef1f nodeName:}" failed. No retries permitted until 2026-07-09 20:07:40.933676937 +0000 UTC m=+251.351346737 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/14a1d22f-43ff-4eb8-a89d-920551b2ef1f-metrics-certs") pod "network-metrics-daemon-ksd8v" (UID: "14a1d22f-43ff-4eb8-a89d-920551b2ef1f") : secret "metrics-daemon-secret" not found
Jul 09 20:06:07.427447 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:06:07.427395    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-vhphk" podUID="1f3c5da7-a9c8-4238-840b-df34177f1096"
Jul 09 20:06:07.764172 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:06:07.764143    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vhphk"
Jul 09 20:06:09.254974 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:06:09.254917    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-ksd8v" podUID="14a1d22f-43ff-4eb8-a89d-920551b2ef1f"
Jul 09 20:06:12.258548 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:06:12.258513    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1f3c5da7-a9c8-4238-840b-df34177f1096-metrics-tls\") pod \"dns-default-vhphk\" (UID: \"1f3c5da7-a9c8-4238-840b-df34177f1096\") " pod="openshift-dns/dns-default-vhphk"
Jul 09 20:06:12.259037 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:06:12.258633    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 20:06:12.259037 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:06:12.258686    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f3c5da7-a9c8-4238-840b-df34177f1096-metrics-tls podName:1f3c5da7-a9c8-4238-840b-df34177f1096 nodeName:}" failed. No retries permitted until 2026-07-09 20:08:14.258672758 +0000 UTC m=+284.676342545 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1f3c5da7-a9c8-4238-840b-df34177f1096-metrics-tls") pod "dns-default-vhphk" (UID: "1f3c5da7-a9c8-4238-840b-df34177f1096") : secret "dns-default-metrics-tls" not found
Jul 09 20:06:20.236610 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:06:20.236566    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ksd8v"
Jul 09 20:07:15.178472 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:15.178426    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-t9d7w"]
Jul 09 20:07:15.180590 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:15.180564    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cf856758c-t9d7w"
Jul 09 20:07:15.183863 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:15.183832    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Jul 09 20:07:15.183863 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:15.183851    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-lf9jv\""
Jul 09 20:07:15.185137 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:15.185124    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Jul 09 20:07:15.189682 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:15.189661    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-b66d5d5c8-9kl6t"]
Jul 09 20:07:15.189833 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:15.189814    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3b4c1edc-a174-4e8e-9f15-eccad2f85d52-nginx-conf\") pod \"networking-console-plugin-6cf856758c-t9d7w\" (UID: \"3b4c1edc-a174-4e8e-9f15-eccad2f85d52\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-t9d7w"
Jul 09 20:07:15.189874 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:15.189848    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3b4c1edc-a174-4e8e-9f15-eccad2f85d52-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-t9d7w\" (UID: \"3b4c1edc-a174-4e8e-9f15-eccad2f85d52\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-t9d7w"
Jul 09 20:07:15.191182 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:15.191169    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-b66d5d5c8-9kl6t"
Jul 09 20:07:15.194515 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:15.194490    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Jul 09 20:07:15.195324 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:15.195291    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Jul 09 20:07:15.195756 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:15.195737    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 09 20:07:15.196852 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:15.196829    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 09 20:07:15.197226 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:15.197213    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-r7nlg\""
Jul 09 20:07:15.197684 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:15.197672    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Jul 09 20:07:15.201987 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:15.201973    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Jul 09 20:07:15.211166 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:15.211144    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-t9d7w"]
Jul 09 20:07:15.218055 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:15.218036    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-b66d5d5c8-9kl6t"]
Jul 09 20:07:15.277640 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:15.277602    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7866497494-cqs6d"]
Jul 09 20:07:15.279523 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:15.279506    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7866497494-cqs6d"
Jul 09 20:07:15.283166 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:15.283134    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Jul 09 20:07:15.283294 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:15.283201    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Jul 09 20:07:15.283294 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:15.283211    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-l22ns\""
Jul 09 20:07:15.283294 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:15.283227    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Jul 09 20:07:15.283475 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:15.283429    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Jul 09 20:07:15.288568 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:15.288542    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Jul 09 20:07:15.290400 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:15.290379    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5773e57-8d05-422d-84b1-7c98763683f7-metrics-certs\") pod \"router-default-b66d5d5c8-9kl6t\" (UID: \"e5773e57-8d05-422d-84b1-7c98763683f7\") " pod="openshift-ingress/router-default-b66d5d5c8-9kl6t"
Jul 09 20:07:15.290504 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:15.290410    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dn2n5\" (UniqueName: \"kubernetes.io/projected/e5773e57-8d05-422d-84b1-7c98763683f7-kube-api-access-dn2n5\") pod \"router-default-b66d5d5c8-9kl6t\" (UID: \"e5773e57-8d05-422d-84b1-7c98763683f7\") " pod="openshift-ingress/router-default-b66d5d5c8-9kl6t"
Jul 09 20:07:15.290504 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:15.290439    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/e5773e57-8d05-422d-84b1-7c98763683f7-default-certificate\") pod \"router-default-b66d5d5c8-9kl6t\" (UID: \"e5773e57-8d05-422d-84b1-7c98763683f7\") " pod="openshift-ingress/router-default-b66d5d5c8-9kl6t"
Jul 09 20:07:15.290504 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:15.290491    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tz8nd\" (UniqueName: \"kubernetes.io/projected/578878d0-e6b3-4725-a698-796cfc169fa6-kube-api-access-tz8nd\") pod \"console-operator-7866497494-cqs6d\" (UID: \"578878d0-e6b3-4725-a698-796cfc169fa6\") " pod="openshift-console-operator/console-operator-7866497494-cqs6d"
Jul 09 20:07:15.290658 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:15.290518    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5773e57-8d05-422d-84b1-7c98763683f7-service-ca-bundle\") pod \"router-default-b66d5d5c8-9kl6t\" (UID: \"e5773e57-8d05-422d-84b1-7c98763683f7\") " pod="openshift-ingress/router-default-b66d5d5c8-9kl6t"
Jul 09 20:07:15.290658 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:15.290578    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/578878d0-e6b3-4725-a698-796cfc169fa6-trusted-ca\") pod \"console-operator-7866497494-cqs6d\" (UID: \"578878d0-e6b3-4725-a698-796cfc169fa6\") " pod="openshift-console-operator/console-operator-7866497494-cqs6d"
Jul 09 20:07:15.290658 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:15.290620    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/578878d0-e6b3-4725-a698-796cfc169fa6-serving-cert\") pod \"console-operator-7866497494-cqs6d\" (UID: \"578878d0-e6b3-4725-a698-796cfc169fa6\") " pod="openshift-console-operator/console-operator-7866497494-cqs6d"
Jul 09 20:07:15.290800 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:15.290668    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3b4c1edc-a174-4e8e-9f15-eccad2f85d52-nginx-conf\") pod \"networking-console-plugin-6cf856758c-t9d7w\" (UID: \"3b4c1edc-a174-4e8e-9f15-eccad2f85d52\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-t9d7w"
Jul 09 20:07:15.290800 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:15.290697    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/578878d0-e6b3-4725-a698-796cfc169fa6-config\") pod \"console-operator-7866497494-cqs6d\" (UID: \"578878d0-e6b3-4725-a698-796cfc169fa6\") " pod="openshift-console-operator/console-operator-7866497494-cqs6d"
Jul 09 20:07:15.290800 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:15.290735    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3b4c1edc-a174-4e8e-9f15-eccad2f85d52-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-t9d7w\" (UID: \"3b4c1edc-a174-4e8e-9f15-eccad2f85d52\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-t9d7w"
Jul 09 20:07:15.290800 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:15.290760    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/e5773e57-8d05-422d-84b1-7c98763683f7-stats-auth\") pod \"router-default-b66d5d5c8-9kl6t\" (UID: \"e5773e57-8d05-422d-84b1-7c98763683f7\") " pod="openshift-ingress/router-default-b66d5d5c8-9kl6t"
Jul 09 20:07:15.290930 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:07:15.290862    2562 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 09 20:07:15.290930 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:07:15.290925    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b4c1edc-a174-4e8e-9f15-eccad2f85d52-networking-console-plugin-cert podName:3b4c1edc-a174-4e8e-9f15-eccad2f85d52 nodeName:}" failed. No retries permitted until 2026-07-09 20:07:15.790904706 +0000 UTC m=+226.208574499 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3b4c1edc-a174-4e8e-9f15-eccad2f85d52-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-t9d7w" (UID: "3b4c1edc-a174-4e8e-9f15-eccad2f85d52") : secret "networking-console-plugin-cert" not found
Jul 09 20:07:15.292107 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:15.291925    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3b4c1edc-a174-4e8e-9f15-eccad2f85d52-nginx-conf\") pod \"networking-console-plugin-6cf856758c-t9d7w\" (UID: \"3b4c1edc-a174-4e8e-9f15-eccad2f85d52\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-t9d7w"
Jul 09 20:07:15.293395 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:15.293379    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7866497494-cqs6d"]
Jul 09 20:07:15.391293 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:15.391263    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5773e57-8d05-422d-84b1-7c98763683f7-metrics-certs\") pod \"router-default-b66d5d5c8-9kl6t\" (UID: \"e5773e57-8d05-422d-84b1-7c98763683f7\") " pod="openshift-ingress/router-default-b66d5d5c8-9kl6t"
Jul 09 20:07:15.391478 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:15.391294    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dn2n5\" (UniqueName: \"kubernetes.io/projected/e5773e57-8d05-422d-84b1-7c98763683f7-kube-api-access-dn2n5\") pod \"router-default-b66d5d5c8-9kl6t\" (UID: \"e5773e57-8d05-422d-84b1-7c98763683f7\") " pod="openshift-ingress/router-default-b66d5d5c8-9kl6t"
Jul 09 20:07:15.391478 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:15.391335    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/e5773e57-8d05-422d-84b1-7c98763683f7-default-certificate\") pod \"router-default-b66d5d5c8-9kl6t\" (UID: \"e5773e57-8d05-422d-84b1-7c98763683f7\") " pod="openshift-ingress/router-default-b66d5d5c8-9kl6t"
Jul 09 20:07:15.391478 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:15.391385    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tz8nd\" (UniqueName: \"kubernetes.io/projected/578878d0-e6b3-4725-a698-796cfc169fa6-kube-api-access-tz8nd\") pod \"console-operator-7866497494-cqs6d\" (UID: \"578878d0-e6b3-4725-a698-796cfc169fa6\") " pod="openshift-console-operator/console-operator-7866497494-cqs6d"
Jul 09 20:07:15.391478 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:07:15.391422    2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 09 20:07:15.391478 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:07:15.391481    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5773e57-8d05-422d-84b1-7c98763683f7-metrics-certs podName:e5773e57-8d05-422d-84b1-7c98763683f7 nodeName:}" failed. No retries permitted until 2026-07-09 20:07:15.891465177 +0000 UTC m=+226.309134963 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e5773e57-8d05-422d-84b1-7c98763683f7-metrics-certs") pod "router-default-b66d5d5c8-9kl6t" (UID: "e5773e57-8d05-422d-84b1-7c98763683f7") : secret "router-metrics-certs-default" not found
Jul 09 20:07:15.391735 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:15.391421    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5773e57-8d05-422d-84b1-7c98763683f7-service-ca-bundle\") pod \"router-default-b66d5d5c8-9kl6t\" (UID: \"e5773e57-8d05-422d-84b1-7c98763683f7\") " pod="openshift-ingress/router-default-b66d5d5c8-9kl6t"
Jul 09 20:07:15.391735 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:07:15.391528    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e5773e57-8d05-422d-84b1-7c98763683f7-service-ca-bundle podName:e5773e57-8d05-422d-84b1-7c98763683f7 nodeName:}" failed. No retries permitted until 2026-07-09 20:07:15.891509961 +0000 UTC m=+226.309179780 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e5773e57-8d05-422d-84b1-7c98763683f7-service-ca-bundle") pod "router-default-b66d5d5c8-9kl6t" (UID: "e5773e57-8d05-422d-84b1-7c98763683f7") : configmap references non-existent config key: service-ca.crt
Jul 09 20:07:15.391735 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:15.391598    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/578878d0-e6b3-4725-a698-796cfc169fa6-trusted-ca\") pod \"console-operator-7866497494-cqs6d\" (UID: \"578878d0-e6b3-4725-a698-796cfc169fa6\") " pod="openshift-console-operator/console-operator-7866497494-cqs6d"
Jul 09 20:07:15.391735 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:15.391627    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/578878d0-e6b3-4725-a698-796cfc169fa6-serving-cert\") pod \"console-operator-7866497494-cqs6d\" (UID: \"578878d0-e6b3-4725-a698-796cfc169fa6\") " pod="openshift-console-operator/console-operator-7866497494-cqs6d"
Jul 09 20:07:15.391735 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:15.391660    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/578878d0-e6b3-4725-a698-796cfc169fa6-config\") pod \"console-operator-7866497494-cqs6d\" (UID: \"578878d0-e6b3-4725-a698-796cfc169fa6\") " pod="openshift-console-operator/console-operator-7866497494-cqs6d"
Jul 09 20:07:15.391735 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:15.391693    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/e5773e57-8d05-422d-84b1-7c98763683f7-stats-auth\") pod \"router-default-b66d5d5c8-9kl6t\" (UID: \"e5773e57-8d05-422d-84b1-7c98763683f7\") " pod="openshift-ingress/router-default-b66d5d5c8-9kl6t"
Jul 09 20:07:15.392396 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:15.392375    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/578878d0-e6b3-4725-a698-796cfc169fa6-config\") pod \"console-operator-7866497494-cqs6d\" (UID: \"578878d0-e6b3-4725-a698-796cfc169fa6\") " pod="openshift-console-operator/console-operator-7866497494-cqs6d"
Jul 09 20:07:15.392776 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:15.392751    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/578878d0-e6b3-4725-a698-796cfc169fa6-trusted-ca\") pod \"console-operator-7866497494-cqs6d\" (UID: \"578878d0-e6b3-4725-a698-796cfc169fa6\") " pod="openshift-console-operator/console-operator-7866497494-cqs6d"
Jul 09 20:07:15.393958 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:15.393932    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/578878d0-e6b3-4725-a698-796cfc169fa6-serving-cert\") pod \"console-operator-7866497494-cqs6d\" (UID: \"578878d0-e6b3-4725-a698-796cfc169fa6\") " pod="openshift-console-operator/console-operator-7866497494-cqs6d"
Jul 09 20:07:15.394176 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:15.394159    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/e5773e57-8d05-422d-84b1-7c98763683f7-default-certificate\") pod \"router-default-b66d5d5c8-9kl6t\" (UID: \"e5773e57-8d05-422d-84b1-7c98763683f7\") " pod="openshift-ingress/router-default-b66d5d5c8-9kl6t"
Jul 09 20:07:15.394246 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:15.394156    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/e5773e57-8d05-422d-84b1-7c98763683f7-stats-auth\") pod \"router-default-b66d5d5c8-9kl6t\" (UID: \"e5773e57-8d05-422d-84b1-7c98763683f7\") " pod="openshift-ingress/router-default-b66d5d5c8-9kl6t"
Jul 09 20:07:15.404244 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:15.404224    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dn2n5\" (UniqueName: \"kubernetes.io/projected/e5773e57-8d05-422d-84b1-7c98763683f7-kube-api-access-dn2n5\") pod \"router-default-b66d5d5c8-9kl6t\" (UID: \"e5773e57-8d05-422d-84b1-7c98763683f7\") " pod="openshift-ingress/router-default-b66d5d5c8-9kl6t"
Jul 09 20:07:15.404350 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:15.404290    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tz8nd\" (UniqueName: \"kubernetes.io/projected/578878d0-e6b3-4725-a698-796cfc169fa6-kube-api-access-tz8nd\") pod \"console-operator-7866497494-cqs6d\" (UID: \"578878d0-e6b3-4725-a698-796cfc169fa6\") " pod="openshift-console-operator/console-operator-7866497494-cqs6d"
Jul 09 20:07:15.589206 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:15.589156    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7866497494-cqs6d"
Jul 09 20:07:15.709701 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:15.709664    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7866497494-cqs6d"]
Jul 09 20:07:15.714249 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:07:15.714225    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod578878d0_e6b3_4725_a698_796cfc169fa6.slice/crio-7f2a1ed43edf0a572497b57cba6058697580b8745ae3a425554d91e75832b2d8 WatchSource:0}: Error finding container 7f2a1ed43edf0a572497b57cba6058697580b8745ae3a425554d91e75832b2d8: Status 404 returned error can't find the container with id 7f2a1ed43edf0a572497b57cba6058697580b8745ae3a425554d91e75832b2d8
Jul 09 20:07:15.795745 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:15.795701    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3b4c1edc-a174-4e8e-9f15-eccad2f85d52-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-t9d7w\" (UID: \"3b4c1edc-a174-4e8e-9f15-eccad2f85d52\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-t9d7w"
Jul 09 20:07:15.795949 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:07:15.795841    2562 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 09 20:07:15.795949 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:07:15.795914    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b4c1edc-a174-4e8e-9f15-eccad2f85d52-networking-console-plugin-cert podName:3b4c1edc-a174-4e8e-9f15-eccad2f85d52 nodeName:}" failed. No retries permitted until 2026-07-09 20:07:16.795893869 +0000 UTC m=+227.213563679 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3b4c1edc-a174-4e8e-9f15-eccad2f85d52-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-t9d7w" (UID: "3b4c1edc-a174-4e8e-9f15-eccad2f85d52") : secret "networking-console-plugin-cert" not found
Jul 09 20:07:15.896149 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:15.896041    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5773e57-8d05-422d-84b1-7c98763683f7-metrics-certs\") pod \"router-default-b66d5d5c8-9kl6t\" (UID: \"e5773e57-8d05-422d-84b1-7c98763683f7\") " pod="openshift-ingress/router-default-b66d5d5c8-9kl6t"
Jul 09 20:07:15.896149 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:15.896100    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5773e57-8d05-422d-84b1-7c98763683f7-service-ca-bundle\") pod \"router-default-b66d5d5c8-9kl6t\" (UID: \"e5773e57-8d05-422d-84b1-7c98763683f7\") " pod="openshift-ingress/router-default-b66d5d5c8-9kl6t"
Jul 09 20:07:15.896418 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:07:15.896207    2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 09 20:07:15.896418 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:07:15.896235    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e5773e57-8d05-422d-84b1-7c98763683f7-service-ca-bundle podName:e5773e57-8d05-422d-84b1-7c98763683f7 nodeName:}" failed. No retries permitted until 2026-07-09 20:07:16.896217444 +0000 UTC m=+227.313887245 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e5773e57-8d05-422d-84b1-7c98763683f7-service-ca-bundle") pod "router-default-b66d5d5c8-9kl6t" (UID: "e5773e57-8d05-422d-84b1-7c98763683f7") : configmap references non-existent config key: service-ca.crt
Jul 09 20:07:15.896418 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:07:15.896285    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5773e57-8d05-422d-84b1-7c98763683f7-metrics-certs podName:e5773e57-8d05-422d-84b1-7c98763683f7 nodeName:}" failed. No retries permitted until 2026-07-09 20:07:16.896266695 +0000 UTC m=+227.313936482 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e5773e57-8d05-422d-84b1-7c98763683f7-metrics-certs") pod "router-default-b66d5d5c8-9kl6t" (UID: "e5773e57-8d05-422d-84b1-7c98763683f7") : secret "router-metrics-certs-default" not found
Jul 09 20:07:15.926965 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:15.926931    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-cqs6d" event={"ID":"578878d0-e6b3-4725-a698-796cfc169fa6","Type":"ContainerStarted","Data":"7f2a1ed43edf0a572497b57cba6058697580b8745ae3a425554d91e75832b2d8"}
Jul 09 20:07:16.802379 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:16.802330    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3b4c1edc-a174-4e8e-9f15-eccad2f85d52-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-t9d7w\" (UID: \"3b4c1edc-a174-4e8e-9f15-eccad2f85d52\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-t9d7w"
Jul 09 20:07:16.802782 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:07:16.802499    2562 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 09 20:07:16.802782 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:07:16.802576    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b4c1edc-a174-4e8e-9f15-eccad2f85d52-networking-console-plugin-cert podName:3b4c1edc-a174-4e8e-9f15-eccad2f85d52 nodeName:}" failed. No retries permitted until 2026-07-09 20:07:18.802560325 +0000 UTC m=+229.220230113 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3b4c1edc-a174-4e8e-9f15-eccad2f85d52-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-t9d7w" (UID: "3b4c1edc-a174-4e8e-9f15-eccad2f85d52") : secret "networking-console-plugin-cert" not found
Jul 09 20:07:16.903319 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:16.903265    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5773e57-8d05-422d-84b1-7c98763683f7-metrics-certs\") pod \"router-default-b66d5d5c8-9kl6t\" (UID: \"e5773e57-8d05-422d-84b1-7c98763683f7\") " pod="openshift-ingress/router-default-b66d5d5c8-9kl6t"
Jul 09 20:07:16.903319 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:16.903326    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5773e57-8d05-422d-84b1-7c98763683f7-service-ca-bundle\") pod \"router-default-b66d5d5c8-9kl6t\" (UID: \"e5773e57-8d05-422d-84b1-7c98763683f7\") " pod="openshift-ingress/router-default-b66d5d5c8-9kl6t"
Jul 09 20:07:16.903548 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:07:16.903413    2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 09 20:07:16.903548 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:07:16.903444    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e5773e57-8d05-422d-84b1-7c98763683f7-service-ca-bundle podName:e5773e57-8d05-422d-84b1-7c98763683f7 nodeName:}" failed. No retries permitted until 2026-07-09 20:07:18.903429668 +0000 UTC m=+229.321099469 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e5773e57-8d05-422d-84b1-7c98763683f7-service-ca-bundle") pod "router-default-b66d5d5c8-9kl6t" (UID: "e5773e57-8d05-422d-84b1-7c98763683f7") : configmap references non-existent config key: service-ca.crt
Jul 09 20:07:16.903548 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:07:16.903466    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5773e57-8d05-422d-84b1-7c98763683f7-metrics-certs podName:e5773e57-8d05-422d-84b1-7c98763683f7 nodeName:}" failed. No retries permitted until 2026-07-09 20:07:18.903452233 +0000 UTC m=+229.321122024 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e5773e57-8d05-422d-84b1-7c98763683f7-metrics-certs") pod "router-default-b66d5d5c8-9kl6t" (UID: "e5773e57-8d05-422d-84b1-7c98763683f7") : secret "router-metrics-certs-default" not found
Jul 09 20:07:17.934248 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:17.934226    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-cqs6d_578878d0-e6b3-4725-a698-796cfc169fa6/console-operator/0.log"
Jul 09 20:07:17.934663 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:17.934263    2562 generic.go:356] "Generic (PLEG): container finished" podID="578878d0-e6b3-4725-a698-796cfc169fa6" containerID="f83de38b2eb705b47cfed4573c15deee3b80e9cc0c7b903c16fce1da155f94d7" exitCode=255
Jul 09 20:07:17.934663 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:17.934343    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-cqs6d" event={"ID":"578878d0-e6b3-4725-a698-796cfc169fa6","Type":"ContainerDied","Data":"f83de38b2eb705b47cfed4573c15deee3b80e9cc0c7b903c16fce1da155f94d7"}
Jul 09 20:07:17.934663 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:17.934617    2562 scope.go:117] "RemoveContainer" containerID="f83de38b2eb705b47cfed4573c15deee3b80e9cc0c7b903c16fce1da155f94d7"
Jul 09 20:07:18.820043 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:18.820011    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3b4c1edc-a174-4e8e-9f15-eccad2f85d52-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-t9d7w\" (UID: \"3b4c1edc-a174-4e8e-9f15-eccad2f85d52\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-t9d7w"
Jul 09 20:07:18.820239 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:07:18.820131    2562 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 09 20:07:18.820239 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:07:18.820186    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b4c1edc-a174-4e8e-9f15-eccad2f85d52-networking-console-plugin-cert podName:3b4c1edc-a174-4e8e-9f15-eccad2f85d52 nodeName:}" failed. No retries permitted until 2026-07-09 20:07:22.820172073 +0000 UTC m=+233.237841860 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3b4c1edc-a174-4e8e-9f15-eccad2f85d52-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-t9d7w" (UID: "3b4c1edc-a174-4e8e-9f15-eccad2f85d52") : secret "networking-console-plugin-cert" not found
Jul 09 20:07:18.920822 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:18.920773    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5773e57-8d05-422d-84b1-7c98763683f7-metrics-certs\") pod \"router-default-b66d5d5c8-9kl6t\" (UID: \"e5773e57-8d05-422d-84b1-7c98763683f7\") " pod="openshift-ingress/router-default-b66d5d5c8-9kl6t"
Jul 09 20:07:18.920822 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:18.920833    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5773e57-8d05-422d-84b1-7c98763683f7-service-ca-bundle\") pod \"router-default-b66d5d5c8-9kl6t\" (UID: \"e5773e57-8d05-422d-84b1-7c98763683f7\") " pod="openshift-ingress/router-default-b66d5d5c8-9kl6t"
Jul 09 20:07:18.921081 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:07:18.920943    2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 09 20:07:18.921081 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:07:18.920959    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e5773e57-8d05-422d-84b1-7c98763683f7-service-ca-bundle podName:e5773e57-8d05-422d-84b1-7c98763683f7 nodeName:}" failed. No retries permitted until 2026-07-09 20:07:22.920943377 +0000 UTC m=+233.338613164 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e5773e57-8d05-422d-84b1-7c98763683f7-service-ca-bundle") pod "router-default-b66d5d5c8-9kl6t" (UID: "e5773e57-8d05-422d-84b1-7c98763683f7") : configmap references non-existent config key: service-ca.crt
Jul 09 20:07:18.921081 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:07:18.921017    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5773e57-8d05-422d-84b1-7c98763683f7-metrics-certs podName:e5773e57-8d05-422d-84b1-7c98763683f7 nodeName:}" failed. No retries permitted until 2026-07-09 20:07:22.921000913 +0000 UTC m=+233.338670705 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e5773e57-8d05-422d-84b1-7c98763683f7-metrics-certs") pod "router-default-b66d5d5c8-9kl6t" (UID: "e5773e57-8d05-422d-84b1-7c98763683f7") : secret "router-metrics-certs-default" not found
Jul 09 20:07:18.938758 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:18.938726    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-cqs6d_578878d0-e6b3-4725-a698-796cfc169fa6/console-operator/1.log"
Jul 09 20:07:18.939132 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:18.939112    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-cqs6d_578878d0-e6b3-4725-a698-796cfc169fa6/console-operator/0.log"
Jul 09 20:07:18.939189 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:18.939149    2562 generic.go:356] "Generic (PLEG): container finished" podID="578878d0-e6b3-4725-a698-796cfc169fa6" containerID="207e7bb147cdd6b535445ed48c67b14cea1756b3637f54206d52d869f3f3394c" exitCode=255
Jul 09 20:07:18.939234 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:18.939217    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-cqs6d" event={"ID":"578878d0-e6b3-4725-a698-796cfc169fa6","Type":"ContainerDied","Data":"207e7bb147cdd6b535445ed48c67b14cea1756b3637f54206d52d869f3f3394c"}
Jul 09 20:07:18.939289 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:18.939277    2562 scope.go:117] "RemoveContainer" containerID="f83de38b2eb705b47cfed4573c15deee3b80e9cc0c7b903c16fce1da155f94d7"
Jul 09 20:07:18.939520 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:18.939496    2562 scope.go:117] "RemoveContainer" containerID="207e7bb147cdd6b535445ed48c67b14cea1756b3637f54206d52d869f3f3394c"
Jul 09 20:07:18.939677 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:07:18.939652    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-cqs6d_openshift-console-operator(578878d0-e6b3-4725-a698-796cfc169fa6)\"" pod="openshift-console-operator/console-operator-7866497494-cqs6d" podUID="578878d0-e6b3-4725-a698-796cfc169fa6"
Jul 09 20:07:19.942719 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:19.942691    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-cqs6d_578878d0-e6b3-4725-a698-796cfc169fa6/console-operator/1.log"
Jul 09 20:07:19.943121 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:19.943009    2562 scope.go:117] "RemoveContainer" containerID="207e7bb147cdd6b535445ed48c67b14cea1756b3637f54206d52d869f3f3394c"
Jul 09 20:07:19.943171 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:07:19.943151    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-cqs6d_openshift-console-operator(578878d0-e6b3-4725-a698-796cfc169fa6)\"" pod="openshift-console-operator/console-operator-7866497494-cqs6d" podUID="578878d0-e6b3-4725-a698-796cfc169fa6"
Jul 09 20:07:20.057029 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:20.056993    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-7qx9m"]
Jul 09 20:07:20.059632 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:20.059607    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7qx9m"
Jul 09 20:07:20.062765 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:20.062743    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 09 20:07:20.062907 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:20.062781    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 09 20:07:20.062907 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:20.062781    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 09 20:07:20.062907 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:20.062807    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-6wftc\""
Jul 09 20:07:20.062907 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:20.062781    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 09 20:07:20.070408 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:20.070389    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-7qx9m"]
Jul 09 20:07:20.128011 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:20.127981    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/09094efa-769d-4669-944e-a76a98aa185e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7qx9m\" (UID: \"09094efa-769d-4669-944e-a76a98aa185e\") " pod="openshift-insights/insights-runtime-extractor-7qx9m"
Jul 09 20:07:20.128200 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:20.128065    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/09094efa-769d-4669-944e-a76a98aa185e-crio-socket\") pod \"insights-runtime-extractor-7qx9m\" (UID: \"09094efa-769d-4669-944e-a76a98aa185e\") " pod="openshift-insights/insights-runtime-extractor-7qx9m"
Jul 09 20:07:20.128200 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:20.128099    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/09094efa-769d-4669-944e-a76a98aa185e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7qx9m\" (UID: \"09094efa-769d-4669-944e-a76a98aa185e\") " pod="openshift-insights/insights-runtime-extractor-7qx9m"
Jul 09 20:07:20.128200 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:20.128116    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zt6ph\" (UniqueName: \"kubernetes.io/projected/09094efa-769d-4669-944e-a76a98aa185e-kube-api-access-zt6ph\") pod \"insights-runtime-extractor-7qx9m\" (UID: \"09094efa-769d-4669-944e-a76a98aa185e\") " pod="openshift-insights/insights-runtime-extractor-7qx9m"
Jul 09 20:07:20.128200 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:20.128168    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/09094efa-769d-4669-944e-a76a98aa185e-data-volume\") pod \"insights-runtime-extractor-7qx9m\" (UID: \"09094efa-769d-4669-944e-a76a98aa185e\") " pod="openshift-insights/insights-runtime-extractor-7qx9m"
Jul 09 20:07:20.229467 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:20.229436    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/09094efa-769d-4669-944e-a76a98aa185e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7qx9m\" (UID: \"09094efa-769d-4669-944e-a76a98aa185e\") " pod="openshift-insights/insights-runtime-extractor-7qx9m"
Jul 09 20:07:20.229626 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:20.229498    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/09094efa-769d-4669-944e-a76a98aa185e-crio-socket\") pod \"insights-runtime-extractor-7qx9m\" (UID: \"09094efa-769d-4669-944e-a76a98aa185e\") " pod="openshift-insights/insights-runtime-extractor-7qx9m"
Jul 09 20:07:20.229626 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:20.229518    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/09094efa-769d-4669-944e-a76a98aa185e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7qx9m\" (UID: \"09094efa-769d-4669-944e-a76a98aa185e\") " pod="openshift-insights/insights-runtime-extractor-7qx9m"
Jul 09 20:07:20.229626 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:20.229534    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zt6ph\" (UniqueName: \"kubernetes.io/projected/09094efa-769d-4669-944e-a76a98aa185e-kube-api-access-zt6ph\") pod \"insights-runtime-extractor-7qx9m\" (UID: \"09094efa-769d-4669-944e-a76a98aa185e\") " pod="openshift-insights/insights-runtime-extractor-7qx9m"
Jul 09 20:07:20.229626 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:20.229550    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/09094efa-769d-4669-944e-a76a98aa185e-data-volume\") pod \"insights-runtime-extractor-7qx9m\" (UID: \"09094efa-769d-4669-944e-a76a98aa185e\") " pod="openshift-insights/insights-runtime-extractor-7qx9m"
Jul 09 20:07:20.229626 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:07:20.229596    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 09 20:07:20.229626 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:07:20.229620    2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 09 20:07:20.229858 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:07:20.229675    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/09094efa-769d-4669-944e-a76a98aa185e-insights-runtime-extractor-tls podName:09094efa-769d-4669-944e-a76a98aa185e nodeName:}" failed. No retries permitted until 2026-07-09 20:07:20.729655004 +0000 UTC m=+231.147324792 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/09094efa-769d-4669-944e-a76a98aa185e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7qx9m" (UID: "09094efa-769d-4669-944e-a76a98aa185e") : secret "insights-runtime-extractor-tls" not found
Jul 09 20:07:20.229858 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:20.229677    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/09094efa-769d-4669-944e-a76a98aa185e-crio-socket\") pod \"insights-runtime-extractor-7qx9m\" (UID: \"09094efa-769d-4669-944e-a76a98aa185e\") " pod="openshift-insights/insights-runtime-extractor-7qx9m"
Jul 09 20:07:20.229858 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:07:20.229689    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/09094efa-769d-4669-944e-a76a98aa185e-kube-rbac-proxy-cm podName:09094efa-769d-4669-944e-a76a98aa185e nodeName:}" failed. No retries permitted until 2026-07-09 20:07:20.729683059 +0000 UTC m=+231.147352846 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/09094efa-769d-4669-944e-a76a98aa185e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7qx9m" (UID: "09094efa-769d-4669-944e-a76a98aa185e") : configmap "kube-rbac-proxy" not found
Jul 09 20:07:20.229963 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:20.229880    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/09094efa-769d-4669-944e-a76a98aa185e-data-volume\") pod \"insights-runtime-extractor-7qx9m\" (UID: \"09094efa-769d-4669-944e-a76a98aa185e\") " pod="openshift-insights/insights-runtime-extractor-7qx9m"
Jul 09 20:07:20.238138 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:20.238111    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zt6ph\" (UniqueName: \"kubernetes.io/projected/09094efa-769d-4669-944e-a76a98aa185e-kube-api-access-zt6ph\") pod \"insights-runtime-extractor-7qx9m\" (UID: \"09094efa-769d-4669-944e-a76a98aa185e\") " pod="openshift-insights/insights-runtime-extractor-7qx9m"
Jul 09 20:07:20.734024 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:20.733993    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/09094efa-769d-4669-944e-a76a98aa185e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7qx9m\" (UID: \"09094efa-769d-4669-944e-a76a98aa185e\") " pod="openshift-insights/insights-runtime-extractor-7qx9m"
Jul 09 20:07:20.734199 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:20.734062    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/09094efa-769d-4669-944e-a76a98aa185e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7qx9m\" (UID: \"09094efa-769d-4669-944e-a76a98aa185e\") " pod="openshift-insights/insights-runtime-extractor-7qx9m"
Jul 09 20:07:20.734199 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:07:20.734143    2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 09 20:07:20.734199 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:07:20.734167    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 09 20:07:20.734199 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:07:20.734197    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/09094efa-769d-4669-944e-a76a98aa185e-kube-rbac-proxy-cm podName:09094efa-769d-4669-944e-a76a98aa185e nodeName:}" failed. No retries permitted until 2026-07-09 20:07:21.734183488 +0000 UTC m=+232.151853275 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/09094efa-769d-4669-944e-a76a98aa185e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7qx9m" (UID: "09094efa-769d-4669-944e-a76a98aa185e") : configmap "kube-rbac-proxy" not found
Jul 09 20:07:20.734364 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:07:20.734232    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/09094efa-769d-4669-944e-a76a98aa185e-insights-runtime-extractor-tls podName:09094efa-769d-4669-944e-a76a98aa185e nodeName:}" failed. No retries permitted until 2026-07-09 20:07:21.734217545 +0000 UTC m=+232.151887333 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/09094efa-769d-4669-944e-a76a98aa185e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7qx9m" (UID: "09094efa-769d-4669-944e-a76a98aa185e") : secret "insights-runtime-extractor-tls" not found
Jul 09 20:07:21.167177 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:21.167088    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-8gtgc"]
Jul 09 20:07:21.169093 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:21.169077    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-8gtgc"
Jul 09 20:07:21.172973 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:21.172949    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Jul 09 20:07:21.173210 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:21.173196    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Jul 09 20:07:21.173325 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:21.173289    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-4dxmd\""
Jul 09 20:07:21.179335 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:21.179314    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-8gtgc"]
Jul 09 20:07:21.236623 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:21.236586    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fj2hj\" (UniqueName: \"kubernetes.io/projected/89b4a7d6-9699-455b-af5f-473a56d4f1fc-kube-api-access-fj2hj\") pod \"migrator-7f59b54f6c-8gtgc\" (UID: \"89b4a7d6-9699-455b-af5f-473a56d4f1fc\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-8gtgc"
Jul 09 20:07:21.337278 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:21.337237    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fj2hj\" (UniqueName: \"kubernetes.io/projected/89b4a7d6-9699-455b-af5f-473a56d4f1fc-kube-api-access-fj2hj\") pod \"migrator-7f59b54f6c-8gtgc\" (UID: \"89b4a7d6-9699-455b-af5f-473a56d4f1fc\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-8gtgc"
Jul 09 20:07:21.345113 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:21.345092    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fj2hj\" (UniqueName: \"kubernetes.io/projected/89b4a7d6-9699-455b-af5f-473a56d4f1fc-kube-api-access-fj2hj\") pod \"migrator-7f59b54f6c-8gtgc\" (UID: \"89b4a7d6-9699-455b-af5f-473a56d4f1fc\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-8gtgc"
Jul 09 20:07:21.478451 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:21.478417    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-8gtgc"
Jul 09 20:07:21.488188 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:21.488157    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-qmg2j"]
Jul 09 20:07:21.490457 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:21.490438    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-qmg2j"
Jul 09 20:07:21.493772 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:21.493481    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 09 20:07:21.493772 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:21.493620    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-s5ssh\""
Jul 09 20:07:21.493772 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:21.493655    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 09 20:07:21.493772 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:21.493716    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 09 20:07:21.494081 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:21.493922    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 09 20:07:21.498703 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:21.498674    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-qmg2j"]
Jul 09 20:07:21.539727 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:21.539698    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/52d4510f-bde3-4471-a0a8-04015a4cff02-signing-key\") pod \"service-ca-79f68b9f4f-qmg2j\" (UID: \"52d4510f-bde3-4471-a0a8-04015a4cff02\") " pod="openshift-service-ca/service-ca-79f68b9f4f-qmg2j"
Jul 09 20:07:21.539889 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:21.539735    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/52d4510f-bde3-4471-a0a8-04015a4cff02-signing-cabundle\") pod \"service-ca-79f68b9f4f-qmg2j\" (UID: \"52d4510f-bde3-4471-a0a8-04015a4cff02\") " pod="openshift-service-ca/service-ca-79f68b9f4f-qmg2j"
Jul 09 20:07:21.539889 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:21.539779    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mf6wh\" (UniqueName: \"kubernetes.io/projected/52d4510f-bde3-4471-a0a8-04015a4cff02-kube-api-access-mf6wh\") pod \"service-ca-79f68b9f4f-qmg2j\" (UID: \"52d4510f-bde3-4471-a0a8-04015a4cff02\") " pod="openshift-service-ca/service-ca-79f68b9f4f-qmg2j"
Jul 09 20:07:21.599953 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:21.599924    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-8gtgc"]
Jul 09 20:07:21.602438 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:07:21.602411    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89b4a7d6_9699_455b_af5f_473a56d4f1fc.slice/crio-6964d83fb9edd8e073f8fdbd9d1211daa4ec2539a74962568eabbec3dc63640e WatchSource:0}: Error finding container 6964d83fb9edd8e073f8fdbd9d1211daa4ec2539a74962568eabbec3dc63640e: Status 404 returned error can't find the container with id 6964d83fb9edd8e073f8fdbd9d1211daa4ec2539a74962568eabbec3dc63640e
Jul 09 20:07:21.640313 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:21.640281    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/52d4510f-bde3-4471-a0a8-04015a4cff02-signing-key\") pod \"service-ca-79f68b9f4f-qmg2j\" (UID: \"52d4510f-bde3-4471-a0a8-04015a4cff02\") " pod="openshift-service-ca/service-ca-79f68b9f4f-qmg2j"
Jul 09 20:07:21.640468 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:21.640329    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/52d4510f-bde3-4471-a0a8-04015a4cff02-signing-cabundle\") pod \"service-ca-79f68b9f4f-qmg2j\" (UID: \"52d4510f-bde3-4471-a0a8-04015a4cff02\") " pod="openshift-service-ca/service-ca-79f68b9f4f-qmg2j"
Jul 09 20:07:21.640468 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:21.640347    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mf6wh\" (UniqueName: \"kubernetes.io/projected/52d4510f-bde3-4471-a0a8-04015a4cff02-kube-api-access-mf6wh\") pod \"service-ca-79f68b9f4f-qmg2j\" (UID: \"52d4510f-bde3-4471-a0a8-04015a4cff02\") " pod="openshift-service-ca/service-ca-79f68b9f4f-qmg2j"
Jul 09 20:07:21.640948 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:21.640924    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/52d4510f-bde3-4471-a0a8-04015a4cff02-signing-cabundle\") pod \"service-ca-79f68b9f4f-qmg2j\" (UID: \"52d4510f-bde3-4471-a0a8-04015a4cff02\") " pod="openshift-service-ca/service-ca-79f68b9f4f-qmg2j"
Jul 09 20:07:21.642723 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:21.642697    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/52d4510f-bde3-4471-a0a8-04015a4cff02-signing-key\") pod \"service-ca-79f68b9f4f-qmg2j\" (UID: \"52d4510f-bde3-4471-a0a8-04015a4cff02\") " pod="openshift-service-ca/service-ca-79f68b9f4f-qmg2j"
Jul 09 20:07:21.650948 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:21.650928    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mf6wh\" (UniqueName: \"kubernetes.io/projected/52d4510f-bde3-4471-a0a8-04015a4cff02-kube-api-access-mf6wh\") pod \"service-ca-79f68b9f4f-qmg2j\" (UID: \"52d4510f-bde3-4471-a0a8-04015a4cff02\") " pod="openshift-service-ca/service-ca-79f68b9f4f-qmg2j"
Jul 09 20:07:21.741227 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:21.741149    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/09094efa-769d-4669-944e-a76a98aa185e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7qx9m\" (UID: \"09094efa-769d-4669-944e-a76a98aa185e\") " pod="openshift-insights/insights-runtime-extractor-7qx9m"
Jul 09 20:07:21.741227 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:21.741226    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/09094efa-769d-4669-944e-a76a98aa185e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7qx9m\" (UID: \"09094efa-769d-4669-944e-a76a98aa185e\") " pod="openshift-insights/insights-runtime-extractor-7qx9m"
Jul 09 20:07:21.741420 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:07:21.741281    2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 09 20:07:21.741420 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:07:21.741357    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 09 20:07:21.741420 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:07:21.741383    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/09094efa-769d-4669-944e-a76a98aa185e-kube-rbac-proxy-cm podName:09094efa-769d-4669-944e-a76a98aa185e nodeName:}" failed. No retries permitted until 2026-07-09 20:07:23.74136682 +0000 UTC m=+234.159036607 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/09094efa-769d-4669-944e-a76a98aa185e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7qx9m" (UID: "09094efa-769d-4669-944e-a76a98aa185e") : configmap "kube-rbac-proxy" not found
Jul 09 20:07:21.741420 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:07:21.741399    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/09094efa-769d-4669-944e-a76a98aa185e-insights-runtime-extractor-tls podName:09094efa-769d-4669-944e-a76a98aa185e nodeName:}" failed. No retries permitted until 2026-07-09 20:07:23.741392715 +0000 UTC m=+234.159062502 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/09094efa-769d-4669-944e-a76a98aa185e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7qx9m" (UID: "09094efa-769d-4669-944e-a76a98aa185e") : secret "insights-runtime-extractor-tls" not found
Jul 09 20:07:21.802784 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:21.802753    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-qmg2j"
Jul 09 20:07:21.920144 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:21.920106    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-qmg2j"]
Jul 09 20:07:21.923509 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:07:21.923486    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52d4510f_bde3_4471_a0a8_04015a4cff02.slice/crio-8cbb980ab451fa34e48feccebaeca9ad6cabc277c3a9dfb2c1ed453aec113011 WatchSource:0}: Error finding container 8cbb980ab451fa34e48feccebaeca9ad6cabc277c3a9dfb2c1ed453aec113011: Status 404 returned error can't find the container with id 8cbb980ab451fa34e48feccebaeca9ad6cabc277c3a9dfb2c1ed453aec113011
Jul 09 20:07:21.948688 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:21.948655    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-qmg2j" event={"ID":"52d4510f-bde3-4471-a0a8-04015a4cff02","Type":"ContainerStarted","Data":"8cbb980ab451fa34e48feccebaeca9ad6cabc277c3a9dfb2c1ed453aec113011"}
Jul 09 20:07:21.949656 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:21.949635    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-8gtgc" event={"ID":"89b4a7d6-9699-455b-af5f-473a56d4f1fc","Type":"ContainerStarted","Data":"6964d83fb9edd8e073f8fdbd9d1211daa4ec2539a74962568eabbec3dc63640e"}
Jul 09 20:07:22.039607 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:22.039535    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-8r78k_dddbfeb6-339f-4aa6-b74f-26dfe9438a2c/dns-node-resolver/0.log"
Jul 09 20:07:22.851064 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:22.851021    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3b4c1edc-a174-4e8e-9f15-eccad2f85d52-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-t9d7w\" (UID: \"3b4c1edc-a174-4e8e-9f15-eccad2f85d52\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-t9d7w"
Jul 09 20:07:22.851547 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:07:22.851218    2562 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 09 20:07:22.851547 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:07:22.851313    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b4c1edc-a174-4e8e-9f15-eccad2f85d52-networking-console-plugin-cert podName:3b4c1edc-a174-4e8e-9f15-eccad2f85d52 nodeName:}" failed. No retries permitted until 2026-07-09 20:07:30.851279436 +0000 UTC m=+241.268949245 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3b4c1edc-a174-4e8e-9f15-eccad2f85d52-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-t9d7w" (UID: "3b4c1edc-a174-4e8e-9f15-eccad2f85d52") : secret "networking-console-plugin-cert" not found
Jul 09 20:07:22.951919 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:22.951887    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5773e57-8d05-422d-84b1-7c98763683f7-metrics-certs\") pod \"router-default-b66d5d5c8-9kl6t\" (UID: \"e5773e57-8d05-422d-84b1-7c98763683f7\") " pod="openshift-ingress/router-default-b66d5d5c8-9kl6t"
Jul 09 20:07:22.952094 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:22.951943    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5773e57-8d05-422d-84b1-7c98763683f7-service-ca-bundle\") pod \"router-default-b66d5d5c8-9kl6t\" (UID: \"e5773e57-8d05-422d-84b1-7c98763683f7\") " pod="openshift-ingress/router-default-b66d5d5c8-9kl6t"
Jul 09 20:07:22.952094 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:07:22.952059    2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 09 20:07:22.952094 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:07:22.952083    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e5773e57-8d05-422d-84b1-7c98763683f7-service-ca-bundle podName:e5773e57-8d05-422d-84b1-7c98763683f7 nodeName:}" failed. No retries permitted until 2026-07-09 20:07:30.952068341 +0000 UTC m=+241.369738128 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e5773e57-8d05-422d-84b1-7c98763683f7-service-ca-bundle") pod "router-default-b66d5d5c8-9kl6t" (UID: "e5773e57-8d05-422d-84b1-7c98763683f7") : configmap references non-existent config key: service-ca.crt
Jul 09 20:07:22.952255 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:07:22.952119    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5773e57-8d05-422d-84b1-7c98763683f7-metrics-certs podName:e5773e57-8d05-422d-84b1-7c98763683f7 nodeName:}" failed. No retries permitted until 2026-07-09 20:07:30.952100295 +0000 UTC m=+241.369770097 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e5773e57-8d05-422d-84b1-7c98763683f7-metrics-certs") pod "router-default-b66d5d5c8-9kl6t" (UID: "e5773e57-8d05-422d-84b1-7c98763683f7") : secret "router-metrics-certs-default" not found
Jul 09 20:07:23.038995 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:23.038972    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-n2tvb_785ecd20-a92f-4924-b2f3-663697381982/node-ca/0.log"
Jul 09 20:07:23.760468 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:23.760365    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/09094efa-769d-4669-944e-a76a98aa185e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7qx9m\" (UID: \"09094efa-769d-4669-944e-a76a98aa185e\") " pod="openshift-insights/insights-runtime-extractor-7qx9m"
Jul 09 20:07:23.760669 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:23.760493    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/09094efa-769d-4669-944e-a76a98aa185e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7qx9m\" (UID: \"09094efa-769d-4669-944e-a76a98aa185e\") " pod="openshift-insights/insights-runtime-extractor-7qx9m"
Jul 09 20:07:23.760669 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:07:23.760556    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 09 20:07:23.760669 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:07:23.760647    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/09094efa-769d-4669-944e-a76a98aa185e-insights-runtime-extractor-tls podName:09094efa-769d-4669-944e-a76a98aa185e nodeName:}" failed. No retries permitted until 2026-07-09 20:07:27.760618471 +0000 UTC m=+238.178288259 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/09094efa-769d-4669-944e-a76a98aa185e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7qx9m" (UID: "09094efa-769d-4669-944e-a76a98aa185e") : secret "insights-runtime-extractor-tls" not found
Jul 09 20:07:23.760669 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:07:23.760644    2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 09 20:07:23.760890 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:07:23.760688    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/09094efa-769d-4669-944e-a76a98aa185e-kube-rbac-proxy-cm podName:09094efa-769d-4669-944e-a76a98aa185e nodeName:}" failed. No retries permitted until 2026-07-09 20:07:27.760679442 +0000 UTC m=+238.178349229 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/09094efa-769d-4669-944e-a76a98aa185e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7qx9m" (UID: "09094efa-769d-4669-944e-a76a98aa185e") : configmap "kube-rbac-proxy" not found
Jul 09 20:07:23.959242 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:23.959200    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-qmg2j" event={"ID":"52d4510f-bde3-4471-a0a8-04015a4cff02","Type":"ContainerStarted","Data":"b0e0ac1879e90909dc11847b41265005df70f7b8a67c22bcde4506a848fc4154"}
Jul 09 20:07:23.960915 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:23.960886    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-8gtgc" event={"ID":"89b4a7d6-9699-455b-af5f-473a56d4f1fc","Type":"ContainerStarted","Data":"09eae1badd0b8f1ed13a413ab75120d91a38aa91d982c3d08031440a20ab5b83"}
Jul 09 20:07:23.961068 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:23.960927    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-8gtgc" event={"ID":"89b4a7d6-9699-455b-af5f-473a56d4f1fc","Type":"ContainerStarted","Data":"c91ab808a07772a8f4d22417486dc493ddbbcc20f0f08425d4931c7ae3e5d5e2"}
Jul 09 20:07:23.977391 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:23.977329    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-79f68b9f4f-qmg2j" podStartSLOduration=1.091683363 podStartE2EDuration="2.977288663s" totalImagesPullingTime="1.8856053s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 20:07:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 20:07:23.97714886 +0000 UTC m=+234.394818670" watchObservedRunningTime="2026-07-09 20:07:23.977288663 +0000 UTC m=+234.394958477"
Jul 09 20:07:23.993436 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:23.993382    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-8gtgc" podStartSLOduration=1.470284261 podStartE2EDuration="2.993367217s" totalImagesPullingTime="1.523082956s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 20:07:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 20:07:23.99295466 +0000 UTC m=+234.410624460" watchObservedRunningTime="2026-07-09 20:07:23.993367217 +0000 UTC m=+234.411037025"
Jul 09 20:07:24.039554 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:24.039487    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-pwjnv_fd8d29db-3983-4f37-9476-6dca7b1b33b7/serve-healthcheck-canary/0.log"
Jul 09 20:07:25.590042 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:25.590003    2562 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7866497494-cqs6d"
Jul 09 20:07:25.590458 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:25.590062    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7866497494-cqs6d"
Jul 09 20:07:25.590458 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:25.590441    2562 scope.go:117] "RemoveContainer" containerID="207e7bb147cdd6b535445ed48c67b14cea1756b3637f54206d52d869f3f3394c"
Jul 09 20:07:25.590613 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:07:25.590592    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-cqs6d_openshift-console-operator(578878d0-e6b3-4725-a698-796cfc169fa6)\"" pod="openshift-console-operator/console-operator-7866497494-cqs6d" podUID="578878d0-e6b3-4725-a698-796cfc169fa6"
Jul 09 20:07:27.793561 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:27.793516    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/09094efa-769d-4669-944e-a76a98aa185e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7qx9m\" (UID: \"09094efa-769d-4669-944e-a76a98aa185e\") " pod="openshift-insights/insights-runtime-extractor-7qx9m"
Jul 09 20:07:27.794034 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:27.793601    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/09094efa-769d-4669-944e-a76a98aa185e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7qx9m\" (UID: \"09094efa-769d-4669-944e-a76a98aa185e\") " pod="openshift-insights/insights-runtime-extractor-7qx9m"
Jul 09 20:07:27.794034 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:07:27.793661    2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 09 20:07:27.794034 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:07:27.793688    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 09 20:07:27.794034 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:07:27.793744    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/09094efa-769d-4669-944e-a76a98aa185e-kube-rbac-proxy-cm podName:09094efa-769d-4669-944e-a76a98aa185e nodeName:}" failed. No retries permitted until 2026-07-09 20:07:35.793722791 +0000 UTC m=+246.211392583 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/09094efa-769d-4669-944e-a76a98aa185e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7qx9m" (UID: "09094efa-769d-4669-944e-a76a98aa185e") : configmap "kube-rbac-proxy" not found
Jul 09 20:07:27.794034 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:07:27.793766    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/09094efa-769d-4669-944e-a76a98aa185e-insights-runtime-extractor-tls podName:09094efa-769d-4669-944e-a76a98aa185e nodeName:}" failed. No retries permitted until 2026-07-09 20:07:35.79375621 +0000 UTC m=+246.211425999 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/09094efa-769d-4669-944e-a76a98aa185e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7qx9m" (UID: "09094efa-769d-4669-944e-a76a98aa185e") : secret "insights-runtime-extractor-tls" not found
Jul 09 20:07:30.919536 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:30.919498    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3b4c1edc-a174-4e8e-9f15-eccad2f85d52-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-t9d7w\" (UID: \"3b4c1edc-a174-4e8e-9f15-eccad2f85d52\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-t9d7w"
Jul 09 20:07:30.919931 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:07:30.919629    2562 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 09 20:07:30.919931 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:07:30.919684    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b4c1edc-a174-4e8e-9f15-eccad2f85d52-networking-console-plugin-cert podName:3b4c1edc-a174-4e8e-9f15-eccad2f85d52 nodeName:}" failed. No retries permitted until 2026-07-09 20:07:46.91966932 +0000 UTC m=+257.337339107 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3b4c1edc-a174-4e8e-9f15-eccad2f85d52-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-t9d7w" (UID: "3b4c1edc-a174-4e8e-9f15-eccad2f85d52") : secret "networking-console-plugin-cert" not found
Jul 09 20:07:31.020567 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:31.020522    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5773e57-8d05-422d-84b1-7c98763683f7-service-ca-bundle\") pod \"router-default-b66d5d5c8-9kl6t\" (UID: \"e5773e57-8d05-422d-84b1-7c98763683f7\") " pod="openshift-ingress/router-default-b66d5d5c8-9kl6t"
Jul 09 20:07:31.020715 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:31.020638    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5773e57-8d05-422d-84b1-7c98763683f7-metrics-certs\") pod \"router-default-b66d5d5c8-9kl6t\" (UID: \"e5773e57-8d05-422d-84b1-7c98763683f7\") " pod="openshift-ingress/router-default-b66d5d5c8-9kl6t"
Jul 09 20:07:31.021209 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:31.021181    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5773e57-8d05-422d-84b1-7c98763683f7-service-ca-bundle\") pod \"router-default-b66d5d5c8-9kl6t\" (UID: \"e5773e57-8d05-422d-84b1-7c98763683f7\") " pod="openshift-ingress/router-default-b66d5d5c8-9kl6t"
Jul 09 20:07:31.023018 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:31.022987    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5773e57-8d05-422d-84b1-7c98763683f7-metrics-certs\") pod \"router-default-b66d5d5c8-9kl6t\" (UID: \"e5773e57-8d05-422d-84b1-7c98763683f7\") " pod="openshift-ingress/router-default-b66d5d5c8-9kl6t"
Jul 09 20:07:31.101465 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:31.101434    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-r7nlg\""
Jul 09 20:07:31.108919 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:31.108894    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-b66d5d5c8-9kl6t"
Jul 09 20:07:31.239293 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:31.239267    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-b66d5d5c8-9kl6t"]
Jul 09 20:07:31.241869 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:07:31.241842    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5773e57_8d05_422d_84b1_7c98763683f7.slice/crio-50868fc949712b8391e17e582a8c622e3e86f2f8adb3ad1a8c67af0c3a3a5f30 WatchSource:0}: Error finding container 50868fc949712b8391e17e582a8c622e3e86f2f8adb3ad1a8c67af0c3a3a5f30: Status 404 returned error can't find the container with id 50868fc949712b8391e17e582a8c622e3e86f2f8adb3ad1a8c67af0c3a3a5f30
Jul 09 20:07:31.994537 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:31.994495    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-b66d5d5c8-9kl6t" event={"ID":"e5773e57-8d05-422d-84b1-7c98763683f7","Type":"ContainerStarted","Data":"de5fd72133078b6eb3fa8ee849c1edbe07bc36c03dd5b38eb48f5263a915dda8"}
Jul 09 20:07:31.995002 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:31.994552    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-b66d5d5c8-9kl6t" event={"ID":"e5773e57-8d05-422d-84b1-7c98763683f7","Type":"ContainerStarted","Data":"50868fc949712b8391e17e582a8c622e3e86f2f8adb3ad1a8c67af0c3a3a5f30"}
Jul 09 20:07:32.012633 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:32.012584    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-b66d5d5c8-9kl6t" podStartSLOduration=17.01256834 podStartE2EDuration="17.01256834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 20:07:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 20:07:32.012180958 +0000 UTC m=+242.429850767" watchObservedRunningTime="2026-07-09 20:07:32.01256834 +0000 UTC m=+242.430238143"
Jul 09 20:07:32.109387 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:32.109343    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-b66d5d5c8-9kl6t"
Jul 09 20:07:32.111981 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:32.111958    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-b66d5d5c8-9kl6t"
Jul 09 20:07:32.998209 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:32.998179    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-b66d5d5c8-9kl6t"
Jul 09 20:07:32.999469 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:32.999448    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-b66d5d5c8-9kl6t"
Jul 09 20:07:35.862834 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:35.862793    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/09094efa-769d-4669-944e-a76a98aa185e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7qx9m\" (UID: \"09094efa-769d-4669-944e-a76a98aa185e\") " pod="openshift-insights/insights-runtime-extractor-7qx9m"
Jul 09 20:07:35.863251 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:35.862856    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/09094efa-769d-4669-944e-a76a98aa185e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7qx9m\" (UID: \"09094efa-769d-4669-944e-a76a98aa185e\") " pod="openshift-insights/insights-runtime-extractor-7qx9m"
Jul 09 20:07:35.863375 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:35.863355    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/09094efa-769d-4669-944e-a76a98aa185e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7qx9m\" (UID: \"09094efa-769d-4669-944e-a76a98aa185e\") " pod="openshift-insights/insights-runtime-extractor-7qx9m"
Jul 09 20:07:35.865210 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:35.865190    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/09094efa-769d-4669-944e-a76a98aa185e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7qx9m\" (UID: \"09094efa-769d-4669-944e-a76a98aa185e\") " pod="openshift-insights/insights-runtime-extractor-7qx9m"
Jul 09 20:07:35.971156 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:35.971124    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-6wftc\""
Jul 09 20:07:35.979503 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:35.979482    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7qx9m"
Jul 09 20:07:36.103987 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:36.103949    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-7qx9m"]
Jul 09 20:07:36.106717 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:07:36.106689    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09094efa_769d_4669_944e_a76a98aa185e.slice/crio-1ba7517ea9396babc5061a2c60063dec45bc2768fe74384d920f9617680fca19 WatchSource:0}: Error finding container 1ba7517ea9396babc5061a2c60063dec45bc2768fe74384d920f9617680fca19: Status 404 returned error can't find the container with id 1ba7517ea9396babc5061a2c60063dec45bc2768fe74384d920f9617680fca19
Jul 09 20:07:37.012464 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:37.012421    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7qx9m" event={"ID":"09094efa-769d-4669-944e-a76a98aa185e","Type":"ContainerStarted","Data":"1ba7517ea9396babc5061a2c60063dec45bc2768fe74384d920f9617680fca19"}
Jul 09 20:07:39.236607 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:39.236575    2562 scope.go:117] "RemoveContainer" containerID="207e7bb147cdd6b535445ed48c67b14cea1756b3637f54206d52d869f3f3394c"
Jul 09 20:07:40.023864 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:40.023836    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-cqs6d_578878d0-e6b3-4725-a698-796cfc169fa6/console-operator/1.log"
Jul 09 20:07:40.024052 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:40.023937    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-cqs6d" event={"ID":"578878d0-e6b3-4725-a698-796cfc169fa6","Type":"ContainerStarted","Data":"6e0a8acd52546335ce58214173e91e6eeea03c78cd375eb91824f650c3c2bfb3"}
Jul 09 20:07:40.024254 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:40.024237    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7866497494-cqs6d"
Jul 09 20:07:40.025728 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:40.025703    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7qx9m" event={"ID":"09094efa-769d-4669-944e-a76a98aa185e","Type":"ContainerStarted","Data":"fdb5415b6bb2448288ada7e701e1371aace414bfe628eb1fe2a4e75b37a73680"}
Jul 09 20:07:40.041621 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:40.041578    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-7866497494-cqs6d" podStartSLOduration=22.942526762 podStartE2EDuration="25.041563578s" totalImagesPullingTime="2.099036816s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 20:07:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 20:07:40.040537861 +0000 UTC m=+250.458207670" watchObservedRunningTime="2026-07-09 20:07:40.041563578 +0000 UTC m=+250.459233387"
Jul 09 20:07:40.712794 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:40.712769    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7866497494-cqs6d"
Jul 09 20:07:41.004778 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:41.004736    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14a1d22f-43ff-4eb8-a89d-920551b2ef1f-metrics-certs\") pod \"network-metrics-daemon-ksd8v\" (UID: \"14a1d22f-43ff-4eb8-a89d-920551b2ef1f\") " pod="openshift-multus/network-metrics-daemon-ksd8v"
Jul 09 20:07:41.007206 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:41.007177    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14a1d22f-43ff-4eb8-a89d-920551b2ef1f-metrics-certs\") pod \"network-metrics-daemon-ksd8v\" (UID: \"14a1d22f-43ff-4eb8-a89d-920551b2ef1f\") " pod="openshift-multus/network-metrics-daemon-ksd8v"
Jul 09 20:07:41.030293 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:41.030259    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7qx9m" event={"ID":"09094efa-769d-4669-944e-a76a98aa185e","Type":"ContainerStarted","Data":"ad4f63bfdc8e4d1f85e3e1f9636221e9c0e3e190bd5eb4d84d4f4154da4b62a0"}
Jul 09 20:07:41.240725 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:41.240696    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-s29qd\""
Jul 09 20:07:41.248198 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:41.248161    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ksd8v"
Jul 09 20:07:41.394164 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:41.394122    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-ksd8v"]
Jul 09 20:07:41.397526 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:07:41.397496    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14a1d22f_43ff_4eb8_a89d_920551b2ef1f.slice/crio-06782762328c9799a2ba70902c78cc7233675ffe79a70668dd5f322f211d17c5 WatchSource:0}: Error finding container 06782762328c9799a2ba70902c78cc7233675ffe79a70668dd5f322f211d17c5: Status 404 returned error can't find the container with id 06782762328c9799a2ba70902c78cc7233675ffe79a70668dd5f322f211d17c5
Jul 09 20:07:42.035140 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:42.035098    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ksd8v" event={"ID":"14a1d22f-43ff-4eb8-a89d-920551b2ef1f","Type":"ContainerStarted","Data":"06782762328c9799a2ba70902c78cc7233675ffe79a70668dd5f322f211d17c5"}
Jul 09 20:07:44.043247 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:44.043205    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7qx9m" event={"ID":"09094efa-769d-4669-944e-a76a98aa185e","Type":"ContainerStarted","Data":"07930f62fbed4612c983393806883183b87d05e3346a9a6613aed05119cd0be3"}
Jul 09 20:07:44.044846 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:44.044825    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ksd8v" event={"ID":"14a1d22f-43ff-4eb8-a89d-920551b2ef1f","Type":"ContainerStarted","Data":"42c6eb253b025fa199254886c7e91ddc1b2c7a545b59f42603c883dc33eadcc1"}
Jul 09 20:07:44.044955 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:44.044864    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ksd8v" event={"ID":"14a1d22f-43ff-4eb8-a89d-920551b2ef1f","Type":"ContainerStarted","Data":"16738aeb2682124fbf0fd00c2f340f0a8ab5e85ad270e9e46bb4bc68c6af0dec"}
Jul 09 20:07:44.061863 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:44.061822    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-7qx9m" podStartSLOduration=16.842552491 podStartE2EDuration="24.061807204s" totalImagesPullingTime="7.219254713s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 20:07:20 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 20:07:44.061383765 +0000 UTC m=+254.479053587" watchObservedRunningTime="2026-07-09 20:07:44.061807204 +0000 UTC m=+254.479477012"
Jul 09 20:07:46.951941 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:46.951906    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3b4c1edc-a174-4e8e-9f15-eccad2f85d52-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-t9d7w\" (UID: \"3b4c1edc-a174-4e8e-9f15-eccad2f85d52\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-t9d7w"
Jul 09 20:07:46.954399 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:46.954369    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3b4c1edc-a174-4e8e-9f15-eccad2f85d52-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-t9d7w\" (UID: \"3b4c1edc-a174-4e8e-9f15-eccad2f85d52\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-t9d7w"
Jul 09 20:07:46.992234 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:46.992207    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-lf9jv\""
Jul 09 20:07:46.999705 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:46.999684    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cf856758c-t9d7w"
Jul 09 20:07:47.125460 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:47.125408    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-ksd8v" podStartSLOduration=255.059405163 podStartE2EDuration="4m17.125387381s" totalImagesPullingTime="2.065982218s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 20:03:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 20:07:44.077525517 +0000 UTC m=+254.495195325" watchObservedRunningTime="2026-07-09 20:07:47.125387381 +0000 UTC m=+257.543057191"
Jul 09 20:07:47.125996 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:47.125974    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-t9d7w"]
Jul 09 20:07:47.128546 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:07:47.128521    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b4c1edc_a174_4e8e_9f15_eccad2f85d52.slice/crio-7d74c1f33206e08007a1e286633b76127525e8de0c80e08c7b6e8d2d065e607e WatchSource:0}: Error finding container 7d74c1f33206e08007a1e286633b76127525e8de0c80e08c7b6e8d2d065e607e: Status 404 returned error can't find the container with id 7d74c1f33206e08007a1e286633b76127525e8de0c80e08c7b6e8d2d065e607e
Jul 09 20:07:48.060710 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:48.060651    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cf856758c-t9d7w" event={"ID":"3b4c1edc-a174-4e8e-9f15-eccad2f85d52","Type":"ContainerStarted","Data":"7d74c1f33206e08007a1e286633b76127525e8de0c80e08c7b6e8d2d065e607e"}
Jul 09 20:07:49.064821 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:49.064783    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cf856758c-t9d7w" event={"ID":"3b4c1edc-a174-4e8e-9f15-eccad2f85d52","Type":"ContainerStarted","Data":"b6c4ebaaf79a1e14ba97f7c7b3dfd93184a81ce00d8ec0b3cb110342366d2804"}
Jul 09 20:07:49.081060 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:49.081002    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6cf856758c-t9d7w" podStartSLOduration=32.837628177 podStartE2EDuration="34.080986089s" totalImagesPullingTime="1.243357912s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 20:07:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 20:07:49.079813382 +0000 UTC m=+259.497483191" watchObservedRunningTime="2026-07-09 20:07:49.080986089 +0000 UTC m=+259.498655972"
Jul 09 20:07:49.811325 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:49.811279    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-mh8xw"]
Jul 09 20:07:49.814633 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:49.814610    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-mh8xw"
Jul 09 20:07:49.817345 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:49.817322    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Jul 09 20:07:49.817466 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:49.817379    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-jtdnv\""
Jul 09 20:07:49.822514 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:49.822467    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-mh8xw"]
Jul 09 20:07:49.975228 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:49.975197    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/cad231e2-9767-458e-af67-8ca2505a24c6-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-mh8xw\" (UID: \"cad231e2-9767-458e-af67-8ca2505a24c6\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-mh8xw"
Jul 09 20:07:50.075812 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:50.075735    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/cad231e2-9767-458e-af67-8ca2505a24c6-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-mh8xw\" (UID: \"cad231e2-9767-458e-af67-8ca2505a24c6\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-mh8xw"
Jul 09 20:07:50.077971 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:50.077950    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/cad231e2-9767-458e-af67-8ca2505a24c6-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-mh8xw\" (UID: \"cad231e2-9767-458e-af67-8ca2505a24c6\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-mh8xw"
Jul 09 20:07:50.124621 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:50.124584    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-mh8xw"
Jul 09 20:07:50.284376 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:50.284342    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-mh8xw"]
Jul 09 20:07:50.286951 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:07:50.286921    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcad231e2_9767_458e_af67_8ca2505a24c6.slice/crio-82a11b4456286f69c7576c989284549bf5a558223579d0a93a7a50b6b36d637c WatchSource:0}: Error finding container 82a11b4456286f69c7576c989284549bf5a558223579d0a93a7a50b6b36d637c: Status 404 returned error can't find the container with id 82a11b4456286f69c7576c989284549bf5a558223579d0a93a7a50b6b36d637c
Jul 09 20:07:51.076549 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:51.076511    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-mh8xw" event={"ID":"cad231e2-9767-458e-af67-8ca2505a24c6","Type":"ContainerStarted","Data":"82a11b4456286f69c7576c989284549bf5a558223579d0a93a7a50b6b36d637c"}
Jul 09 20:07:52.081471 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:52.081432    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-mh8xw" event={"ID":"cad231e2-9767-458e-af67-8ca2505a24c6","Type":"ContainerStarted","Data":"fbac6592e03e5b03742b277bb922ceca140adbeb321af3c8c35da8d23f903298"}
Jul 09 20:07:52.081908 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:52.081639    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-mh8xw"
Jul 09 20:07:52.086154 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:52.086135    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-mh8xw"
Jul 09 20:07:52.113940 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:52.113895    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-mh8xw" podStartSLOduration=1.935325165 podStartE2EDuration="3.113880892s" totalImagesPullingTime="1.178555727s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 20:07:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 20:07:52.097233659 +0000 UTC m=+262.514903468" watchObservedRunningTime="2026-07-09 20:07:52.113880892 +0000 UTC m=+262.531550700"
Jul 09 20:07:58.200587 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:58.200542    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-ww6bh"]
Jul 09 20:07:58.203919 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:58.203893    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-ww6bh"
Jul 09 20:07:58.207026 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:58.207001    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Jul 09 20:07:58.207169 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:58.207083    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Jul 09 20:07:58.207590 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:58.207562    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 09 20:07:58.208695 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:58.208673    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 09 20:07:58.208880 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:58.208865    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-r8r6l\""
Jul 09 20:07:58.208921 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:58.208899    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 09 20:07:58.216872 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:58.216851    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-ww6bh"]
Jul 09 20:07:58.227570 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:58.227546    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-grp5l"]
Jul 09 20:07:58.231026 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:58.231006    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-grp5l"
Jul 09 20:07:58.234322 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:58.233921    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 09 20:07:58.234322 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:58.233995    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-pnzg7\""
Jul 09 20:07:58.234322 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:58.233923    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 09 20:07:58.234322 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:58.234176    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 09 20:07:58.346443 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:58.346406    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/da490631-59e9-45ed-9083-c8949f2ca1af-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-ww6bh\" (UID: \"da490631-59e9-45ed-9083-c8949f2ca1af\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-ww6bh"
Jul 09 20:07:58.346629 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:58.346465    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/09dfd499-f51c-4d1b-ad6e-9fad60d7798e-node-exporter-tls\") pod \"node-exporter-grp5l\" (UID: \"09dfd499-f51c-4d1b-ad6e-9fad60d7798e\") " pod="openshift-monitoring/node-exporter-grp5l"
Jul 09 20:07:58.346629 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:58.346514    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z44sb\" (UniqueName: \"kubernetes.io/projected/da490631-59e9-45ed-9083-c8949f2ca1af-kube-api-access-z44sb\") pod \"openshift-state-metrics-84bd68b896-ww6bh\" (UID: \"da490631-59e9-45ed-9083-c8949f2ca1af\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-ww6bh"
Jul 09 20:07:58.346629 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:58.346548    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/09dfd499-f51c-4d1b-ad6e-9fad60d7798e-sys\") pod \"node-exporter-grp5l\" (UID: \"09dfd499-f51c-4d1b-ad6e-9fad60d7798e\") " pod="openshift-monitoring/node-exporter-grp5l"
Jul 09 20:07:58.346629 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:58.346572    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/09dfd499-f51c-4d1b-ad6e-9fad60d7798e-root\") pod \"node-exporter-grp5l\" (UID: \"09dfd499-f51c-4d1b-ad6e-9fad60d7798e\") " pod="openshift-monitoring/node-exporter-grp5l"
Jul 09 20:07:58.346629 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:58.346596    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/09dfd499-f51c-4d1b-ad6e-9fad60d7798e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-grp5l\" (UID: \"09dfd499-f51c-4d1b-ad6e-9fad60d7798e\") " pod="openshift-monitoring/node-exporter-grp5l"
Jul 09 20:07:58.346825 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:58.346673    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/09dfd499-f51c-4d1b-ad6e-9fad60d7798e-node-exporter-wtmp\") pod \"node-exporter-grp5l\" (UID: \"09dfd499-f51c-4d1b-ad6e-9fad60d7798e\") " pod="openshift-monitoring/node-exporter-grp5l"
Jul 09 20:07:58.346825 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:58.346724    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/da490631-59e9-45ed-9083-c8949f2ca1af-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-ww6bh\" (UID: \"da490631-59e9-45ed-9083-c8949f2ca1af\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-ww6bh"
Jul 09 20:07:58.346825 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:58.346758    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/09dfd499-f51c-4d1b-ad6e-9fad60d7798e-node-exporter-textfile\") pod \"node-exporter-grp5l\" (UID: \"09dfd499-f51c-4d1b-ad6e-9fad60d7798e\") " pod="openshift-monitoring/node-exporter-grp5l"
Jul 09 20:07:58.346825 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:58.346781    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkzgp\" (UniqueName: \"kubernetes.io/projected/09dfd499-f51c-4d1b-ad6e-9fad60d7798e-kube-api-access-fkzgp\") pod \"node-exporter-grp5l\" (UID: \"09dfd499-f51c-4d1b-ad6e-9fad60d7798e\") " pod="openshift-monitoring/node-exporter-grp5l"
Jul 09 20:07:58.346825 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:58.346817    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/09dfd499-f51c-4d1b-ad6e-9fad60d7798e-node-exporter-accelerators-collector-config\") pod \"node-exporter-grp5l\" (UID: \"09dfd499-f51c-4d1b-ad6e-9fad60d7798e\") " pod="openshift-monitoring/node-exporter-grp5l"
Jul 09 20:07:58.346990 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:58.346883    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/09dfd499-f51c-4d1b-ad6e-9fad60d7798e-metrics-client-ca\") pod \"node-exporter-grp5l\" (UID: \"09dfd499-f51c-4d1b-ad6e-9fad60d7798e\") " pod="openshift-monitoring/node-exporter-grp5l"
Jul 09 20:07:58.346990 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:58.346928    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/da490631-59e9-45ed-9083-c8949f2ca1af-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-ww6bh\" (UID: \"da490631-59e9-45ed-9083-c8949f2ca1af\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-ww6bh"
Jul 09 20:07:58.447638 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:58.447603    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z44sb\" (UniqueName: \"kubernetes.io/projected/da490631-59e9-45ed-9083-c8949f2ca1af-kube-api-access-z44sb\") pod \"openshift-state-metrics-84bd68b896-ww6bh\" (UID: \"da490631-59e9-45ed-9083-c8949f2ca1af\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-ww6bh"
Jul 09 20:07:58.447638 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:58.447642    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/09dfd499-f51c-4d1b-ad6e-9fad60d7798e-sys\") pod \"node-exporter-grp5l\" (UID: \"09dfd499-f51c-4d1b-ad6e-9fad60d7798e\") " pod="openshift-monitoring/node-exporter-grp5l"
Jul 09 20:07:58.447866 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:58.447667    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/09dfd499-f51c-4d1b-ad6e-9fad60d7798e-root\") pod \"node-exporter-grp5l\" (UID: \"09dfd499-f51c-4d1b-ad6e-9fad60d7798e\") " pod="openshift-monitoring/node-exporter-grp5l"
Jul 09 20:07:58.447866 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:58.447691    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/09dfd499-f51c-4d1b-ad6e-9fad60d7798e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-grp5l\" (UID: \"09dfd499-f51c-4d1b-ad6e-9fad60d7798e\") " pod="openshift-monitoring/node-exporter-grp5l"
Jul 09 20:07:58.447866 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:58.447728    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/09dfd499-f51c-4d1b-ad6e-9fad60d7798e-node-exporter-wtmp\") pod \"node-exporter-grp5l\" (UID: \"09dfd499-f51c-4d1b-ad6e-9fad60d7798e\") " pod="openshift-monitoring/node-exporter-grp5l"
Jul 09 20:07:58.447866 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:58.447735    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/09dfd499-f51c-4d1b-ad6e-9fad60d7798e-sys\") pod \"node-exporter-grp5l\" (UID: \"09dfd499-f51c-4d1b-ad6e-9fad60d7798e\") " pod="openshift-monitoring/node-exporter-grp5l"
Jul 09 20:07:58.447866 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:58.447752    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/da490631-59e9-45ed-9083-c8949f2ca1af-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-ww6bh\" (UID: \"da490631-59e9-45ed-9083-c8949f2ca1af\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-ww6bh"
Jul 09 20:07:58.447866 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:58.447802    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/09dfd499-f51c-4d1b-ad6e-9fad60d7798e-node-exporter-textfile\") pod \"node-exporter-grp5l\" (UID: \"09dfd499-f51c-4d1b-ad6e-9fad60d7798e\") " pod="openshift-monitoring/node-exporter-grp5l"
Jul 09 20:07:58.447866 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:58.447831    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkzgp\" (UniqueName: \"kubernetes.io/projected/09dfd499-f51c-4d1b-ad6e-9fad60d7798e-kube-api-access-fkzgp\") pod \"node-exporter-grp5l\" (UID: \"09dfd499-f51c-4d1b-ad6e-9fad60d7798e\") " pod="openshift-monitoring/node-exporter-grp5l"
Jul 09 20:07:58.447866 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:58.447866    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/09dfd499-f51c-4d1b-ad6e-9fad60d7798e-node-exporter-accelerators-collector-config\") pod \"node-exporter-grp5l\" (UID: \"09dfd499-f51c-4d1b-ad6e-9fad60d7798e\") " pod="openshift-monitoring/node-exporter-grp5l"
Jul 09 20:07:58.448245 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:58.447924    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/09dfd499-f51c-4d1b-ad6e-9fad60d7798e-metrics-client-ca\") pod \"node-exporter-grp5l\" (UID: \"09dfd499-f51c-4d1b-ad6e-9fad60d7798e\") " pod="openshift-monitoring/node-exporter-grp5l"
Jul 09 20:07:58.448245 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:58.447956    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/da490631-59e9-45ed-9083-c8949f2ca1af-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-ww6bh\" (UID: \"da490631-59e9-45ed-9083-c8949f2ca1af\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-ww6bh"
Jul 09 20:07:58.448245 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:58.448002    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/da490631-59e9-45ed-9083-c8949f2ca1af-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-ww6bh\" (UID: \"da490631-59e9-45ed-9083-c8949f2ca1af\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-ww6bh"
Jul 09 20:07:58.448245 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:58.448041    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/09dfd499-f51c-4d1b-ad6e-9fad60d7798e-node-exporter-tls\") pod \"node-exporter-grp5l\" (UID: \"09dfd499-f51c-4d1b-ad6e-9fad60d7798e\") " pod="openshift-monitoring/node-exporter-grp5l"
Jul 09 20:07:58.448467 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:58.448250    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/09dfd499-f51c-4d1b-ad6e-9fad60d7798e-root\") pod \"node-exporter-grp5l\" (UID: \"09dfd499-f51c-4d1b-ad6e-9fad60d7798e\") " pod="openshift-monitoring/node-exporter-grp5l"
Jul 09 20:07:58.448467 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:58.448410    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/09dfd499-f51c-4d1b-ad6e-9fad60d7798e-node-exporter-textfile\") pod \"node-exporter-grp5l\" (UID: \"09dfd499-f51c-4d1b-ad6e-9fad60d7798e\") " pod="openshift-monitoring/node-exporter-grp5l"
Jul 09 20:07:58.448467 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:58.448433    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/09dfd499-f51c-4d1b-ad6e-9fad60d7798e-node-exporter-wtmp\") pod \"node-exporter-grp5l\" (UID: \"09dfd499-f51c-4d1b-ad6e-9fad60d7798e\") " pod="openshift-monitoring/node-exporter-grp5l"
Jul 09 20:07:58.449004 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:58.448903    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/09dfd499-f51c-4d1b-ad6e-9fad60d7798e-metrics-client-ca\") pod \"node-exporter-grp5l\" (UID: \"09dfd499-f51c-4d1b-ad6e-9fad60d7798e\") " pod="openshift-monitoring/node-exporter-grp5l"
Jul 09 20:07:58.449683 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:58.449537    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/09dfd499-f51c-4d1b-ad6e-9fad60d7798e-node-exporter-accelerators-collector-config\") pod \"node-exporter-grp5l\" (UID: \"09dfd499-f51c-4d1b-ad6e-9fad60d7798e\") " pod="openshift-monitoring/node-exporter-grp5l"
Jul 09 20:07:58.450237 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:58.450216    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/da490631-59e9-45ed-9083-c8949f2ca1af-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-ww6bh\" (UID: \"da490631-59e9-45ed-9083-c8949f2ca1af\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-ww6bh"
Jul 09 20:07:58.452279 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:58.451387    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/da490631-59e9-45ed-9083-c8949f2ca1af-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-ww6bh\" (UID: \"da490631-59e9-45ed-9083-c8949f2ca1af\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-ww6bh"
Jul 09 20:07:58.452279 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:58.451520    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/09dfd499-f51c-4d1b-ad6e-9fad60d7798e-node-exporter-tls\") pod \"node-exporter-grp5l\" (UID: \"09dfd499-f51c-4d1b-ad6e-9fad60d7798e\") " pod="openshift-monitoring/node-exporter-grp5l"
Jul 09 20:07:58.452809 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:58.452783    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/09dfd499-f51c-4d1b-ad6e-9fad60d7798e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-grp5l\" (UID: \"09dfd499-f51c-4d1b-ad6e-9fad60d7798e\") " pod="openshift-monitoring/node-exporter-grp5l"
Jul 09 20:07:58.453100 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:58.453071    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/da490631-59e9-45ed-9083-c8949f2ca1af-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-ww6bh\" (UID: \"da490631-59e9-45ed-9083-c8949f2ca1af\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-ww6bh"
Jul 09 20:07:58.456350 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:58.456282    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z44sb\" (UniqueName: \"kubernetes.io/projected/da490631-59e9-45ed-9083-c8949f2ca1af-kube-api-access-z44sb\") pod \"openshift-state-metrics-84bd68b896-ww6bh\" (UID: \"da490631-59e9-45ed-9083-c8949f2ca1af\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-ww6bh"
Jul 09 20:07:58.456628 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:58.456609    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkzgp\" (UniqueName: \"kubernetes.io/projected/09dfd499-f51c-4d1b-ad6e-9fad60d7798e-kube-api-access-fkzgp\") pod \"node-exporter-grp5l\" (UID: \"09dfd499-f51c-4d1b-ad6e-9fad60d7798e\") " pod="openshift-monitoring/node-exporter-grp5l"
Jul 09 20:07:58.515871 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:58.515837    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-ww6bh"
Jul 09 20:07:58.548992 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:58.548955    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-grp5l"
Jul 09 20:07:58.562805 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:07:58.562772    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09dfd499_f51c_4d1b_ad6e_9fad60d7798e.slice/crio-c52e040f29ae25dd68e8cedfd6d812057e50f304f3d6de3666e58d9a4cc633ce WatchSource:0}: Error finding container c52e040f29ae25dd68e8cedfd6d812057e50f304f3d6de3666e58d9a4cc633ce: Status 404 returned error can't find the container with id c52e040f29ae25dd68e8cedfd6d812057e50f304f3d6de3666e58d9a4cc633ce
Jul 09 20:07:58.645015 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:58.644991    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-ww6bh"]
Jul 09 20:07:58.647342 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:07:58.647312    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda490631_59e9_45ed_9083_c8949f2ca1af.slice/crio-c2776e7df3e593d04a5078b0c318830470168fa398d6b713a5e12a08c53dab11 WatchSource:0}: Error finding container c2776e7df3e593d04a5078b0c318830470168fa398d6b713a5e12a08c53dab11: Status 404 returned error can't find the container with id c2776e7df3e593d04a5078b0c318830470168fa398d6b713a5e12a08c53dab11
Jul 09 20:07:59.105605 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:59.105519    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-grp5l" event={"ID":"09dfd499-f51c-4d1b-ad6e-9fad60d7798e","Type":"ContainerStarted","Data":"c52e040f29ae25dd68e8cedfd6d812057e50f304f3d6de3666e58d9a4cc633ce"}
Jul 09 20:07:59.107335 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:59.107289    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-ww6bh" event={"ID":"da490631-59e9-45ed-9083-c8949f2ca1af","Type":"ContainerStarted","Data":"7e65fb1922d8cde11a0b33e65f60db4da0add51b6cfc3653b7644a0120a18606"}
Jul 09 20:07:59.107451 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:59.107358    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-ww6bh" event={"ID":"da490631-59e9-45ed-9083-c8949f2ca1af","Type":"ContainerStarted","Data":"e00080766ea4aab486737826f763080eae87a245d75eef936b11ef5e736b0aa7"}
Jul 09 20:07:59.107451 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:07:59.107377    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-ww6bh" event={"ID":"da490631-59e9-45ed-9083-c8949f2ca1af","Type":"ContainerStarted","Data":"c2776e7df3e593d04a5078b0c318830470168fa398d6b713a5e12a08c53dab11"}
Jul 09 20:08:01.117229 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:08:01.117198    2562 generic.go:356] "Generic (PLEG): container finished" podID="09dfd499-f51c-4d1b-ad6e-9fad60d7798e" containerID="cb77779ae5a6825f3afee3ae9e231971372a7f3f3e501ed850e886e27b2d3279" exitCode=0
Jul 09 20:08:01.117739 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:08:01.117274    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-grp5l" event={"ID":"09dfd499-f51c-4d1b-ad6e-9fad60d7798e","Type":"ContainerDied","Data":"cb77779ae5a6825f3afee3ae9e231971372a7f3f3e501ed850e886e27b2d3279"}
Jul 09 20:08:01.119738 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:08:01.119711    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-ww6bh" event={"ID":"da490631-59e9-45ed-9083-c8949f2ca1af","Type":"ContainerStarted","Data":"f92af628746950e9d2d1534817dd25ab09f789bba848ffc48bad2598c8a67277"}
Jul 09 20:08:01.158405 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:08:01.158358    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-ww6bh" podStartSLOduration=1.840555418 podStartE2EDuration="3.158341292s" totalImagesPullingTime="1.317785874s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 20:07:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 20:08:01.157521753 +0000 UTC m=+271.575191566" watchObservedRunningTime="2026-07-09 20:08:01.158341292 +0000 UTC m=+271.576011101"
Jul 09 20:08:02.125478 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:08:02.125435    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-grp5l" event={"ID":"09dfd499-f51c-4d1b-ad6e-9fad60d7798e","Type":"ContainerStarted","Data":"5155f7de8e1a5abed02424aabcca70be527a984879adbfd1384a76239cd61e9f"}
Jul 09 20:08:02.125961 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:08:02.125497    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-grp5l" event={"ID":"09dfd499-f51c-4d1b-ad6e-9fad60d7798e","Type":"ContainerStarted","Data":"94b5a280ad94ac2d22acb1a594ff36ca21b9d82dbf71359b5e0b9850822d4911"}
Jul 09 20:08:02.146322 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:08:02.146259    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-grp5l" podStartSLOduration=2.559867883 podStartE2EDuration="4.146244206s" totalImagesPullingTime="1.521974152s" totalInitContainerRuntime="64.402171ms" isStatefulPod=true podCreationTimestamp="2026-07-09 20:07:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 20:08:02.144111074 +0000 UTC m=+272.561780884" watchObservedRunningTime="2026-07-09 20:08:02.146244206 +0000 UTC m=+272.563914015"
Jul 09 20:08:03.052067 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:08:03.052032    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5cd9fd56c6-6z7pr"]
Jul 09 20:08:03.055248 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:08:03.055226    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5cd9fd56c6-6z7pr"
Jul 09 20:08:03.058086 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:08:03.058065    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 09 20:08:03.059818 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:08:03.059792    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 09 20:08:03.059939 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:08:03.059832    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 09 20:08:03.059939 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:08:03.059848    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 09 20:08:03.059939 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:08:03.059866    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-twwfz\""
Jul 09 20:08:03.060213 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:08:03.060198    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 09 20:08:03.060320 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:08:03.060230    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 09 20:08:03.060320 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:08:03.060276    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 09 20:08:03.065209 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:08:03.065188    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5cd9fd56c6-6z7pr"]
Jul 09 20:08:03.065653 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:08:03.065637    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 09 20:08:03.182876 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:08:03.182845    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d16a8e79-e4ff-4474-9e21-617024bae5a7-trusted-ca-bundle\") pod \"console-5cd9fd56c6-6z7pr\" (UID: \"d16a8e79-e4ff-4474-9e21-617024bae5a7\") " pod="openshift-console/console-5cd9fd56c6-6z7pr"
Jul 09 20:08:03.183359 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:08:03.182910    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zk54r\" (UniqueName: \"kubernetes.io/projected/d16a8e79-e4ff-4474-9e21-617024bae5a7-kube-api-access-zk54r\") pod \"console-5cd9fd56c6-6z7pr\" (UID: \"d16a8e79-e4ff-4474-9e21-617024bae5a7\") " pod="openshift-console/console-5cd9fd56c6-6z7pr"
Jul 09 20:08:03.183359 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:08:03.182934    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d16a8e79-e4ff-4474-9e21-617024bae5a7-console-config\") pod \"console-5cd9fd56c6-6z7pr\" (UID: \"d16a8e79-e4ff-4474-9e21-617024bae5a7\") " pod="openshift-console/console-5cd9fd56c6-6z7pr"
Jul 09 20:08:03.183359 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:08:03.182995    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d16a8e79-e4ff-4474-9e21-617024bae5a7-service-ca\") pod \"console-5cd9fd56c6-6z7pr\" (UID: \"d16a8e79-e4ff-4474-9e21-617024bae5a7\") " pod="openshift-console/console-5cd9fd56c6-6z7pr"
Jul 09 20:08:03.183359 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:08:03.183044    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d16a8e79-e4ff-4474-9e21-617024bae5a7-console-serving-cert\") pod \"console-5cd9fd56c6-6z7pr\" (UID: \"d16a8e79-e4ff-4474-9e21-617024bae5a7\") " pod="openshift-console/console-5cd9fd56c6-6z7pr"
Jul 09 20:08:03.183359 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:08:03.183088    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d16a8e79-e4ff-4474-9e21-617024bae5a7-oauth-serving-cert\") pod \"console-5cd9fd56c6-6z7pr\" (UID: \"d16a8e79-e4ff-4474-9e21-617024bae5a7\") " pod="openshift-console/console-5cd9fd56c6-6z7pr"
Jul 09 20:08:03.183359 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:08:03.183129    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d16a8e79-e4ff-4474-9e21-617024bae5a7-console-oauth-config\") pod \"console-5cd9fd56c6-6z7pr\" (UID: \"d16a8e79-e4ff-4474-9e21-617024bae5a7\") " pod="openshift-console/console-5cd9fd56c6-6z7pr"
Jul 09 20:08:03.284108 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:08:03.284066    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d16a8e79-e4ff-4474-9e21-617024bae5a7-console-oauth-config\") pod \"console-5cd9fd56c6-6z7pr\" (UID: \"d16a8e79-e4ff-4474-9e21-617024bae5a7\") " pod="openshift-console/console-5cd9fd56c6-6z7pr"
Jul 09 20:08:03.284331 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:08:03.284132    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d16a8e79-e4ff-4474-9e21-617024bae5a7-trusted-ca-bundle\") pod \"console-5cd9fd56c6-6z7pr\" (UID: \"d16a8e79-e4ff-4474-9e21-617024bae5a7\") " pod="openshift-console/console-5cd9fd56c6-6z7pr"
Jul 09 20:08:03.284331 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:08:03.284215    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zk54r\" (UniqueName: \"kubernetes.io/projected/d16a8e79-e4ff-4474-9e21-617024bae5a7-kube-api-access-zk54r\") pod \"console-5cd9fd56c6-6z7pr\" (UID: \"d16a8e79-e4ff-4474-9e21-617024bae5a7\") " pod="openshift-console/console-5cd9fd56c6-6z7pr"
Jul 09 20:08:03.284331 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:08:03.284242    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d16a8e79-e4ff-4474-9e21-617024bae5a7-console-config\") pod \"console-5cd9fd56c6-6z7pr\" (UID: \"d16a8e79-e4ff-4474-9e21-617024bae5a7\") " pod="openshift-console/console-5cd9fd56c6-6z7pr"
Jul 09 20:08:03.284519 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:08:03.284374    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d16a8e79-e4ff-4474-9e21-617024bae5a7-service-ca\") pod \"console-5cd9fd56c6-6z7pr\" (UID: \"d16a8e79-e4ff-4474-9e21-617024bae5a7\") " pod="openshift-console/console-5cd9fd56c6-6z7pr"
Jul 09 20:08:03.284519 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:08:03.284440    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d16a8e79-e4ff-4474-9e21-617024bae5a7-console-serving-cert\") pod \"console-5cd9fd56c6-6z7pr\" (UID: \"d16a8e79-e4ff-4474-9e21-617024bae5a7\") " pod="openshift-console/console-5cd9fd56c6-6z7pr"
Jul 09 20:08:03.284519 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:08:03.284490    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d16a8e79-e4ff-4474-9e21-617024bae5a7-oauth-serving-cert\") pod \"console-5cd9fd56c6-6z7pr\" (UID: \"d16a8e79-e4ff-4474-9e21-617024bae5a7\") " pod="openshift-console/console-5cd9fd56c6-6z7pr"
Jul 09 20:08:03.285147 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:08:03.285124    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d16a8e79-e4ff-4474-9e21-617024bae5a7-trusted-ca-bundle\") pod \"console-5cd9fd56c6-6z7pr\" (UID: \"d16a8e79-e4ff-4474-9e21-617024bae5a7\") " pod="openshift-console/console-5cd9fd56c6-6z7pr"
Jul 09 20:08:03.285251 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:08:03.285178    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d16a8e79-e4ff-4474-9e21-617024bae5a7-console-config\") pod \"console-5cd9fd56c6-6z7pr\" (UID: \"d16a8e79-e4ff-4474-9e21-617024bae5a7\") " pod="openshift-console/console-5cd9fd56c6-6z7pr"
Jul 09 20:08:03.285251 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:08:03.285221    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d16a8e79-e4ff-4474-9e21-617024bae5a7-service-ca\") pod \"console-5cd9fd56c6-6z7pr\" (UID: \"d16a8e79-e4ff-4474-9e21-617024bae5a7\") " pod="openshift-console/console-5cd9fd56c6-6z7pr"
Jul 09 20:08:03.285251 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:08:03.285220    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d16a8e79-e4ff-4474-9e21-617024bae5a7-oauth-serving-cert\") pod \"console-5cd9fd56c6-6z7pr\" (UID: \"d16a8e79-e4ff-4474-9e21-617024bae5a7\") " pod="openshift-console/console-5cd9fd56c6-6z7pr"
Jul 09 20:08:03.286941 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:08:03.286910    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d16a8e79-e4ff-4474-9e21-617024bae5a7-console-oauth-config\") pod \"console-5cd9fd56c6-6z7pr\" (UID: \"d16a8e79-e4ff-4474-9e21-617024bae5a7\") " pod="openshift-console/console-5cd9fd56c6-6z7pr"
Jul 09 20:08:03.287036 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:08:03.286921    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d16a8e79-e4ff-4474-9e21-617024bae5a7-console-serving-cert\") pod \"console-5cd9fd56c6-6z7pr\" (UID: \"d16a8e79-e4ff-4474-9e21-617024bae5a7\") " pod="openshift-console/console-5cd9fd56c6-6z7pr"
Jul 09 20:08:03.291928 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:08:03.291906    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zk54r\" (UniqueName: \"kubernetes.io/projected/d16a8e79-e4ff-4474-9e21-617024bae5a7-kube-api-access-zk54r\") pod \"console-5cd9fd56c6-6z7pr\" (UID: \"d16a8e79-e4ff-4474-9e21-617024bae5a7\") " pod="openshift-console/console-5cd9fd56c6-6z7pr"
Jul 09 20:08:03.365521 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:08:03.365446    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5cd9fd56c6-6z7pr"
Jul 09 20:08:03.490914 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:08:03.490890    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5cd9fd56c6-6z7pr"]
Jul 09 20:08:03.493259 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:08:03.493233    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd16a8e79_e4ff_4474_9e21_617024bae5a7.slice/crio-23136b89bb5ebd3192f851252c715b4f6d0e4f94f9a9dd2c8719c379ef28ff1f WatchSource:0}: Error finding container 23136b89bb5ebd3192f851252c715b4f6d0e4f94f9a9dd2c8719c379ef28ff1f: Status 404 returned error can't find the container with id 23136b89bb5ebd3192f851252c715b4f6d0e4f94f9a9dd2c8719c379ef28ff1f
Jul 09 20:08:04.136683 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:08:04.136647    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5cd9fd56c6-6z7pr" event={"ID":"d16a8e79-e4ff-4474-9e21-617024bae5a7","Type":"ContainerStarted","Data":"23136b89bb5ebd3192f851252c715b4f6d0e4f94f9a9dd2c8719c379ef28ff1f"}
Jul 09 20:08:07.152175 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:08:07.152129    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5cd9fd56c6-6z7pr" event={"ID":"d16a8e79-e4ff-4474-9e21-617024bae5a7","Type":"ContainerStarted","Data":"efe249e39e76e62ecfd5a7f52b30f7791cdfb29e875a75d29143b2b50b399212"}
Jul 09 20:08:07.169675 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:08:07.169625    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5cd9fd56c6-6z7pr" podStartSLOduration=1.021746276 podStartE2EDuration="4.169609314s" totalImagesPullingTime="3.147863038s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 20:08:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 20:08:07.167990524 +0000 UTC m=+277.585660345" watchObservedRunningTime="2026-07-09 20:08:07.169609314 +0000 UTC m=+277.587279183"
Jul 09 20:08:10.765235 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:08:10.765183    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-vhphk" podUID="1f3c5da7-a9c8-4238-840b-df34177f1096"
Jul 09 20:08:11.166992 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:08:11.166916    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vhphk"
Jul 09 20:08:13.366265 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:08:13.366230    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5cd9fd56c6-6z7pr"
Jul 09 20:08:13.366660 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:08:13.366310    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5cd9fd56c6-6z7pr"
Jul 09 20:08:13.370922 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:08:13.370900    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5cd9fd56c6-6z7pr"
Jul 09 20:08:14.181210 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:08:14.181180    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5cd9fd56c6-6z7pr"
Jul 09 20:08:14.271122 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:08:14.271087    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1f3c5da7-a9c8-4238-840b-df34177f1096-metrics-tls\") pod \"dns-default-vhphk\" (UID: \"1f3c5da7-a9c8-4238-840b-df34177f1096\") " pod="openshift-dns/dns-default-vhphk"
Jul 09 20:08:14.273534 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:08:14.273512    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1f3c5da7-a9c8-4238-840b-df34177f1096-metrics-tls\") pod \"dns-default-vhphk\" (UID: \"1f3c5da7-a9c8-4238-840b-df34177f1096\") " pod="openshift-dns/dns-default-vhphk"
Jul 09 20:08:14.471210 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:08:14.471186    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-9lrpv\""
Jul 09 20:08:14.478823 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:08:14.478798    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vhphk"
Jul 09 20:08:14.627377 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:08:14.627219    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-vhphk"]
Jul 09 20:08:14.630011 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:08:14.629969    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f3c5da7_a9c8_4238_840b_df34177f1096.slice/crio-a9aa55645b3610cf1dfe40e3cec18055f666d10f175d3eaf6491a471e241fddb WatchSource:0}: Error finding container a9aa55645b3610cf1dfe40e3cec18055f666d10f175d3eaf6491a471e241fddb: Status 404 returned error can't find the container with id a9aa55645b3610cf1dfe40e3cec18055f666d10f175d3eaf6491a471e241fddb
Jul 09 20:08:15.181378 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:08:15.181339    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vhphk" event={"ID":"1f3c5da7-a9c8-4238-840b-df34177f1096","Type":"ContainerStarted","Data":"a9aa55645b3610cf1dfe40e3cec18055f666d10f175d3eaf6491a471e241fddb"}
Jul 09 20:08:17.190741 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:08:17.190695    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vhphk" event={"ID":"1f3c5da7-a9c8-4238-840b-df34177f1096","Type":"ContainerStarted","Data":"8d2241ce7a335cfe9409192548d9cbc3f6d86caf7ec19c1ab7fa69e1e9cd3882"}
Jul 09 20:08:17.191186 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:08:17.190755    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vhphk" event={"ID":"1f3c5da7-a9c8-4238-840b-df34177f1096","Type":"ContainerStarted","Data":"a44f1b8c845ab7f92d57c2669817f9366e71af4ab36ef5a66ade0397f61b228e"}
Jul 09 20:08:17.191186 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:08:17.190831    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-vhphk"
Jul 09 20:08:17.206984 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:08:17.206939    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-vhphk" podStartSLOduration=251.166542479 podStartE2EDuration="4m13.206925255s" totalImagesPullingTime="2.040382776s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 20:04:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 20:08:17.206441622 +0000 UTC m=+287.624111443" watchObservedRunningTime="2026-07-09 20:08:17.206925255 +0000 UTC m=+287.624595063"
Jul 09 20:08:27.196782 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:08:27.196746    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-vhphk"
Jul 09 20:08:30.030556 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:08:30.030521    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-cqs6d_578878d0-e6b3-4725-a698-796cfc169fa6/console-operator/1.log"
Jul 09 20:08:30.030966 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:08:30.030741    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-cqs6d_578878d0-e6b3-4725-a698-796cfc169fa6/console-operator/1.log"
Jul 09 20:08:30.036264 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:08:30.036246    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hds4g_4b4468fd-1708-487b-8d69-bb7d817e25de/ovn-acl-logging/0.log"
Jul 09 20:08:30.036417 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:08:30.036265    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hds4g_4b4468fd-1708-487b-8d69-bb7d817e25de/ovn-acl-logging/0.log"
Jul 09 20:08:30.040084 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:08:30.040062    2562 kubelet.go:1648] "Image garbage collection succeeded"
Jul 09 20:09:28.123492 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:09:28.123449    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-79d967c4d6-5sc6v"]
Jul 09 20:09:28.126696 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:09:28.126674    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79d967c4d6-5sc6v"
Jul 09 20:09:28.139788 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:09:28.139745    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-79d967c4d6-5sc6v"]
Jul 09 20:09:28.230227 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:09:28.230181    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a987c9b8-b785-454f-9a49-196717ab5b70-service-ca\") pod \"console-79d967c4d6-5sc6v\" (UID: \"a987c9b8-b785-454f-9a49-196717ab5b70\") " pod="openshift-console/console-79d967c4d6-5sc6v"
Jul 09 20:09:28.230227 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:09:28.230232    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a987c9b8-b785-454f-9a49-196717ab5b70-console-serving-cert\") pod \"console-79d967c4d6-5sc6v\" (UID: \"a987c9b8-b785-454f-9a49-196717ab5b70\") " pod="openshift-console/console-79d967c4d6-5sc6v"
Jul 09 20:09:28.230480 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:09:28.230250    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a987c9b8-b785-454f-9a49-196717ab5b70-console-config\") pod \"console-79d967c4d6-5sc6v\" (UID: \"a987c9b8-b785-454f-9a49-196717ab5b70\") " pod="openshift-console/console-79d967c4d6-5sc6v"
Jul 09 20:09:28.230480 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:09:28.230369    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6h6q\" (UniqueName: \"kubernetes.io/projected/a987c9b8-b785-454f-9a49-196717ab5b70-kube-api-access-d6h6q\") pod \"console-79d967c4d6-5sc6v\" (UID: \"a987c9b8-b785-454f-9a49-196717ab5b70\") " pod="openshift-console/console-79d967c4d6-5sc6v"
Jul 09 20:09:28.230480 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:09:28.230427    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a987c9b8-b785-454f-9a49-196717ab5b70-trusted-ca-bundle\") pod \"console-79d967c4d6-5sc6v\" (UID: \"a987c9b8-b785-454f-9a49-196717ab5b70\") " pod="openshift-console/console-79d967c4d6-5sc6v"
Jul 09 20:09:28.230480 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:09:28.230452    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a987c9b8-b785-454f-9a49-196717ab5b70-oauth-serving-cert\") pod \"console-79d967c4d6-5sc6v\" (UID: \"a987c9b8-b785-454f-9a49-196717ab5b70\") " pod="openshift-console/console-79d967c4d6-5sc6v"
Jul 09 20:09:28.230480 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:09:28.230470    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a987c9b8-b785-454f-9a49-196717ab5b70-console-oauth-config\") pod \"console-79d967c4d6-5sc6v\" (UID: \"a987c9b8-b785-454f-9a49-196717ab5b70\") " pod="openshift-console/console-79d967c4d6-5sc6v"
Jul 09 20:09:28.331583 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:09:28.331546    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a987c9b8-b785-454f-9a49-196717ab5b70-trusted-ca-bundle\") pod \"console-79d967c4d6-5sc6v\" (UID: \"a987c9b8-b785-454f-9a49-196717ab5b70\") " pod="openshift-console/console-79d967c4d6-5sc6v"
Jul 09 20:09:28.331783 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:09:28.331593    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a987c9b8-b785-454f-9a49-196717ab5b70-oauth-serving-cert\") pod \"console-79d967c4d6-5sc6v\" (UID: \"a987c9b8-b785-454f-9a49-196717ab5b70\") " pod="openshift-console/console-79d967c4d6-5sc6v"
Jul 09 20:09:28.331783 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:09:28.331611    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a987c9b8-b785-454f-9a49-196717ab5b70-console-oauth-config\") pod \"console-79d967c4d6-5sc6v\" (UID: \"a987c9b8-b785-454f-9a49-196717ab5b70\") " pod="openshift-console/console-79d967c4d6-5sc6v"
Jul 09 20:09:28.331783 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:09:28.331645    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a987c9b8-b785-454f-9a49-196717ab5b70-service-ca\") pod \"console-79d967c4d6-5sc6v\" (UID: \"a987c9b8-b785-454f-9a49-196717ab5b70\") " pod="openshift-console/console-79d967c4d6-5sc6v"
Jul 09 20:09:28.331783 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:09:28.331753    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a987c9b8-b785-454f-9a49-196717ab5b70-console-serving-cert\") pod \"console-79d967c4d6-5sc6v\" (UID: \"a987c9b8-b785-454f-9a49-196717ab5b70\") " pod="openshift-console/console-79d967c4d6-5sc6v"
Jul 09 20:09:28.331783 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:09:28.331769    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a987c9b8-b785-454f-9a49-196717ab5b70-console-config\") pod \"console-79d967c4d6-5sc6v\" (UID: \"a987c9b8-b785-454f-9a49-196717ab5b70\") " pod="openshift-console/console-79d967c4d6-5sc6v"
Jul 09 20:09:28.332042 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:09:28.331812    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6h6q\" (UniqueName: \"kubernetes.io/projected/a987c9b8-b785-454f-9a49-196717ab5b70-kube-api-access-d6h6q\") pod \"console-79d967c4d6-5sc6v\" (UID: \"a987c9b8-b785-454f-9a49-196717ab5b70\") " pod="openshift-console/console-79d967c4d6-5sc6v"
Jul 09 20:09:28.332422 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:09:28.332379    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a987c9b8-b785-454f-9a49-196717ab5b70-oauth-serving-cert\") pod \"console-79d967c4d6-5sc6v\" (UID: \"a987c9b8-b785-454f-9a49-196717ab5b70\") " pod="openshift-console/console-79d967c4d6-5sc6v"
Jul 09 20:09:28.332539 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:09:28.332508    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a987c9b8-b785-454f-9a49-196717ab5b70-console-config\") pod \"console-79d967c4d6-5sc6v\" (UID: \"a987c9b8-b785-454f-9a49-196717ab5b70\") " pod="openshift-console/console-79d967c4d6-5sc6v"
Jul 09 20:09:28.332595 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:09:28.332578    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a987c9b8-b785-454f-9a49-196717ab5b70-trusted-ca-bundle\") pod \"console-79d967c4d6-5sc6v\" (UID: \"a987c9b8-b785-454f-9a49-196717ab5b70\") " pod="openshift-console/console-79d967c4d6-5sc6v"
Jul 09 20:09:28.332971 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:09:28.332947    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a987c9b8-b785-454f-9a49-196717ab5b70-service-ca\") pod \"console-79d967c4d6-5sc6v\" (UID: \"a987c9b8-b785-454f-9a49-196717ab5b70\") " pod="openshift-console/console-79d967c4d6-5sc6v"
Jul 09 20:09:28.334454 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:09:28.334426    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a987c9b8-b785-454f-9a49-196717ab5b70-console-oauth-config\") pod \"console-79d967c4d6-5sc6v\" (UID: \"a987c9b8-b785-454f-9a49-196717ab5b70\") " pod="openshift-console/console-79d967c4d6-5sc6v"
Jul 09 20:09:28.334554 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:09:28.334524    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a987c9b8-b785-454f-9a49-196717ab5b70-console-serving-cert\") pod \"console-79d967c4d6-5sc6v\" (UID: \"a987c9b8-b785-454f-9a49-196717ab5b70\") " pod="openshift-console/console-79d967c4d6-5sc6v"
Jul 09 20:09:28.340101 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:09:28.340076    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6h6q\" (UniqueName: \"kubernetes.io/projected/a987c9b8-b785-454f-9a49-196717ab5b70-kube-api-access-d6h6q\") pod \"console-79d967c4d6-5sc6v\" (UID: \"a987c9b8-b785-454f-9a49-196717ab5b70\") " pod="openshift-console/console-79d967c4d6-5sc6v"
Jul 09 20:09:28.443738 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:09:28.443655    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79d967c4d6-5sc6v"
Jul 09 20:09:28.580912 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:09:28.580879    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-79d967c4d6-5sc6v"]
Jul 09 20:09:28.583890 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:09:28.583865    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda987c9b8_b785_454f_9a49_196717ab5b70.slice/crio-8d6b25ed5a2cd84ae421b44580ec72a45de6163eaa45d9032eec8cb899f0e7bd WatchSource:0}: Error finding container 8d6b25ed5a2cd84ae421b44580ec72a45de6163eaa45d9032eec8cb899f0e7bd: Status 404 returned error can't find the container with id 8d6b25ed5a2cd84ae421b44580ec72a45de6163eaa45d9032eec8cb899f0e7bd
Jul 09 20:09:29.445521 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:09:29.445482    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79d967c4d6-5sc6v" event={"ID":"a987c9b8-b785-454f-9a49-196717ab5b70","Type":"ContainerStarted","Data":"d037f338cfd983e02237fec371a8b5149d9517cb92a5112cd7cbc614f480d8b2"}
Jul 09 20:09:29.445918 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:09:29.445543    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79d967c4d6-5sc6v" event={"ID":"a987c9b8-b785-454f-9a49-196717ab5b70","Type":"ContainerStarted","Data":"8d6b25ed5a2cd84ae421b44580ec72a45de6163eaa45d9032eec8cb899f0e7bd"}
Jul 09 20:09:29.462815 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:09:29.462768    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-79d967c4d6-5sc6v" podStartSLOduration=1.462752839 podStartE2EDuration="1.462752839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 20:09:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 20:09:29.462654523 +0000 UTC m=+359.880324333" watchObservedRunningTime="2026-07-09 20:09:29.462752839 +0000 UTC m=+359.880422647"
Jul 09 20:09:37.184624 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:09:37.184582    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-79d967c4d6-5sc6v"]
Jul 09 20:09:38.444653 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:09:38.444612    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-79d967c4d6-5sc6v"
Jul 09 20:09:48.546893 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:09:48.546851    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5cd9fd56c6-6z7pr"]
Jul 09 20:10:02.205395 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:10:02.205326    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-79d967c4d6-5sc6v" podUID="a987c9b8-b785-454f-9a49-196717ab5b70" containerName="console" containerID="cri-o://d037f338cfd983e02237fec371a8b5149d9517cb92a5112cd7cbc614f480d8b2" gracePeriod=15
Jul 09 20:10:02.452130 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:10:02.452104    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79d967c4d6-5sc6v"
Jul 09 20:10:02.566220 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:10:02.566122    2562 generic.go:356] "Generic (PLEG): container finished" podID="a987c9b8-b785-454f-9a49-196717ab5b70" containerID="d037f338cfd983e02237fec371a8b5149d9517cb92a5112cd7cbc614f480d8b2" exitCode=0
Jul 09 20:10:02.566220 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:10:02.566163    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79d967c4d6-5sc6v"
Jul 09 20:10:02.566220 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:10:02.566205    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79d967c4d6-5sc6v" event={"ID":"a987c9b8-b785-454f-9a49-196717ab5b70","Type":"ContainerDied","Data":"d037f338cfd983e02237fec371a8b5149d9517cb92a5112cd7cbc614f480d8b2"}
Jul 09 20:10:02.566512 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:10:02.566255    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79d967c4d6-5sc6v" event={"ID":"a987c9b8-b785-454f-9a49-196717ab5b70","Type":"ContainerDied","Data":"8d6b25ed5a2cd84ae421b44580ec72a45de6163eaa45d9032eec8cb899f0e7bd"}
Jul 09 20:10:02.566512 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:10:02.566280    2562 scope.go:117] "RemoveContainer" containerID="d037f338cfd983e02237fec371a8b5149d9517cb92a5112cd7cbc614f480d8b2"
Jul 09 20:10:02.574645 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:10:02.574628    2562 scope.go:117] "RemoveContainer" containerID="d037f338cfd983e02237fec371a8b5149d9517cb92a5112cd7cbc614f480d8b2"
Jul 09 20:10:02.574894 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:10:02.574873    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d037f338cfd983e02237fec371a8b5149d9517cb92a5112cd7cbc614f480d8b2\": container with ID starting with d037f338cfd983e02237fec371a8b5149d9517cb92a5112cd7cbc614f480d8b2 not found: ID does not exist" containerID="d037f338cfd983e02237fec371a8b5149d9517cb92a5112cd7cbc614f480d8b2"
Jul 09 20:10:02.574945 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:10:02.574901    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d037f338cfd983e02237fec371a8b5149d9517cb92a5112cd7cbc614f480d8b2"} err="failed to get container status \"d037f338cfd983e02237fec371a8b5149d9517cb92a5112cd7cbc614f480d8b2\": rpc error: code = NotFound desc = could not find container \"d037f338cfd983e02237fec371a8b5149d9517cb92a5112cd7cbc614f480d8b2\": container with ID starting with d037f338cfd983e02237fec371a8b5149d9517cb92a5112cd7cbc614f480d8b2 not found: ID does not exist"
Jul 09 20:10:02.602175 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:10:02.602140    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a987c9b8-b785-454f-9a49-196717ab5b70-console-config\") pod \"a987c9b8-b785-454f-9a49-196717ab5b70\" (UID: \"a987c9b8-b785-454f-9a49-196717ab5b70\") "
Jul 09 20:10:02.602359 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:10:02.602192    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a987c9b8-b785-454f-9a49-196717ab5b70-trusted-ca-bundle\") pod \"a987c9b8-b785-454f-9a49-196717ab5b70\" (UID: \"a987c9b8-b785-454f-9a49-196717ab5b70\") "
Jul 09 20:10:02.602359 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:10:02.602216    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a987c9b8-b785-454f-9a49-196717ab5b70-console-serving-cert\") pod \"a987c9b8-b785-454f-9a49-196717ab5b70\" (UID: \"a987c9b8-b785-454f-9a49-196717ab5b70\") "
Jul 09 20:10:02.602359 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:10:02.602278    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a987c9b8-b785-454f-9a49-196717ab5b70-console-oauth-config\") pod \"a987c9b8-b785-454f-9a49-196717ab5b70\" (UID: \"a987c9b8-b785-454f-9a49-196717ab5b70\") "
Jul 09 20:10:02.602359 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:10:02.602327    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d6h6q\" (UniqueName: \"kubernetes.io/projected/a987c9b8-b785-454f-9a49-196717ab5b70-kube-api-access-d6h6q\") pod \"a987c9b8-b785-454f-9a49-196717ab5b70\" (UID: \"a987c9b8-b785-454f-9a49-196717ab5b70\") "
Jul 09 20:10:02.602559 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:10:02.602366    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a987c9b8-b785-454f-9a49-196717ab5b70-oauth-serving-cert\") pod \"a987c9b8-b785-454f-9a49-196717ab5b70\" (UID: \"a987c9b8-b785-454f-9a49-196717ab5b70\") "
Jul 09 20:10:02.602559 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:10:02.602400    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a987c9b8-b785-454f-9a49-196717ab5b70-service-ca\") pod \"a987c9b8-b785-454f-9a49-196717ab5b70\" (UID: \"a987c9b8-b785-454f-9a49-196717ab5b70\") "
Jul 09 20:10:02.602640 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:10:02.602556    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a987c9b8-b785-454f-9a49-196717ab5b70-console-config" (OuterVolumeSpecName: "console-config") pod "a987c9b8-b785-454f-9a49-196717ab5b70" (UID: "a987c9b8-b785-454f-9a49-196717ab5b70"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 20:10:02.602840 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:10:02.602813    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a987c9b8-b785-454f-9a49-196717ab5b70-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "a987c9b8-b785-454f-9a49-196717ab5b70" (UID: "a987c9b8-b785-454f-9a49-196717ab5b70"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 20:10:02.602965 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:10:02.602840    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a987c9b8-b785-454f-9a49-196717ab5b70-service-ca" (OuterVolumeSpecName: "service-ca") pod "a987c9b8-b785-454f-9a49-196717ab5b70" (UID: "a987c9b8-b785-454f-9a49-196717ab5b70"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 20:10:02.602965 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:10:02.602872    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a987c9b8-b785-454f-9a49-196717ab5b70-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "a987c9b8-b785-454f-9a49-196717ab5b70" (UID: "a987c9b8-b785-454f-9a49-196717ab5b70"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 20:10:02.604654 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:10:02.604630    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a987c9b8-b785-454f-9a49-196717ab5b70-kube-api-access-d6h6q" (OuterVolumeSpecName: "kube-api-access-d6h6q") pod "a987c9b8-b785-454f-9a49-196717ab5b70" (UID: "a987c9b8-b785-454f-9a49-196717ab5b70"). InnerVolumeSpecName "kube-api-access-d6h6q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 20:10:02.604750 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:10:02.604654    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a987c9b8-b785-454f-9a49-196717ab5b70-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "a987c9b8-b785-454f-9a49-196717ab5b70" (UID: "a987c9b8-b785-454f-9a49-196717ab5b70"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 20:10:02.604810 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:10:02.604765    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a987c9b8-b785-454f-9a49-196717ab5b70-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "a987c9b8-b785-454f-9a49-196717ab5b70" (UID: "a987c9b8-b785-454f-9a49-196717ab5b70"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 20:10:02.703294 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:10:02.703247    2562 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a987c9b8-b785-454f-9a49-196717ab5b70-console-oauth-config\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:10:02.703294 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:10:02.703294    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6h6q\" (UniqueName: \"kubernetes.io/projected/a987c9b8-b785-454f-9a49-196717ab5b70-kube-api-access-d6h6q\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:10:02.703294 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:10:02.703331    2562 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a987c9b8-b785-454f-9a49-196717ab5b70-oauth-serving-cert\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:10:02.703571 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:10:02.703345    2562 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a987c9b8-b785-454f-9a49-196717ab5b70-service-ca\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:10:02.703571 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:10:02.703357    2562 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a987c9b8-b785-454f-9a49-196717ab5b70-console-config\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:10:02.703571 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:10:02.703367    2562 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a987c9b8-b785-454f-9a49-196717ab5b70-trusted-ca-bundle\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:10:02.703571 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:10:02.703378    2562 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a987c9b8-b785-454f-9a49-196717ab5b70-console-serving-cert\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:10:02.888515 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:10:02.888489    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-79d967c4d6-5sc6v"]
Jul 09 20:10:02.891594 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:10:02.891570    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-79d967c4d6-5sc6v"]
Jul 09 20:10:04.245273 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:10:04.245238    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a987c9b8-b785-454f-9a49-196717ab5b70" path="/var/lib/kubelet/pods/a987c9b8-b785-454f-9a49-196717ab5b70/volumes"
Jul 09 20:10:13.569269 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:10:13.569205    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5cd9fd56c6-6z7pr" podUID="d16a8e79-e4ff-4474-9e21-617024bae5a7" containerName="console" containerID="cri-o://efe249e39e76e62ecfd5a7f52b30f7791cdfb29e875a75d29143b2b50b399212" gracePeriod=15
Jul 09 20:10:13.810255 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:10:13.810231    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5cd9fd56c6-6z7pr"
Jul 09 20:10:13.895254 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:10:13.895156    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d16a8e79-e4ff-4474-9e21-617024bae5a7-console-serving-cert\") pod \"d16a8e79-e4ff-4474-9e21-617024bae5a7\" (UID: \"d16a8e79-e4ff-4474-9e21-617024bae5a7\") "
Jul 09 20:10:13.895254 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:10:13.895210    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d16a8e79-e4ff-4474-9e21-617024bae5a7-console-oauth-config\") pod \"d16a8e79-e4ff-4474-9e21-617024bae5a7\" (UID: \"d16a8e79-e4ff-4474-9e21-617024bae5a7\") "
Jul 09 20:10:13.895254 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:10:13.895231    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d16a8e79-e4ff-4474-9e21-617024bae5a7-console-config\") pod \"d16a8e79-e4ff-4474-9e21-617024bae5a7\" (UID: \"d16a8e79-e4ff-4474-9e21-617024bae5a7\") "
Jul 09 20:10:13.895254 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:10:13.895248    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d16a8e79-e4ff-4474-9e21-617024bae5a7-service-ca\") pod \"d16a8e79-e4ff-4474-9e21-617024bae5a7\" (UID: \"d16a8e79-e4ff-4474-9e21-617024bae5a7\") "
Jul 09 20:10:13.895601 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:10:13.895285    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zk54r\" (UniqueName: \"kubernetes.io/projected/d16a8e79-e4ff-4474-9e21-617024bae5a7-kube-api-access-zk54r\") pod \"d16a8e79-e4ff-4474-9e21-617024bae5a7\" (UID: \"d16a8e79-e4ff-4474-9e21-617024bae5a7\") "
Jul 09 20:10:13.895601 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:10:13.895348    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d16a8e79-e4ff-4474-9e21-617024bae5a7-oauth-serving-cert\") pod \"d16a8e79-e4ff-4474-9e21-617024bae5a7\" (UID: \"d16a8e79-e4ff-4474-9e21-617024bae5a7\") "
Jul 09 20:10:13.895601 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:10:13.895367    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d16a8e79-e4ff-4474-9e21-617024bae5a7-trusted-ca-bundle\") pod \"d16a8e79-e4ff-4474-9e21-617024bae5a7\" (UID: \"d16a8e79-e4ff-4474-9e21-617024bae5a7\") "
Jul 09 20:10:13.895808 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:10:13.895757    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d16a8e79-e4ff-4474-9e21-617024bae5a7-service-ca" (OuterVolumeSpecName: "service-ca") pod "d16a8e79-e4ff-4474-9e21-617024bae5a7" (UID: "d16a8e79-e4ff-4474-9e21-617024bae5a7"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 20:10:13.895808 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:10:13.895705    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d16a8e79-e4ff-4474-9e21-617024bae5a7-console-config" (OuterVolumeSpecName: "console-config") pod "d16a8e79-e4ff-4474-9e21-617024bae5a7" (UID: "d16a8e79-e4ff-4474-9e21-617024bae5a7"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 20:10:13.895979 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:10:13.895812    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d16a8e79-e4ff-4474-9e21-617024bae5a7-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "d16a8e79-e4ff-4474-9e21-617024bae5a7" (UID: "d16a8e79-e4ff-4474-9e21-617024bae5a7"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 20:10:13.895979 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:10:13.895876    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d16a8e79-e4ff-4474-9e21-617024bae5a7-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "d16a8e79-e4ff-4474-9e21-617024bae5a7" (UID: "d16a8e79-e4ff-4474-9e21-617024bae5a7"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 20:10:13.897435 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:10:13.897406    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d16a8e79-e4ff-4474-9e21-617024bae5a7-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "d16a8e79-e4ff-4474-9e21-617024bae5a7" (UID: "d16a8e79-e4ff-4474-9e21-617024bae5a7"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 20:10:13.897526 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:10:13.897466    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d16a8e79-e4ff-4474-9e21-617024bae5a7-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "d16a8e79-e4ff-4474-9e21-617024bae5a7" (UID: "d16a8e79-e4ff-4474-9e21-617024bae5a7"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 20:10:13.897526 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:10:13.897504    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d16a8e79-e4ff-4474-9e21-617024bae5a7-kube-api-access-zk54r" (OuterVolumeSpecName: "kube-api-access-zk54r") pod "d16a8e79-e4ff-4474-9e21-617024bae5a7" (UID: "d16a8e79-e4ff-4474-9e21-617024bae5a7"). InnerVolumeSpecName "kube-api-access-zk54r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 20:10:13.995982 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:10:13.995943    2562 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d16a8e79-e4ff-4474-9e21-617024bae5a7-console-oauth-config\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:10:13.995982 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:10:13.995972    2562 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d16a8e79-e4ff-4474-9e21-617024bae5a7-console-config\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:10:13.995982 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:10:13.995981    2562 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d16a8e79-e4ff-4474-9e21-617024bae5a7-service-ca\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:10:13.995982 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:10:13.995991    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zk54r\" (UniqueName: \"kubernetes.io/projected/d16a8e79-e4ff-4474-9e21-617024bae5a7-kube-api-access-zk54r\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:10:13.996243 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:10:13.996000    2562 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d16a8e79-e4ff-4474-9e21-617024bae5a7-oauth-serving-cert\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:10:13.996243 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:10:13.996008    2562 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d16a8e79-e4ff-4474-9e21-617024bae5a7-trusted-ca-bundle\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:10:13.996243 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:10:13.996016    2562 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d16a8e79-e4ff-4474-9e21-617024bae5a7-console-serving-cert\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:10:14.610670 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:10:14.610574    2562 generic.go:356] "Generic (PLEG): container finished" podID="d16a8e79-e4ff-4474-9e21-617024bae5a7" containerID="efe249e39e76e62ecfd5a7f52b30f7791cdfb29e875a75d29143b2b50b399212" exitCode=0
Jul 09 20:10:14.610670 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:10:14.610619    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5cd9fd56c6-6z7pr"
Jul 09 20:10:14.611191 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:10:14.610662    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5cd9fd56c6-6z7pr" event={"ID":"d16a8e79-e4ff-4474-9e21-617024bae5a7","Type":"ContainerDied","Data":"efe249e39e76e62ecfd5a7f52b30f7791cdfb29e875a75d29143b2b50b399212"}
Jul 09 20:10:14.611191 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:10:14.610718    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5cd9fd56c6-6z7pr" event={"ID":"d16a8e79-e4ff-4474-9e21-617024bae5a7","Type":"ContainerDied","Data":"23136b89bb5ebd3192f851252c715b4f6d0e4f94f9a9dd2c8719c379ef28ff1f"}
Jul 09 20:10:14.611191 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:10:14.610747    2562 scope.go:117] "RemoveContainer" containerID="efe249e39e76e62ecfd5a7f52b30f7791cdfb29e875a75d29143b2b50b399212"
Jul 09 20:10:14.618753 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:10:14.618736    2562 scope.go:117] "RemoveContainer" containerID="efe249e39e76e62ecfd5a7f52b30f7791cdfb29e875a75d29143b2b50b399212"
Jul 09 20:10:14.619018 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:10:14.619000    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efe249e39e76e62ecfd5a7f52b30f7791cdfb29e875a75d29143b2b50b399212\": container with ID starting with efe249e39e76e62ecfd5a7f52b30f7791cdfb29e875a75d29143b2b50b399212 not found: ID does not exist" containerID="efe249e39e76e62ecfd5a7f52b30f7791cdfb29e875a75d29143b2b50b399212"
Jul 09 20:10:14.619079 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:10:14.619031    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efe249e39e76e62ecfd5a7f52b30f7791cdfb29e875a75d29143b2b50b399212"} err="failed to get container status \"efe249e39e76e62ecfd5a7f52b30f7791cdfb29e875a75d29143b2b50b399212\": rpc error: code = NotFound desc = could not find container \"efe249e39e76e62ecfd5a7f52b30f7791cdfb29e875a75d29143b2b50b399212\": container with ID starting with efe249e39e76e62ecfd5a7f52b30f7791cdfb29e875a75d29143b2b50b399212 not found: ID does not exist"
Jul 09 20:10:14.629058 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:10:14.629037    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5cd9fd56c6-6z7pr"]
Jul 09 20:10:14.632628 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:10:14.632584    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5cd9fd56c6-6z7pr"]
Jul 09 20:10:16.240519 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:10:16.240480    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d16a8e79-e4ff-4474-9e21-617024bae5a7" path="/var/lib/kubelet/pods/d16a8e79-e4ff-4474-9e21-617024bae5a7/volumes"
Jul 09 20:10:39.664018 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:10:39.663982    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-584d967bd-27qcc"]
Jul 09 20:10:39.664430 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:10:39.664227    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d16a8e79-e4ff-4474-9e21-617024bae5a7" containerName="console"
Jul 09 20:10:39.664430 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:10:39.664238    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d16a8e79-e4ff-4474-9e21-617024bae5a7" containerName="console"
Jul 09 20:10:39.664430 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:10:39.664258    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a987c9b8-b785-454f-9a49-196717ab5b70" containerName="console"
Jul 09 20:10:39.664430 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:10:39.664263    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a987c9b8-b785-454f-9a49-196717ab5b70" containerName="console"
Jul 09 20:10:39.664430 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:10:39.664321    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="d16a8e79-e4ff-4474-9e21-617024bae5a7" containerName="console"
Jul 09 20:10:39.664430 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:10:39.664332    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="a987c9b8-b785-454f-9a49-196717ab5b70" containerName="console"
Jul 09 20:10:39.668450 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:10:39.668431    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-584d967bd-27qcc"
Jul 09 20:10:39.671476 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:10:39.671457    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 09 20:10:39.671575 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:10:39.671540    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 09 20:10:39.673061 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:10:39.673039    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 09 20:10:39.673190 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:10:39.673092    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 09 20:10:39.673190 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:10:39.673102    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 09 20:10:39.673190 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:10:39.673102    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 09 20:10:39.673190 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:10:39.673098    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 09 20:10:39.675551 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:10:39.675498    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/99856b53-daa7-4f90-ba0a-762b199e9638-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-584d967bd-27qcc\" (UID: \"99856b53-daa7-4f90-ba0a-762b199e9638\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-584d967bd-27qcc"
Jul 09 20:10:39.675681 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:10:39.675559    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/99856b53-daa7-4f90-ba0a-762b199e9638-additional-service-ca\") pod \"cluster-proxy-proxy-agent-584d967bd-27qcc\" (UID: \"99856b53-daa7-4f90-ba0a-762b199e9638\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-584d967bd-27qcc"
Jul 09 20:10:39.675681 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:10:39.675588    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/99856b53-daa7-4f90-ba0a-762b199e9638-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-584d967bd-27qcc\" (UID: \"99856b53-daa7-4f90-ba0a-762b199e9638\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-584d967bd-27qcc"
Jul 09 20:10:39.675805 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:10:39.675676    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/99856b53-daa7-4f90-ba0a-762b199e9638-hub\") pod \"cluster-proxy-proxy-agent-584d967bd-27qcc\" (UID: \"99856b53-daa7-4f90-ba0a-762b199e9638\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-584d967bd-27qcc"
Jul 09 20:10:39.675805 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:10:39.675734    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vd7x\" (UniqueName: \"kubernetes.io/projected/99856b53-daa7-4f90-ba0a-762b199e9638-kube-api-access-2vd7x\") pod \"cluster-proxy-proxy-agent-584d967bd-27qcc\" (UID: \"99856b53-daa7-4f90-ba0a-762b199e9638\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-584d967bd-27qcc"
Jul 09 20:10:39.675909 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:10:39.675810    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/99856b53-daa7-4f90-ba0a-762b199e9638-ca\") pod \"cluster-proxy-proxy-agent-584d967bd-27qcc\" (UID: \"99856b53-daa7-4f90-ba0a-762b199e9638\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-584d967bd-27qcc"
Jul 09 20:10:39.676660 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:10:39.676636    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-584d967bd-27qcc"]
Jul 09 20:10:39.776788 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:10:39.776753    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vd7x\" (UniqueName: \"kubernetes.io/projected/99856b53-daa7-4f90-ba0a-762b199e9638-kube-api-access-2vd7x\") pod \"cluster-proxy-proxy-agent-584d967bd-27qcc\" (UID: \"99856b53-daa7-4f90-ba0a-762b199e9638\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-584d967bd-27qcc"
Jul 09 20:10:39.776788 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:10:39.776789    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/99856b53-daa7-4f90-ba0a-762b199e9638-ca\") pod \"cluster-proxy-proxy-agent-584d967bd-27qcc\" (UID: \"99856b53-daa7-4f90-ba0a-762b199e9638\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-584d967bd-27qcc"
Jul 09 20:10:39.777188 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:10:39.776851    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/99856b53-daa7-4f90-ba0a-762b199e9638-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-584d967bd-27qcc\" (UID: \"99856b53-daa7-4f90-ba0a-762b199e9638\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-584d967bd-27qcc"
Jul 09 20:10:39.777188 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:10:39.776876    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/99856b53-daa7-4f90-ba0a-762b199e9638-additional-service-ca\") pod \"cluster-proxy-proxy-agent-584d967bd-27qcc\" (UID: \"99856b53-daa7-4f90-ba0a-762b199e9638\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-584d967bd-27qcc"
Jul 09 20:10:39.777188 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:10:39.776899    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/99856b53-daa7-4f90-ba0a-762b199e9638-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-584d967bd-27qcc\" (UID: \"99856b53-daa7-4f90-ba0a-762b199e9638\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-584d967bd-27qcc"
Jul 09 20:10:39.777378 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:10:39.777322    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/99856b53-daa7-4f90-ba0a-762b199e9638-hub\") pod \"cluster-proxy-proxy-agent-584d967bd-27qcc\" (UID: \"99856b53-daa7-4f90-ba0a-762b199e9638\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-584d967bd-27qcc"
Jul 09 20:10:39.778112 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:10:39.778081    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/99856b53-daa7-4f90-ba0a-762b199e9638-additional-service-ca\") pod \"cluster-proxy-proxy-agent-584d967bd-27qcc\" (UID: \"99856b53-daa7-4f90-ba0a-762b199e9638\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-584d967bd-27qcc"
Jul 09 20:10:39.779713 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:10:39.779688    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/99856b53-daa7-4f90-ba0a-762b199e9638-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-584d967bd-27qcc\" (UID: \"99856b53-daa7-4f90-ba0a-762b199e9638\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-584d967bd-27qcc"
Jul 09 20:10:39.779850 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:10:39.779787    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/99856b53-daa7-4f90-ba0a-762b199e9638-ca\") pod \"cluster-proxy-proxy-agent-584d967bd-27qcc\" (UID: \"99856b53-daa7-4f90-ba0a-762b199e9638\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-584d967bd-27qcc"
Jul 09 20:10:39.780078 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:10:39.779848    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/99856b53-daa7-4f90-ba0a-762b199e9638-hub\") pod \"cluster-proxy-proxy-agent-584d967bd-27qcc\" (UID: \"99856b53-daa7-4f90-ba0a-762b199e9638\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-584d967bd-27qcc"
Jul 09 20:10:39.780078 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:10:39.779963    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/99856b53-daa7-4f90-ba0a-762b199e9638-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-584d967bd-27qcc\" (UID: \"99856b53-daa7-4f90-ba0a-762b199e9638\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-584d967bd-27qcc"
Jul 09 20:10:39.787358 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:10:39.787282    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vd7x\" (UniqueName: \"kubernetes.io/projected/99856b53-daa7-4f90-ba0a-762b199e9638-kube-api-access-2vd7x\") pod \"cluster-proxy-proxy-agent-584d967bd-27qcc\" (UID: \"99856b53-daa7-4f90-ba0a-762b199e9638\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-584d967bd-27qcc"
Jul 09 20:10:39.987712 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:10:39.987670    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-584d967bd-27qcc"
Jul 09 20:10:40.116365 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:10:40.116321    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-584d967bd-27qcc"]
Jul 09 20:10:40.119100 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:10:40.119070    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod99856b53_daa7_4f90_ba0a_762b199e9638.slice/crio-83efe4e90845ff02a9842ba35956fe8264a60c89a5443a03645a6657613e4d75 WatchSource:0}: Error finding container 83efe4e90845ff02a9842ba35956fe8264a60c89a5443a03645a6657613e4d75: Status 404 returned error can't find the container with id 83efe4e90845ff02a9842ba35956fe8264a60c89a5443a03645a6657613e4d75
Jul 09 20:10:40.121094 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:10:40.121076    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 09 20:10:40.705647 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:10:40.705610    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-584d967bd-27qcc" event={"ID":"99856b53-daa7-4f90-ba0a-762b199e9638","Type":"ContainerStarted","Data":"83efe4e90845ff02a9842ba35956fe8264a60c89a5443a03645a6657613e4d75"}
Jul 09 20:10:44.724906 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:10:44.724868    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-584d967bd-27qcc" event={"ID":"99856b53-daa7-4f90-ba0a-762b199e9638","Type":"ContainerStarted","Data":"83636940fd5b32208de3b026abaa73a3f45bed84c06956b66db1a2db5567eeaa"}
Jul 09 20:10:44.725458 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:10:44.724919    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-584d967bd-27qcc" event={"ID":"99856b53-daa7-4f90-ba0a-762b199e9638","Type":"ContainerStarted","Data":"786206f6048d85f8c8b9b55759d9cbe96edca1f8da32a607adba5fc98e1fa49e"}
Jul 09 20:10:44.725458 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:10:44.724936    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-584d967bd-27qcc" event={"ID":"99856b53-daa7-4f90-ba0a-762b199e9638","Type":"ContainerStarted","Data":"9d1794fb17142f6a25c0a141eaedc5b218bfdfe3b7aa910065668bc3cb0e0d54"}
Jul 09 20:10:44.744672 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:10:44.744621    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-584d967bd-27qcc" podStartSLOduration=1.4893021690000001 podStartE2EDuration="5.744605502s" totalImagesPullingTime="4.255303333s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 20:10:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 20:10:44.7422657 +0000 UTC m=+435.159935511" watchObservedRunningTime="2026-07-09 20:10:44.744605502 +0000 UTC m=+435.162275377"
Jul 09 20:11:18.791815 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:11:18.791777    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vj4qk"]
Jul 09 20:11:18.796623 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:11:18.796603    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vj4qk"
Jul 09 20:11:18.799723 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:11:18.799691    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 09 20:11:18.801161 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:11:18.801135    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 09 20:11:18.801481 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:11:18.801462    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-fqltr\""
Jul 09 20:11:18.803835 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:11:18.803809    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vj4qk"]
Jul 09 20:11:18.861434 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:11:18.861396    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eb47f6d7-b5d5-4ed4-94f0-8635b1b89199-tmp\") pod \"openshift-lws-operator-fd8ccff4c-vj4qk\" (UID: \"eb47f6d7-b5d5-4ed4-94f0-8635b1b89199\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vj4qk"
Jul 09 20:11:18.861605 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:11:18.861451    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5sr8\" (UniqueName: \"kubernetes.io/projected/eb47f6d7-b5d5-4ed4-94f0-8635b1b89199-kube-api-access-l5sr8\") pod \"openshift-lws-operator-fd8ccff4c-vj4qk\" (UID: \"eb47f6d7-b5d5-4ed4-94f0-8635b1b89199\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vj4qk"
Jul 09 20:11:18.962290 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:11:18.962251    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5sr8\" (UniqueName: \"kubernetes.io/projected/eb47f6d7-b5d5-4ed4-94f0-8635b1b89199-kube-api-access-l5sr8\") pod \"openshift-lws-operator-fd8ccff4c-vj4qk\" (UID: \"eb47f6d7-b5d5-4ed4-94f0-8635b1b89199\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vj4qk"
Jul 09 20:11:18.962487 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:11:18.962355    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eb47f6d7-b5d5-4ed4-94f0-8635b1b89199-tmp\") pod \"openshift-lws-operator-fd8ccff4c-vj4qk\" (UID: \"eb47f6d7-b5d5-4ed4-94f0-8635b1b89199\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vj4qk"
Jul 09 20:11:18.962685 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:11:18.962669    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eb47f6d7-b5d5-4ed4-94f0-8635b1b89199-tmp\") pod \"openshift-lws-operator-fd8ccff4c-vj4qk\" (UID: \"eb47f6d7-b5d5-4ed4-94f0-8635b1b89199\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vj4qk"
Jul 09 20:11:18.970855 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:11:18.970824    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5sr8\" (UniqueName: \"kubernetes.io/projected/eb47f6d7-b5d5-4ed4-94f0-8635b1b89199-kube-api-access-l5sr8\") pod \"openshift-lws-operator-fd8ccff4c-vj4qk\" (UID: \"eb47f6d7-b5d5-4ed4-94f0-8635b1b89199\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vj4qk"
Jul 09 20:11:19.111453 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:11:19.111368    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vj4qk"
Jul 09 20:11:19.238886 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:11:19.238853    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vj4qk"]
Jul 09 20:11:19.241706 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:11:19.241680    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeb47f6d7_b5d5_4ed4_94f0_8635b1b89199.slice/crio-359fd1c365594b9fe060ecd8d039dbf9e87844c873cd64e9891825ca1458be88 WatchSource:0}: Error finding container 359fd1c365594b9fe060ecd8d039dbf9e87844c873cd64e9891825ca1458be88: Status 404 returned error can't find the container with id 359fd1c365594b9fe060ecd8d039dbf9e87844c873cd64e9891825ca1458be88
Jul 09 20:11:19.847089 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:11:19.847057    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vj4qk" event={"ID":"eb47f6d7-b5d5-4ed4-94f0-8635b1b89199","Type":"ContainerStarted","Data":"359fd1c365594b9fe060ecd8d039dbf9e87844c873cd64e9891825ca1458be88"}
Jul 09 20:11:21.856466 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:11:21.856425    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vj4qk" event={"ID":"eb47f6d7-b5d5-4ed4-94f0-8635b1b89199","Type":"ContainerStarted","Data":"01bc8eaaaebe7b55829e6909d81bfe7885fdf6141768de9027ab133ba9acd24b"}
Jul 09 20:11:21.873864 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:11:21.873802    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-vj4qk" podStartSLOduration=1.3918005230000001 podStartE2EDuration="3.873782927s" totalImagesPullingTime="2.481982404s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 20:11:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 20:11:21.873669552 +0000 UTC m=+472.291339364" watchObservedRunningTime="2026-07-09 20:11:21.873782927 +0000 UTC m=+472.291452735"
Jul 09 20:11:40.013808 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:11:40.013772    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-64f986597c-htv4s"]
Jul 09 20:11:40.017056 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:11:40.017034    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-64f986597c-htv4s"
Jul 09 20:11:40.023036 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:11:40.023005    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-wc5fj\""
Jul 09 20:11:40.023036 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:11:40.023025    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 09 20:11:40.023224 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:11:40.023164    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 09 20:11:40.023224 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:11:40.023182    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 09 20:11:40.023661 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:11:40.023246    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 09 20:11:40.062563 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:11:40.062524    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-64f986597c-htv4s"]
Jul 09 20:11:40.135322 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:11:40.135281    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/48e85b41-042f-4b1f-a2bc-b7a246286070-tmp\") pod \"opendatahub-operator-controller-manager-64f986597c-htv4s\" (UID: \"48e85b41-042f-4b1f-a2bc-b7a246286070\") " pod="opendatahub/opendatahub-operator-controller-manager-64f986597c-htv4s"
Jul 09 20:11:40.135483 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:11:40.135333    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/48e85b41-042f-4b1f-a2bc-b7a246286070-webhook-cert\") pod \"opendatahub-operator-controller-manager-64f986597c-htv4s\" (UID: \"48e85b41-042f-4b1f-a2bc-b7a246286070\") " pod="opendatahub/opendatahub-operator-controller-manager-64f986597c-htv4s"
Jul 09 20:11:40.135483 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:11:40.135395    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/48e85b41-042f-4b1f-a2bc-b7a246286070-apiservice-cert\") pod \"opendatahub-operator-controller-manager-64f986597c-htv4s\" (UID: \"48e85b41-042f-4b1f-a2bc-b7a246286070\") " pod="opendatahub/opendatahub-operator-controller-manager-64f986597c-htv4s"
Jul 09 20:11:40.135483 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:11:40.135421    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tktgq\" (UniqueName: \"kubernetes.io/projected/48e85b41-042f-4b1f-a2bc-b7a246286070-kube-api-access-tktgq\") pod \"opendatahub-operator-controller-manager-64f986597c-htv4s\" (UID: \"48e85b41-042f-4b1f-a2bc-b7a246286070\") " pod="opendatahub/opendatahub-operator-controller-manager-64f986597c-htv4s"
Jul 09 20:11:40.135483 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:11:40.135456    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/48e85b41-042f-4b1f-a2bc-b7a246286070-manifests\") pod \"opendatahub-operator-controller-manager-64f986597c-htv4s\" (UID: \"48e85b41-042f-4b1f-a2bc-b7a246286070\") " pod="opendatahub/opendatahub-operator-controller-manager-64f986597c-htv4s"
Jul 09 20:11:40.236617 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:11:40.236582    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/48e85b41-042f-4b1f-a2bc-b7a246286070-apiservice-cert\") pod \"opendatahub-operator-controller-manager-64f986597c-htv4s\" (UID: \"48e85b41-042f-4b1f-a2bc-b7a246286070\") " pod="opendatahub/opendatahub-operator-controller-manager-64f986597c-htv4s"
Jul 09 20:11:40.236828 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:11:40.236629    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tktgq\" (UniqueName: \"kubernetes.io/projected/48e85b41-042f-4b1f-a2bc-b7a246286070-kube-api-access-tktgq\") pod \"opendatahub-operator-controller-manager-64f986597c-htv4s\" (UID: \"48e85b41-042f-4b1f-a2bc-b7a246286070\") " pod="opendatahub/opendatahub-operator-controller-manager-64f986597c-htv4s"
Jul 09 20:11:40.236828 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:11:40.236671    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/48e85b41-042f-4b1f-a2bc-b7a246286070-manifests\") pod \"opendatahub-operator-controller-manager-64f986597c-htv4s\" (UID: \"48e85b41-042f-4b1f-a2bc-b7a246286070\") " pod="opendatahub/opendatahub-operator-controller-manager-64f986597c-htv4s"
Jul 09 20:11:40.236828 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:11:40.236702    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/48e85b41-042f-4b1f-a2bc-b7a246286070-tmp\") pod \"opendatahub-operator-controller-manager-64f986597c-htv4s\" (UID: \"48e85b41-042f-4b1f-a2bc-b7a246286070\") " pod="opendatahub/opendatahub-operator-controller-manager-64f986597c-htv4s"
Jul 09 20:11:40.236828 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:11:40.236727    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/48e85b41-042f-4b1f-a2bc-b7a246286070-webhook-cert\") pod \"opendatahub-operator-controller-manager-64f986597c-htv4s\" (UID: \"48e85b41-042f-4b1f-a2bc-b7a246286070\") " pod="opendatahub/opendatahub-operator-controller-manager-64f986597c-htv4s"
Jul 09 20:11:40.237176 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:11:40.237147    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/48e85b41-042f-4b1f-a2bc-b7a246286070-manifests\") pod \"opendatahub-operator-controller-manager-64f986597c-htv4s\" (UID: \"48e85b41-042f-4b1f-a2bc-b7a246286070\") " pod="opendatahub/opendatahub-operator-controller-manager-64f986597c-htv4s"
Jul 09 20:11:40.239106 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:11:40.239070    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/48e85b41-042f-4b1f-a2bc-b7a246286070-tmp\") pod \"opendatahub-operator-controller-manager-64f986597c-htv4s\" (UID: \"48e85b41-042f-4b1f-a2bc-b7a246286070\") " pod="opendatahub/opendatahub-operator-controller-manager-64f986597c-htv4s"
Jul 09 20:11:40.239477 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:11:40.239445    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/48e85b41-042f-4b1f-a2bc-b7a246286070-webhook-cert\") pod \"opendatahub-operator-controller-manager-64f986597c-htv4s\" (UID: \"48e85b41-042f-4b1f-a2bc-b7a246286070\") " pod="opendatahub/opendatahub-operator-controller-manager-64f986597c-htv4s"
Jul 09 20:11:40.239560 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:11:40.239487    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/48e85b41-042f-4b1f-a2bc-b7a246286070-apiservice-cert\") pod \"opendatahub-operator-controller-manager-64f986597c-htv4s\" (UID: \"48e85b41-042f-4b1f-a2bc-b7a246286070\") " pod="opendatahub/opendatahub-operator-controller-manager-64f986597c-htv4s"
Jul 09 20:11:40.247677 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:11:40.247655    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tktgq\" (UniqueName: \"kubernetes.io/projected/48e85b41-042f-4b1f-a2bc-b7a246286070-kube-api-access-tktgq\") pod \"opendatahub-operator-controller-manager-64f986597c-htv4s\" (UID: \"48e85b41-042f-4b1f-a2bc-b7a246286070\") " pod="opendatahub/opendatahub-operator-controller-manager-64f986597c-htv4s"
Jul 09 20:11:40.327342 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:11:40.327235    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-64f986597c-htv4s"
Jul 09 20:11:40.467793 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:11:40.467771    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-64f986597c-htv4s"]
Jul 09 20:11:40.470074 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:11:40.470049    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48e85b41_042f_4b1f_a2bc_b7a246286070.slice/crio-52c31e66d57d892a846439af022afc21596ecffcdff6729e84b7613b641e402b WatchSource:0}: Error finding container 52c31e66d57d892a846439af022afc21596ecffcdff6729e84b7613b641e402b: Status 404 returned error can't find the container with id 52c31e66d57d892a846439af022afc21596ecffcdff6729e84b7613b641e402b
Jul 09 20:11:40.929789 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:11:40.929749    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-64f986597c-htv4s" event={"ID":"48e85b41-042f-4b1f-a2bc-b7a246286070","Type":"ContainerStarted","Data":"52c31e66d57d892a846439af022afc21596ecffcdff6729e84b7613b641e402b"}
Jul 09 20:11:43.944672 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:11:43.944584    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-64f986597c-htv4s" event={"ID":"48e85b41-042f-4b1f-a2bc-b7a246286070","Type":"ContainerStarted","Data":"77f2ac3cbbc4d213a1763cd6ff886fea383daef44e1c99bc6659c902754219c4"}
Jul 09 20:11:44.950573 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:11:44.950488    2562 generic.go:356] "Generic (PLEG): container finished" podID="48e85b41-042f-4b1f-a2bc-b7a246286070" containerID="77f2ac3cbbc4d213a1763cd6ff886fea383daef44e1c99bc6659c902754219c4" exitCode=0
Jul 09 20:11:44.950957 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:11:44.950573    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-64f986597c-htv4s" event={"ID":"48e85b41-042f-4b1f-a2bc-b7a246286070","Type":"ContainerDied","Data":"77f2ac3cbbc4d213a1763cd6ff886fea383daef44e1c99bc6659c902754219c4"}
Jul 09 20:11:45.956725 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:11:45.956687    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-64f986597c-htv4s" event={"ID":"48e85b41-042f-4b1f-a2bc-b7a246286070","Type":"ContainerStarted","Data":"8a4c14a630ae4886d55ec868c1dcbe621c3d35246819983a46718895aa7f1e27"}
Jul 09 20:11:45.957202 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:11:45.956818    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-64f986597c-htv4s"
Jul 09 20:11:45.987600 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:11:45.987547    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-64f986597c-htv4s" podStartSLOduration=2.498046938 podStartE2EDuration="6.987531325s" totalImagesPullingTime="3.39148556s" totalInitContainerRuntime="1.097998827s" isStatefulPod=false podCreationTimestamp="2026-07-09 20:11:39 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 20:11:45.987211492 +0000 UTC m=+496.404881303" watchObservedRunningTime="2026-07-09 20:11:45.987531325 +0000 UTC m=+496.405201134"
Jul 09 20:11:56.964111 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:11:56.964081    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-64f986597c-htv4s"
Jul 09 20:11:59.121885 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:11:59.121848    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-8d5dd4c74-cgxgl"]
Jul 09 20:11:59.129210 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:11:59.129182    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-8d5dd4c74-cgxgl"
Jul 09 20:11:59.133386 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:11:59.133366    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 09 20:11:59.134028 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:11:59.134008    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 09 20:11:59.134124 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:11:59.134029    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-zh5gl\""
Jul 09 20:11:59.144836 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:11:59.144813    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-8d5dd4c74-cgxgl"]
Jul 09 20:11:59.186646 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:11:59.186615    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4bd539c1-f6e7-4b25-a378-29b483b20851-tmp\") pod \"kube-auth-proxy-8d5dd4c74-cgxgl\" (UID: \"4bd539c1-f6e7-4b25-a378-29b483b20851\") " pod="openshift-ingress/kube-auth-proxy-8d5dd4c74-cgxgl"
Jul 09 20:11:59.186798 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:11:59.186668    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4bd539c1-f6e7-4b25-a378-29b483b20851-tls-certs\") pod \"kube-auth-proxy-8d5dd4c74-cgxgl\" (UID: \"4bd539c1-f6e7-4b25-a378-29b483b20851\") " pod="openshift-ingress/kube-auth-proxy-8d5dd4c74-cgxgl"
Jul 09 20:11:59.186798 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:11:59.186723    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlt4j\" (UniqueName: \"kubernetes.io/projected/4bd539c1-f6e7-4b25-a378-29b483b20851-kube-api-access-nlt4j\") pod \"kube-auth-proxy-8d5dd4c74-cgxgl\" (UID: \"4bd539c1-f6e7-4b25-a378-29b483b20851\") " pod="openshift-ingress/kube-auth-proxy-8d5dd4c74-cgxgl"
Jul 09 20:11:59.287975 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:11:59.287946    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4bd539c1-f6e7-4b25-a378-29b483b20851-tmp\") pod \"kube-auth-proxy-8d5dd4c74-cgxgl\" (UID: \"4bd539c1-f6e7-4b25-a378-29b483b20851\") " pod="openshift-ingress/kube-auth-proxy-8d5dd4c74-cgxgl"
Jul 09 20:11:59.288172 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:11:59.287993    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4bd539c1-f6e7-4b25-a378-29b483b20851-tls-certs\") pod \"kube-auth-proxy-8d5dd4c74-cgxgl\" (UID: \"4bd539c1-f6e7-4b25-a378-29b483b20851\") " pod="openshift-ingress/kube-auth-proxy-8d5dd4c74-cgxgl"
Jul 09 20:11:59.288172 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:11:59.288015    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlt4j\" (UniqueName: \"kubernetes.io/projected/4bd539c1-f6e7-4b25-a378-29b483b20851-kube-api-access-nlt4j\") pod \"kube-auth-proxy-8d5dd4c74-cgxgl\" (UID: \"4bd539c1-f6e7-4b25-a378-29b483b20851\") " pod="openshift-ingress/kube-auth-proxy-8d5dd4c74-cgxgl"
Jul 09 20:11:59.290343 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:11:59.290297    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4bd539c1-f6e7-4b25-a378-29b483b20851-tmp\") pod \"kube-auth-proxy-8d5dd4c74-cgxgl\" (UID: \"4bd539c1-f6e7-4b25-a378-29b483b20851\") " pod="openshift-ingress/kube-auth-proxy-8d5dd4c74-cgxgl"
Jul 09 20:11:59.290554 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:11:59.290536    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4bd539c1-f6e7-4b25-a378-29b483b20851-tls-certs\") pod \"kube-auth-proxy-8d5dd4c74-cgxgl\" (UID: \"4bd539c1-f6e7-4b25-a378-29b483b20851\") " pod="openshift-ingress/kube-auth-proxy-8d5dd4c74-cgxgl"
Jul 09 20:11:59.311871 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:11:59.311847    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlt4j\" (UniqueName: \"kubernetes.io/projected/4bd539c1-f6e7-4b25-a378-29b483b20851-kube-api-access-nlt4j\") pod \"kube-auth-proxy-8d5dd4c74-cgxgl\" (UID: \"4bd539c1-f6e7-4b25-a378-29b483b20851\") " pod="openshift-ingress/kube-auth-proxy-8d5dd4c74-cgxgl"
Jul 09 20:11:59.433339 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:11:59.433222    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-swz8d"]
Jul 09 20:11:59.435598 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:11:59.435574    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-swz8d"
Jul 09 20:11:59.438255 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:11:59.438237    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-8d5dd4c74-cgxgl"
Jul 09 20:11:59.439495 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:11:59.439478    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Jul 09 20:11:59.439785 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:11:59.439762    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-6p2vk\""
Jul 09 20:11:59.453017 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:11:59.452979    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-swz8d"]
Jul 09 20:11:59.490214 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:11:59.490155    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffpn5\" (UniqueName: \"kubernetes.io/projected/3344c0a8-3dcd-4c0b-879e-2491d0aea694-kube-api-access-ffpn5\") pod \"odh-model-controller-7998c67dc4-swz8d\" (UID: \"3344c0a8-3dcd-4c0b-879e-2491d0aea694\") " pod="opendatahub/odh-model-controller-7998c67dc4-swz8d"
Jul 09 20:11:59.490396 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:11:59.490241    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3344c0a8-3dcd-4c0b-879e-2491d0aea694-cert\") pod \"odh-model-controller-7998c67dc4-swz8d\" (UID: \"3344c0a8-3dcd-4c0b-879e-2491d0aea694\") " pod="opendatahub/odh-model-controller-7998c67dc4-swz8d"
Jul 09 20:11:59.571328 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:11:59.571288    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-8d5dd4c74-cgxgl"]
Jul 09 20:11:59.575069 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:11:59.575036    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bd539c1_f6e7_4b25_a378_29b483b20851.slice/crio-4ac19bdddec5b51a0100b519637d4f4e7d5ca4ab9a88c4f961180c81499135f9 WatchSource:0}: Error finding container 4ac19bdddec5b51a0100b519637d4f4e7d5ca4ab9a88c4f961180c81499135f9: Status 404 returned error can't find the container with id 4ac19bdddec5b51a0100b519637d4f4e7d5ca4ab9a88c4f961180c81499135f9
Jul 09 20:11:59.590715 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:11:59.590680    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffpn5\" (UniqueName: \"kubernetes.io/projected/3344c0a8-3dcd-4c0b-879e-2491d0aea694-kube-api-access-ffpn5\") pod \"odh-model-controller-7998c67dc4-swz8d\" (UID: \"3344c0a8-3dcd-4c0b-879e-2491d0aea694\") " pod="opendatahub/odh-model-controller-7998c67dc4-swz8d"
Jul 09 20:11:59.590829 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:11:59.590747    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3344c0a8-3dcd-4c0b-879e-2491d0aea694-cert\") pod \"odh-model-controller-7998c67dc4-swz8d\" (UID: \"3344c0a8-3dcd-4c0b-879e-2491d0aea694\") " pod="opendatahub/odh-model-controller-7998c67dc4-swz8d"
Jul 09 20:11:59.590949 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:11:59.590930    2562 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Jul 09 20:11:59.591028 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:11:59.591016    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3344c0a8-3dcd-4c0b-879e-2491d0aea694-cert podName:3344c0a8-3dcd-4c0b-879e-2491d0aea694 nodeName:}" failed. No retries permitted until 2026-07-09 20:12:00.090993798 +0000 UTC m=+510.508663591 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/3344c0a8-3dcd-4c0b-879e-2491d0aea694-cert") pod "odh-model-controller-7998c67dc4-swz8d" (UID: "3344c0a8-3dcd-4c0b-879e-2491d0aea694") : secret "odh-model-controller-webhook-cert" not found
Jul 09 20:11:59.599855 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:11:59.599832    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffpn5\" (UniqueName: \"kubernetes.io/projected/3344c0a8-3dcd-4c0b-879e-2491d0aea694-kube-api-access-ffpn5\") pod \"odh-model-controller-7998c67dc4-swz8d\" (UID: \"3344c0a8-3dcd-4c0b-879e-2491d0aea694\") " pod="opendatahub/odh-model-controller-7998c67dc4-swz8d"
Jul 09 20:12:00.012564 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:12:00.012525    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-8d5dd4c74-cgxgl" event={"ID":"4bd539c1-f6e7-4b25-a378-29b483b20851","Type":"ContainerStarted","Data":"4ac19bdddec5b51a0100b519637d4f4e7d5ca4ab9a88c4f961180c81499135f9"}
Jul 09 20:12:00.093896 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:12:00.093858    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3344c0a8-3dcd-4c0b-879e-2491d0aea694-cert\") pod \"odh-model-controller-7998c67dc4-swz8d\" (UID: \"3344c0a8-3dcd-4c0b-879e-2491d0aea694\") " pod="opendatahub/odh-model-controller-7998c67dc4-swz8d"
Jul 09 20:12:00.096247 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:12:00.096226    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3344c0a8-3dcd-4c0b-879e-2491d0aea694-cert\") pod \"odh-model-controller-7998c67dc4-swz8d\" (UID: \"3344c0a8-3dcd-4c0b-879e-2491d0aea694\") " pod="opendatahub/odh-model-controller-7998c67dc4-swz8d"
Jul 09 20:12:00.346739 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:12:00.346657    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-swz8d"
Jul 09 20:12:00.484940 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:12:00.484907    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-h7nkt"]
Jul 09 20:12:00.487904 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:12:00.487875    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-h7nkt"
Jul 09 20:12:00.491186 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:12:00.491161    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Jul 09 20:12:00.491186 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:12:00.491173    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Jul 09 20:12:00.491493 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:12:00.491475    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 09 20:12:00.491565 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:12:00.491533    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-dh55v\""
Jul 09 20:12:00.491677 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:12:00.491663    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Jul 09 20:12:00.496192 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:12:00.496161    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1783dc8e-fec8-4ec5-b5a4-3629c52ac8db-local-certs\") pod \"istiod-openshift-gateway-7574459656-h7nkt\" (UID: \"1783dc8e-fec8-4ec5-b5a4-3629c52ac8db\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-h7nkt"
Jul 09 20:12:00.496336 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:12:00.496203    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1783dc8e-fec8-4ec5-b5a4-3629c52ac8db-istio-token\") pod \"istiod-openshift-gateway-7574459656-h7nkt\" (UID: \"1783dc8e-fec8-4ec5-b5a4-3629c52ac8db\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-h7nkt"
Jul 09 20:12:00.496336 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:12:00.496234    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1783dc8e-fec8-4ec5-b5a4-3629c52ac8db-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-h7nkt\" (UID: \"1783dc8e-fec8-4ec5-b5a4-3629c52ac8db\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-h7nkt"
Jul 09 20:12:00.496445 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:12:00.496332    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lh7sv\" (UniqueName: \"kubernetes.io/projected/1783dc8e-fec8-4ec5-b5a4-3629c52ac8db-kube-api-access-lh7sv\") pod \"istiod-openshift-gateway-7574459656-h7nkt\" (UID: \"1783dc8e-fec8-4ec5-b5a4-3629c52ac8db\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-h7nkt"
Jul 09 20:12:00.496445 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:12:00.496400    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1783dc8e-fec8-4ec5-b5a4-3629c52ac8db-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-h7nkt\" (UID: \"1783dc8e-fec8-4ec5-b5a4-3629c52ac8db\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-h7nkt"
Jul 09 20:12:00.496445 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:12:00.496430    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1783dc8e-fec8-4ec5-b5a4-3629c52ac8db-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-h7nkt\" (UID: \"1783dc8e-fec8-4ec5-b5a4-3629c52ac8db\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-h7nkt"
Jul 09 20:12:00.496594 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:12:00.496523    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1783dc8e-fec8-4ec5-b5a4-3629c52ac8db-cacerts\") pod \"istiod-openshift-gateway-7574459656-h7nkt\" (UID: \"1783dc8e-fec8-4ec5-b5a4-3629c52ac8db\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-h7nkt"
Jul 09 20:12:00.501460 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:12:00.501434    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-h7nkt"]
Jul 09 20:12:00.515929 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:12:00.515903    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-swz8d"]
Jul 09 20:12:00.519634 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:12:00.519605    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3344c0a8_3dcd_4c0b_879e_2491d0aea694.slice/crio-6882c4b5f61eab6bf7da733a711fc4a25ef2c2976eafdd8154e76a550aa24542 WatchSource:0}: Error finding container 6882c4b5f61eab6bf7da733a711fc4a25ef2c2976eafdd8154e76a550aa24542: Status 404 returned error can't find the container with id 6882c4b5f61eab6bf7da733a711fc4a25ef2c2976eafdd8154e76a550aa24542
Jul 09 20:12:00.597369 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:12:00.597277    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1783dc8e-fec8-4ec5-b5a4-3629c52ac8db-cacerts\") pod \"istiod-openshift-gateway-7574459656-h7nkt\" (UID: \"1783dc8e-fec8-4ec5-b5a4-3629c52ac8db\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-h7nkt"
Jul 09 20:12:00.597369 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:12:00.597327    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1783dc8e-fec8-4ec5-b5a4-3629c52ac8db-local-certs\") pod \"istiod-openshift-gateway-7574459656-h7nkt\" (UID: \"1783dc8e-fec8-4ec5-b5a4-3629c52ac8db\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-h7nkt"
Jul 09 20:12:00.597613 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:12:00.597442    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1783dc8e-fec8-4ec5-b5a4-3629c52ac8db-istio-token\") pod \"istiod-openshift-gateway-7574459656-h7nkt\" (UID: \"1783dc8e-fec8-4ec5-b5a4-3629c52ac8db\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-h7nkt"
Jul 09 20:12:00.597613 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:12:00.597476    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1783dc8e-fec8-4ec5-b5a4-3629c52ac8db-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-h7nkt\" (UID: \"1783dc8e-fec8-4ec5-b5a4-3629c52ac8db\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-h7nkt"
Jul 09 20:12:00.597613 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:12:00.597499    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lh7sv\" (UniqueName: \"kubernetes.io/projected/1783dc8e-fec8-4ec5-b5a4-3629c52ac8db-kube-api-access-lh7sv\") pod \"istiod-openshift-gateway-7574459656-h7nkt\" (UID: \"1783dc8e-fec8-4ec5-b5a4-3629c52ac8db\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-h7nkt"
Jul 09 20:12:00.597613 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:12:00.597567    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1783dc8e-fec8-4ec5-b5a4-3629c52ac8db-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-h7nkt\" (UID: \"1783dc8e-fec8-4ec5-b5a4-3629c52ac8db\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-h7nkt"
Jul 09 20:12:00.597613 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:12:00.597589    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1783dc8e-fec8-4ec5-b5a4-3629c52ac8db-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-h7nkt\" (UID: \"1783dc8e-fec8-4ec5-b5a4-3629c52ac8db\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-h7nkt"
Jul 09 20:12:00.598374 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:12:00.598345    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1783dc8e-fec8-4ec5-b5a4-3629c52ac8db-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-h7nkt\" (UID: \"1783dc8e-fec8-4ec5-b5a4-3629c52ac8db\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-h7nkt"
Jul 09 20:12:00.599786 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:12:00.599760    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1783dc8e-fec8-4ec5-b5a4-3629c52ac8db-local-certs\") pod \"istiod-openshift-gateway-7574459656-h7nkt\" (UID: \"1783dc8e-fec8-4ec5-b5a4-3629c52ac8db\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-h7nkt"
Jul 09 20:12:00.599904 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:12:00.599783    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1783dc8e-fec8-4ec5-b5a4-3629c52ac8db-cacerts\") pod \"istiod-openshift-gateway-7574459656-h7nkt\" (UID: \"1783dc8e-fec8-4ec5-b5a4-3629c52ac8db\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-h7nkt"
Jul 09 20:12:00.600144 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:12:00.600126    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1783dc8e-fec8-4ec5-b5a4-3629c52ac8db-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-h7nkt\" (UID: \"1783dc8e-fec8-4ec5-b5a4-3629c52ac8db\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-h7nkt"
Jul 09 20:12:00.600250 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:12:00.600231    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1783dc8e-fec8-4ec5-b5a4-3629c52ac8db-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-h7nkt\" (UID: \"1783dc8e-fec8-4ec5-b5a4-3629c52ac8db\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-h7nkt"
Jul 09 20:12:00.606410 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:12:00.606389    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1783dc8e-fec8-4ec5-b5a4-3629c52ac8db-istio-token\") pod \"istiod-openshift-gateway-7574459656-h7nkt\" (UID: \"1783dc8e-fec8-4ec5-b5a4-3629c52ac8db\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-h7nkt"
Jul 09 20:12:00.606528 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:12:00.606512    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lh7sv\" (UniqueName: \"kubernetes.io/projected/1783dc8e-fec8-4ec5-b5a4-3629c52ac8db-kube-api-access-lh7sv\") pod \"istiod-openshift-gateway-7574459656-h7nkt\" (UID: \"1783dc8e-fec8-4ec5-b5a4-3629c52ac8db\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-h7nkt"
Jul 09 20:12:00.803239 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:12:00.803205    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-h7nkt"
Jul 09 20:12:00.955368 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:12:00.955286    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-h7nkt"]
Jul 09 20:12:00.957955 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:12:00.957932    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1783dc8e_fec8_4ec5_b5a4_3629c52ac8db.slice/crio-18a83d3026d88fdf8caa17c0ac93018b59aba49a63c640cc0a72af5c4740fdf3 WatchSource:0}: Error finding container 18a83d3026d88fdf8caa17c0ac93018b59aba49a63c640cc0a72af5c4740fdf3: Status 404 returned error can't find the container with id 18a83d3026d88fdf8caa17c0ac93018b59aba49a63c640cc0a72af5c4740fdf3
Jul 09 20:12:01.018028 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:12:01.017989    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-h7nkt" event={"ID":"1783dc8e-fec8-4ec5-b5a4-3629c52ac8db","Type":"ContainerStarted","Data":"18a83d3026d88fdf8caa17c0ac93018b59aba49a63c640cc0a72af5c4740fdf3"}
Jul 09 20:12:01.019401 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:12:01.019359    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-swz8d" event={"ID":"3344c0a8-3dcd-4c0b-879e-2491d0aea694","Type":"ContainerStarted","Data":"6882c4b5f61eab6bf7da733a711fc4a25ef2c2976eafdd8154e76a550aa24542"}
Jul 09 20:12:05.097563 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:12:05.097521    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 09 20:12:05.097863 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:12:05.097593    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 09 20:12:06.043790 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:12:06.043689    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-swz8d" event={"ID":"3344c0a8-3dcd-4c0b-879e-2491d0aea694","Type":"ContainerStarted","Data":"73065b9e1fde95646ffdd84dcd405cbacfa6a120a939ce8153a74d627916677d"}
Jul 09 20:12:06.043790 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:12:06.043758    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-swz8d"
Jul 09 20:12:06.045339 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:12:06.045294    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-8d5dd4c74-cgxgl" event={"ID":"4bd539c1-f6e7-4b25-a378-29b483b20851","Type":"ContainerStarted","Data":"2e8e80f984b29d7fa12b3a1fb0b3b86eb79b9f149923ef3c3f1643f58f6a7b4f"}
Jul 09 20:12:06.046991 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:12:06.046967    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-h7nkt" event={"ID":"1783dc8e-fec8-4ec5-b5a4-3629c52ac8db","Type":"ContainerStarted","Data":"b04e98ad80e732ca6bf7478afba351aa6d3443949bbcd83c463985cc8536ad4c"}
Jul 09 20:12:06.047170 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:12:06.047155    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-h7nkt"
Jul 09 20:12:06.048704 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:12:06.048675    2562 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-h7nkt container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 09 20:12:06.048798 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:12:06.048728    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-h7nkt" podUID="1783dc8e-fec8-4ec5-b5a4-3629c52ac8db" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 20:12:06.119806 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:12:06.119748    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-h7nkt" podStartSLOduration=1.9824994249999999 podStartE2EDuration="6.119733204s" totalImagesPullingTime="4.137233779s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 20:12:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 20:12:06.119343456 +0000 UTC m=+516.537013266" watchObservedRunningTime="2026-07-09 20:12:06.119733204 +0000 UTC m=+516.537403013"
Jul 09 20:12:06.120926 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:12:06.120893    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-swz8d" podStartSLOduration=2.538422724 podStartE2EDuration="7.120880554s" totalImagesPullingTime="4.58245783s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 20:11:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 20:12:06.085227643 +0000 UTC m=+516.502897452" watchObservedRunningTime="2026-07-09 20:12:06.120880554 +0000 UTC m=+516.538550363"
Jul 09 20:12:06.140002 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:12:06.139941    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-8d5dd4c74-cgxgl" podStartSLOduration=1.6259611189999998 podStartE2EDuration="7.139926664s" totalImagesPullingTime="5.513965545s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 20:11:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 20:12:06.138619473 +0000 UTC m=+516.556289282" watchObservedRunningTime="2026-07-09 20:12:06.139926664 +0000 UTC m=+516.557596473"
Jul 09 20:12:06.802049 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:12:06.802014    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-cqlmf"]
Jul 09 20:12:06.805456 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:12:06.805433    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-cqlmf"
Jul 09 20:12:06.824642 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:12:06.824612    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-cqlmf"]
Jul 09 20:12:06.841998 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:12:06.841965    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b1381926-955e-4bd3-a477-bcfafa860d42-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-cqlmf\" (UID: \"b1381926-955e-4bd3-a477-bcfafa860d42\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-cqlmf"
Jul 09 20:12:06.841998 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:12:06.842001    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b1381926-955e-4bd3-a477-bcfafa860d42-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-cqlmf\" (UID: \"b1381926-955e-4bd3-a477-bcfafa860d42\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-cqlmf"
Jul 09 20:12:06.842180 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:12:06.842021    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b1381926-955e-4bd3-a477-bcfafa860d42-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-cqlmf\" (UID: \"b1381926-955e-4bd3-a477-bcfafa860d42\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-cqlmf"
Jul 09 20:12:06.842180 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:12:06.842064    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b1381926-955e-4bd3-a477-bcfafa860d42-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-cqlmf\" (UID: \"b1381926-955e-4bd3-a477-bcfafa860d42\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-cqlmf"
Jul 09 20:12:06.842180 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:12:06.842097    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1381926-955e-4bd3-a477-bcfafa860d42-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-cqlmf\" (UID: \"b1381926-955e-4bd3-a477-bcfafa860d42\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-cqlmf"
Jul 09 20:12:06.842180 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:12:06.842155    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1381926-955e-4bd3-a477-bcfafa860d42-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-cqlmf\" (UID: \"b1381926-955e-4bd3-a477-bcfafa860d42\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-cqlmf"
Jul 09 20:12:06.842318 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:12:06.842202    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgggb\" (UniqueName: \"kubernetes.io/projected/b1381926-955e-4bd3-a477-bcfafa860d42-kube-api-access-sgggb\") pod \"istiod-openshift-gateway-94bb8fbfd-cqlmf\" (UID: \"b1381926-955e-4bd3-a477-bcfafa860d42\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-cqlmf"
Jul 09 20:12:06.942957 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:12:06.942922    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgggb\" (UniqueName: \"kubernetes.io/projected/b1381926-955e-4bd3-a477-bcfafa860d42-kube-api-access-sgggb\") pod \"istiod-openshift-gateway-94bb8fbfd-cqlmf\" (UID: \"b1381926-955e-4bd3-a477-bcfafa860d42\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-cqlmf"
Jul 09 20:12:06.943157 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:12:06.942969    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b1381926-955e-4bd3-a477-bcfafa860d42-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-cqlmf\" (UID: \"b1381926-955e-4bd3-a477-bcfafa860d42\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-cqlmf"
Jul 09 20:12:06.943157 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:12:06.942989    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b1381926-955e-4bd3-a477-bcfafa860d42-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-cqlmf\" (UID: \"b1381926-955e-4bd3-a477-bcfafa860d42\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-cqlmf"
Jul 09 20:12:06.943157 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:12:06.943008    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b1381926-955e-4bd3-a477-bcfafa860d42-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-cqlmf\" (UID: \"b1381926-955e-4bd3-a477-bcfafa860d42\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-cqlmf"
Jul 09 20:12:06.943157 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:12:06.943023    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b1381926-955e-4bd3-a477-bcfafa860d42-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-cqlmf\" (UID: \"b1381926-955e-4bd3-a477-bcfafa860d42\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-cqlmf"
Jul 09 20:12:06.943157 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:12:06.943131    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1381926-955e-4bd3-a477-bcfafa860d42-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-cqlmf\" (UID: \"b1381926-955e-4bd3-a477-bcfafa860d42\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-cqlmf"
Jul 09 20:12:06.943456 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:12:06.943251    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1381926-955e-4bd3-a477-bcfafa860d42-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-cqlmf\" (UID: \"b1381926-955e-4bd3-a477-bcfafa860d42\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-cqlmf"
Jul 09 20:12:06.943761 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:12:06.943734    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b1381926-955e-4bd3-a477-bcfafa860d42-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-cqlmf\" (UID: \"b1381926-955e-4bd3-a477-bcfafa860d42\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-cqlmf"
Jul 09 20:12:06.945544 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:12:06.945514    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1381926-955e-4bd3-a477-bcfafa860d42-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-cqlmf\" (UID: \"b1381926-955e-4bd3-a477-bcfafa860d42\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-cqlmf"
Jul 09 20:12:06.945671 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:12:06.945573    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b1381926-955e-4bd3-a477-bcfafa860d42-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-cqlmf\" (UID: \"b1381926-955e-4bd3-a477-bcfafa860d42\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-cqlmf"
Jul 09 20:12:06.945829 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:12:06.945809    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b1381926-955e-4bd3-a477-bcfafa860d42-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-cqlmf\" (UID: \"b1381926-955e-4bd3-a477-bcfafa860d42\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-cqlmf"
Jul 09 20:12:06.945889 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:12:06.945867    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b1381926-955e-4bd3-a477-bcfafa860d42-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-cqlmf\" (UID: \"b1381926-955e-4bd3-a477-bcfafa860d42\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-cqlmf"
Jul 09 20:12:06.951345 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:12:06.951325    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1381926-955e-4bd3-a477-bcfafa860d42-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-cqlmf\" (UID: \"b1381926-955e-4bd3-a477-bcfafa860d42\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-cqlmf"
Jul 09 20:12:06.951345 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:12:06.951336    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgggb\" (UniqueName: \"kubernetes.io/projected/b1381926-955e-4bd3-a477-bcfafa860d42-kube-api-access-sgggb\") pod \"istiod-openshift-gateway-94bb8fbfd-cqlmf\" (UID: \"b1381926-955e-4bd3-a477-bcfafa860d42\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-cqlmf"
Jul 09 20:12:07.052005 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:12:07.051981    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-h7nkt"
Jul 09 20:12:07.115073 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:12:07.114977    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-cqlmf"
Jul 09 20:12:07.251435 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:12:07.251405    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-cqlmf"]
Jul 09 20:12:07.254360 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:12:07.254330    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1381926_955e_4bd3_a477_bcfafa860d42.slice/crio-913c179286fcd0c775a4832629196b269dfc4ec90b493234c83d2ab85c303419 WatchSource:0}: Error finding container 913c179286fcd0c775a4832629196b269dfc4ec90b493234c83d2ab85c303419: Status 404 returned error can't find the container with id 913c179286fcd0c775a4832629196b269dfc4ec90b493234c83d2ab85c303419
Jul 09 20:12:07.257964 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:12:07.257935    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 09 20:12:07.258066 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:12:07.258003    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 09 20:12:08.057138 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:12:08.057102    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-cqlmf" event={"ID":"b1381926-955e-4bd3-a477-bcfafa860d42","Type":"ContainerStarted","Data":"312e6f1351379120e4cc8d12404b54c42097b08ca374bfefcad0506d079b5349"}
Jul 09 20:12:08.057343 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:12:08.057157    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-cqlmf" event={"ID":"b1381926-955e-4bd3-a477-bcfafa860d42","Type":"ContainerStarted","Data":"913c179286fcd0c775a4832629196b269dfc4ec90b493234c83d2ab85c303419"}
Jul 09 20:12:08.133084 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:12:08.133008    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-cqlmf" podStartSLOduration=2.132989763 podStartE2EDuration="2.132989763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 20:12:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 20:12:08.132981897 +0000 UTC m=+518.550651708" watchObservedRunningTime="2026-07-09 20:12:08.132989763 +0000 UTC m=+518.550659572"
Jul 09 20:12:09.061374 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:12:09.061336    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-cqlmf"
Jul 09 20:12:09.063076 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:12:09.063053    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-cqlmf"
Jul 09 20:12:09.192698 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:12:09.192668    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-h7nkt"]
Jul 09 20:12:09.192854 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:12:09.192833    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-h7nkt" podUID="1783dc8e-fec8-4ec5-b5a4-3629c52ac8db" containerName="discovery" containerID="cri-o://b04e98ad80e732ca6bf7478afba351aa6d3443949bbcd83c463985cc8536ad4c" gracePeriod=30
Jul 09 20:12:09.448929 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:12:09.448897    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-h7nkt"
Jul 09 20:12:09.464362 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:12:09.464340    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1783dc8e-fec8-4ec5-b5a4-3629c52ac8db-istio-kubeconfig\") pod \"1783dc8e-fec8-4ec5-b5a4-3629c52ac8db\" (UID: \"1783dc8e-fec8-4ec5-b5a4-3629c52ac8db\") "
Jul 09 20:12:09.464528 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:12:09.464375    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1783dc8e-fec8-4ec5-b5a4-3629c52ac8db-istio-csr-ca-configmap\") pod \"1783dc8e-fec8-4ec5-b5a4-3629c52ac8db\" (UID: \"1783dc8e-fec8-4ec5-b5a4-3629c52ac8db\") "
Jul 09 20:12:09.464528 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:12:09.464395    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lh7sv\" (UniqueName: \"kubernetes.io/projected/1783dc8e-fec8-4ec5-b5a4-3629c52ac8db-kube-api-access-lh7sv\") pod \"1783dc8e-fec8-4ec5-b5a4-3629c52ac8db\" (UID: \"1783dc8e-fec8-4ec5-b5a4-3629c52ac8db\") "
Jul 09 20:12:09.464528 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:12:09.464427    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1783dc8e-fec8-4ec5-b5a4-3629c52ac8db-istio-token\") pod \"1783dc8e-fec8-4ec5-b5a4-3629c52ac8db\" (UID: \"1783dc8e-fec8-4ec5-b5a4-3629c52ac8db\") "
Jul 09 20:12:09.464528 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:12:09.464469    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1783dc8e-fec8-4ec5-b5a4-3629c52ac8db-istio-csr-dns-cert\") pod \"1783dc8e-fec8-4ec5-b5a4-3629c52ac8db\" (UID: \"1783dc8e-fec8-4ec5-b5a4-3629c52ac8db\") "
Jul 09 20:12:09.464528 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:12:09.464496    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1783dc8e-fec8-4ec5-b5a4-3629c52ac8db-cacerts\") pod \"1783dc8e-fec8-4ec5-b5a4-3629c52ac8db\" (UID: \"1783dc8e-fec8-4ec5-b5a4-3629c52ac8db\") "
Jul 09 20:12:09.464785 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:12:09.464566    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1783dc8e-fec8-4ec5-b5a4-3629c52ac8db-local-certs\") pod \"1783dc8e-fec8-4ec5-b5a4-3629c52ac8db\" (UID: \"1783dc8e-fec8-4ec5-b5a4-3629c52ac8db\") "
Jul 09 20:12:09.465188 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:12:09.465156    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1783dc8e-fec8-4ec5-b5a4-3629c52ac8db-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "1783dc8e-fec8-4ec5-b5a4-3629c52ac8db" (UID: "1783dc8e-fec8-4ec5-b5a4-3629c52ac8db"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 20:12:09.467429 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:12:09.467364    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1783dc8e-fec8-4ec5-b5a4-3629c52ac8db-kube-api-access-lh7sv" (OuterVolumeSpecName: "kube-api-access-lh7sv") pod "1783dc8e-fec8-4ec5-b5a4-3629c52ac8db" (UID: "1783dc8e-fec8-4ec5-b5a4-3629c52ac8db"). InnerVolumeSpecName "kube-api-access-lh7sv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 20:12:09.467738 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:12:09.467700    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1783dc8e-fec8-4ec5-b5a4-3629c52ac8db-local-certs" (OuterVolumeSpecName: "local-certs") pod "1783dc8e-fec8-4ec5-b5a4-3629c52ac8db" (UID: "1783dc8e-fec8-4ec5-b5a4-3629c52ac8db"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 20:12:09.467829 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:12:09.467795    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1783dc8e-fec8-4ec5-b5a4-3629c52ac8db-istio-token" (OuterVolumeSpecName: "istio-token") pod "1783dc8e-fec8-4ec5-b5a4-3629c52ac8db" (UID: "1783dc8e-fec8-4ec5-b5a4-3629c52ac8db"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 20:12:09.467829 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:12:09.467797    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1783dc8e-fec8-4ec5-b5a4-3629c52ac8db-cacerts" (OuterVolumeSpecName: "cacerts") pod "1783dc8e-fec8-4ec5-b5a4-3629c52ac8db" (UID: "1783dc8e-fec8-4ec5-b5a4-3629c52ac8db"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 20:12:09.467945 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:12:09.467840    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1783dc8e-fec8-4ec5-b5a4-3629c52ac8db-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "1783dc8e-fec8-4ec5-b5a4-3629c52ac8db" (UID: "1783dc8e-fec8-4ec5-b5a4-3629c52ac8db"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 20:12:09.468104 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:12:09.468089    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1783dc8e-fec8-4ec5-b5a4-3629c52ac8db-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "1783dc8e-fec8-4ec5-b5a4-3629c52ac8db" (UID: "1783dc8e-fec8-4ec5-b5a4-3629c52ac8db"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 20:12:09.566153 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:12:09.566119    2562 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1783dc8e-fec8-4ec5-b5a4-3629c52ac8db-local-certs\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:12:09.566153 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:12:09.566151    2562 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1783dc8e-fec8-4ec5-b5a4-3629c52ac8db-istio-kubeconfig\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:12:09.566153 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:12:09.566163    2562 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1783dc8e-fec8-4ec5-b5a4-3629c52ac8db-istio-csr-ca-configmap\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:12:09.566443 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:12:09.566175    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lh7sv\" (UniqueName: \"kubernetes.io/projected/1783dc8e-fec8-4ec5-b5a4-3629c52ac8db-kube-api-access-lh7sv\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:12:09.566443 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:12:09.566188    2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1783dc8e-fec8-4ec5-b5a4-3629c52ac8db-istio-token\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:12:09.566443 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:12:09.566199    2562 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1783dc8e-fec8-4ec5-b5a4-3629c52ac8db-istio-csr-dns-cert\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:12:09.566443 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:12:09.566211    2562 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1783dc8e-fec8-4ec5-b5a4-3629c52ac8db-cacerts\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:12:10.066092 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:12:10.066055    2562 generic.go:356] "Generic (PLEG): container finished" podID="1783dc8e-fec8-4ec5-b5a4-3629c52ac8db" containerID="b04e98ad80e732ca6bf7478afba351aa6d3443949bbcd83c463985cc8536ad4c" exitCode=0
Jul 09 20:12:10.066503 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:12:10.066120    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-h7nkt"
Jul 09 20:12:10.066503 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:12:10.066143    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-h7nkt" event={"ID":"1783dc8e-fec8-4ec5-b5a4-3629c52ac8db","Type":"ContainerDied","Data":"b04e98ad80e732ca6bf7478afba351aa6d3443949bbcd83c463985cc8536ad4c"}
Jul 09 20:12:10.066503 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:12:10.066215    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-h7nkt" event={"ID":"1783dc8e-fec8-4ec5-b5a4-3629c52ac8db","Type":"ContainerDied","Data":"18a83d3026d88fdf8caa17c0ac93018b59aba49a63c640cc0a72af5c4740fdf3"}
Jul 09 20:12:10.066503 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:12:10.066238    2562 scope.go:117] "RemoveContainer" containerID="b04e98ad80e732ca6bf7478afba351aa6d3443949bbcd83c463985cc8536ad4c"
Jul 09 20:12:10.074930 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:12:10.074914    2562 scope.go:117] "RemoveContainer" containerID="b04e98ad80e732ca6bf7478afba351aa6d3443949bbcd83c463985cc8536ad4c"
Jul 09 20:12:10.075170 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:12:10.075150    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b04e98ad80e732ca6bf7478afba351aa6d3443949bbcd83c463985cc8536ad4c\": container with ID starting with b04e98ad80e732ca6bf7478afba351aa6d3443949bbcd83c463985cc8536ad4c not found: ID does not exist" containerID="b04e98ad80e732ca6bf7478afba351aa6d3443949bbcd83c463985cc8536ad4c"
Jul 09 20:12:10.075207 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:12:10.075178    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b04e98ad80e732ca6bf7478afba351aa6d3443949bbcd83c463985cc8536ad4c"} err="failed to get container status \"b04e98ad80e732ca6bf7478afba351aa6d3443949bbcd83c463985cc8536ad4c\": rpc error: code = NotFound desc = could not find container \"b04e98ad80e732ca6bf7478afba351aa6d3443949bbcd83c463985cc8536ad4c\": container with ID starting with b04e98ad80e732ca6bf7478afba351aa6d3443949bbcd83c463985cc8536ad4c not found: ID does not exist"
Jul 09 20:12:10.088554 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:12:10.088531    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-h7nkt"]
Jul 09 20:12:10.098090 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:12:10.098067    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-h7nkt"]
Jul 09 20:12:10.241397 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:12:10.241362    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1783dc8e-fec8-4ec5-b5a4-3629c52ac8db" path="/var/lib/kubelet/pods/1783dc8e-fec8-4ec5-b5a4-3629c52ac8db/volumes"
Jul 09 20:12:17.053939 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:12:17.053903    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-swz8d"
Jul 09 20:13:00.283083 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:00.283038    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-v844x"]
Jul 09 20:13:00.283530 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:00.283390    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1783dc8e-fec8-4ec5-b5a4-3629c52ac8db" containerName="discovery"
Jul 09 20:13:00.283530 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:00.283403    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="1783dc8e-fec8-4ec5-b5a4-3629c52ac8db" containerName="discovery"
Jul 09 20:13:00.283530 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:00.283460    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="1783dc8e-fec8-4ec5-b5a4-3629c52ac8db" containerName="discovery"
Jul 09 20:13:00.285270 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:00.285251    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-v844x"
Jul 09 20:13:00.288416 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:00.288396    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 09 20:13:00.288531 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:00.288455    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-dnsls\""
Jul 09 20:13:00.288531 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:00.288495    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 09 20:13:00.296501 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:00.296480    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-v844x"]
Jul 09 20:13:00.377756 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:00.377717    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvtfn\" (UniqueName: \"kubernetes.io/projected/f5a46fce-57cf-4ee2-a8a3-90daede618e1-kube-api-access-qvtfn\") pod \"kuadrant-operator-catalog-v844x\" (UID: \"f5a46fce-57cf-4ee2-a8a3-90daede618e1\") " pod="kuadrant-system/kuadrant-operator-catalog-v844x"
Jul 09 20:13:00.479143 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:00.479099    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvtfn\" (UniqueName: \"kubernetes.io/projected/f5a46fce-57cf-4ee2-a8a3-90daede618e1-kube-api-access-qvtfn\") pod \"kuadrant-operator-catalog-v844x\" (UID: \"f5a46fce-57cf-4ee2-a8a3-90daede618e1\") " pod="kuadrant-system/kuadrant-operator-catalog-v844x"
Jul 09 20:13:00.488419 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:00.488388    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvtfn\" (UniqueName: \"kubernetes.io/projected/f5a46fce-57cf-4ee2-a8a3-90daede618e1-kube-api-access-qvtfn\") pod \"kuadrant-operator-catalog-v844x\" (UID: \"f5a46fce-57cf-4ee2-a8a3-90daede618e1\") " pod="kuadrant-system/kuadrant-operator-catalog-v844x"
Jul 09 20:13:00.594854 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:00.594753    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-v844x"
Jul 09 20:13:00.654968 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:00.654933    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-v844x"]
Jul 09 20:13:00.728893 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:00.728855    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-v844x"]
Jul 09 20:13:00.730062 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:13:00.730034    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5a46fce_57cf_4ee2_a8a3_90daede618e1.slice/crio-ecc64641739769ddcf7066e4615cc70e7bff46ad5db7f6b24c371c980c362e6f WatchSource:0}: Error finding container ecc64641739769ddcf7066e4615cc70e7bff46ad5db7f6b24c371c980c362e6f: Status 404 returned error can't find the container with id ecc64641739769ddcf7066e4615cc70e7bff46ad5db7f6b24c371c980c362e6f
Jul 09 20:13:00.863020 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:00.862937    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-dlkf9"]
Jul 09 20:13:00.865727 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:00.865708    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-dlkf9"
Jul 09 20:13:00.873977 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:00.873951    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-dlkf9"]
Jul 09 20:13:00.983414 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:00.983384    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwdxj\" (UniqueName: \"kubernetes.io/projected/203f88dd-f8b7-4843-a4b2-246a3583cb83-kube-api-access-qwdxj\") pod \"kuadrant-operator-catalog-dlkf9\" (UID: \"203f88dd-f8b7-4843-a4b2-246a3583cb83\") " pod="kuadrant-system/kuadrant-operator-catalog-dlkf9"
Jul 09 20:13:01.091504 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:01.091469    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwdxj\" (UniqueName: \"kubernetes.io/projected/203f88dd-f8b7-4843-a4b2-246a3583cb83-kube-api-access-qwdxj\") pod \"kuadrant-operator-catalog-dlkf9\" (UID: \"203f88dd-f8b7-4843-a4b2-246a3583cb83\") " pod="kuadrant-system/kuadrant-operator-catalog-dlkf9"
Jul 09 20:13:01.100122 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:01.100094    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwdxj\" (UniqueName: \"kubernetes.io/projected/203f88dd-f8b7-4843-a4b2-246a3583cb83-kube-api-access-qwdxj\") pod \"kuadrant-operator-catalog-dlkf9\" (UID: \"203f88dd-f8b7-4843-a4b2-246a3583cb83\") " pod="kuadrant-system/kuadrant-operator-catalog-dlkf9"
Jul 09 20:13:01.175901 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:01.175813    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-dlkf9"
Jul 09 20:13:01.272383 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:01.272342    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-v844x" event={"ID":"f5a46fce-57cf-4ee2-a8a3-90daede618e1","Type":"ContainerStarted","Data":"ecc64641739769ddcf7066e4615cc70e7bff46ad5db7f6b24c371c980c362e6f"}
Jul 09 20:13:01.305035 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:01.305003    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-dlkf9"]
Jul 09 20:13:01.309018 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:13:01.308990    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod203f88dd_f8b7_4843_a4b2_246a3583cb83.slice/crio-069bcdb6870f55bd7c72fff12dd528a4397f3749194cf1aeb8272ce92ed01cbb WatchSource:0}: Error finding container 069bcdb6870f55bd7c72fff12dd528a4397f3749194cf1aeb8272ce92ed01cbb: Status 404 returned error can't find the container with id 069bcdb6870f55bd7c72fff12dd528a4397f3749194cf1aeb8272ce92ed01cbb
Jul 09 20:13:02.278248 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:02.278203    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-dlkf9" event={"ID":"203f88dd-f8b7-4843-a4b2-246a3583cb83","Type":"ContainerStarted","Data":"069bcdb6870f55bd7c72fff12dd528a4397f3749194cf1aeb8272ce92ed01cbb"}
Jul 09 20:13:03.284565 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:03.284502    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-v844x" event={"ID":"f5a46fce-57cf-4ee2-a8a3-90daede618e1","Type":"ContainerStarted","Data":"31fcb8b55cc1c1e6791a9f287ec8318a969b7f4c4231b0fb31352fb16332a0f9"}
Jul 09 20:13:03.284565 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:03.284540    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-v844x" podUID="f5a46fce-57cf-4ee2-a8a3-90daede618e1" containerName="registry-server" containerID="cri-o://31fcb8b55cc1c1e6791a9f287ec8318a969b7f4c4231b0fb31352fb16332a0f9" gracePeriod=2
Jul 09 20:13:03.286039 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:03.286014    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-dlkf9" event={"ID":"203f88dd-f8b7-4843-a4b2-246a3583cb83","Type":"ContainerStarted","Data":"b724b074a12ee5d8a1e9ed1bc9b9295212190f5f958c05dcc18c0eaffb9082e7"}
Jul 09 20:13:03.302717 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:03.302640    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-v844x" podStartSLOduration=1.145267875 podStartE2EDuration="3.302625287s" totalImagesPullingTime="2.157357412s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 20:13:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 20:13:03.301225368 +0000 UTC m=+573.718895177" watchObservedRunningTime="2026-07-09 20:13:03.302625287 +0000 UTC m=+573.720295096"
Jul 09 20:13:03.316773 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:03.316728    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-dlkf9" podStartSLOduration=1.7362214059999999 podStartE2EDuration="3.316714139s" totalImagesPullingTime="1.580492733s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 20:13:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 20:13:03.315740781 +0000 UTC m=+573.733410591" watchObservedRunningTime="2026-07-09 20:13:03.316714139 +0000 UTC m=+573.734383956"
Jul 09 20:13:03.526766 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:03.526743    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-v844x"
Jul 09 20:13:03.712525 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:03.712496    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qvtfn\" (UniqueName: \"kubernetes.io/projected/f5a46fce-57cf-4ee2-a8a3-90daede618e1-kube-api-access-qvtfn\") pod \"f5a46fce-57cf-4ee2-a8a3-90daede618e1\" (UID: \"f5a46fce-57cf-4ee2-a8a3-90daede618e1\") "
Jul 09 20:13:03.714644 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:03.714612    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5a46fce-57cf-4ee2-a8a3-90daede618e1-kube-api-access-qvtfn" (OuterVolumeSpecName: "kube-api-access-qvtfn") pod "f5a46fce-57cf-4ee2-a8a3-90daede618e1" (UID: "f5a46fce-57cf-4ee2-a8a3-90daede618e1"). InnerVolumeSpecName "kube-api-access-qvtfn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 20:13:03.813980 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:03.813949    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvtfn\" (UniqueName: \"kubernetes.io/projected/f5a46fce-57cf-4ee2-a8a3-90daede618e1-kube-api-access-qvtfn\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:13:04.294995 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:04.294957    2562 generic.go:356] "Generic (PLEG): container finished" podID="f5a46fce-57cf-4ee2-a8a3-90daede618e1" containerID="31fcb8b55cc1c1e6791a9f287ec8318a969b7f4c4231b0fb31352fb16332a0f9" exitCode=0
Jul 09 20:13:04.295431 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:04.295032    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-v844x"
Jul 09 20:13:04.295431 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:04.295036    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-v844x" event={"ID":"f5a46fce-57cf-4ee2-a8a3-90daede618e1","Type":"ContainerDied","Data":"31fcb8b55cc1c1e6791a9f287ec8318a969b7f4c4231b0fb31352fb16332a0f9"}
Jul 09 20:13:04.295431 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:04.295110    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-v844x" event={"ID":"f5a46fce-57cf-4ee2-a8a3-90daede618e1","Type":"ContainerDied","Data":"ecc64641739769ddcf7066e4615cc70e7bff46ad5db7f6b24c371c980c362e6f"}
Jul 09 20:13:04.295431 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:04.295138    2562 scope.go:117] "RemoveContainer" containerID="31fcb8b55cc1c1e6791a9f287ec8318a969b7f4c4231b0fb31352fb16332a0f9"
Jul 09 20:13:04.305392 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:04.305368    2562 scope.go:117] "RemoveContainer" containerID="31fcb8b55cc1c1e6791a9f287ec8318a969b7f4c4231b0fb31352fb16332a0f9"
Jul 09 20:13:04.305933 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:13:04.305914    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31fcb8b55cc1c1e6791a9f287ec8318a969b7f4c4231b0fb31352fb16332a0f9\": container with ID starting with 31fcb8b55cc1c1e6791a9f287ec8318a969b7f4c4231b0fb31352fb16332a0f9 not found: ID does not exist" containerID="31fcb8b55cc1c1e6791a9f287ec8318a969b7f4c4231b0fb31352fb16332a0f9"
Jul 09 20:13:04.305987 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:04.305942    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31fcb8b55cc1c1e6791a9f287ec8318a969b7f4c4231b0fb31352fb16332a0f9"} err="failed to get container status \"31fcb8b55cc1c1e6791a9f287ec8318a969b7f4c4231b0fb31352fb16332a0f9\": rpc error: code = NotFound desc = could not find container \"31fcb8b55cc1c1e6791a9f287ec8318a969b7f4c4231b0fb31352fb16332a0f9\": container with ID starting with 31fcb8b55cc1c1e6791a9f287ec8318a969b7f4c4231b0fb31352fb16332a0f9 not found: ID does not exist"
Jul 09 20:13:04.316477 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:04.316455    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-v844x"]
Jul 09 20:13:04.327147 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:04.327128    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-v844x"]
Jul 09 20:13:06.241078 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:06.241038    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5a46fce-57cf-4ee2-a8a3-90daede618e1" path="/var/lib/kubelet/pods/f5a46fce-57cf-4ee2-a8a3-90daede618e1/volumes"
Jul 09 20:13:11.175987 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:11.175941    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-dlkf9"
Jul 09 20:13:11.176439 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:11.176016    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-dlkf9"
Jul 09 20:13:11.198694 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:11.198667    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-dlkf9"
Jul 09 20:13:11.346496 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:11.346469    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-dlkf9"
Jul 09 20:13:30.062526 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:30.062497    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-cqs6d_578878d0-e6b3-4725-a698-796cfc169fa6/console-operator/1.log"
Jul 09 20:13:30.062944 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:30.062663    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-cqs6d_578878d0-e6b3-4725-a698-796cfc169fa6/console-operator/1.log"
Jul 09 20:13:30.068545 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:30.068522    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hds4g_4b4468fd-1708-487b-8d69-bb7d817e25de/ovn-acl-logging/0.log"
Jul 09 20:13:30.068700 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:30.068525    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hds4g_4b4468fd-1708-487b-8d69-bb7d817e25de/ovn-acl-logging/0.log"
Jul 09 20:13:33.801196 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:33.801158    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c4dr6"]
Jul 09 20:13:33.801828 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:33.801607    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5a46fce-57cf-4ee2-a8a3-90daede618e1" containerName="registry-server"
Jul 09 20:13:33.801828 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:33.801625    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5a46fce-57cf-4ee2-a8a3-90daede618e1" containerName="registry-server"
Jul 09 20:13:33.801828 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:33.801741    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5a46fce-57cf-4ee2-a8a3-90daede618e1" containerName="registry-server"
Jul 09 20:13:33.806071 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:33.806049    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c4dr6"
Jul 09 20:13:33.809917 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:33.809896    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-btptb\""
Jul 09 20:13:33.831593 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:33.831566    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c4dr6"]
Jul 09 20:13:33.852315 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:33.852284    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8s7g\" (UniqueName: \"kubernetes.io/projected/ba15e096-32a9-4ddc-89fd-d2d0e5faf3c2-kube-api-access-f8s7g\") pod \"kuadrant-operator-controller-manager-8684c9ff49-c4dr6\" (UID: \"ba15e096-32a9-4ddc-89fd-d2d0e5faf3c2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c4dr6"
Jul 09 20:13:33.852426 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:33.852367    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ba15e096-32a9-4ddc-89fd-d2d0e5faf3c2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-c4dr6\" (UID: \"ba15e096-32a9-4ddc-89fd-d2d0e5faf3c2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c4dr6"
Jul 09 20:13:33.953769 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:33.953736    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8s7g\" (UniqueName: \"kubernetes.io/projected/ba15e096-32a9-4ddc-89fd-d2d0e5faf3c2-kube-api-access-f8s7g\") pod \"kuadrant-operator-controller-manager-8684c9ff49-c4dr6\" (UID: \"ba15e096-32a9-4ddc-89fd-d2d0e5faf3c2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c4dr6"
Jul 09 20:13:33.953943 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:33.953796    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ba15e096-32a9-4ddc-89fd-d2d0e5faf3c2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-c4dr6\" (UID: \"ba15e096-32a9-4ddc-89fd-d2d0e5faf3c2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c4dr6"
Jul 09 20:13:33.954151 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:33.954136    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ba15e096-32a9-4ddc-89fd-d2d0e5faf3c2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-c4dr6\" (UID: \"ba15e096-32a9-4ddc-89fd-d2d0e5faf3c2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c4dr6"
Jul 09 20:13:33.966028 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:33.965997    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8s7g\" (UniqueName: \"kubernetes.io/projected/ba15e096-32a9-4ddc-89fd-d2d0e5faf3c2-kube-api-access-f8s7g\") pod \"kuadrant-operator-controller-manager-8684c9ff49-c4dr6\" (UID: \"ba15e096-32a9-4ddc-89fd-d2d0e5faf3c2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c4dr6"
Jul 09 20:13:34.115477 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:34.115396    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c4dr6"
Jul 09 20:13:34.249546 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:34.249482    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c4dr6"]
Jul 09 20:13:34.252397 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:13:34.252361    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba15e096_32a9_4ddc_89fd_d2d0e5faf3c2.slice/crio-8ff42533d983c9524647d2b9cd2a5d79cc2c9f32b7c3a3f846bc7d58cb561a23 WatchSource:0}: Error finding container 8ff42533d983c9524647d2b9cd2a5d79cc2c9f32b7c3a3f846bc7d58cb561a23: Status 404 returned error can't find the container with id 8ff42533d983c9524647d2b9cd2a5d79cc2c9f32b7c3a3f846bc7d58cb561a23
Jul 09 20:13:34.422590 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:34.422491    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c4dr6" event={"ID":"ba15e096-32a9-4ddc-89fd-d2d0e5faf3c2","Type":"ContainerStarted","Data":"8ff42533d983c9524647d2b9cd2a5d79cc2c9f32b7c3a3f846bc7d58cb561a23"}
Jul 09 20:13:35.607565 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:35.607528    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xs9kh"]
Jul 09 20:13:35.610720 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:35.610701    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xs9kh"
Jul 09 20:13:35.613660 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:35.613634    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-wkqg6\""
Jul 09 20:13:35.620173 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:35.620151    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xs9kh"]
Jul 09 20:13:35.668529 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:35.668493    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6629\" (UniqueName: \"kubernetes.io/projected/f7c8e3d7-c021-476b-9174-9896787e3a4d-kube-api-access-b6629\") pod \"limitador-operator-controller-manager-859d7b78bd-xs9kh\" (UID: \"f7c8e3d7-c021-476b-9174-9896787e3a4d\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xs9kh"
Jul 09 20:13:35.769943 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:35.769900    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6629\" (UniqueName: \"kubernetes.io/projected/f7c8e3d7-c021-476b-9174-9896787e3a4d-kube-api-access-b6629\") pod \"limitador-operator-controller-manager-859d7b78bd-xs9kh\" (UID: \"f7c8e3d7-c021-476b-9174-9896787e3a4d\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xs9kh"
Jul 09 20:13:35.788190 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:35.788155    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6629\" (UniqueName: \"kubernetes.io/projected/f7c8e3d7-c021-476b-9174-9896787e3a4d-kube-api-access-b6629\") pod \"limitador-operator-controller-manager-859d7b78bd-xs9kh\" (UID: \"f7c8e3d7-c021-476b-9174-9896787e3a4d\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xs9kh"
Jul 09 20:13:35.921824 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:35.921733    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xs9kh"
Jul 09 20:13:36.076408 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:36.076379    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xs9kh"]
Jul 09 20:13:36.079823 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:13:36.079797    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7c8e3d7_c021_476b_9174_9896787e3a4d.slice/crio-9bdaf6b6ce9df41ad4943b15d0d23a8a2c0e6e449e2529ad6bbedc0ebb7a3a09 WatchSource:0}: Error finding container 9bdaf6b6ce9df41ad4943b15d0d23a8a2c0e6e449e2529ad6bbedc0ebb7a3a09: Status 404 returned error can't find the container with id 9bdaf6b6ce9df41ad4943b15d0d23a8a2c0e6e449e2529ad6bbedc0ebb7a3a09
Jul 09 20:13:36.434817 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:36.434779    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xs9kh" event={"ID":"f7c8e3d7-c021-476b-9174-9896787e3a4d","Type":"ContainerStarted","Data":"9bdaf6b6ce9df41ad4943b15d0d23a8a2c0e6e449e2529ad6bbedc0ebb7a3a09"}
Jul 09 20:13:38.445685 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:38.445585    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xs9kh" event={"ID":"f7c8e3d7-c021-476b-9174-9896787e3a4d","Type":"ContainerStarted","Data":"7b141b65dff28bc0c35e9428d79fef0b8fe08d4d33d7926f99f241d1df301101"}
Jul 09 20:13:38.445685 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:38.445671    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xs9kh"
Jul 09 20:13:38.463930 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:38.463880    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xs9kh" podStartSLOduration=1.365208899 podStartE2EDuration="3.463863541s" totalImagesPullingTime="2.098654642s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 20:13:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 20:13:38.463458648 +0000 UTC m=+608.881128459" watchObservedRunningTime="2026-07-09 20:13:38.463863541 +0000 UTC m=+608.881533350"
Jul 09 20:13:41.461693 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:41.461650    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c4dr6" event={"ID":"ba15e096-32a9-4ddc-89fd-d2d0e5faf3c2","Type":"ContainerStarted","Data":"600894623fbfd7f22408f6ea0570cbe05b0867dc2f751276e6f0a17badb99845"}
Jul 09 20:13:41.462094 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:41.461770    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c4dr6"
Jul 09 20:13:41.495036 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:41.494980    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c4dr6" podStartSLOduration=1.903200883 podStartE2EDuration="8.494964029s" totalImagesPullingTime="6.591763146s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 20:13:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 20:13:41.491683568 +0000 UTC m=+611.909353377" watchObservedRunningTime="2026-07-09 20:13:41.494964029 +0000 UTC m=+611.912633837"
Jul 09 20:13:49.452748 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:49.452717    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xs9kh"
Jul 09 20:13:52.467997 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:52.467920    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c4dr6"
Jul 09 20:13:53.938643 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:53.938610    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-8dbcp"]
Jul 09 20:13:53.941781 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:53.941762    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-8dbcp"
Jul 09 20:13:53.962852 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:53.962829    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-8dbcp"]
Jul 09 20:13:54.135623 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:54.135577    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n84hm\" (UniqueName: \"kubernetes.io/projected/7a1a7dd1-e074-4057-9b17-47221a719a88-kube-api-access-n84hm\") pod \"kuadrant-operator-controller-manager-55467d9f7-8dbcp\" (UID: \"7a1a7dd1-e074-4057-9b17-47221a719a88\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-8dbcp"
Jul 09 20:13:54.135803 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:54.135721    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7a1a7dd1-e074-4057-9b17-47221a719a88-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-8dbcp\" (UID: \"7a1a7dd1-e074-4057-9b17-47221a719a88\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-8dbcp"
Jul 09 20:13:54.199574 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:54.199499    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-8dbcp"]
Jul 09 20:13:54.199755 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:13:54.199737    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[extensions-socket-volume kube-api-access-n84hm], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-8dbcp" podUID="7a1a7dd1-e074-4057-9b17-47221a719a88"
Jul 09 20:13:54.202419 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:54.202379    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-8dbcp"]
Jul 09 20:13:54.210607 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:54.210586    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c4dr6"]
Jul 09 20:13:54.210787 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:54.210764    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c4dr6" podUID="ba15e096-32a9-4ddc-89fd-d2d0e5faf3c2" containerName="manager" containerID="cri-o://600894623fbfd7f22408f6ea0570cbe05b0867dc2f751276e6f0a17badb99845" gracePeriod=2
Jul 09 20:13:54.224999 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:54.224976    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-9xwjz"]
Jul 09 20:13:54.228289 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:54.228269    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c4dr6"]
Jul 09 20:13:54.228425 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:54.228385    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-9xwjz"
Jul 09 20:13:54.236402 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:54.236339    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7a1a7dd1-e074-4057-9b17-47221a719a88-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-8dbcp\" (UID: \"7a1a7dd1-e074-4057-9b17-47221a719a88\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-8dbcp"
Jul 09 20:13:54.236520 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:54.236424    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n84hm\" (UniqueName: \"kubernetes.io/projected/7a1a7dd1-e074-4057-9b17-47221a719a88-kube-api-access-n84hm\") pod \"kuadrant-operator-controller-manager-55467d9f7-8dbcp\" (UID: \"7a1a7dd1-e074-4057-9b17-47221a719a88\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-8dbcp"
Jul 09 20:13:54.237139 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:54.237114    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7a1a7dd1-e074-4057-9b17-47221a719a88-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-8dbcp\" (UID: \"7a1a7dd1-e074-4057-9b17-47221a719a88\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-8dbcp"
Jul 09 20:13:54.237275 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:13:54.237239    2562 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"ba15e096-32a9-4ddc-89fd-d2d0e5faf3c2\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-8684c9ff49-c4dr6\": pods \"kuadrant-operator-controller-manager-8684c9ff49-c4dr6\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c4dr6"
Jul 09 20:13:54.243132 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:54.243110    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kt2mp"]
Jul 09 20:13:54.243403 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:54.243390    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba15e096-32a9-4ddc-89fd-d2d0e5faf3c2" containerName="manager"
Jul 09 20:13:54.243448 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:54.243404    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba15e096-32a9-4ddc-89fd-d2d0e5faf3c2" containerName="manager"
Jul 09 20:13:54.243481 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:54.243467    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba15e096-32a9-4ddc-89fd-d2d0e5faf3c2" containerName="manager"
Jul 09 20:13:54.246267 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:54.246245    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xs9kh"]
Jul 09 20:13:54.246501 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:54.246392    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kt2mp"
Jul 09 20:13:54.246614 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:54.246580    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xs9kh" podUID="f7c8e3d7-c021-476b-9174-9896787e3a4d" containerName="manager" containerID="cri-o://7b141b65dff28bc0c35e9428d79fef0b8fe08d4d33d7926f99f241d1df301101" gracePeriod=2
Jul 09 20:13:54.246742 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:13:54.246647    2562 projected.go:196] Error preparing data for projected volume kube-api-access-n84hm for pod kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-8dbcp: failed to fetch token: pod "kuadrant-operator-controller-manager-55467d9f7-8dbcp" not found
Jul 09 20:13:54.246742 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:13:54.246704    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7a1a7dd1-e074-4057-9b17-47221a719a88-kube-api-access-n84hm podName:7a1a7dd1-e074-4057-9b17-47221a719a88 nodeName:}" failed. No retries permitted until 2026-07-09 20:13:54.746683657 +0000 UTC m=+625.164353461 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-n84hm" (UniqueName: "kubernetes.io/projected/7a1a7dd1-e074-4057-9b17-47221a719a88-kube-api-access-n84hm") pod "kuadrant-operator-controller-manager-55467d9f7-8dbcp" (UID: "7a1a7dd1-e074-4057-9b17-47221a719a88") : failed to fetch token: pod "kuadrant-operator-controller-manager-55467d9f7-8dbcp" not found
Jul 09 20:13:54.249670 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:54.249642    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-9xwjz"]
Jul 09 20:13:54.257211 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:54.257176    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kt2mp"]
Jul 09 20:13:54.262708 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:54.262683    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xs9kh"]
Jul 09 20:13:54.269469 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:13:54.269430    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-c4dr6\" is forbidden: User \"system:node:ip-10-0-140-63.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-63.ec2.internal' and this object" podUID="ba15e096-32a9-4ddc-89fd-d2d0e5faf3c2" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c4dr6"
Jul 09 20:13:54.272711 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:13:54.272684    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-xs9kh\" is forbidden: User \"system:node:ip-10-0-140-63.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-63.ec2.internal' and this object" podUID="f7c8e3d7-c021-476b-9174-9896787e3a4d" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xs9kh"
Jul 09 20:13:54.283209 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:54.283180    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-g4vck"]
Jul 09 20:13:54.283706 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:54.283676    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7c8e3d7-c021-476b-9174-9896787e3a4d" containerName="manager"
Jul 09 20:13:54.283706 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:54.283700    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7c8e3d7-c021-476b-9174-9896787e3a4d" containerName="manager"
Jul 09 20:13:54.283804 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:54.283784    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7c8e3d7-c021-476b-9174-9896787e3a4d" containerName="manager"
Jul 09 20:13:54.289793 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:54.289771    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-g4vck"
Jul 09 20:13:54.298478 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:54.298449    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-g4vck"]
Jul 09 20:13:54.324863 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:13:54.324824    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-xs9kh\" is forbidden: User \"system:node:ip-10-0-140-63.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-63.ec2.internal' and this object" podUID="f7c8e3d7-c021-476b-9174-9896787e3a4d" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xs9kh"
Jul 09 20:13:54.337236 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:54.337215    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgg5p\" (UniqueName: \"kubernetes.io/projected/16ae5f64-9beb-4641-b98f-7fb01c577ad3-kube-api-access-zgg5p\") pod \"kuadrant-operator-controller-manager-55467d9f7-9xwjz\" (UID: \"16ae5f64-9beb-4641-b98f-7fb01c577ad3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-9xwjz"
Jul 09 20:13:54.337344 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:54.337260    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/16ae5f64-9beb-4641-b98f-7fb01c577ad3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-9xwjz\" (UID: \"16ae5f64-9beb-4641-b98f-7fb01c577ad3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-9xwjz"
Jul 09 20:13:54.438380 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:54.438282    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/16ae5f64-9beb-4641-b98f-7fb01c577ad3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-9xwjz\" (UID: \"16ae5f64-9beb-4641-b98f-7fb01c577ad3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-9xwjz"
Jul 09 20:13:54.438632 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:54.438614    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/16ae5f64-9beb-4641-b98f-7fb01c577ad3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-9xwjz\" (UID: \"16ae5f64-9beb-4641-b98f-7fb01c577ad3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-9xwjz"
Jul 09 20:13:54.438712 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:54.438688    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5lz8\" (UniqueName: \"kubernetes.io/projected/9643463c-0818-4009-a263-eab02ba00943-kube-api-access-j5lz8\") pod \"kuadrant-operator-controller-manager-8684c9ff49-kt2mp\" (UID: \"9643463c-0818-4009-a263-eab02ba00943\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kt2mp"
Jul 09 20:13:54.438769 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:54.438753    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgg5p\" (UniqueName: \"kubernetes.io/projected/16ae5f64-9beb-4641-b98f-7fb01c577ad3-kube-api-access-zgg5p\") pod \"kuadrant-operator-controller-manager-55467d9f7-9xwjz\" (UID: \"16ae5f64-9beb-4641-b98f-7fb01c577ad3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-9xwjz"
Jul 09 20:13:54.438814 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:54.438788    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9643463c-0818-4009-a263-eab02ba00943-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-kt2mp\" (UID: \"9643463c-0818-4009-a263-eab02ba00943\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kt2mp"
Jul 09 20:13:54.438856 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:54.438838    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5c9wb\" (UniqueName: \"kubernetes.io/projected/f5ed03c8-ae4f-42cd-89e1-4573415a3463-kube-api-access-5c9wb\") pod \"limitador-operator-controller-manager-859d7b78bd-g4vck\" (UID: \"f5ed03c8-ae4f-42cd-89e1-4573415a3463\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-g4vck"
Jul 09 20:13:54.448140 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:54.448112    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgg5p\" (UniqueName: \"kubernetes.io/projected/16ae5f64-9beb-4641-b98f-7fb01c577ad3-kube-api-access-zgg5p\") pod \"kuadrant-operator-controller-manager-55467d9f7-9xwjz\" (UID: \"16ae5f64-9beb-4641-b98f-7fb01c577ad3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-9xwjz"
Jul 09 20:13:54.485992 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:54.485962    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c4dr6"
Jul 09 20:13:54.487967 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:54.487948    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xs9kh"
Jul 09 20:13:54.488905 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:13:54.488883    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-xs9kh\" is forbidden: User \"system:node:ip-10-0-140-63.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-63.ec2.internal' and this object" podUID="f7c8e3d7-c021-476b-9174-9896787e3a4d" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xs9kh"
Jul 09 20:13:54.491499 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:13:54.491475    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-xs9kh\" is forbidden: User \"system:node:ip-10-0-140-63.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-63.ec2.internal' and this object" podUID="f7c8e3d7-c021-476b-9174-9896787e3a4d" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xs9kh"
Jul 09 20:13:54.523941 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:54.523913    2562 generic.go:356] "Generic (PLEG): container finished" podID="f7c8e3d7-c021-476b-9174-9896787e3a4d" containerID="7b141b65dff28bc0c35e9428d79fef0b8fe08d4d33d7926f99f241d1df301101" exitCode=0
Jul 09 20:13:54.524071 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:54.523968    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xs9kh"
Jul 09 20:13:54.524071 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:54.524020    2562 scope.go:117] "RemoveContainer" containerID="7b141b65dff28bc0c35e9428d79fef0b8fe08d4d33d7926f99f241d1df301101"
Jul 09 20:13:54.525582 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:54.525551    2562 generic.go:356] "Generic (PLEG): container finished" podID="ba15e096-32a9-4ddc-89fd-d2d0e5faf3c2" containerID="600894623fbfd7f22408f6ea0570cbe05b0867dc2f751276e6f0a17badb99845" exitCode=0
Jul 09 20:13:54.525674 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:54.525601    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-c4dr6"
Jul 09 20:13:54.525731 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:54.525720    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-8dbcp"
Jul 09 20:13:54.526989 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:13:54.526960    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-xs9kh\" is forbidden: User \"system:node:ip-10-0-140-63.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-63.ec2.internal' and this object" podUID="f7c8e3d7-c021-476b-9174-9896787e3a4d" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xs9kh"
Jul 09 20:13:54.529798 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:13:54.529772    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-xs9kh\" is forbidden: User \"system:node:ip-10-0-140-63.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-63.ec2.internal' and this object" podUID="f7c8e3d7-c021-476b-9174-9896787e3a4d" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xs9kh"
Jul 09 20:13:54.532352 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:54.532330    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-8dbcp"
Jul 09 20:13:54.533714 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:54.533698    2562 scope.go:117] "RemoveContainer" containerID="7b141b65dff28bc0c35e9428d79fef0b8fe08d4d33d7926f99f241d1df301101"
Jul 09 20:13:54.533960 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:13:54.533942    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b141b65dff28bc0c35e9428d79fef0b8fe08d4d33d7926f99f241d1df301101\": container with ID starting with 7b141b65dff28bc0c35e9428d79fef0b8fe08d4d33d7926f99f241d1df301101 not found: ID does not exist" containerID="7b141b65dff28bc0c35e9428d79fef0b8fe08d4d33d7926f99f241d1df301101"
Jul 09 20:13:54.534008 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:54.533971    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b141b65dff28bc0c35e9428d79fef0b8fe08d4d33d7926f99f241d1df301101"} err="failed to get container status \"7b141b65dff28bc0c35e9428d79fef0b8fe08d4d33d7926f99f241d1df301101\": rpc error: code = NotFound desc = could not find container \"7b141b65dff28bc0c35e9428d79fef0b8fe08d4d33d7926f99f241d1df301101\": container with ID starting with 7b141b65dff28bc0c35e9428d79fef0b8fe08d4d33d7926f99f241d1df301101 not found: ID does not exist"
Jul 09 20:13:54.534008 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:54.533991    2562 scope.go:117] "RemoveContainer" containerID="600894623fbfd7f22408f6ea0570cbe05b0867dc2f751276e6f0a17badb99845"
Jul 09 20:13:54.537052 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:13:54.537025    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-xs9kh\" is forbidden: User \"system:node:ip-10-0-140-63.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-63.ec2.internal' and this object" podUID="f7c8e3d7-c021-476b-9174-9896787e3a4d" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xs9kh"
Jul 09 20:13:54.539346 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:54.539329    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ba15e096-32a9-4ddc-89fd-d2d0e5faf3c2-extensions-socket-volume\") pod \"ba15e096-32a9-4ddc-89fd-d2d0e5faf3c2\" (UID: \"ba15e096-32a9-4ddc-89fd-d2d0e5faf3c2\") "
Jul 09 20:13:54.539449 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:54.539399    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b6629\" (UniqueName: \"kubernetes.io/projected/f7c8e3d7-c021-476b-9174-9896787e3a4d-kube-api-access-b6629\") pod \"f7c8e3d7-c021-476b-9174-9896787e3a4d\" (UID: \"f7c8e3d7-c021-476b-9174-9896787e3a4d\") "
Jul 09 20:13:54.539570 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:54.539455    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f8s7g\" (UniqueName: \"kubernetes.io/projected/ba15e096-32a9-4ddc-89fd-d2d0e5faf3c2-kube-api-access-f8s7g\") pod \"ba15e096-32a9-4ddc-89fd-d2d0e5faf3c2\" (UID: \"ba15e096-32a9-4ddc-89fd-d2d0e5faf3c2\") "
Jul 09 20:13:54.539570 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:54.539490    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7a1a7dd1-e074-4057-9b17-47221a719a88-extensions-socket-volume\") pod \"7a1a7dd1-e074-4057-9b17-47221a719a88\" (UID: \"7a1a7dd1-e074-4057-9b17-47221a719a88\") "
Jul 09 20:13:54.539686 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:54.539650    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5lz8\" (UniqueName: \"kubernetes.io/projected/9643463c-0818-4009-a263-eab02ba00943-kube-api-access-j5lz8\") pod \"kuadrant-operator-controller-manager-8684c9ff49-kt2mp\" (UID: \"9643463c-0818-4009-a263-eab02ba00943\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kt2mp"
Jul 09 20:13:54.539740 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:54.539708    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9643463c-0818-4009-a263-eab02ba00943-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-kt2mp\" (UID: \"9643463c-0818-4009-a263-eab02ba00943\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kt2mp"
Jul 09 20:13:54.539793 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:13:54.539733    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-8dbcp\" is forbidden: User \"system:node:ip-10-0-140-63.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-63.ec2.internal' and this object" podUID="7a1a7dd1-e074-4057-9b17-47221a719a88" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-8dbcp"
Jul 09 20:13:54.539793 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:54.539760    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5c9wb\" (UniqueName: \"kubernetes.io/projected/f5ed03c8-ae4f-42cd-89e1-4573415a3463-kube-api-access-5c9wb\") pod \"limitador-operator-controller-manager-859d7b78bd-g4vck\" (UID: \"f5ed03c8-ae4f-42cd-89e1-4573415a3463\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-g4vck"
Jul 09 20:13:54.539793 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:54.539776    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a1a7dd1-e074-4057-9b17-47221a719a88-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "7a1a7dd1-e074-4057-9b17-47221a719a88" (UID: "7a1a7dd1-e074-4057-9b17-47221a719a88"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 20:13:54.539958 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:54.539827    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n84hm\" (UniqueName: \"kubernetes.io/projected/7a1a7dd1-e074-4057-9b17-47221a719a88-kube-api-access-n84hm\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:13:54.539958 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:54.539876    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba15e096-32a9-4ddc-89fd-d2d0e5faf3c2-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "ba15e096-32a9-4ddc-89fd-d2d0e5faf3c2" (UID: "ba15e096-32a9-4ddc-89fd-d2d0e5faf3c2"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 20:13:54.540129 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:54.540111    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9643463c-0818-4009-a263-eab02ba00943-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-kt2mp\" (UID: \"9643463c-0818-4009-a263-eab02ba00943\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kt2mp"
Jul 09 20:13:54.541515 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:54.541493    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba15e096-32a9-4ddc-89fd-d2d0e5faf3c2-kube-api-access-f8s7g" (OuterVolumeSpecName: "kube-api-access-f8s7g") pod "ba15e096-32a9-4ddc-89fd-d2d0e5faf3c2" (UID: "ba15e096-32a9-4ddc-89fd-d2d0e5faf3c2"). InnerVolumeSpecName "kube-api-access-f8s7g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 20:13:54.541848 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:54.541831    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7c8e3d7-c021-476b-9174-9896787e3a4d-kube-api-access-b6629" (OuterVolumeSpecName: "kube-api-access-b6629") pod "f7c8e3d7-c021-476b-9174-9896787e3a4d" (UID: "f7c8e3d7-c021-476b-9174-9896787e3a4d"). InnerVolumeSpecName "kube-api-access-b6629". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 20:13:54.542379 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:54.542363    2562 scope.go:117] "RemoveContainer" containerID="600894623fbfd7f22408f6ea0570cbe05b0867dc2f751276e6f0a17badb99845"
Jul 09 20:13:54.542601 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:13:54.542585    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"600894623fbfd7f22408f6ea0570cbe05b0867dc2f751276e6f0a17badb99845\": container with ID starting with 600894623fbfd7f22408f6ea0570cbe05b0867dc2f751276e6f0a17badb99845 not found: ID does not exist" containerID="600894623fbfd7f22408f6ea0570cbe05b0867dc2f751276e6f0a17badb99845"
Jul 09 20:13:54.542658 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:54.542610    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"600894623fbfd7f22408f6ea0570cbe05b0867dc2f751276e6f0a17badb99845"} err="failed to get container status \"600894623fbfd7f22408f6ea0570cbe05b0867dc2f751276e6f0a17badb99845\": rpc error: code = NotFound desc = could not find container \"600894623fbfd7f22408f6ea0570cbe05b0867dc2f751276e6f0a17badb99845\": container with ID starting with 600894623fbfd7f22408f6ea0570cbe05b0867dc2f751276e6f0a17badb99845 not found: ID does not exist"
Jul 09 20:13:54.558372 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:54.558340    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5c9wb\" (UniqueName: \"kubernetes.io/projected/f5ed03c8-ae4f-42cd-89e1-4573415a3463-kube-api-access-5c9wb\") pod \"limitador-operator-controller-manager-859d7b78bd-g4vck\" (UID: \"f5ed03c8-ae4f-42cd-89e1-4573415a3463\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-g4vck"
Jul 09 20:13:54.558483 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:54.558428    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5lz8\" (UniqueName: \"kubernetes.io/projected/9643463c-0818-4009-a263-eab02ba00943-kube-api-access-j5lz8\") pod \"kuadrant-operator-controller-manager-8684c9ff49-kt2mp\" (UID: \"9643463c-0818-4009-a263-eab02ba00943\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kt2mp"
Jul 09 20:13:54.640709 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:54.640679    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8s7g\" (UniqueName: \"kubernetes.io/projected/ba15e096-32a9-4ddc-89fd-d2d0e5faf3c2-kube-api-access-f8s7g\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:13:54.640709 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:54.640702    2562 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7a1a7dd1-e074-4057-9b17-47221a719a88-extensions-socket-volume\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:13:54.640709 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:54.640711    2562 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ba15e096-32a9-4ddc-89fd-d2d0e5faf3c2-extensions-socket-volume\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:13:54.640709 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:54.640719    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6629\" (UniqueName: \"kubernetes.io/projected/f7c8e3d7-c021-476b-9174-9896787e3a4d-kube-api-access-b6629\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:13:54.672667 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:54.672639    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-9xwjz"
Jul 09 20:13:54.680956 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:54.680933    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kt2mp"
Jul 09 20:13:54.688855 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:54.688831    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-g4vck"
Jul 09 20:13:54.835492 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:54.835461    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kt2mp"]
Jul 09 20:13:54.839498 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:13:54.839471    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9643463c_0818_4009_a263_eab02ba00943.slice/crio-754c0a6336d9163788881a00f066b1f51355a882d85e3ab221fb7b1a8d3681ed WatchSource:0}: Error finding container 754c0a6336d9163788881a00f066b1f51355a882d85e3ab221fb7b1a8d3681ed: Status 404 returned error can't find the container with id 754c0a6336d9163788881a00f066b1f51355a882d85e3ab221fb7b1a8d3681ed
Jul 09 20:13:54.839723 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:13:54.839700    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-xs9kh\" is forbidden: User \"system:node:ip-10-0-140-63.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-63.ec2.internal' and this object" podUID="f7c8e3d7-c021-476b-9174-9896787e3a4d" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xs9kh"
Jul 09 20:13:54.842502 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:13:54.842464    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-8dbcp\" is forbidden: User \"system:node:ip-10-0-140-63.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-63.ec2.internal' and this object" podUID="7a1a7dd1-e074-4057-9b17-47221a719a88" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-8dbcp"
Jul 09 20:13:54.845829 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:13:54.845805    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-8dbcp\" is forbidden: User \"system:node:ip-10-0-140-63.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-63.ec2.internal' and this object" podUID="7a1a7dd1-e074-4057-9b17-47221a719a88" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-8dbcp"
Jul 09 20:13:54.850691 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:13:54.850669    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-xs9kh\" is forbidden: User \"system:node:ip-10-0-140-63.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-63.ec2.internal' and this object" podUID="f7c8e3d7-c021-476b-9174-9896787e3a4d" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xs9kh"
Jul 09 20:13:55.060054 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:55.060027    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-g4vck"]
Jul 09 20:13:55.061834 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:13:55.061809    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16ae5f64_9beb_4641_b98f_7fb01c577ad3.slice/crio-44d3587b17a3223f004db2ec47976d2d33dd43633eb2d3df68c53ef59972441d WatchSource:0}: Error finding container 44d3587b17a3223f004db2ec47976d2d33dd43633eb2d3df68c53ef59972441d: Status 404 returned error can't find the container with id 44d3587b17a3223f004db2ec47976d2d33dd43633eb2d3df68c53ef59972441d
Jul 09 20:13:55.061834 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:55.061828    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-9xwjz"]
Jul 09 20:13:55.062409 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:13:55.062386    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5ed03c8_ae4f_42cd_89e1_4573415a3463.slice/crio-b3899c86ff3c8f6e171a3f200a2a0784fd59841a80e132ffeabc5cf37931ac57 WatchSource:0}: Error finding container b3899c86ff3c8f6e171a3f200a2a0784fd59841a80e132ffeabc5cf37931ac57: Status 404 returned error can't find the container with id b3899c86ff3c8f6e171a3f200a2a0784fd59841a80e132ffeabc5cf37931ac57
Jul 09 20:13:55.541812 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:55.541774    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kt2mp" event={"ID":"9643463c-0818-4009-a263-eab02ba00943","Type":"ContainerStarted","Data":"c4d9eed6936d8e2c92d2ae4325d24347c8846dfbbcdbd04b0cdd2f73c2c35f07"}
Jul 09 20:13:55.541977 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:55.541851    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kt2mp" event={"ID":"9643463c-0818-4009-a263-eab02ba00943","Type":"ContainerStarted","Data":"754c0a6336d9163788881a00f066b1f51355a882d85e3ab221fb7b1a8d3681ed"}
Jul 09 20:13:55.541977 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:55.541895    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kt2mp"
Jul 09 20:13:55.543333 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:55.543293    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-g4vck" event={"ID":"f5ed03c8-ae4f-42cd-89e1-4573415a3463","Type":"ContainerStarted","Data":"f1b610f34aed2d407cd3de43efb48010b6c5f8f0ce053c1f7bdcd49bfab72dae"}
Jul 09 20:13:55.543466 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:55.543354    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-g4vck"
Jul 09 20:13:55.543466 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:55.543376    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-g4vck" event={"ID":"f5ed03c8-ae4f-42cd-89e1-4573415a3463","Type":"ContainerStarted","Data":"b3899c86ff3c8f6e171a3f200a2a0784fd59841a80e132ffeabc5cf37931ac57"}
Jul 09 20:13:55.544653 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:13:55.544627    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-xs9kh\" is forbidden: User \"system:node:ip-10-0-140-63.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-63.ec2.internal' and this object" podUID="f7c8e3d7-c021-476b-9174-9896787e3a4d" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xs9kh"
Jul 09 20:13:55.544822 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:55.544803    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-9xwjz" event={"ID":"16ae5f64-9beb-4641-b98f-7fb01c577ad3","Type":"ContainerStarted","Data":"d6209b0d4d5a1736101aff6c397c58a2d59b9e3f777c7eb09c6368ee33acd37b"}
Jul 09 20:13:55.544880 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:55.544828    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-8dbcp"
Jul 09 20:13:55.544880 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:55.544843    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-9xwjz" event={"ID":"16ae5f64-9beb-4641-b98f-7fb01c577ad3","Type":"ContainerStarted","Data":"44d3587b17a3223f004db2ec47976d2d33dd43633eb2d3df68c53ef59972441d"}
Jul 09 20:13:55.544880 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:55.544875    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-9xwjz"
Jul 09 20:13:55.568290 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:55.568247    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kt2mp" podStartSLOduration=1.568234935 podStartE2EDuration="1.568234935s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 20:13:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 20:13:55.567558331 +0000 UTC m=+625.985228141" watchObservedRunningTime="2026-07-09 20:13:55.568234935 +0000 UTC m=+625.985905130"
Jul 09 20:13:55.570415 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:13:55.570390    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-8dbcp\" is forbidden: User \"system:node:ip-10-0-140-63.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-63.ec2.internal' and this object" podUID="7a1a7dd1-e074-4057-9b17-47221a719a88" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-8dbcp"
Jul 09 20:13:55.591612 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:55.591562    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-g4vck" podStartSLOduration=1.591546606 podStartE2EDuration="1.591546606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 20:13:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 20:13:55.591165325 +0000 UTC m=+626.008835135" watchObservedRunningTime="2026-07-09 20:13:55.591546606 +0000 UTC m=+626.009216416"
Jul 09 20:13:55.594579 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:13:55.594546    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-8dbcp\" is forbidden: User \"system:node:ip-10-0-140-63.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-63.ec2.internal' and this object" podUID="7a1a7dd1-e074-4057-9b17-47221a719a88" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-8dbcp"
Jul 09 20:13:55.618018 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:55.617960    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-9xwjz" podStartSLOduration=1.617939389 podStartE2EDuration="1.617939389s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 20:13:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 20:13:55.616932007 +0000 UTC m=+626.034601841" watchObservedRunningTime="2026-07-09 20:13:55.617939389 +0000 UTC m=+626.035609406"
Jul 09 20:13:55.622021 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:13:55.621988    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-xs9kh\" is forbidden: User \"system:node:ip-10-0-140-63.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-63.ec2.internal' and this object" podUID="f7c8e3d7-c021-476b-9174-9896787e3a4d" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xs9kh"
Jul 09 20:13:56.241286 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:56.241252    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a1a7dd1-e074-4057-9b17-47221a719a88" path="/var/lib/kubelet/pods/7a1a7dd1-e074-4057-9b17-47221a719a88/volumes"
Jul 09 20:13:56.241682 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:56.241556    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba15e096-32a9-4ddc-89fd-d2d0e5faf3c2" path="/var/lib/kubelet/pods/ba15e096-32a9-4ddc-89fd-d2d0e5faf3c2/volumes"
Jul 09 20:13:56.241888 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:13:56.241875    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7c8e3d7-c021-476b-9174-9896787e3a4d" path="/var/lib/kubelet/pods/f7c8e3d7-c021-476b-9174-9896787e3a4d/volumes"
Jul 09 20:14:06.551408 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:06.551374    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-g4vck"
Jul 09 20:14:06.551920 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:06.551447    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-9xwjz"
Jul 09 20:14:06.551920 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:06.551478    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kt2mp"
Jul 09 20:14:06.626767 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:06.626730    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kt2mp"]
Jul 09 20:14:06.627037 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:06.627002    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kt2mp" podUID="9643463c-0818-4009-a263-eab02ba00943" containerName="manager" containerID="cri-o://c4d9eed6936d8e2c92d2ae4325d24347c8846dfbbcdbd04b0cdd2f73c2c35f07" gracePeriod=10
Jul 09 20:14:06.875910 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:06.875886    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kt2mp"
Jul 09 20:14:06.947231 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:06.947198    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9643463c-0818-4009-a263-eab02ba00943-extensions-socket-volume\") pod \"9643463c-0818-4009-a263-eab02ba00943\" (UID: \"9643463c-0818-4009-a263-eab02ba00943\") "
Jul 09 20:14:06.947434 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:06.947294    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j5lz8\" (UniqueName: \"kubernetes.io/projected/9643463c-0818-4009-a263-eab02ba00943-kube-api-access-j5lz8\") pod \"9643463c-0818-4009-a263-eab02ba00943\" (UID: \"9643463c-0818-4009-a263-eab02ba00943\") "
Jul 09 20:14:06.947597 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:06.947571    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9643463c-0818-4009-a263-eab02ba00943-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "9643463c-0818-4009-a263-eab02ba00943" (UID: "9643463c-0818-4009-a263-eab02ba00943"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 20:14:06.949208 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:06.949185    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9643463c-0818-4009-a263-eab02ba00943-kube-api-access-j5lz8" (OuterVolumeSpecName: "kube-api-access-j5lz8") pod "9643463c-0818-4009-a263-eab02ba00943" (UID: "9643463c-0818-4009-a263-eab02ba00943"). InnerVolumeSpecName "kube-api-access-j5lz8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 20:14:07.047849 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:07.047804    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5lz8\" (UniqueName: \"kubernetes.io/projected/9643463c-0818-4009-a263-eab02ba00943-kube-api-access-j5lz8\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:14:07.047849 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:07.047831    2562 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9643463c-0818-4009-a263-eab02ba00943-extensions-socket-volume\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:14:07.602473 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:07.602434    2562 generic.go:356] "Generic (PLEG): container finished" podID="9643463c-0818-4009-a263-eab02ba00943" containerID="c4d9eed6936d8e2c92d2ae4325d24347c8846dfbbcdbd04b0cdd2f73c2c35f07" exitCode=0
Jul 09 20:14:07.602891 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:07.602480    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kt2mp" event={"ID":"9643463c-0818-4009-a263-eab02ba00943","Type":"ContainerDied","Data":"c4d9eed6936d8e2c92d2ae4325d24347c8846dfbbcdbd04b0cdd2f73c2c35f07"}
Jul 09 20:14:07.602891 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:07.602494    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kt2mp"
Jul 09 20:14:07.602891 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:07.602535    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kt2mp" event={"ID":"9643463c-0818-4009-a263-eab02ba00943","Type":"ContainerDied","Data":"754c0a6336d9163788881a00f066b1f51355a882d85e3ab221fb7b1a8d3681ed"}
Jul 09 20:14:07.602891 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:07.602556    2562 scope.go:117] "RemoveContainer" containerID="c4d9eed6936d8e2c92d2ae4325d24347c8846dfbbcdbd04b0cdd2f73c2c35f07"
Jul 09 20:14:07.611283 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:07.611261    2562 scope.go:117] "RemoveContainer" containerID="c4d9eed6936d8e2c92d2ae4325d24347c8846dfbbcdbd04b0cdd2f73c2c35f07"
Jul 09 20:14:07.611592 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:14:07.611573    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4d9eed6936d8e2c92d2ae4325d24347c8846dfbbcdbd04b0cdd2f73c2c35f07\": container with ID starting with c4d9eed6936d8e2c92d2ae4325d24347c8846dfbbcdbd04b0cdd2f73c2c35f07 not found: ID does not exist" containerID="c4d9eed6936d8e2c92d2ae4325d24347c8846dfbbcdbd04b0cdd2f73c2c35f07"
Jul 09 20:14:07.611643 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:07.611601    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4d9eed6936d8e2c92d2ae4325d24347c8846dfbbcdbd04b0cdd2f73c2c35f07"} err="failed to get container status \"c4d9eed6936d8e2c92d2ae4325d24347c8846dfbbcdbd04b0cdd2f73c2c35f07\": rpc error: code = NotFound desc = could not find container \"c4d9eed6936d8e2c92d2ae4325d24347c8846dfbbcdbd04b0cdd2f73c2c35f07\": container with ID starting with c4d9eed6936d8e2c92d2ae4325d24347c8846dfbbcdbd04b0cdd2f73c2c35f07 not found: ID does not exist"
Jul 09 20:14:07.624155 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:07.624128    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kt2mp"]
Jul 09 20:14:07.629542 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:07.629521    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kt2mp"]
Jul 09 20:14:08.241953 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:08.241921    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9643463c-0818-4009-a263-eab02ba00943" path="/var/lib/kubelet/pods/9643463c-0818-4009-a263-eab02ba00943/volumes"
Jul 09 20:14:22.996727 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:22.996691    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-phr4z"]
Jul 09 20:14:22.997168 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:22.997020    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9643463c-0818-4009-a263-eab02ba00943" containerName="manager"
Jul 09 20:14:22.997168 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:22.997032    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="9643463c-0818-4009-a263-eab02ba00943" containerName="manager"
Jul 09 20:14:22.997168 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:22.997082    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="9643463c-0818-4009-a263-eab02ba00943" containerName="manager"
Jul 09 20:14:23.000320 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:23.000278    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-phr4z"
Jul 09 20:14:23.003596 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:23.003572    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-d9295\""
Jul 09 20:14:23.015680 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:23.015654    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-phr4z"]
Jul 09 20:14:23.076546 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:23.076506    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fa2660d-7348-4743-9447-c477bb97a722-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-phr4z\" (UID: \"5fa2660d-7348-4743-9447-c477bb97a722\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-phr4z"
Jul 09 20:14:23.076750 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:23.076580    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fa2660d-7348-4743-9447-c477bb97a722-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-phr4z\" (UID: \"5fa2660d-7348-4743-9447-c477bb97a722\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-phr4z"
Jul 09 20:14:23.076750 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:23.076621    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fa2660d-7348-4743-9447-c477bb97a722-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-phr4z\" (UID: \"5fa2660d-7348-4743-9447-c477bb97a722\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-phr4z"
Jul 09 20:14:23.076750 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:23.076663    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwdzr\" (UniqueName: \"kubernetes.io/projected/5fa2660d-7348-4743-9447-c477bb97a722-kube-api-access-bwdzr\") pod \"maas-default-gateway-openshift-default-56654dcf7-phr4z\" (UID: \"5fa2660d-7348-4743-9447-c477bb97a722\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-phr4z"
Jul 09 20:14:23.076750 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:23.076689    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fa2660d-7348-4743-9447-c477bb97a722-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-phr4z\" (UID: \"5fa2660d-7348-4743-9447-c477bb97a722\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-phr4z"
Jul 09 20:14:23.076750 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:23.076718    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fa2660d-7348-4743-9447-c477bb97a722-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-phr4z\" (UID: \"5fa2660d-7348-4743-9447-c477bb97a722\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-phr4z"
Jul 09 20:14:23.076750 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:23.076744    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fa2660d-7348-4743-9447-c477bb97a722-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-phr4z\" (UID: \"5fa2660d-7348-4743-9447-c477bb97a722\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-phr4z"
Jul 09 20:14:23.077010 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:23.076802    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fa2660d-7348-4743-9447-c477bb97a722-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-phr4z\" (UID: \"5fa2660d-7348-4743-9447-c477bb97a722\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-phr4z"
Jul 09 20:14:23.077010 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:23.076832    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fa2660d-7348-4743-9447-c477bb97a722-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-phr4z\" (UID: \"5fa2660d-7348-4743-9447-c477bb97a722\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-phr4z"
Jul 09 20:14:23.178320 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:23.178263    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fa2660d-7348-4743-9447-c477bb97a722-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-phr4z\" (UID: \"5fa2660d-7348-4743-9447-c477bb97a722\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-phr4z"
Jul 09 20:14:23.178501 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:23.178333    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fa2660d-7348-4743-9447-c477bb97a722-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-phr4z\" (UID: \"5fa2660d-7348-4743-9447-c477bb97a722\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-phr4z"
Jul 09 20:14:23.178501 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:23.178365    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fa2660d-7348-4743-9447-c477bb97a722-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-phr4z\" (UID: \"5fa2660d-7348-4743-9447-c477bb97a722\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-phr4z"
Jul 09 20:14:23.178501 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:23.178398    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fa2660d-7348-4743-9447-c477bb97a722-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-phr4z\" (UID: \"5fa2660d-7348-4743-9447-c477bb97a722\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-phr4z"
Jul 09 20:14:23.178501 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:23.178428    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fa2660d-7348-4743-9447-c477bb97a722-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-phr4z\" (UID: \"5fa2660d-7348-4743-9447-c477bb97a722\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-phr4z"
Jul 09 20:14:23.178501 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:23.178480    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fa2660d-7348-4743-9447-c477bb97a722-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-phr4z\" (UID: \"5fa2660d-7348-4743-9447-c477bb97a722\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-phr4z"
Jul 09 20:14:23.178744 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:23.178533    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fa2660d-7348-4743-9447-c477bb97a722-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-phr4z\" (UID: \"5fa2660d-7348-4743-9447-c477bb97a722\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-phr4z"
Jul 09 20:14:23.178744 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:23.178569    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fa2660d-7348-4743-9447-c477bb97a722-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-phr4z\" (UID: \"5fa2660d-7348-4743-9447-c477bb97a722\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-phr4z"
Jul 09 20:14:23.178744 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:23.178618    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwdzr\" (UniqueName: \"kubernetes.io/projected/5fa2660d-7348-4743-9447-c477bb97a722-kube-api-access-bwdzr\") pod \"maas-default-gateway-openshift-default-56654dcf7-phr4z\" (UID: \"5fa2660d-7348-4743-9447-c477bb97a722\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-phr4z"
Jul 09 20:14:23.178900 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:23.178827    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fa2660d-7348-4743-9447-c477bb97a722-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-phr4z\" (UID: \"5fa2660d-7348-4743-9447-c477bb97a722\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-phr4z"
Jul 09 20:14:23.179067 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:23.179042    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fa2660d-7348-4743-9447-c477bb97a722-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-phr4z\" (UID: \"5fa2660d-7348-4743-9447-c477bb97a722\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-phr4z"
Jul 09 20:14:23.179205 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:23.179073    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fa2660d-7348-4743-9447-c477bb97a722-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-phr4z\" (UID: \"5fa2660d-7348-4743-9447-c477bb97a722\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-phr4z"
Jul 09 20:14:23.179205 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:23.179112    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fa2660d-7348-4743-9447-c477bb97a722-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-phr4z\" (UID: \"5fa2660d-7348-4743-9447-c477bb97a722\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-phr4z"
Jul 09 20:14:23.179491 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:23.179461    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fa2660d-7348-4743-9447-c477bb97a722-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-phr4z\" (UID: \"5fa2660d-7348-4743-9447-c477bb97a722\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-phr4z"
Jul 09 20:14:23.180628 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:23.180610    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fa2660d-7348-4743-9447-c477bb97a722-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-phr4z\" (UID: \"5fa2660d-7348-4743-9447-c477bb97a722\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-phr4z"
Jul 09 20:14:23.181128 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:23.181109    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fa2660d-7348-4743-9447-c477bb97a722-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-phr4z\" (UID: \"5fa2660d-7348-4743-9447-c477bb97a722\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-phr4z"
Jul 09 20:14:23.190368 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:23.190342    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fa2660d-7348-4743-9447-c477bb97a722-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-phr4z\" (UID: \"5fa2660d-7348-4743-9447-c477bb97a722\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-phr4z"
Jul 09 20:14:23.190511 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:23.190495    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwdzr\" (UniqueName: \"kubernetes.io/projected/5fa2660d-7348-4743-9447-c477bb97a722-kube-api-access-bwdzr\") pod \"maas-default-gateway-openshift-default-56654dcf7-phr4z\" (UID: \"5fa2660d-7348-4743-9447-c477bb97a722\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-phr4z"
Jul 09 20:14:23.310921 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:23.310836    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-phr4z"
Jul 09 20:14:23.448910 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:23.448885    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-phr4z"]
Jul 09 20:14:23.451368 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:14:23.451340    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fa2660d_7348_4743_9447_c477bb97a722.slice/crio-6ae173b8e5e3d20089885258055c8d44f5e548a4de09ce8cbdea5660896c832f WatchSource:0}: Error finding container 6ae173b8e5e3d20089885258055c8d44f5e548a4de09ce8cbdea5660896c832f: Status 404 returned error can't find the container with id 6ae173b8e5e3d20089885258055c8d44f5e548a4de09ce8cbdea5660896c832f
Jul 09 20:14:23.676295 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:23.676198    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-phr4z" event={"ID":"5fa2660d-7348-4743-9447-c477bb97a722","Type":"ContainerStarted","Data":"6ae173b8e5e3d20089885258055c8d44f5e548a4de09ce8cbdea5660896c832f"}
Jul 09 20:14:23.949399 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:23.949293    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-9xwjz"]
Jul 09 20:14:23.949589 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:23.949555    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-9xwjz" podUID="16ae5f64-9beb-4641-b98f-7fb01c577ad3" containerName="manager" containerID="cri-o://d6209b0d4d5a1736101aff6c397c58a2d59b9e3f777c7eb09c6368ee33acd37b" gracePeriod=10
Jul 09 20:14:24.206173 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:24.206143    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-9xwjz"
Jul 09 20:14:24.288964 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:24.288931    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zgg5p\" (UniqueName: \"kubernetes.io/projected/16ae5f64-9beb-4641-b98f-7fb01c577ad3-kube-api-access-zgg5p\") pod \"16ae5f64-9beb-4641-b98f-7fb01c577ad3\" (UID: \"16ae5f64-9beb-4641-b98f-7fb01c577ad3\") "
Jul 09 20:14:24.289168 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:24.289064    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/16ae5f64-9beb-4641-b98f-7fb01c577ad3-extensions-socket-volume\") pod \"16ae5f64-9beb-4641-b98f-7fb01c577ad3\" (UID: \"16ae5f64-9beb-4641-b98f-7fb01c577ad3\") "
Jul 09 20:14:24.289475 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:24.289437    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16ae5f64-9beb-4641-b98f-7fb01c577ad3-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "16ae5f64-9beb-4641-b98f-7fb01c577ad3" (UID: "16ae5f64-9beb-4641-b98f-7fb01c577ad3"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 20:14:24.291513 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:24.291486    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16ae5f64-9beb-4641-b98f-7fb01c577ad3-kube-api-access-zgg5p" (OuterVolumeSpecName: "kube-api-access-zgg5p") pod "16ae5f64-9beb-4641-b98f-7fb01c577ad3" (UID: "16ae5f64-9beb-4641-b98f-7fb01c577ad3"). InnerVolumeSpecName "kube-api-access-zgg5p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 20:14:24.389962 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:24.389920    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgg5p\" (UniqueName: \"kubernetes.io/projected/16ae5f64-9beb-4641-b98f-7fb01c577ad3-kube-api-access-zgg5p\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:14:24.389962 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:24.389961    2562 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/16ae5f64-9beb-4641-b98f-7fb01c577ad3-extensions-socket-volume\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:14:24.683616 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:24.683523    2562 generic.go:356] "Generic (PLEG): container finished" podID="16ae5f64-9beb-4641-b98f-7fb01c577ad3" containerID="d6209b0d4d5a1736101aff6c397c58a2d59b9e3f777c7eb09c6368ee33acd37b" exitCode=0
Jul 09 20:14:24.683616 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:24.683587    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-9xwjz"
Jul 09 20:14:24.683616 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:24.683595    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-9xwjz" event={"ID":"16ae5f64-9beb-4641-b98f-7fb01c577ad3","Type":"ContainerDied","Data":"d6209b0d4d5a1736101aff6c397c58a2d59b9e3f777c7eb09c6368ee33acd37b"}
Jul 09 20:14:24.683886 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:24.683654    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-9xwjz" event={"ID":"16ae5f64-9beb-4641-b98f-7fb01c577ad3","Type":"ContainerDied","Data":"44d3587b17a3223f004db2ec47976d2d33dd43633eb2d3df68c53ef59972441d"}
Jul 09 20:14:24.683886 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:24.683683    2562 scope.go:117] "RemoveContainer" containerID="d6209b0d4d5a1736101aff6c397c58a2d59b9e3f777c7eb09c6368ee33acd37b"
Jul 09 20:14:24.693482 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:24.693463    2562 scope.go:117] "RemoveContainer" containerID="d6209b0d4d5a1736101aff6c397c58a2d59b9e3f777c7eb09c6368ee33acd37b"
Jul 09 20:14:24.693807 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:14:24.693784    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6209b0d4d5a1736101aff6c397c58a2d59b9e3f777c7eb09c6368ee33acd37b\": container with ID starting with d6209b0d4d5a1736101aff6c397c58a2d59b9e3f777c7eb09c6368ee33acd37b not found: ID does not exist" containerID="d6209b0d4d5a1736101aff6c397c58a2d59b9e3f777c7eb09c6368ee33acd37b"
Jul 09 20:14:24.693881 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:24.693813    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6209b0d4d5a1736101aff6c397c58a2d59b9e3f777c7eb09c6368ee33acd37b"} err="failed to get container status \"d6209b0d4d5a1736101aff6c397c58a2d59b9e3f777c7eb09c6368ee33acd37b\": rpc error: code = NotFound desc = could not find container \"d6209b0d4d5a1736101aff6c397c58a2d59b9e3f777c7eb09c6368ee33acd37b\": container with ID starting with d6209b0d4d5a1736101aff6c397c58a2d59b9e3f777c7eb09c6368ee33acd37b not found: ID does not exist"
Jul 09 20:14:24.719669 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:24.719629    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-9xwjz"]
Jul 09 20:14:24.722289 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:24.722258    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-9xwjz"]
Jul 09 20:14:25.850144 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:25.850109    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 09 20:14:25.850456 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:25.850182    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 09 20:14:25.850456 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:25.850214    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 09 20:14:26.241520 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:26.241489    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="16ae5f64-9beb-4641-b98f-7fb01c577ad3" path="/var/lib/kubelet/pods/16ae5f64-9beb-4641-b98f-7fb01c577ad3/volumes"
Jul 09 20:14:26.697101 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:26.697011    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-phr4z" event={"ID":"5fa2660d-7348-4743-9447-c477bb97a722","Type":"ContainerStarted","Data":"b8790a3773483366bfdb0cfefcdd97c64c4f965bbdcbce6f95cd14a495d9ae9a"}
Jul 09 20:14:26.719175 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:26.719126    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-phr4z" podStartSLOduration=2.322771737 podStartE2EDuration="4.719108895s" totalImagesPullingTime="2.396337158s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 20:14:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 20:14:26.71781357 +0000 UTC m=+657.135483402" watchObservedRunningTime="2026-07-09 20:14:26.719108895 +0000 UTC m=+657.136778703"
Jul 09 20:14:27.311417 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:27.311385    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-phr4z"
Jul 09 20:14:27.316048 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:27.316021    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-phr4z"
Jul 09 20:14:27.702646 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:27.702549    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-phr4z"
Jul 09 20:14:27.703629 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:27.703609    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-phr4z"
Jul 09 20:14:41.944145 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:41.944082    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-z5kk6"]
Jul 09 20:14:41.944683 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:41.944629    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16ae5f64-9beb-4641-b98f-7fb01c577ad3" containerName="manager"
Jul 09 20:14:41.944683 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:41.944649    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="16ae5f64-9beb-4641-b98f-7fb01c577ad3" containerName="manager"
Jul 09 20:14:41.944803 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:41.944759    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="16ae5f64-9beb-4641-b98f-7fb01c577ad3" containerName="manager"
Jul 09 20:14:41.949081 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:41.949059    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-z5kk6"
Jul 09 20:14:41.952652 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:41.952631    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-vqwng\""
Jul 09 20:14:41.952827 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:41.952811    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 09 20:14:41.968495 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:41.968463    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-z5kk6"]
Jul 09 20:14:42.045323 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:42.045269    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7z8h\" (UniqueName: \"kubernetes.io/projected/bfbdb936-8837-476a-84ee-b7ac307c8130-kube-api-access-h7z8h\") pod \"limitador-limitador-64c94fd64f-z5kk6\" (UID: \"bfbdb936-8837-476a-84ee-b7ac307c8130\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-z5kk6"
Jul 09 20:14:42.045506 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:42.045417    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/bfbdb936-8837-476a-84ee-b7ac307c8130-config-file\") pod \"limitador-limitador-64c94fd64f-z5kk6\" (UID: \"bfbdb936-8837-476a-84ee-b7ac307c8130\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-z5kk6"
Jul 09 20:14:42.059488 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:42.059460    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-z5kk6"]
Jul 09 20:14:42.146124 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:42.146087    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/bfbdb936-8837-476a-84ee-b7ac307c8130-config-file\") pod \"limitador-limitador-64c94fd64f-z5kk6\" (UID: \"bfbdb936-8837-476a-84ee-b7ac307c8130\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-z5kk6"
Jul 09 20:14:42.146124 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:42.146126    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7z8h\" (UniqueName: \"kubernetes.io/projected/bfbdb936-8837-476a-84ee-b7ac307c8130-kube-api-access-h7z8h\") pod \"limitador-limitador-64c94fd64f-z5kk6\" (UID: \"bfbdb936-8837-476a-84ee-b7ac307c8130\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-z5kk6"
Jul 09 20:14:42.146819 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:42.146794    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/bfbdb936-8837-476a-84ee-b7ac307c8130-config-file\") pod \"limitador-limitador-64c94fd64f-z5kk6\" (UID: \"bfbdb936-8837-476a-84ee-b7ac307c8130\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-z5kk6"
Jul 09 20:14:42.154263 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:42.154240    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7z8h\" (UniqueName: \"kubernetes.io/projected/bfbdb936-8837-476a-84ee-b7ac307c8130-kube-api-access-h7z8h\") pod \"limitador-limitador-64c94fd64f-z5kk6\" (UID: \"bfbdb936-8837-476a-84ee-b7ac307c8130\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-z5kk6"
Jul 09 20:14:42.259297 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:42.259261    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-z5kk6"
Jul 09 20:14:42.388254 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:42.388228    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-z5kk6"]
Jul 09 20:14:42.390648 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:14:42.390619    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfbdb936_8837_476a_84ee_b7ac307c8130.slice/crio-cb6cdb51a6d16adb03893250dca4c59f271d0b848687c4b0fff8f96bb142935d WatchSource:0}: Error finding container cb6cdb51a6d16adb03893250dca4c59f271d0b848687c4b0fff8f96bb142935d: Status 404 returned error can't find the container with id cb6cdb51a6d16adb03893250dca4c59f271d0b848687c4b0fff8f96bb142935d
Jul 09 20:14:42.767484 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:42.767449    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-z5kk6" event={"ID":"bfbdb936-8837-476a-84ee-b7ac307c8130","Type":"ContainerStarted","Data":"cb6cdb51a6d16adb03893250dca4c59f271d0b848687c4b0fff8f96bb142935d"}
Jul 09 20:14:45.782911 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:45.782871    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-z5kk6" event={"ID":"bfbdb936-8837-476a-84ee-b7ac307c8130","Type":"ContainerStarted","Data":"920e2f5ea73159b1b112ea361e85f25f7cc237a9c14916bdbd0862b95c219479"}
Jul 09 20:14:45.783324 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:45.782971    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-z5kk6"
Jul 09 20:14:45.802595 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:45.802542    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-z5kk6" podStartSLOduration=2.286534206 podStartE2EDuration="4.802521759s" totalImagesPullingTime="2.515987553s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 20:14:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 20:14:45.800758453 +0000 UTC m=+676.218428263" watchObservedRunningTime="2026-07-09 20:14:45.802521759 +0000 UTC m=+676.220191571"
Jul 09 20:14:56.711161 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:56.711124    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-z5kk6"]
Jul 09 20:14:56.711688 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:56.711356    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-z5kk6" podUID="bfbdb936-8837-476a-84ee-b7ac307c8130" containerName="limitador" containerID="cri-o://920e2f5ea73159b1b112ea361e85f25f7cc237a9c14916bdbd0862b95c219479" gracePeriod=30
Jul 09 20:14:56.712094 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:56.712072    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-z5kk6"
Jul 09 20:14:57.252911 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:57.252885    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-z5kk6"
Jul 09 20:14:57.370608 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:57.370521    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/bfbdb936-8837-476a-84ee-b7ac307c8130-config-file\") pod \"bfbdb936-8837-476a-84ee-b7ac307c8130\" (UID: \"bfbdb936-8837-476a-84ee-b7ac307c8130\") "
Jul 09 20:14:57.370608 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:57.370594    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h7z8h\" (UniqueName: \"kubernetes.io/projected/bfbdb936-8837-476a-84ee-b7ac307c8130-kube-api-access-h7z8h\") pod \"bfbdb936-8837-476a-84ee-b7ac307c8130\" (UID: \"bfbdb936-8837-476a-84ee-b7ac307c8130\") "
Jul 09 20:14:57.370894 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:57.370873    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfbdb936-8837-476a-84ee-b7ac307c8130-config-file" (OuterVolumeSpecName: "config-file") pod "bfbdb936-8837-476a-84ee-b7ac307c8130" (UID: "bfbdb936-8837-476a-84ee-b7ac307c8130"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 20:14:57.372739 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:57.372710    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfbdb936-8837-476a-84ee-b7ac307c8130-kube-api-access-h7z8h" (OuterVolumeSpecName: "kube-api-access-h7z8h") pod "bfbdb936-8837-476a-84ee-b7ac307c8130" (UID: "bfbdb936-8837-476a-84ee-b7ac307c8130"). InnerVolumeSpecName "kube-api-access-h7z8h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 20:14:57.472216 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:57.472165    2562 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/bfbdb936-8837-476a-84ee-b7ac307c8130-config-file\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:14:57.472216 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:57.472211    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7z8h\" (UniqueName: \"kubernetes.io/projected/bfbdb936-8837-476a-84ee-b7ac307c8130-kube-api-access-h7z8h\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:14:57.836422 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:57.836388    2562 generic.go:356] "Generic (PLEG): container finished" podID="bfbdb936-8837-476a-84ee-b7ac307c8130" containerID="920e2f5ea73159b1b112ea361e85f25f7cc237a9c14916bdbd0862b95c219479" exitCode=0
Jul 09 20:14:57.836860 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:57.836436    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-z5kk6"
Jul 09 20:14:57.836860 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:57.836464    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-z5kk6" event={"ID":"bfbdb936-8837-476a-84ee-b7ac307c8130","Type":"ContainerDied","Data":"920e2f5ea73159b1b112ea361e85f25f7cc237a9c14916bdbd0862b95c219479"}
Jul 09 20:14:57.836860 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:57.836511    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-z5kk6" event={"ID":"bfbdb936-8837-476a-84ee-b7ac307c8130","Type":"ContainerDied","Data":"cb6cdb51a6d16adb03893250dca4c59f271d0b848687c4b0fff8f96bb142935d"}
Jul 09 20:14:57.836860 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:57.836536    2562 scope.go:117] "RemoveContainer" containerID="920e2f5ea73159b1b112ea361e85f25f7cc237a9c14916bdbd0862b95c219479"
Jul 09 20:14:57.845628 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:57.845611    2562 scope.go:117] "RemoveContainer" containerID="920e2f5ea73159b1b112ea361e85f25f7cc237a9c14916bdbd0862b95c219479"
Jul 09 20:14:57.845902 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:14:57.845885    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"920e2f5ea73159b1b112ea361e85f25f7cc237a9c14916bdbd0862b95c219479\": container with ID starting with 920e2f5ea73159b1b112ea361e85f25f7cc237a9c14916bdbd0862b95c219479 not found: ID does not exist" containerID="920e2f5ea73159b1b112ea361e85f25f7cc237a9c14916bdbd0862b95c219479"
Jul 09 20:14:57.845949 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:57.845911    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"920e2f5ea73159b1b112ea361e85f25f7cc237a9c14916bdbd0862b95c219479"} err="failed to get container status \"920e2f5ea73159b1b112ea361e85f25f7cc237a9c14916bdbd0862b95c219479\": rpc error: code = NotFound desc = could not find container \"920e2f5ea73159b1b112ea361e85f25f7cc237a9c14916bdbd0862b95c219479\": container with ID starting with 920e2f5ea73159b1b112ea361e85f25f7cc237a9c14916bdbd0862b95c219479 not found: ID does not exist"
Jul 09 20:14:57.857448 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:57.857407    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-z5kk6"]
Jul 09 20:14:57.861594 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:57.861566    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-z5kk6"]
Jul 09 20:14:58.241651 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:58.241613    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bfbdb936-8837-476a-84ee-b7ac307c8130" path="/var/lib/kubelet/pods/bfbdb936-8837-476a-84ee-b7ac307c8130/volumes"
Jul 09 20:14:59.878982 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:59.878943    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-k2rgs"]
Jul 09 20:14:59.879461 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:59.879441    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfbdb936-8837-476a-84ee-b7ac307c8130" containerName="limitador"
Jul 09 20:14:59.879544 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:59.879464    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfbdb936-8837-476a-84ee-b7ac307c8130" containerName="limitador"
Jul 09 20:14:59.879613 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:59.879601    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="bfbdb936-8837-476a-84ee-b7ac307c8130" containerName="limitador"
Jul 09 20:14:59.882192 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:59.882170    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-k2rgs"
Jul 09 20:14:59.885266 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:59.885247    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Jul 09 20:14:59.885388 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:59.885326    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 09 20:14:59.886870 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:59.886847    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-8bspp\""
Jul 09 20:14:59.886956 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:59.886855    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 09 20:14:59.891754 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:59.891733    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-k2rgs"]
Jul 09 20:14:59.894954 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:59.894934    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmqr8\" (UniqueName: \"kubernetes.io/projected/737886eb-3b43-45da-b81d-ea174d47a0ae-kube-api-access-gmqr8\") pod \"postgres-69fb4594cf-k2rgs\" (UID: \"737886eb-3b43-45da-b81d-ea174d47a0ae\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-k2rgs"
Jul 09 20:14:59.895054 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:59.894963    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/737886eb-3b43-45da-b81d-ea174d47a0ae-data\") pod \"postgres-69fb4594cf-k2rgs\" (UID: \"737886eb-3b43-45da-b81d-ea174d47a0ae\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-k2rgs"
Jul 09 20:14:59.995790 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:59.995760    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/737886eb-3b43-45da-b81d-ea174d47a0ae-data\") pod \"postgres-69fb4594cf-k2rgs\" (UID: \"737886eb-3b43-45da-b81d-ea174d47a0ae\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-k2rgs"
Jul 09 20:14:59.995945 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:59.995853    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmqr8\" (UniqueName: \"kubernetes.io/projected/737886eb-3b43-45da-b81d-ea174d47a0ae-kube-api-access-gmqr8\") pod \"postgres-69fb4594cf-k2rgs\" (UID: \"737886eb-3b43-45da-b81d-ea174d47a0ae\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-k2rgs"
Jul 09 20:14:59.996126 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:14:59.996106    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/737886eb-3b43-45da-b81d-ea174d47a0ae-data\") pod \"postgres-69fb4594cf-k2rgs\" (UID: \"737886eb-3b43-45da-b81d-ea174d47a0ae\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-k2rgs"
Jul 09 20:15:00.004457 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:00.004433    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmqr8\" (UniqueName: \"kubernetes.io/projected/737886eb-3b43-45da-b81d-ea174d47a0ae-kube-api-access-gmqr8\") pod \"postgres-69fb4594cf-k2rgs\" (UID: \"737886eb-3b43-45da-b81d-ea174d47a0ae\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-k2rgs"
Jul 09 20:15:00.193822 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:00.193727    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-k2rgs"
Jul 09 20:15:00.328647 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:00.328603    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-k2rgs"]
Jul 09 20:15:00.330996 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:15:00.330965    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod737886eb_3b43_45da_b81d_ea174d47a0ae.slice/crio-8bc04393376f7a73f2dceae0f1c630666488180c1df6e365144bcb9b5843cafe WatchSource:0}: Error finding container 8bc04393376f7a73f2dceae0f1c630666488180c1df6e365144bcb9b5843cafe: Status 404 returned error can't find the container with id 8bc04393376f7a73f2dceae0f1c630666488180c1df6e365144bcb9b5843cafe
Jul 09 20:15:00.851946 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:00.851904    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-k2rgs" event={"ID":"737886eb-3b43-45da-b81d-ea174d47a0ae","Type":"ContainerStarted","Data":"8bc04393376f7a73f2dceae0f1c630666488180c1df6e365144bcb9b5843cafe"}
Jul 09 20:15:05.878908 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:05.878872    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-k2rgs" event={"ID":"737886eb-3b43-45da-b81d-ea174d47a0ae","Type":"ContainerStarted","Data":"b6595d452895bf90c5c3fafb3db5541121696168d5e9ee6b26abc57f128d4722"}
Jul 09 20:15:05.879284 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:05.878935    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-k2rgs"
Jul 09 20:15:05.897149 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:05.897098    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-k2rgs" podStartSLOduration=2.008149605 podStartE2EDuration="6.89708279s" totalImagesPullingTime="4.888933185s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 20:14:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 20:15:05.89647478 +0000 UTC m=+696.314144615" watchObservedRunningTime="2026-07-09 20:15:05.89708279 +0000 UTC m=+696.314752598"
Jul 09 20:15:11.913348 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:11.913297    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-k2rgs"
Jul 09 20:15:12.797486 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:12.797452    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-686f995674-gmg4v"]
Jul 09 20:15:12.803806 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:12.803783    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-686f995674-gmg4v"
Jul 09 20:15:12.807151 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:12.807128    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-z9bp7\""
Jul 09 20:15:12.807802 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:12.807782    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-686f995674-gmg4v"]
Jul 09 20:15:12.904503 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:12.904468    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpzdq\" (UniqueName: \"kubernetes.io/projected/1c08a3bc-01b9-444f-ab63-94c20791471e-kube-api-access-xpzdq\") pod \"authorino-686f995674-gmg4v\" (UID: \"1c08a3bc-01b9-444f-ab63-94c20791471e\") " pod="kuadrant-system/authorino-686f995674-gmg4v"
Jul 09 20:15:13.004918 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:13.004886    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpzdq\" (UniqueName: \"kubernetes.io/projected/1c08a3bc-01b9-444f-ab63-94c20791471e-kube-api-access-xpzdq\") pod \"authorino-686f995674-gmg4v\" (UID: \"1c08a3bc-01b9-444f-ab63-94c20791471e\") " pod="kuadrant-system/authorino-686f995674-gmg4v"
Jul 09 20:15:13.013413 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:13.013384    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpzdq\" (UniqueName: \"kubernetes.io/projected/1c08a3bc-01b9-444f-ab63-94c20791471e-kube-api-access-xpzdq\") pod \"authorino-686f995674-gmg4v\" (UID: \"1c08a3bc-01b9-444f-ab63-94c20791471e\") " pod="kuadrant-system/authorino-686f995674-gmg4v"
Jul 09 20:15:13.083219 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:13.083136    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-686f995674-gmg4v"]
Jul 09 20:15:13.083437 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:13.083422    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-686f995674-gmg4v"
Jul 09 20:15:13.110098 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:13.110068    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-8445478c58-h7lzl"]
Jul 09 20:15:13.115734 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:13.115700    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8445478c58-h7lzl"
Jul 09 20:15:13.119107 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:13.119083    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Jul 09 20:15:13.120795 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:13.120766    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-8445478c58-h7lzl"]
Jul 09 20:15:13.206808 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:13.206775    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k65hg\" (UniqueName: \"kubernetes.io/projected/701712cd-0859-4007-9d29-48f3135733f0-kube-api-access-k65hg\") pod \"authorino-8445478c58-h7lzl\" (UID: \"701712cd-0859-4007-9d29-48f3135733f0\") " pod="kuadrant-system/authorino-8445478c58-h7lzl"
Jul 09 20:15:13.206954 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:13.206826    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/701712cd-0859-4007-9d29-48f3135733f0-tls-cert\") pod \"authorino-8445478c58-h7lzl\" (UID: \"701712cd-0859-4007-9d29-48f3135733f0\") " pod="kuadrant-system/authorino-8445478c58-h7lzl"
Jul 09 20:15:13.215222 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:13.215197    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-686f995674-gmg4v"]
Jul 09 20:15:13.217799 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:15:13.217773    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1c08a3bc_01b9_444f_ab63_94c20791471e.slice/crio-e5517b6eb46337f0e5b42909350b41a6fa3c5236fcab277dddfa9c8586b67efa WatchSource:0}: Error finding container e5517b6eb46337f0e5b42909350b41a6fa3c5236fcab277dddfa9c8586b67efa: Status 404 returned error can't find the container with id e5517b6eb46337f0e5b42909350b41a6fa3c5236fcab277dddfa9c8586b67efa
Jul 09 20:15:13.307497 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:13.307467    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k65hg\" (UniqueName: \"kubernetes.io/projected/701712cd-0859-4007-9d29-48f3135733f0-kube-api-access-k65hg\") pod \"authorino-8445478c58-h7lzl\" (UID: \"701712cd-0859-4007-9d29-48f3135733f0\") " pod="kuadrant-system/authorino-8445478c58-h7lzl"
Jul 09 20:15:13.307679 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:13.307520    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/701712cd-0859-4007-9d29-48f3135733f0-tls-cert\") pod \"authorino-8445478c58-h7lzl\" (UID: \"701712cd-0859-4007-9d29-48f3135733f0\") " pod="kuadrant-system/authorino-8445478c58-h7lzl"
Jul 09 20:15:13.309950 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:13.309925    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/701712cd-0859-4007-9d29-48f3135733f0-tls-cert\") pod \"authorino-8445478c58-h7lzl\" (UID: \"701712cd-0859-4007-9d29-48f3135733f0\") " pod="kuadrant-system/authorino-8445478c58-h7lzl"
Jul 09 20:15:13.315434 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:13.315414    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k65hg\" (UniqueName: \"kubernetes.io/projected/701712cd-0859-4007-9d29-48f3135733f0-kube-api-access-k65hg\") pod \"authorino-8445478c58-h7lzl\" (UID: \"701712cd-0859-4007-9d29-48f3135733f0\") " pod="kuadrant-system/authorino-8445478c58-h7lzl"
Jul 09 20:15:13.427387 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:13.427287    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8445478c58-h7lzl"
Jul 09 20:15:13.554106 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:13.554082    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-8445478c58-h7lzl"]
Jul 09 20:15:13.556126 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:15:13.556105    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod701712cd_0859_4007_9d29_48f3135733f0.slice/crio-30ddc6c5960460f6f3632f657313f4a5ab3555fd1499735858f51fcd77f60a09 WatchSource:0}: Error finding container 30ddc6c5960460f6f3632f657313f4a5ab3555fd1499735858f51fcd77f60a09: Status 404 returned error can't find the container with id 30ddc6c5960460f6f3632f657313f4a5ab3555fd1499735858f51fcd77f60a09
Jul 09 20:15:13.915040 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:13.915006    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8445478c58-h7lzl" event={"ID":"701712cd-0859-4007-9d29-48f3135733f0","Type":"ContainerStarted","Data":"30ddc6c5960460f6f3632f657313f4a5ab3555fd1499735858f51fcd77f60a09"}
Jul 09 20:15:13.916243 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:13.916198    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-686f995674-gmg4v" event={"ID":"1c08a3bc-01b9-444f-ab63-94c20791471e","Type":"ContainerStarted","Data":"e5517b6eb46337f0e5b42909350b41a6fa3c5236fcab277dddfa9c8586b67efa"}
Jul 09 20:15:16.934518 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:16.934459    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8445478c58-h7lzl" event={"ID":"701712cd-0859-4007-9d29-48f3135733f0","Type":"ContainerStarted","Data":"3c4d424cd88580dbb90e8c3d0604167d534f457a6a3cf51881cce8fe5b374bd3"}
Jul 09 20:15:16.937060 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:16.937013    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-686f995674-gmg4v" event={"ID":"1c08a3bc-01b9-444f-ab63-94c20791471e","Type":"ContainerStarted","Data":"7f5b6e124eab6dfe60327dd45b6622ccec6b6deca682eaa2ea59d373b3007cab"}
Jul 09 20:15:16.937060 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:16.937028    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-686f995674-gmg4v" podUID="1c08a3bc-01b9-444f-ab63-94c20791471e" containerName="authorino" containerID="cri-o://7f5b6e124eab6dfe60327dd45b6622ccec6b6deca682eaa2ea59d373b3007cab" gracePeriod=30
Jul 09 20:15:16.950736 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:16.950679    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-8445478c58-h7lzl" podStartSLOduration=0.70057801 podStartE2EDuration="3.950663822s" totalImagesPullingTime="3.250085812s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 20:15:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 20:15:16.950098023 +0000 UTC m=+707.367767859" watchObservedRunningTime="2026-07-09 20:15:16.950663822 +0000 UTC m=+707.368333630"
Jul 09 20:15:17.188924 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:17.188897    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-686f995674-gmg4v"
Jul 09 20:15:17.246414 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:17.246384    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xpzdq\" (UniqueName: \"kubernetes.io/projected/1c08a3bc-01b9-444f-ab63-94c20791471e-kube-api-access-xpzdq\") pod \"1c08a3bc-01b9-444f-ab63-94c20791471e\" (UID: \"1c08a3bc-01b9-444f-ab63-94c20791471e\") "
Jul 09 20:15:17.248395 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:17.248373    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c08a3bc-01b9-444f-ab63-94c20791471e-kube-api-access-xpzdq" (OuterVolumeSpecName: "kube-api-access-xpzdq") pod "1c08a3bc-01b9-444f-ab63-94c20791471e" (UID: "1c08a3bc-01b9-444f-ab63-94c20791471e"). InnerVolumeSpecName "kube-api-access-xpzdq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 20:15:17.347231 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:17.347158    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpzdq\" (UniqueName: \"kubernetes.io/projected/1c08a3bc-01b9-444f-ab63-94c20791471e-kube-api-access-xpzdq\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:15:17.943278 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:17.943231    2562 generic.go:356] "Generic (PLEG): container finished" podID="1c08a3bc-01b9-444f-ab63-94c20791471e" containerID="7f5b6e124eab6dfe60327dd45b6622ccec6b6deca682eaa2ea59d373b3007cab" exitCode=0
Jul 09 20:15:17.943278 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:17.943271    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-686f995674-gmg4v"
Jul 09 20:15:17.943799 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:17.943328    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-686f995674-gmg4v" event={"ID":"1c08a3bc-01b9-444f-ab63-94c20791471e","Type":"ContainerDied","Data":"7f5b6e124eab6dfe60327dd45b6622ccec6b6deca682eaa2ea59d373b3007cab"}
Jul 09 20:15:17.943799 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:17.943364    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-686f995674-gmg4v" event={"ID":"1c08a3bc-01b9-444f-ab63-94c20791471e","Type":"ContainerDied","Data":"e5517b6eb46337f0e5b42909350b41a6fa3c5236fcab277dddfa9c8586b67efa"}
Jul 09 20:15:17.943799 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:17.943381    2562 scope.go:117] "RemoveContainer" containerID="7f5b6e124eab6dfe60327dd45b6622ccec6b6deca682eaa2ea59d373b3007cab"
Jul 09 20:15:17.951941 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:17.951925    2562 scope.go:117] "RemoveContainer" containerID="7f5b6e124eab6dfe60327dd45b6622ccec6b6deca682eaa2ea59d373b3007cab"
Jul 09 20:15:17.952211 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:15:17.952194    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f5b6e124eab6dfe60327dd45b6622ccec6b6deca682eaa2ea59d373b3007cab\": container with ID starting with 7f5b6e124eab6dfe60327dd45b6622ccec6b6deca682eaa2ea59d373b3007cab not found: ID does not exist" containerID="7f5b6e124eab6dfe60327dd45b6622ccec6b6deca682eaa2ea59d373b3007cab"
Jul 09 20:15:17.952263 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:17.952220    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f5b6e124eab6dfe60327dd45b6622ccec6b6deca682eaa2ea59d373b3007cab"} err="failed to get container status \"7f5b6e124eab6dfe60327dd45b6622ccec6b6deca682eaa2ea59d373b3007cab\": rpc error: code = NotFound desc = could not find container \"7f5b6e124eab6dfe60327dd45b6622ccec6b6deca682eaa2ea59d373b3007cab\": container with ID starting with 7f5b6e124eab6dfe60327dd45b6622ccec6b6deca682eaa2ea59d373b3007cab not found: ID does not exist"
Jul 09 20:15:17.964291 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:17.964267    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-686f995674-gmg4v"]
Jul 09 20:15:17.967621 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:17.967599    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-686f995674-gmg4v"]
Jul 09 20:15:18.241650 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:18.241615    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c08a3bc-01b9-444f-ab63-94c20791471e" path="/var/lib/kubelet/pods/1c08a3bc-01b9-444f-ab63-94c20791471e/volumes"
Jul 09 20:15:21.702197 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:21.702118    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5dd56bd4fc-jhlq9"]
Jul 09 20:15:21.702631 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:21.702471    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c08a3bc-01b9-444f-ab63-94c20791471e" containerName="authorino"
Jul 09 20:15:21.702631 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:21.702483    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c08a3bc-01b9-444f-ab63-94c20791471e" containerName="authorino"
Jul 09 20:15:21.702631 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:21.702560    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c08a3bc-01b9-444f-ab63-94c20791471e" containerName="authorino"
Jul 09 20:15:21.721767 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:21.721736    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5dd56bd4fc-jhlq9"]
Jul 09 20:15:21.721911 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:21.721874    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5dd56bd4fc-jhlq9"
Jul 09 20:15:21.724906 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:21.724883    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 09 20:15:21.725060 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:21.725039    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 09 20:15:21.725135 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:21.725120    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-wnrwq\""
Jul 09 20:15:21.785762 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:21.785733    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9rbm\" (UniqueName: \"kubernetes.io/projected/46cf03a5-9281-4860-9a05-07438c7fe0d6-kube-api-access-t9rbm\") pod \"maas-controller-5dd56bd4fc-jhlq9\" (UID: \"46cf03a5-9281-4860-9a05-07438c7fe0d6\") " pod="opendatahub/maas-controller-5dd56bd4fc-jhlq9"
Jul 09 20:15:21.785939 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:21.785773    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/46cf03a5-9281-4860-9a05-07438c7fe0d6-webhook-cert\") pod \"maas-controller-5dd56bd4fc-jhlq9\" (UID: \"46cf03a5-9281-4860-9a05-07438c7fe0d6\") " pod="opendatahub/maas-controller-5dd56bd4fc-jhlq9"
Jul 09 20:15:21.875192 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:21.875158    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5ff76ccd7-nf8jp"]
Jul 09 20:15:21.877772 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:21.877746    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5ff76ccd7-nf8jp"
Jul 09 20:15:21.887238 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:21.887211    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9rbm\" (UniqueName: \"kubernetes.io/projected/46cf03a5-9281-4860-9a05-07438c7fe0d6-kube-api-access-t9rbm\") pod \"maas-controller-5dd56bd4fc-jhlq9\" (UID: \"46cf03a5-9281-4860-9a05-07438c7fe0d6\") " pod="opendatahub/maas-controller-5dd56bd4fc-jhlq9"
Jul 09 20:15:21.887396 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:21.887259    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/46cf03a5-9281-4860-9a05-07438c7fe0d6-webhook-cert\") pod \"maas-controller-5dd56bd4fc-jhlq9\" (UID: \"46cf03a5-9281-4860-9a05-07438c7fe0d6\") " pod="opendatahub/maas-controller-5dd56bd4fc-jhlq9"
Jul 09 20:15:21.890149 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:21.890120    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/46cf03a5-9281-4860-9a05-07438c7fe0d6-webhook-cert\") pod \"maas-controller-5dd56bd4fc-jhlq9\" (UID: \"46cf03a5-9281-4860-9a05-07438c7fe0d6\") " pod="opendatahub/maas-controller-5dd56bd4fc-jhlq9"
Jul 09 20:15:21.890321 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:21.890284    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5ff76ccd7-nf8jp"]
Jul 09 20:15:21.895730 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:21.895709    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9rbm\" (UniqueName: \"kubernetes.io/projected/46cf03a5-9281-4860-9a05-07438c7fe0d6-kube-api-access-t9rbm\") pod \"maas-controller-5dd56bd4fc-jhlq9\" (UID: \"46cf03a5-9281-4860-9a05-07438c7fe0d6\") " pod="opendatahub/maas-controller-5dd56bd4fc-jhlq9"
Jul 09 20:15:21.988286 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:21.988196    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bz4t\" (UniqueName: \"kubernetes.io/projected/fe9e3119-d39d-4455-b935-74781c0ad41e-kube-api-access-2bz4t\") pod \"maas-controller-5ff76ccd7-nf8jp\" (UID: \"fe9e3119-d39d-4455-b935-74781c0ad41e\") " pod="opendatahub/maas-controller-5ff76ccd7-nf8jp"
Jul 09 20:15:21.988286 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:21.988260    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fe9e3119-d39d-4455-b935-74781c0ad41e-webhook-cert\") pod \"maas-controller-5ff76ccd7-nf8jp\" (UID: \"fe9e3119-d39d-4455-b935-74781c0ad41e\") " pod="opendatahub/maas-controller-5ff76ccd7-nf8jp"
Jul 09 20:15:22.035516 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:22.035484    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5dd56bd4fc-jhlq9"
Jul 09 20:15:22.089283 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:22.089249    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fe9e3119-d39d-4455-b935-74781c0ad41e-webhook-cert\") pod \"maas-controller-5ff76ccd7-nf8jp\" (UID: \"fe9e3119-d39d-4455-b935-74781c0ad41e\") " pod="opendatahub/maas-controller-5ff76ccd7-nf8jp"
Jul 09 20:15:22.089446 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:22.089354    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bz4t\" (UniqueName: \"kubernetes.io/projected/fe9e3119-d39d-4455-b935-74781c0ad41e-kube-api-access-2bz4t\") pod \"maas-controller-5ff76ccd7-nf8jp\" (UID: \"fe9e3119-d39d-4455-b935-74781c0ad41e\") " pod="opendatahub/maas-controller-5ff76ccd7-nf8jp"
Jul 09 20:15:22.091711 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:22.091688    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fe9e3119-d39d-4455-b935-74781c0ad41e-webhook-cert\") pod \"maas-controller-5ff76ccd7-nf8jp\" (UID: \"fe9e3119-d39d-4455-b935-74781c0ad41e\") " pod="opendatahub/maas-controller-5ff76ccd7-nf8jp"
Jul 09 20:15:22.098434 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:22.098405    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bz4t\" (UniqueName: \"kubernetes.io/projected/fe9e3119-d39d-4455-b935-74781c0ad41e-kube-api-access-2bz4t\") pod \"maas-controller-5ff76ccd7-nf8jp\" (UID: \"fe9e3119-d39d-4455-b935-74781c0ad41e\") " pod="opendatahub/maas-controller-5ff76ccd7-nf8jp"
Jul 09 20:15:22.166717 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:22.166688    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5dd56bd4fc-jhlq9"]
Jul 09 20:15:22.168490 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:15:22.168460    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46cf03a5_9281_4860_9a05_07438c7fe0d6.slice/crio-e58ad676b4692851f33f35f659b698c3da9c59ba89f0e2b01792f492f705aaa1 WatchSource:0}: Error finding container e58ad676b4692851f33f35f659b698c3da9c59ba89f0e2b01792f492f705aaa1: Status 404 returned error can't find the container with id e58ad676b4692851f33f35f659b698c3da9c59ba89f0e2b01792f492f705aaa1
Jul 09 20:15:22.187267 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:22.187240    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5ff76ccd7-nf8jp"
Jul 09 20:15:22.319346 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:22.319319    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5ff76ccd7-nf8jp"]
Jul 09 20:15:22.321355 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:15:22.321327    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe9e3119_d39d_4455_b935_74781c0ad41e.slice/crio-3c8acebcc75493c6a584d144749e23d3eef289ecdb792e5a7d585da8bc3b214e WatchSource:0}: Error finding container 3c8acebcc75493c6a584d144749e23d3eef289ecdb792e5a7d585da8bc3b214e: Status 404 returned error can't find the container with id 3c8acebcc75493c6a584d144749e23d3eef289ecdb792e5a7d585da8bc3b214e
Jul 09 20:15:22.976588 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:22.976547    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5ff76ccd7-nf8jp" event={"ID":"fe9e3119-d39d-4455-b935-74781c0ad41e","Type":"ContainerStarted","Data":"3c8acebcc75493c6a584d144749e23d3eef289ecdb792e5a7d585da8bc3b214e"}
Jul 09 20:15:22.978776 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:22.978745    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5dd56bd4fc-jhlq9" event={"ID":"46cf03a5-9281-4860-9a05-07438c7fe0d6","Type":"ContainerStarted","Data":"e58ad676b4692851f33f35f659b698c3da9c59ba89f0e2b01792f492f705aaa1"}
Jul 09 20:15:24.997533 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:24.997496    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5ff76ccd7-nf8jp" event={"ID":"fe9e3119-d39d-4455-b935-74781c0ad41e","Type":"ContainerStarted","Data":"a46552a49cd02d82477d8503ff8c72e8f992c386cfe6752f090ed470f9417720"}
Jul 09 20:15:24.997974 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:24.997616    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5ff76ccd7-nf8jp"
Jul 09 20:15:24.999428 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:24.999362    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5dd56bd4fc-jhlq9" event={"ID":"46cf03a5-9281-4860-9a05-07438c7fe0d6","Type":"ContainerStarted","Data":"848663073d54536ea1e22413eed1d6a520a3661650215e16999f03aa33d23097"}
Jul 09 20:15:24.999550 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:24.999505    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5dd56bd4fc-jhlq9"
Jul 09 20:15:25.020316 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:25.020238    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5ff76ccd7-nf8jp" podStartSLOduration=1.586899251 podStartE2EDuration="4.020221475s" totalImagesPullingTime="2.433322224s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 20:15:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 20:15:25.017661996 +0000 UTC m=+715.435331860" watchObservedRunningTime="2026-07-09 20:15:25.020221475 +0000 UTC m=+715.437891283"
Jul 09 20:15:25.037882 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:25.037826    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5dd56bd4fc-jhlq9" podStartSLOduration=1.457312723 podStartE2EDuration="4.037809837s" totalImagesPullingTime="2.580497114s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 20:15:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 20:15:25.036247671 +0000 UTC m=+715.453917493" watchObservedRunningTime="2026-07-09 20:15:25.037809837 +0000 UTC m=+715.455479645"
Jul 09 20:15:36.011092 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:36.011062    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5dd56bd4fc-jhlq9"
Jul 09 20:15:36.011573 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:36.011373    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5ff76ccd7-nf8jp"
Jul 09 20:15:36.067244 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:36.067206    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5dd56bd4fc-jhlq9"]
Jul 09 20:15:36.067468 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:36.067442    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-5dd56bd4fc-jhlq9" podUID="46cf03a5-9281-4860-9a05-07438c7fe0d6" containerName="manager" containerID="cri-o://848663073d54536ea1e22413eed1d6a520a3661650215e16999f03aa33d23097" gracePeriod=30
Jul 09 20:15:36.314695 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:36.314670    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5dd56bd4fc-jhlq9"
Jul 09 20:15:36.415601 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:36.415563    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/46cf03a5-9281-4860-9a05-07438c7fe0d6-webhook-cert\") pod \"46cf03a5-9281-4860-9a05-07438c7fe0d6\" (UID: \"46cf03a5-9281-4860-9a05-07438c7fe0d6\") "
Jul 09 20:15:36.415775 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:36.415617    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t9rbm\" (UniqueName: \"kubernetes.io/projected/46cf03a5-9281-4860-9a05-07438c7fe0d6-kube-api-access-t9rbm\") pod \"46cf03a5-9281-4860-9a05-07438c7fe0d6\" (UID: \"46cf03a5-9281-4860-9a05-07438c7fe0d6\") "
Jul 09 20:15:36.417942 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:36.417912    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46cf03a5-9281-4860-9a05-07438c7fe0d6-kube-api-access-t9rbm" (OuterVolumeSpecName: "kube-api-access-t9rbm") pod "46cf03a5-9281-4860-9a05-07438c7fe0d6" (UID: "46cf03a5-9281-4860-9a05-07438c7fe0d6"). InnerVolumeSpecName "kube-api-access-t9rbm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 20:15:36.418047 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:36.417913    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46cf03a5-9281-4860-9a05-07438c7fe0d6-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "46cf03a5-9281-4860-9a05-07438c7fe0d6" (UID: "46cf03a5-9281-4860-9a05-07438c7fe0d6"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 20:15:36.516558 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:36.516522    2562 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/46cf03a5-9281-4860-9a05-07438c7fe0d6-webhook-cert\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:15:36.516558 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:36.516551    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9rbm\" (UniqueName: \"kubernetes.io/projected/46cf03a5-9281-4860-9a05-07438c7fe0d6-kube-api-access-t9rbm\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:15:37.061587 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:37.061554    2562 generic.go:356] "Generic (PLEG): container finished" podID="46cf03a5-9281-4860-9a05-07438c7fe0d6" containerID="848663073d54536ea1e22413eed1d6a520a3661650215e16999f03aa33d23097" exitCode=0
Jul 09 20:15:37.062110 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:37.061615    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5dd56bd4fc-jhlq9"
Jul 09 20:15:37.062110 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:37.061627    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5dd56bd4fc-jhlq9" event={"ID":"46cf03a5-9281-4860-9a05-07438c7fe0d6","Type":"ContainerDied","Data":"848663073d54536ea1e22413eed1d6a520a3661650215e16999f03aa33d23097"}
Jul 09 20:15:37.062110 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:37.061668    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5dd56bd4fc-jhlq9" event={"ID":"46cf03a5-9281-4860-9a05-07438c7fe0d6","Type":"ContainerDied","Data":"e58ad676b4692851f33f35f659b698c3da9c59ba89f0e2b01792f492f705aaa1"}
Jul 09 20:15:37.062110 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:37.061692    2562 scope.go:117] "RemoveContainer" containerID="848663073d54536ea1e22413eed1d6a520a3661650215e16999f03aa33d23097"
Jul 09 20:15:37.076026 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:37.075896    2562 scope.go:117] "RemoveContainer" containerID="848663073d54536ea1e22413eed1d6a520a3661650215e16999f03aa33d23097"
Jul 09 20:15:37.076281 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:15:37.076255    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"848663073d54536ea1e22413eed1d6a520a3661650215e16999f03aa33d23097\": container with ID starting with 848663073d54536ea1e22413eed1d6a520a3661650215e16999f03aa33d23097 not found: ID does not exist" containerID="848663073d54536ea1e22413eed1d6a520a3661650215e16999f03aa33d23097"
Jul 09 20:15:37.076378 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:37.076294    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"848663073d54536ea1e22413eed1d6a520a3661650215e16999f03aa33d23097"} err="failed to get container status \"848663073d54536ea1e22413eed1d6a520a3661650215e16999f03aa33d23097\": rpc error: code = NotFound desc = could not find container \"848663073d54536ea1e22413eed1d6a520a3661650215e16999f03aa33d23097\": container with ID starting with 848663073d54536ea1e22413eed1d6a520a3661650215e16999f03aa33d23097 not found: ID does not exist"
Jul 09 20:15:37.086629 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:37.086588    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5dd56bd4fc-jhlq9"]
Jul 09 20:15:37.088014 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:37.087993    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5dd56bd4fc-jhlq9"]
Jul 09 20:15:38.241526 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:38.241495    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="46cf03a5-9281-4860-9a05-07438c7fe0d6" path="/var/lib/kubelet/pods/46cf03a5-9281-4860-9a05-07438c7fe0d6/volumes"
Jul 09 20:15:58.088020 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:58.087979    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6bc8548564-gvhbr"]
Jul 09 20:15:58.088498 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:58.088483    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46cf03a5-9281-4860-9a05-07438c7fe0d6" containerName="manager"
Jul 09 20:15:58.088543 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:58.088501    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="46cf03a5-9281-4860-9a05-07438c7fe0d6" containerName="manager"
Jul 09 20:15:58.088610 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:58.088600    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="46cf03a5-9281-4860-9a05-07438c7fe0d6" containerName="manager"
Jul 09 20:15:58.094994 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:58.094963    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6bc8548564-gvhbr"
Jul 09 20:15:58.098085 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:58.098060    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Jul 09 20:15:58.098540 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:58.098517    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-qk7hz\""
Jul 09 20:15:58.102241 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:58.102218    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6bc8548564-gvhbr"]
Jul 09 20:15:58.199839 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:58.199807    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dbf505e3-ecfb-4dc4-92f7-78f52be1c12f-maas-api-tls\") pod \"maas-api-6bc8548564-gvhbr\" (UID: \"dbf505e3-ecfb-4dc4-92f7-78f52be1c12f\") " pod="odh-ai-gateway-infra/maas-api-6bc8548564-gvhbr"
Jul 09 20:15:58.200018 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:58.199849    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ln95\" (UniqueName: \"kubernetes.io/projected/dbf505e3-ecfb-4dc4-92f7-78f52be1c12f-kube-api-access-8ln95\") pod \"maas-api-6bc8548564-gvhbr\" (UID: \"dbf505e3-ecfb-4dc4-92f7-78f52be1c12f\") " pod="odh-ai-gateway-infra/maas-api-6bc8548564-gvhbr"
Jul 09 20:15:58.200018 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:58.199933    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbf505e3-ecfb-4dc4-92f7-78f52be1c12f-openshift-service-ca\") pod \"maas-api-6bc8548564-gvhbr\" (UID: \"dbf505e3-ecfb-4dc4-92f7-78f52be1c12f\") " pod="odh-ai-gateway-infra/maas-api-6bc8548564-gvhbr"
Jul 09 20:15:58.301090 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:58.301046    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbf505e3-ecfb-4dc4-92f7-78f52be1c12f-openshift-service-ca\") pod \"maas-api-6bc8548564-gvhbr\" (UID: \"dbf505e3-ecfb-4dc4-92f7-78f52be1c12f\") " pod="odh-ai-gateway-infra/maas-api-6bc8548564-gvhbr"
Jul 09 20:15:58.301295 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:58.301107    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dbf505e3-ecfb-4dc4-92f7-78f52be1c12f-maas-api-tls\") pod \"maas-api-6bc8548564-gvhbr\" (UID: \"dbf505e3-ecfb-4dc4-92f7-78f52be1c12f\") " pod="odh-ai-gateway-infra/maas-api-6bc8548564-gvhbr"
Jul 09 20:15:58.301295 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:58.301139    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ln95\" (UniqueName: \"kubernetes.io/projected/dbf505e3-ecfb-4dc4-92f7-78f52be1c12f-kube-api-access-8ln95\") pod \"maas-api-6bc8548564-gvhbr\" (UID: \"dbf505e3-ecfb-4dc4-92f7-78f52be1c12f\") " pod="odh-ai-gateway-infra/maas-api-6bc8548564-gvhbr"
Jul 09 20:15:58.301875 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:58.301851    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbf505e3-ecfb-4dc4-92f7-78f52be1c12f-openshift-service-ca\") pod \"maas-api-6bc8548564-gvhbr\" (UID: \"dbf505e3-ecfb-4dc4-92f7-78f52be1c12f\") " pod="odh-ai-gateway-infra/maas-api-6bc8548564-gvhbr"
Jul 09 20:15:58.303608 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:58.303588    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dbf505e3-ecfb-4dc4-92f7-78f52be1c12f-maas-api-tls\") pod \"maas-api-6bc8548564-gvhbr\" (UID: \"dbf505e3-ecfb-4dc4-92f7-78f52be1c12f\") " pod="odh-ai-gateway-infra/maas-api-6bc8548564-gvhbr"
Jul 09 20:15:58.310079 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:58.310057    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ln95\" (UniqueName: \"kubernetes.io/projected/dbf505e3-ecfb-4dc4-92f7-78f52be1c12f-kube-api-access-8ln95\") pod \"maas-api-6bc8548564-gvhbr\" (UID: \"dbf505e3-ecfb-4dc4-92f7-78f52be1c12f\") " pod="odh-ai-gateway-infra/maas-api-6bc8548564-gvhbr"
Jul 09 20:15:58.408496 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:58.408418    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6bc8548564-gvhbr"
Jul 09 20:15:58.585255 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:58.585228    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6bc8548564-gvhbr"]
Jul 09 20:15:58.588037 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:15:58.588011    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbf505e3_ecfb_4dc4_92f7_78f52be1c12f.slice/crio-aa32dd4a52d686b716b958a9320da56058984521560536eec82e8d117f91ea70 WatchSource:0}: Error finding container aa32dd4a52d686b716b958a9320da56058984521560536eec82e8d117f91ea70: Status 404 returned error can't find the container with id aa32dd4a52d686b716b958a9320da56058984521560536eec82e8d117f91ea70
Jul 09 20:15:58.589887 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:58.589863    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 09 20:15:58.641840 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:58.641801    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-jfpjl"]
Jul 09 20:15:58.644871 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:58.644850    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7964457696-jfpjl"
Jul 09 20:15:58.647754 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:58.647730    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Jul 09 20:15:58.647754 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:58.647750    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-j2pqg\""
Jul 09 20:15:58.654523 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:58.654501    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-jfpjl"]
Jul 09 20:15:58.669119 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:58.669036    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-8xb6m"]
Jul 09 20:15:58.671338 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:58.671291    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58f86c7798-8xb6m"
Jul 09 20:15:58.682361 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:58.682335    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-8xb6m"]
Jul 09 20:15:58.704853 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:58.704824    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2548ca62-2892-4717-869c-c78ed5445438-plugins-config-volume\") pod \"payload-processing-58f86c7798-8xb6m\" (UID: \"2548ca62-2892-4717-869c-c78ed5445438\") " pod="openshift-ingress/payload-processing-58f86c7798-8xb6m"
Jul 09 20:15:58.704996 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:58.704865    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3da2fa45-fd15-4a70-ba90-9ed26ea33d79-plugins-config-volume\") pod \"payload-pre-processing-7964457696-jfpjl\" (UID: \"3da2fa45-fd15-4a70-ba90-9ed26ea33d79\") " pod="openshift-ingress/payload-pre-processing-7964457696-jfpjl"
Jul 09 20:15:58.704996 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:58.704961    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wkhd\" (UniqueName: \"kubernetes.io/projected/3da2fa45-fd15-4a70-ba90-9ed26ea33d79-kube-api-access-5wkhd\") pod \"payload-pre-processing-7964457696-jfpjl\" (UID: \"3da2fa45-fd15-4a70-ba90-9ed26ea33d79\") " pod="openshift-ingress/payload-pre-processing-7964457696-jfpjl"
Jul 09 20:15:58.704996 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:58.704989    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4t55z\" (UniqueName: \"kubernetes.io/projected/2548ca62-2892-4717-869c-c78ed5445438-kube-api-access-4t55z\") pod \"payload-processing-58f86c7798-8xb6m\" (UID: \"2548ca62-2892-4717-869c-c78ed5445438\") " pod="openshift-ingress/payload-processing-58f86c7798-8xb6m"
Jul 09 20:15:58.805674 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:58.805642    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wkhd\" (UniqueName: \"kubernetes.io/projected/3da2fa45-fd15-4a70-ba90-9ed26ea33d79-kube-api-access-5wkhd\") pod \"payload-pre-processing-7964457696-jfpjl\" (UID: \"3da2fa45-fd15-4a70-ba90-9ed26ea33d79\") " pod="openshift-ingress/payload-pre-processing-7964457696-jfpjl"
Jul 09 20:15:58.805865 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:58.805678    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4t55z\" (UniqueName: \"kubernetes.io/projected/2548ca62-2892-4717-869c-c78ed5445438-kube-api-access-4t55z\") pod \"payload-processing-58f86c7798-8xb6m\" (UID: \"2548ca62-2892-4717-869c-c78ed5445438\") " pod="openshift-ingress/payload-processing-58f86c7798-8xb6m"
Jul 09 20:15:58.805865 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:58.805744    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2548ca62-2892-4717-869c-c78ed5445438-plugins-config-volume\") pod \"payload-processing-58f86c7798-8xb6m\" (UID: \"2548ca62-2892-4717-869c-c78ed5445438\") " pod="openshift-ingress/payload-processing-58f86c7798-8xb6m"
Jul 09 20:15:58.805865 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:58.805765    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3da2fa45-fd15-4a70-ba90-9ed26ea33d79-plugins-config-volume\") pod \"payload-pre-processing-7964457696-jfpjl\" (UID: \"3da2fa45-fd15-4a70-ba90-9ed26ea33d79\") " pod="openshift-ingress/payload-pre-processing-7964457696-jfpjl"
Jul 09 20:15:58.806703 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:58.806678    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3da2fa45-fd15-4a70-ba90-9ed26ea33d79-plugins-config-volume\") pod \"payload-pre-processing-7964457696-jfpjl\" (UID: \"3da2fa45-fd15-4a70-ba90-9ed26ea33d79\") " pod="openshift-ingress/payload-pre-processing-7964457696-jfpjl"
Jul 09 20:15:58.806764 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:58.806688    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2548ca62-2892-4717-869c-c78ed5445438-plugins-config-volume\") pod \"payload-processing-58f86c7798-8xb6m\" (UID: \"2548ca62-2892-4717-869c-c78ed5445438\") " pod="openshift-ingress/payload-processing-58f86c7798-8xb6m"
Jul 09 20:15:58.814286 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:58.814257    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4t55z\" (UniqueName: \"kubernetes.io/projected/2548ca62-2892-4717-869c-c78ed5445438-kube-api-access-4t55z\") pod \"payload-processing-58f86c7798-8xb6m\" (UID: \"2548ca62-2892-4717-869c-c78ed5445438\") " pod="openshift-ingress/payload-processing-58f86c7798-8xb6m"
Jul 09 20:15:58.814415 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:58.814361    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wkhd\" (UniqueName: \"kubernetes.io/projected/3da2fa45-fd15-4a70-ba90-9ed26ea33d79-kube-api-access-5wkhd\") pod \"payload-pre-processing-7964457696-jfpjl\" (UID: \"3da2fa45-fd15-4a70-ba90-9ed26ea33d79\") " pod="openshift-ingress/payload-pre-processing-7964457696-jfpjl"
Jul 09 20:15:58.956479 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:58.956448    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7964457696-jfpjl"
Jul 09 20:15:58.982456 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:58.982424    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58f86c7798-8xb6m"
Jul 09 20:15:59.102970 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:59.102934    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-jfpjl"]
Jul 09 20:15:59.103910 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:15:59.103878    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3da2fa45_fd15_4a70_ba90_9ed26ea33d79.slice/crio-3a61bb5940c970eb84a26b7132d19b075d9de208b29047e73dbc700ac54da7ba WatchSource:0}: Error finding container 3a61bb5940c970eb84a26b7132d19b075d9de208b29047e73dbc700ac54da7ba: Status 404 returned error can't find the container with id 3a61bb5940c970eb84a26b7132d19b075d9de208b29047e73dbc700ac54da7ba
Jul 09 20:15:59.128081 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:59.128052    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-8xb6m"]
Jul 09 20:15:59.131046 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:15:59.131016    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2548ca62_2892_4717_869c_c78ed5445438.slice/crio-a360271bd63cd3229ab517b2e1e662f9608bb51b05dda21303286c6a3b71ce9e WatchSource:0}: Error finding container a360271bd63cd3229ab517b2e1e662f9608bb51b05dda21303286c6a3b71ce9e: Status 404 returned error can't find the container with id a360271bd63cd3229ab517b2e1e662f9608bb51b05dda21303286c6a3b71ce9e
Jul 09 20:15:59.190332 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:59.188663    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58f86c7798-8xb6m" event={"ID":"2548ca62-2892-4717-869c-c78ed5445438","Type":"ContainerStarted","Data":"a360271bd63cd3229ab517b2e1e662f9608bb51b05dda21303286c6a3b71ce9e"}
Jul 09 20:15:59.193845 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:59.193810    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7964457696-jfpjl" event={"ID":"3da2fa45-fd15-4a70-ba90-9ed26ea33d79","Type":"ContainerStarted","Data":"3a61bb5940c970eb84a26b7132d19b075d9de208b29047e73dbc700ac54da7ba"}
Jul 09 20:15:59.195333 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:15:59.195280    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6bc8548564-gvhbr" event={"ID":"dbf505e3-ecfb-4dc4-92f7-78f52be1c12f","Type":"ContainerStarted","Data":"aa32dd4a52d686b716b958a9320da56058984521560536eec82e8d117f91ea70"}
Jul 09 20:16:02.215398 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:16:02.215348    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7964457696-jfpjl" event={"ID":"3da2fa45-fd15-4a70-ba90-9ed26ea33d79","Type":"ContainerStarted","Data":"1a32ce0b96d489c28b26731be5dfd890eb4e0fba27e1c5c2343d57d600e583a1"}
Jul 09 20:16:02.216232 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:16:02.216203    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7964457696-jfpjl"
Jul 09 20:16:02.218193 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:16:02.218159    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6bc8548564-gvhbr" event={"ID":"dbf505e3-ecfb-4dc4-92f7-78f52be1c12f","Type":"ContainerStarted","Data":"1f780b3c89af7a0c5f095483587c14983b3b62746ef396bebfc6e63390e3c070"}
Jul 09 20:16:02.218709 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:16:02.218683    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6bc8548564-gvhbr"
Jul 09 20:16:02.220632 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:16:02.220601    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58f86c7798-8xb6m" event={"ID":"2548ca62-2892-4717-869c-c78ed5445438","Type":"ContainerStarted","Data":"57605d3d32732e00bac3f0fd17ec66095ae43989cecf15713c4e31137f51f438"}
Jul 09 20:16:02.220730 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:16:02.220673    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-58f86c7798-8xb6m"
Jul 09 20:16:02.234249 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:16:02.234181    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7964457696-jfpjl" podStartSLOduration=2.181902739 podStartE2EDuration="4.234161906s" totalImagesPullingTime="2.052259167s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 20:15:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 20:16:02.23181734 +0000 UTC m=+752.649487150" watchObservedRunningTime="2026-07-09 20:16:02.234161906 +0000 UTC m=+752.651831715"
Jul 09 20:16:02.250173 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:16:02.250129    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6bc8548564-gvhbr" podStartSLOduration=1.681444564 podStartE2EDuration="4.250115559s" totalImagesPullingTime="2.568670995s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 20:15:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 20:16:02.249187165 +0000 UTC m=+752.666856974" watchObservedRunningTime="2026-07-09 20:16:02.250115559 +0000 UTC m=+752.667785368"
Jul 09 20:16:02.266244 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:16:02.266192    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-58f86c7798-8xb6m" podStartSLOduration=2.2376707160000002 podStartE2EDuration="4.266176638s" totalImagesPullingTime="2.028505922s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 20:15:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 20:16:02.265462561 +0000 UTC m=+752.683132371" watchObservedRunningTime="2026-07-09 20:16:02.266176638 +0000 UTC m=+752.683846489"
Jul 09 20:16:06.243823 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:16:06.243789    2562 generic.go:356] "Generic (PLEG): container finished" podID="5fa2660d-7348-4743-9447-c477bb97a722" containerID="b8790a3773483366bfdb0cfefcdd97c64c4f965bbdcbce6f95cd14a495d9ae9a" exitCode=0
Jul 09 20:16:06.244245 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:16:06.243863    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-phr4z" event={"ID":"5fa2660d-7348-4743-9447-c477bb97a722","Type":"ContainerDied","Data":"b8790a3773483366bfdb0cfefcdd97c64c4f965bbdcbce6f95cd14a495d9ae9a"}
Jul 09 20:16:06.244290 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:16:06.244270    2562 scope.go:117] "RemoveContainer" containerID="b8790a3773483366bfdb0cfefcdd97c64c4f965bbdcbce6f95cd14a495d9ae9a"
Jul 09 20:16:06.245279 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:16:06.245253    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 09 20:16:06.245356 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:16:06.245338    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 09 20:16:06.245397 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:16:06.245374    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 09 20:16:07.250732 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:16:07.250700    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-phr4z" event={"ID":"5fa2660d-7348-4743-9447-c477bb97a722","Type":"ContainerStarted","Data":"e53f8aaefc9549ab16f735f3eb84872059a5056655ba2cfe0b39387256ceadc9"}
Jul 09 20:16:07.312443 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:16:07.312004    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-phr4z"
Jul 09 20:16:07.312635 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:16:07.312396    2562 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-phr4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Jul 09 20:16:07.312635 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:16:07.312526    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-phr4z" podUID="5fa2660d-7348-4743-9447-c477bb97a722" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Jul 09 20:16:08.312445 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:16:08.312400    2562 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-phr4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Jul 09 20:16:08.312946 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:16:08.312469    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-phr4z" podUID="5fa2660d-7348-4743-9447-c477bb97a722" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Jul 09 20:16:09.240026 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:16:09.239996    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6bc8548564-gvhbr"
Jul 09 20:16:09.328329 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:16:09.324703    2562 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-phr4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Jul 09 20:16:09.328329 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:16:09.324766    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-phr4z" podUID="5fa2660d-7348-4743-9447-c477bb97a722" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Jul 09 20:16:10.324329 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:16:10.311984    2562 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-phr4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Jul 09 20:16:10.324329 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:16:10.312061    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-phr4z" podUID="5fa2660d-7348-4743-9447-c477bb97a722" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Jul 09 20:16:11.315662 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:16:11.315632    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-phr4z"
Jul 09 20:16:11.316065 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:16:11.315913    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-phr4z"
Jul 09 20:16:11.316678 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:16:11.316663    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-phr4z"
Jul 09 20:16:13.230646 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:16:13.230610    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-58f86c7798-8xb6m"
Jul 09 20:16:14.235295 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:16:14.235263    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7964457696-jfpjl"
Jul 09 20:16:16.308387 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:16:16.308349    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5wbtg"]
Jul 09 20:16:16.311611 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:16:16.311591    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5wbtg"
Jul 09 20:16:16.314565 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:16:16.314542    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Jul 09 20:16:16.314702 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:16:16.314654    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-2qhc4\""
Jul 09 20:16:16.316190 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:16:16.316147    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 09 20:16:16.316540 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:16:16.316198    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 09 20:16:16.319033 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:16:16.319007    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5wbtg"]
Jul 09 20:16:16.399712 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:16:16.399671    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1343760b-26ed-44de-96f5-87bb0e9b67f9-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5wbtg\" (UID: \"1343760b-26ed-44de-96f5-87bb0e9b67f9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5wbtg"
Jul 09 20:16:16.399875 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:16:16.399727    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1343760b-26ed-44de-96f5-87bb0e9b67f9-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5wbtg\" (UID: \"1343760b-26ed-44de-96f5-87bb0e9b67f9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5wbtg"
Jul 09 20:16:16.399875 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:16:16.399794    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1343760b-26ed-44de-96f5-87bb0e9b67f9-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5wbtg\" (UID: \"1343760b-26ed-44de-96f5-87bb0e9b67f9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5wbtg"
Jul 09 20:16:16.399875 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:16:16.399840    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8vq6\" (UniqueName: \"kubernetes.io/projected/1343760b-26ed-44de-96f5-87bb0e9b67f9-kube-api-access-z8vq6\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5wbtg\" (UID: \"1343760b-26ed-44de-96f5-87bb0e9b67f9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5wbtg"
Jul 09 20:16:16.399980 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:16:16.399888    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1343760b-26ed-44de-96f5-87bb0e9b67f9-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5wbtg\" (UID: \"1343760b-26ed-44de-96f5-87bb0e9b67f9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5wbtg"
Jul 09 20:16:16.399980 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:16:16.399960    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1343760b-26ed-44de-96f5-87bb0e9b67f9-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5wbtg\" (UID: \"1343760b-26ed-44de-96f5-87bb0e9b67f9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5wbtg"
Jul 09 20:16:16.501352 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:16:16.501290    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8vq6\" (UniqueName: \"kubernetes.io/projected/1343760b-26ed-44de-96f5-87bb0e9b67f9-kube-api-access-z8vq6\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5wbtg\" (UID: \"1343760b-26ed-44de-96f5-87bb0e9b67f9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5wbtg"
Jul 09 20:16:16.501515 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:16:16.501383    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1343760b-26ed-44de-96f5-87bb0e9b67f9-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5wbtg\" (UID: \"1343760b-26ed-44de-96f5-87bb0e9b67f9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5wbtg"
Jul 09 20:16:16.501515 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:16:16.501470    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1343760b-26ed-44de-96f5-87bb0e9b67f9-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5wbtg\" (UID: \"1343760b-26ed-44de-96f5-87bb0e9b67f9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5wbtg"
Jul 09 20:16:16.501515 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:16:16.501498    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1343760b-26ed-44de-96f5-87bb0e9b67f9-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5wbtg\" (UID: \"1343760b-26ed-44de-96f5-87bb0e9b67f9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5wbtg"
Jul 09 20:16:16.501655 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:16:16.501546    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1343760b-26ed-44de-96f5-87bb0e9b67f9-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5wbtg\" (UID: \"1343760b-26ed-44de-96f5-87bb0e9b67f9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5wbtg"
Jul 09 20:16:16.501655 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:16:16.501572    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1343760b-26ed-44de-96f5-87bb0e9b67f9-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5wbtg\" (UID: \"1343760b-26ed-44de-96f5-87bb0e9b67f9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5wbtg"
Jul 09 20:16:16.501973 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:16:16.501951    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1343760b-26ed-44de-96f5-87bb0e9b67f9-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5wbtg\" (UID: \"1343760b-26ed-44de-96f5-87bb0e9b67f9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5wbtg"
Jul 09 20:16:16.502054 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:16:16.502028    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1343760b-26ed-44de-96f5-87bb0e9b67f9-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5wbtg\" (UID: \"1343760b-26ed-44de-96f5-87bb0e9b67f9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5wbtg"
Jul 09 20:16:16.502112 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:16:16.502072    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1343760b-26ed-44de-96f5-87bb0e9b67f9-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5wbtg\" (UID: \"1343760b-26ed-44de-96f5-87bb0e9b67f9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5wbtg"
Jul 09 20:16:16.503709 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:16:16.503682    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1343760b-26ed-44de-96f5-87bb0e9b67f9-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5wbtg\" (UID: \"1343760b-26ed-44de-96f5-87bb0e9b67f9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5wbtg"
Jul 09 20:16:16.503956 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:16:16.503934    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1343760b-26ed-44de-96f5-87bb0e9b67f9-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5wbtg\" (UID: \"1343760b-26ed-44de-96f5-87bb0e9b67f9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5wbtg"
Jul 09 20:16:16.509024 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:16:16.509005    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8vq6\" (UniqueName: \"kubernetes.io/projected/1343760b-26ed-44de-96f5-87bb0e9b67f9-kube-api-access-z8vq6\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5wbtg\" (UID: \"1343760b-26ed-44de-96f5-87bb0e9b67f9\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5wbtg"
Jul 09 20:16:16.625228 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:16:16.625115    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5wbtg"
Jul 09 20:16:16.762586 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:16:16.762511    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5wbtg"]
Jul 09 20:16:16.765234 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:16:16.765204    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1343760b_26ed_44de_96f5_87bb0e9b67f9.slice/crio-4176d8e677beeb1f3c8d604a0be67d5d1869893e1acac4a87887793da6799635 WatchSource:0}: Error finding container 4176d8e677beeb1f3c8d604a0be67d5d1869893e1acac4a87887793da6799635: Status 404 returned error can't find the container with id 4176d8e677beeb1f3c8d604a0be67d5d1869893e1acac4a87887793da6799635
Jul 09 20:16:17.307662 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:16:17.307591    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5wbtg" event={"ID":"1343760b-26ed-44de-96f5-87bb0e9b67f9","Type":"ContainerStarted","Data":"4176d8e677beeb1f3c8d604a0be67d5d1869893e1acac4a87887793da6799635"}
Jul 09 20:16:18.314169 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:16:18.314075    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5wbtg" event={"ID":"1343760b-26ed-44de-96f5-87bb0e9b67f9","Type":"ContainerStarted","Data":"4d139478814f7dda2b6094147ecb60518c480155fc2c811c01d593933ee8b541"}
Jul 09 20:16:18.333818 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:16:18.333770    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5wbtg" podStartSLOduration=1.1336031069999999 podStartE2EDuration="2.33375449s" totalImagesPullingTime="1.200151383s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 20:16:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 20:16:18.332738708 +0000 UTC m=+768.750408518" watchObservedRunningTime="2026-07-09 20:16:18.33375449 +0000 UTC m=+768.751424298"
Jul 09 20:16:26.626314 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:16:26.626266    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5wbtg"
Jul 09 20:16:26.626715 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:16:26.626391    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5wbtg"
Jul 09 20:16:26.638456 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:16:26.638431    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5wbtg"
Jul 09 20:16:27.006159 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:16:27.006118    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-x455w"]
Jul 09 20:16:27.008780 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:16:27.008760    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-x455w"
Jul 09 20:16:27.011698 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:16:27.011676    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Jul 09 20:16:27.018665 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:16:27.018641    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-x455w"]
Jul 09 20:16:27.097121 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:16:27.097084    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b0b703d-1f9a-4487-b9cd-8ea2dddb4ee5-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-x455w\" (UID: \"3b0b703d-1f9a-4487-b9cd-8ea2dddb4ee5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-x455w"
Jul 09 20:16:27.097294 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:16:27.097209    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b0b703d-1f9a-4487-b9cd-8ea2dddb4ee5-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-x455w\" (UID: \"3b0b703d-1f9a-4487-b9cd-8ea2dddb4ee5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-x455w"
Jul 09 20:16:27.097294 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:16:27.097258    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c42kp\" (UniqueName: \"kubernetes.io/projected/3b0b703d-1f9a-4487-b9cd-8ea2dddb4ee5-kube-api-access-c42kp\") pod \"e2e-distinct-simulated-kserve-59975d69f9-x455w\" (UID: \"3b0b703d-1f9a-4487-b9cd-8ea2dddb4ee5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-x455w"
Jul 09 20:16:27.097425 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:16:27.097363    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b0b703d-1f9a-4487-b9cd-8ea2dddb4ee5-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-x455w\" (UID: \"3b0b703d-1f9a-4487-b9cd-8ea2dddb4ee5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-x455w"
Jul 09 20:16:27.097425 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:16:27.097394    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b0b703d-1f9a-4487-b9cd-8ea2dddb4ee5-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-x455w\" (UID: \"3b0b703d-1f9a-4487-b9cd-8ea2dddb4ee5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-x455w"
Jul 09 20:16:27.097425 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:16:27.097409    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b0b703d-1f9a-4487-b9cd-8ea2dddb4ee5-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-x455w\" (UID: \"3b0b703d-1f9a-4487-b9cd-8ea2dddb4ee5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-x455w"
Jul 09 20:16:27.198618 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:16:27.198576    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b0b703d-1f9a-4487-b9cd-8ea2dddb4ee5-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-x455w\" (UID: \"3b0b703d-1f9a-4487-b9cd-8ea2dddb4ee5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-x455w"
Jul 09 20:16:27.198802 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:16:27.198646    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b0b703d-1f9a-4487-b9cd-8ea2dddb4ee5-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-x455w\" (UID: \"3b0b703d-1f9a-4487-b9cd-8ea2dddb4ee5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-x455w"
Jul 09 20:16:27.198802 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:16:27.198669    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b0b703d-1f9a-4487-b9cd-8ea2dddb4ee5-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-x455w\" (UID: \"3b0b703d-1f9a-4487-b9cd-8ea2dddb4ee5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-x455w"
Jul 09 20:16:27.198802 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:16:27.198732    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b0b703d-1f9a-4487-b9cd-8ea2dddb4ee5-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-x455w\" (UID: \"3b0b703d-1f9a-4487-b9cd-8ea2dddb4ee5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-x455w"
Jul 09 20:16:27.198802 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:16:27.198799    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b0b703d-1f9a-4487-b9cd-8ea2dddb4ee5-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-x455w\" (UID: \"3b0b703d-1f9a-4487-b9cd-8ea2dddb4ee5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-x455w"
Jul 09 20:16:27.198996 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:16:27.198830    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c42kp\" (UniqueName: \"kubernetes.io/projected/3b0b703d-1f9a-4487-b9cd-8ea2dddb4ee5-kube-api-access-c42kp\") pod \"e2e-distinct-simulated-kserve-59975d69f9-x455w\" (UID: \"3b0b703d-1f9a-4487-b9cd-8ea2dddb4ee5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-x455w"
Jul 09 20:16:27.199155 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:16:27.199127    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b0b703d-1f9a-4487-b9cd-8ea2dddb4ee5-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-x455w\" (UID: \"3b0b703d-1f9a-4487-b9cd-8ea2dddb4ee5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-x455w"
Jul 09 20:16:27.199155 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:16:27.199144    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b0b703d-1f9a-4487-b9cd-8ea2dddb4ee5-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-x455w\" (UID: \"3b0b703d-1f9a-4487-b9cd-8ea2dddb4ee5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-x455w"
Jul 09 20:16:27.199407 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:16:27.199382    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b0b703d-1f9a-4487-b9cd-8ea2dddb4ee5-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-x455w\" (UID: \"3b0b703d-1f9a-4487-b9cd-8ea2dddb4ee5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-x455w"
Jul 09 20:16:27.200998 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:16:27.200976    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b0b703d-1f9a-4487-b9cd-8ea2dddb4ee5-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-x455w\" (UID: \"3b0b703d-1f9a-4487-b9cd-8ea2dddb4ee5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-x455w"
Jul 09 20:16:27.201282 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:16:27.201263    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b0b703d-1f9a-4487-b9cd-8ea2dddb4ee5-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-x455w\" (UID: \"3b0b703d-1f9a-4487-b9cd-8ea2dddb4ee5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-x455w"
Jul 09 20:16:27.206742 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:16:27.206719    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c42kp\" (UniqueName: \"kubernetes.io/projected/3b0b703d-1f9a-4487-b9cd-8ea2dddb4ee5-kube-api-access-c42kp\") pod \"e2e-distinct-simulated-kserve-59975d69f9-x455w\" (UID: \"3b0b703d-1f9a-4487-b9cd-8ea2dddb4ee5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-x455w"
Jul 09 20:16:27.320023 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:16:27.319918    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-x455w"
Jul 09 20:16:27.373038 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:16:27.373001    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5wbtg"
Jul 09 20:16:27.461642 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:16:27.461615    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-x455w"]
Jul 09 20:16:27.463141 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:16:27.463107    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b0b703d_1f9a_4487_b9cd_8ea2dddb4ee5.slice/crio-e6286a693b7f702e833e0ff587cc5221cc0d8fb09cfbc9f50747f6621133ed1e WatchSource:0}: Error finding container e6286a693b7f702e833e0ff587cc5221cc0d8fb09cfbc9f50747f6621133ed1e: Status 404 returned error can't find the container with id e6286a693b7f702e833e0ff587cc5221cc0d8fb09cfbc9f50747f6621133ed1e
Jul 09 20:16:28.367486 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:16:28.367446    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-x455w" event={"ID":"3b0b703d-1f9a-4487-b9cd-8ea2dddb4ee5","Type":"ContainerStarted","Data":"9bdc2a0c3a43a4402d701f684b44e5909f982540b7f2dedb4bdeea5691159da0"}
Jul 09 20:16:28.367980 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:16:28.367504    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-x455w" event={"ID":"3b0b703d-1f9a-4487-b9cd-8ea2dddb4ee5","Type":"ContainerStarted","Data":"e6286a693b7f702e833e0ff587cc5221cc0d8fb09cfbc9f50747f6621133ed1e"}
Jul 09 20:16:28.389181 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:16:28.389130    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-x455w" podStartSLOduration=2.154810653 podStartE2EDuration="2.389114038s" totalImagesPullingTime="234.303385ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 20:16:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 20:16:28.38673084 +0000 UTC m=+778.804400649" watchObservedRunningTime="2026-07-09 20:16:28.389114038 +0000 UTC m=+778.806783846"
Jul 09 20:16:37.320917 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:16:37.320872    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-x455w"
Jul 09 20:16:37.321509 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:16:37.320945    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-x455w"
Jul 09 20:16:37.334002 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:16:37.333977    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-x455w"
Jul 09 20:16:37.419763 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:16:37.419735    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-x455w"
Jul 09 20:18:25.700209 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:18:25.700169    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-8445478c58-h7lzl"]
Jul 09 20:18:25.700769 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:18:25.700408    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-8445478c58-h7lzl" podUID="701712cd-0859-4007-9d29-48f3135733f0" containerName="authorino" containerID="cri-o://3c4d424cd88580dbb90e8c3d0604167d534f457a6a3cf51881cce8fe5b374bd3" gracePeriod=30
Jul 09 20:18:25.916699 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:18:25.916662    2562 generic.go:356] "Generic (PLEG): container finished" podID="701712cd-0859-4007-9d29-48f3135733f0" containerID="3c4d424cd88580dbb90e8c3d0604167d534f457a6a3cf51881cce8fe5b374bd3" exitCode=0
Jul 09 20:18:25.916867 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:18:25.916721    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8445478c58-h7lzl" event={"ID":"701712cd-0859-4007-9d29-48f3135733f0","Type":"ContainerDied","Data":"3c4d424cd88580dbb90e8c3d0604167d534f457a6a3cf51881cce8fe5b374bd3"}
Jul 09 20:18:25.966844 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:18:25.966814    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8445478c58-h7lzl"
Jul 09 20:18:26.069552 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:18:26.069518    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k65hg\" (UniqueName: \"kubernetes.io/projected/701712cd-0859-4007-9d29-48f3135733f0-kube-api-access-k65hg\") pod \"701712cd-0859-4007-9d29-48f3135733f0\" (UID: \"701712cd-0859-4007-9d29-48f3135733f0\") "
Jul 09 20:18:26.069730 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:18:26.069588    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/701712cd-0859-4007-9d29-48f3135733f0-tls-cert\") pod \"701712cd-0859-4007-9d29-48f3135733f0\" (UID: \"701712cd-0859-4007-9d29-48f3135733f0\") "
Jul 09 20:18:26.071627 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:18:26.071596    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/701712cd-0859-4007-9d29-48f3135733f0-kube-api-access-k65hg" (OuterVolumeSpecName: "kube-api-access-k65hg") pod "701712cd-0859-4007-9d29-48f3135733f0" (UID: "701712cd-0859-4007-9d29-48f3135733f0"). InnerVolumeSpecName "kube-api-access-k65hg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 20:18:26.079682 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:18:26.079655    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/701712cd-0859-4007-9d29-48f3135733f0-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "701712cd-0859-4007-9d29-48f3135733f0" (UID: "701712cd-0859-4007-9d29-48f3135733f0"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 20:18:26.171205 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:18:26.171153    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k65hg\" (UniqueName: \"kubernetes.io/projected/701712cd-0859-4007-9d29-48f3135733f0-kube-api-access-k65hg\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:18:26.171205 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:18:26.171198    2562 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/701712cd-0859-4007-9d29-48f3135733f0-tls-cert\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:18:26.922786 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:18:26.922749    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8445478c58-h7lzl" event={"ID":"701712cd-0859-4007-9d29-48f3135733f0","Type":"ContainerDied","Data":"30ddc6c5960460f6f3632f657313f4a5ab3555fd1499735858f51fcd77f60a09"}
Jul 09 20:18:26.922786 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:18:26.922774    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8445478c58-h7lzl"
Jul 09 20:18:26.923257 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:18:26.922797    2562 scope.go:117] "RemoveContainer" containerID="3c4d424cd88580dbb90e8c3d0604167d534f457a6a3cf51881cce8fe5b374bd3"
Jul 09 20:18:26.945386 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:18:26.945364    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-8445478c58-h7lzl"]
Jul 09 20:18:26.948026 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:18:26.948007    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-8445478c58-h7lzl"]
Jul 09 20:18:28.243888 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:18:28.243846    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="701712cd-0859-4007-9d29-48f3135733f0" path="/var/lib/kubelet/pods/701712cd-0859-4007-9d29-48f3135733f0/volumes"
Jul 09 20:18:30.090007 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:18:30.089976    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-cqs6d_578878d0-e6b3-4725-a698-796cfc169fa6/console-operator/1.log"
Jul 09 20:18:30.090570 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:18:30.090551    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-cqs6d_578878d0-e6b3-4725-a698-796cfc169fa6/console-operator/1.log"
Jul 09 20:18:30.095833 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:18:30.095807    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hds4g_4b4468fd-1708-487b-8d69-bb7d817e25de/ovn-acl-logging/0.log"
Jul 09 20:18:30.096340 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:18:30.096321    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hds4g_4b4468fd-1708-487b-8d69-bb7d817e25de/ovn-acl-logging/0.log"
Jul 09 20:19:38.516639 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:19:38.516608    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5ff76ccd7-nf8jp"]
Jul 09 20:19:38.517199 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:19:38.516848    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-5ff76ccd7-nf8jp" podUID="fe9e3119-d39d-4455-b935-74781c0ad41e" containerName="manager" containerID="cri-o://a46552a49cd02d82477d8503ff8c72e8f992c386cfe6752f090ed470f9417720" gracePeriod=30
Jul 09 20:19:38.770793 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:19:38.770733    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5ff76ccd7-nf8jp"
Jul 09 20:19:38.905718 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:19:38.905688    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fe9e3119-d39d-4455-b935-74781c0ad41e-webhook-cert\") pod \"fe9e3119-d39d-4455-b935-74781c0ad41e\" (UID: \"fe9e3119-d39d-4455-b935-74781c0ad41e\") "
Jul 09 20:19:38.905718 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:19:38.905723    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2bz4t\" (UniqueName: \"kubernetes.io/projected/fe9e3119-d39d-4455-b935-74781c0ad41e-kube-api-access-2bz4t\") pod \"fe9e3119-d39d-4455-b935-74781c0ad41e\" (UID: \"fe9e3119-d39d-4455-b935-74781c0ad41e\") "
Jul 09 20:19:38.907796 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:19:38.907764    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe9e3119-d39d-4455-b935-74781c0ad41e-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "fe9e3119-d39d-4455-b935-74781c0ad41e" (UID: "fe9e3119-d39d-4455-b935-74781c0ad41e"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 20:19:38.907913 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:19:38.907867    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe9e3119-d39d-4455-b935-74781c0ad41e-kube-api-access-2bz4t" (OuterVolumeSpecName: "kube-api-access-2bz4t") pod "fe9e3119-d39d-4455-b935-74781c0ad41e" (UID: "fe9e3119-d39d-4455-b935-74781c0ad41e"). InnerVolumeSpecName "kube-api-access-2bz4t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 20:19:39.007196 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:19:39.007159    2562 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fe9e3119-d39d-4455-b935-74781c0ad41e-webhook-cert\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:19:39.007196 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:19:39.007192    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bz4t\" (UniqueName: \"kubernetes.io/projected/fe9e3119-d39d-4455-b935-74781c0ad41e-kube-api-access-2bz4t\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:19:39.269037 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:19:39.268998    2562 generic.go:356] "Generic (PLEG): container finished" podID="fe9e3119-d39d-4455-b935-74781c0ad41e" containerID="a46552a49cd02d82477d8503ff8c72e8f992c386cfe6752f090ed470f9417720" exitCode=0
Jul 09 20:19:39.269241 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:19:39.269041    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5ff76ccd7-nf8jp"
Jul 09 20:19:39.269241 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:19:39.269046    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5ff76ccd7-nf8jp" event={"ID":"fe9e3119-d39d-4455-b935-74781c0ad41e","Type":"ContainerDied","Data":"a46552a49cd02d82477d8503ff8c72e8f992c386cfe6752f090ed470f9417720"}
Jul 09 20:19:39.269241 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:19:39.269098    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5ff76ccd7-nf8jp" event={"ID":"fe9e3119-d39d-4455-b935-74781c0ad41e","Type":"ContainerDied","Data":"3c8acebcc75493c6a584d144749e23d3eef289ecdb792e5a7d585da8bc3b214e"}
Jul 09 20:19:39.269241 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:19:39.269126    2562 scope.go:117] "RemoveContainer" containerID="a46552a49cd02d82477d8503ff8c72e8f992c386cfe6752f090ed470f9417720"
Jul 09 20:19:39.278501 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:19:39.278480    2562 scope.go:117] "RemoveContainer" containerID="a46552a49cd02d82477d8503ff8c72e8f992c386cfe6752f090ed470f9417720"
Jul 09 20:19:39.278741 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:19:39.278721    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a46552a49cd02d82477d8503ff8c72e8f992c386cfe6752f090ed470f9417720\": container with ID starting with a46552a49cd02d82477d8503ff8c72e8f992c386cfe6752f090ed470f9417720 not found: ID does not exist" containerID="a46552a49cd02d82477d8503ff8c72e8f992c386cfe6752f090ed470f9417720"
Jul 09 20:19:39.278788 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:19:39.278751    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a46552a49cd02d82477d8503ff8c72e8f992c386cfe6752f090ed470f9417720"} err="failed to get container status \"a46552a49cd02d82477d8503ff8c72e8f992c386cfe6752f090ed470f9417720\": rpc error: code = NotFound desc = could not find container \"a46552a49cd02d82477d8503ff8c72e8f992c386cfe6752f090ed470f9417720\": container with ID starting with a46552a49cd02d82477d8503ff8c72e8f992c386cfe6752f090ed470f9417720 not found: ID does not exist"
Jul 09 20:19:39.290403 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:19:39.290381    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5ff76ccd7-nf8jp"]
Jul 09 20:19:39.293807 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:19:39.293787    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5ff76ccd7-nf8jp"]
Jul 09 20:19:40.076409 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:19:40.076375    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5ff76ccd7-gntk4"]
Jul 09 20:19:40.076905 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:19:40.076887    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="701712cd-0859-4007-9d29-48f3135733f0" containerName="authorino"
Jul 09 20:19:40.076985 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:19:40.076907    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="701712cd-0859-4007-9d29-48f3135733f0" containerName="authorino"
Jul 09 20:19:40.076985 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:19:40.076938    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe9e3119-d39d-4455-b935-74781c0ad41e" containerName="manager"
Jul 09 20:19:40.076985 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:19:40.076946    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe9e3119-d39d-4455-b935-74781c0ad41e" containerName="manager"
Jul 09 20:19:40.077136 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:19:40.077048    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe9e3119-d39d-4455-b935-74781c0ad41e" containerName="manager"
Jul 09 20:19:40.077136 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:19:40.077065    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="701712cd-0859-4007-9d29-48f3135733f0" containerName="authorino"
Jul 09 20:19:40.081238 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:19:40.081216    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5ff76ccd7-gntk4"
Jul 09 20:19:40.085953 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:19:40.085929    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-wnrwq\""
Jul 09 20:19:40.086109 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:19:40.085954    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 09 20:19:40.086109 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:19:40.085952    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 09 20:19:40.091249 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:19:40.091222    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5ff76ccd7-gntk4"]
Jul 09 20:19:40.218714 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:19:40.218673    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbvj8\" (UniqueName: \"kubernetes.io/projected/be8647ac-8106-4927-b1e6-b1c604e87ccd-kube-api-access-xbvj8\") pod \"maas-controller-5ff76ccd7-gntk4\" (UID: \"be8647ac-8106-4927-b1e6-b1c604e87ccd\") " pod="opendatahub/maas-controller-5ff76ccd7-gntk4"
Jul 09 20:19:40.218911 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:19:40.218745    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/be8647ac-8106-4927-b1e6-b1c604e87ccd-webhook-cert\") pod \"maas-controller-5ff76ccd7-gntk4\" (UID: \"be8647ac-8106-4927-b1e6-b1c604e87ccd\") " pod="opendatahub/maas-controller-5ff76ccd7-gntk4"
Jul 09 20:19:40.242001 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:19:40.241968    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe9e3119-d39d-4455-b935-74781c0ad41e" path="/var/lib/kubelet/pods/fe9e3119-d39d-4455-b935-74781c0ad41e/volumes"
Jul 09 20:19:40.319283 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:19:40.319253    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbvj8\" (UniqueName: \"kubernetes.io/projected/be8647ac-8106-4927-b1e6-b1c604e87ccd-kube-api-access-xbvj8\") pod \"maas-controller-5ff76ccd7-gntk4\" (UID: \"be8647ac-8106-4927-b1e6-b1c604e87ccd\") " pod="opendatahub/maas-controller-5ff76ccd7-gntk4"
Jul 09 20:19:40.319476 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:19:40.319323    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/be8647ac-8106-4927-b1e6-b1c604e87ccd-webhook-cert\") pod \"maas-controller-5ff76ccd7-gntk4\" (UID: \"be8647ac-8106-4927-b1e6-b1c604e87ccd\") " pod="opendatahub/maas-controller-5ff76ccd7-gntk4"
Jul 09 20:19:40.322095 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:19:40.322072    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/be8647ac-8106-4927-b1e6-b1c604e87ccd-webhook-cert\") pod \"maas-controller-5ff76ccd7-gntk4\" (UID: \"be8647ac-8106-4927-b1e6-b1c604e87ccd\") " pod="opendatahub/maas-controller-5ff76ccd7-gntk4"
Jul 09 20:19:40.329009 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:19:40.328940    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbvj8\" (UniqueName: \"kubernetes.io/projected/be8647ac-8106-4927-b1e6-b1c604e87ccd-kube-api-access-xbvj8\") pod \"maas-controller-5ff76ccd7-gntk4\" (UID: \"be8647ac-8106-4927-b1e6-b1c604e87ccd\") " pod="opendatahub/maas-controller-5ff76ccd7-gntk4"
Jul 09 20:19:40.392500 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:19:40.392471    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5ff76ccd7-gntk4"
Jul 09 20:19:40.524075 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:19:40.524053    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5ff76ccd7-gntk4"]
Jul 09 20:19:40.526360 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:19:40.526331    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe8647ac_8106_4927_b1e6_b1c604e87ccd.slice/crio-dba3c8ca28ddd3bf3c495552687782490f7cee132dd3e4b3a5ee3d6f0bd0ffee WatchSource:0}: Error finding container dba3c8ca28ddd3bf3c495552687782490f7cee132dd3e4b3a5ee3d6f0bd0ffee: Status 404 returned error can't find the container with id dba3c8ca28ddd3bf3c495552687782490f7cee132dd3e4b3a5ee3d6f0bd0ffee
Jul 09 20:19:41.284961 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:19:41.284877    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5ff76ccd7-gntk4" event={"ID":"be8647ac-8106-4927-b1e6-b1c604e87ccd","Type":"ContainerStarted","Data":"297cf1a34e1be5cbb1092f73aaff1b9940db7ec49e5abd76832693a7a81234b5"}
Jul 09 20:19:41.284961 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:19:41.284949    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5ff76ccd7-gntk4" event={"ID":"be8647ac-8106-4927-b1e6-b1c604e87ccd","Type":"ContainerStarted","Data":"dba3c8ca28ddd3bf3c495552687782490f7cee132dd3e4b3a5ee3d6f0bd0ffee"}
Jul 09 20:19:41.285462 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:19:41.285008    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5ff76ccd7-gntk4"
Jul 09 20:19:41.320260 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:19:41.320209    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5ff76ccd7-gntk4" podStartSLOduration=0.863403585 podStartE2EDuration="1.320194294s" totalImagesPullingTime="456.790709ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 20:19:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 20:19:41.318137946 +0000 UTC m=+971.735807754" watchObservedRunningTime="2026-07-09 20:19:41.320194294 +0000 UTC m=+971.737864103"
Jul 09 20:19:52.295256 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:19:52.295170    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5ff76ccd7-gntk4"
Jul 09 20:23:30.118939 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:23:30.118898    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-cqs6d_578878d0-e6b3-4725-a698-796cfc169fa6/console-operator/1.log"
Jul 09 20:23:30.121480 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:23:30.121192    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-cqs6d_578878d0-e6b3-4725-a698-796cfc169fa6/console-operator/1.log"
Jul 09 20:23:30.125102 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:23:30.125072    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hds4g_4b4468fd-1708-487b-8d69-bb7d817e25de/ovn-acl-logging/0.log"
Jul 09 20:23:30.127387 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:23:30.127365    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hds4g_4b4468fd-1708-487b-8d69-bb7d817e25de/ovn-acl-logging/0.log"
Jul 09 20:28:30.146566 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:28:30.146449    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-cqs6d_578878d0-e6b3-4725-a698-796cfc169fa6/console-operator/1.log"
Jul 09 20:28:30.150428 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:28:30.149482    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-cqs6d_578878d0-e6b3-4725-a698-796cfc169fa6/console-operator/1.log"
Jul 09 20:28:30.152161 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:28:30.152138    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hds4g_4b4468fd-1708-487b-8d69-bb7d817e25de/ovn-acl-logging/0.log"
Jul 09 20:28:30.155075 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:28:30.155058    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hds4g_4b4468fd-1708-487b-8d69-bb7d817e25de/ovn-acl-logging/0.log"
Jul 09 20:30:00.190727 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:30:00.190688    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29727150-brk9x"]
Jul 09 20:30:00.194121 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:30:00.194102    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727150-brk9x"
Jul 09 20:30:00.239914 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:30:00.239880    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjnqt\" (UniqueName: \"kubernetes.io/projected/b6ea142e-d7e3-4a2c-b9c0-f94db91d6110-kube-api-access-gjnqt\") pod \"maas-api-key-cleanup-29727150-brk9x\" (UID: \"b6ea142e-d7e3-4a2c-b9c0-f94db91d6110\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727150-brk9x"
Jul 09 20:30:00.242621 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:30:00.242590    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29727150-brk9x"]
Jul 09 20:30:00.341264 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:30:00.341226    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjnqt\" (UniqueName: \"kubernetes.io/projected/b6ea142e-d7e3-4a2c-b9c0-f94db91d6110-kube-api-access-gjnqt\") pod \"maas-api-key-cleanup-29727150-brk9x\" (UID: \"b6ea142e-d7e3-4a2c-b9c0-f94db91d6110\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727150-brk9x"
Jul 09 20:30:00.350235 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:30:00.350209    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjnqt\" (UniqueName: \"kubernetes.io/projected/b6ea142e-d7e3-4a2c-b9c0-f94db91d6110-kube-api-access-gjnqt\") pod \"maas-api-key-cleanup-29727150-brk9x\" (UID: \"b6ea142e-d7e3-4a2c-b9c0-f94db91d6110\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727150-brk9x"
Jul 09 20:30:00.503876 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:30:00.503844    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727150-brk9x"
Jul 09 20:30:00.632570 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:30:00.632221    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29727150-brk9x"]
Jul 09 20:30:00.635624 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:30:00.635591    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6ea142e_d7e3_4a2c_b9c0_f94db91d6110.slice/crio-42e8d6558c5080f8b759dede731b3fe32b499b75139bbe9a72ee3403c19c8a53 WatchSource:0}: Error finding container 42e8d6558c5080f8b759dede731b3fe32b499b75139bbe9a72ee3403c19c8a53: Status 404 returned error can't find the container with id 42e8d6558c5080f8b759dede731b3fe32b499b75139bbe9a72ee3403c19c8a53
Jul 09 20:30:00.637661 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:30:00.637641    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 09 20:30:01.170857 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:30:01.170822    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727150-brk9x" event={"ID":"b6ea142e-d7e3-4a2c-b9c0-f94db91d6110","Type":"ContainerStarted","Data":"42e8d6558c5080f8b759dede731b3fe32b499b75139bbe9a72ee3403c19c8a53"}
Jul 09 20:30:04.186193 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:30:04.186157    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727150-brk9x" event={"ID":"b6ea142e-d7e3-4a2c-b9c0-f94db91d6110","Type":"ContainerStarted","Data":"b12772417124baf1b00063f4f88a30145cd7d6d4dd41e6950a7965748a461555"}
Jul 09 20:30:04.201815 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:30:04.201766    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727150-brk9x" podStartSLOduration=1.118775601 podStartE2EDuration="4.201750937s" totalImagesPullingTime="3.082975336s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 20:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 20:30:04.200736485 +0000 UTC m=+1594.618406296" watchObservedRunningTime="2026-07-09 20:30:04.201750937 +0000 UTC m=+1594.619420744"
Jul 09 20:31:00.010500 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:31:00.010462    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29727150-brk9x"]
Jul 09 20:31:00.011028 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:31:00.010667    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727150-brk9x" podUID="b6ea142e-d7e3-4a2c-b9c0-f94db91d6110" containerName="cleanup" containerID="cri-o://b12772417124baf1b00063f4f88a30145cd7d6d4dd41e6950a7965748a461555" gracePeriod=30
Jul 09 20:31:30.186636 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:31:30.186608    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727150-brk9x"
Jul 09 20:31:30.250888 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:31:30.250862    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gjnqt\" (UniqueName: \"kubernetes.io/projected/b6ea142e-d7e3-4a2c-b9c0-f94db91d6110-kube-api-access-gjnqt\") pod \"b6ea142e-d7e3-4a2c-b9c0-f94db91d6110\" (UID: \"b6ea142e-d7e3-4a2c-b9c0-f94db91d6110\") "
Jul 09 20:31:30.252882 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:31:30.252855    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6ea142e-d7e3-4a2c-b9c0-f94db91d6110-kube-api-access-gjnqt" (OuterVolumeSpecName: "kube-api-access-gjnqt") pod "b6ea142e-d7e3-4a2c-b9c0-f94db91d6110" (UID: "b6ea142e-d7e3-4a2c-b9c0-f94db91d6110"). InnerVolumeSpecName "kube-api-access-gjnqt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 20:31:30.352106 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:31:30.351990    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjnqt\" (UniqueName: \"kubernetes.io/projected/b6ea142e-d7e3-4a2c-b9c0-f94db91d6110-kube-api-access-gjnqt\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:31:30.596177 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:31:30.596141    2562 generic.go:356] "Generic (PLEG): container finished" podID="b6ea142e-d7e3-4a2c-b9c0-f94db91d6110" containerID="b12772417124baf1b00063f4f88a30145cd7d6d4dd41e6950a7965748a461555" exitCode=137
Jul 09 20:31:30.596381 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:31:30.596189    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727150-brk9x"
Jul 09 20:31:30.596381 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:31:30.596225    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727150-brk9x" event={"ID":"b6ea142e-d7e3-4a2c-b9c0-f94db91d6110","Type":"ContainerDied","Data":"b12772417124baf1b00063f4f88a30145cd7d6d4dd41e6950a7965748a461555"}
Jul 09 20:31:30.596381 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:31:30.596282    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727150-brk9x" event={"ID":"b6ea142e-d7e3-4a2c-b9c0-f94db91d6110","Type":"ContainerDied","Data":"42e8d6558c5080f8b759dede731b3fe32b499b75139bbe9a72ee3403c19c8a53"}
Jul 09 20:31:30.596381 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:31:30.596327    2562 scope.go:117] "RemoveContainer" containerID="b12772417124baf1b00063f4f88a30145cd7d6d4dd41e6950a7965748a461555"
Jul 09 20:31:30.604773 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:31:30.604756    2562 scope.go:117] "RemoveContainer" containerID="b12772417124baf1b00063f4f88a30145cd7d6d4dd41e6950a7965748a461555"
Jul 09 20:31:30.605057 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:31:30.605039    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b12772417124baf1b00063f4f88a30145cd7d6d4dd41e6950a7965748a461555\": container with ID starting with b12772417124baf1b00063f4f88a30145cd7d6d4dd41e6950a7965748a461555 not found: ID does not exist" containerID="b12772417124baf1b00063f4f88a30145cd7d6d4dd41e6950a7965748a461555"
Jul 09 20:31:30.605102 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:31:30.605068    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b12772417124baf1b00063f4f88a30145cd7d6d4dd41e6950a7965748a461555"} err="failed to get container status \"b12772417124baf1b00063f4f88a30145cd7d6d4dd41e6950a7965748a461555\": rpc error: code = NotFound desc = could not find container \"b12772417124baf1b00063f4f88a30145cd7d6d4dd41e6950a7965748a461555\": container with ID starting with b12772417124baf1b00063f4f88a30145cd7d6d4dd41e6950a7965748a461555 not found: ID does not exist"
Jul 09 20:31:30.618067 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:31:30.618039    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29727150-brk9x"]
Jul 09 20:31:30.621816 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:31:30.621791    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29727150-brk9x"]
Jul 09 20:31:32.244255 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:31:32.244222    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b6ea142e-d7e3-4a2c-b9c0-f94db91d6110" path="/var/lib/kubelet/pods/b6ea142e-d7e3-4a2c-b9c0-f94db91d6110/volumes"
Jul 09 20:32:03.371653 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:03.371609    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-f632bdcd-openshift-default-54bcf7678c-8469t"]
Jul 09 20:32:03.372157 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:03.372015    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6ea142e-d7e3-4a2c-b9c0-f94db91d6110" containerName="cleanup"
Jul 09 20:32:03.372157 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:03.372029    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6ea142e-d7e3-4a2c-b9c0-f94db91d6110" containerName="cleanup"
Jul 09 20:32:03.372157 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:03.372131    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6ea142e-d7e3-4a2c-b9c0-f94db91d6110" containerName="cleanup"
Jul 09 20:32:03.375441 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:03.375417    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f632bdcd-openshift-default-54bcf7678c-8469t"
Jul 09 20:32:03.382888 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:03.382867    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-f632bdcd-openshift-default-dockercfg-tpcx6\""
Jul 09 20:32:03.393893 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:03.393864    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-f632bdcd-openshift-default-54bcf7678c-8469t"]
Jul 09 20:32:03.447035 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:03.446991    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81833794-e0f8-4927-91ba-d4567d3c8d11-istio-data\") pod \"e2e-ait-f632bdcd-openshift-default-54bcf7678c-8469t\" (UID: \"81833794-e0f8-4927-91ba-d4567d3c8d11\") " pod="openshift-ingress/e2e-ait-f632bdcd-openshift-default-54bcf7678c-8469t"
Jul 09 20:32:03.447221 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:03.447044    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81833794-e0f8-4927-91ba-d4567d3c8d11-istiod-ca-cert\") pod \"e2e-ait-f632bdcd-openshift-default-54bcf7678c-8469t\" (UID: \"81833794-e0f8-4927-91ba-d4567d3c8d11\") " pod="openshift-ingress/e2e-ait-f632bdcd-openshift-default-54bcf7678c-8469t"
Jul 09 20:32:03.447221 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:03.447108    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81833794-e0f8-4927-91ba-d4567d3c8d11-istio-envoy\") pod \"e2e-ait-f632bdcd-openshift-default-54bcf7678c-8469t\" (UID: \"81833794-e0f8-4927-91ba-d4567d3c8d11\") " pod="openshift-ingress/e2e-ait-f632bdcd-openshift-default-54bcf7678c-8469t"
Jul 09 20:32:03.447221 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:03.447145    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4zv9\" (UniqueName: \"kubernetes.io/projected/81833794-e0f8-4927-91ba-d4567d3c8d11-kube-api-access-s4zv9\") pod \"e2e-ait-f632bdcd-openshift-default-54bcf7678c-8469t\" (UID: \"81833794-e0f8-4927-91ba-d4567d3c8d11\") " pod="openshift-ingress/e2e-ait-f632bdcd-openshift-default-54bcf7678c-8469t"
Jul 09 20:32:03.447221 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:03.447176    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81833794-e0f8-4927-91ba-d4567d3c8d11-workload-socket\") pod \"e2e-ait-f632bdcd-openshift-default-54bcf7678c-8469t\" (UID: \"81833794-e0f8-4927-91ba-d4567d3c8d11\") " pod="openshift-ingress/e2e-ait-f632bdcd-openshift-default-54bcf7678c-8469t"
Jul 09 20:32:03.447221 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:03.447212    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81833794-e0f8-4927-91ba-d4567d3c8d11-istio-token\") pod \"e2e-ait-f632bdcd-openshift-default-54bcf7678c-8469t\" (UID: \"81833794-e0f8-4927-91ba-d4567d3c8d11\") " pod="openshift-ingress/e2e-ait-f632bdcd-openshift-default-54bcf7678c-8469t"
Jul 09 20:32:03.447450 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:03.447257    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81833794-e0f8-4927-91ba-d4567d3c8d11-istio-podinfo\") pod \"e2e-ait-f632bdcd-openshift-default-54bcf7678c-8469t\" (UID: \"81833794-e0f8-4927-91ba-d4567d3c8d11\") " pod="openshift-ingress/e2e-ait-f632bdcd-openshift-default-54bcf7678c-8469t"
Jul 09 20:32:03.447450 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:03.447283    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81833794-e0f8-4927-91ba-d4567d3c8d11-credential-socket\") pod \"e2e-ait-f632bdcd-openshift-default-54bcf7678c-8469t\" (UID: \"81833794-e0f8-4927-91ba-d4567d3c8d11\") " pod="openshift-ingress/e2e-ait-f632bdcd-openshift-default-54bcf7678c-8469t"
Jul 09 20:32:03.447450 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:03.447352    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81833794-e0f8-4927-91ba-d4567d3c8d11-workload-certs\") pod \"e2e-ait-f632bdcd-openshift-default-54bcf7678c-8469t\" (UID: \"81833794-e0f8-4927-91ba-d4567d3c8d11\") " pod="openshift-ingress/e2e-ait-f632bdcd-openshift-default-54bcf7678c-8469t"
Jul 09 20:32:03.548361 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:03.548321    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81833794-e0f8-4927-91ba-d4567d3c8d11-istio-data\") pod \"e2e-ait-f632bdcd-openshift-default-54bcf7678c-8469t\" (UID: \"81833794-e0f8-4927-91ba-d4567d3c8d11\") " pod="openshift-ingress/e2e-ait-f632bdcd-openshift-default-54bcf7678c-8469t"
Jul 09 20:32:03.548539 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:03.548371    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81833794-e0f8-4927-91ba-d4567d3c8d11-istiod-ca-cert\") pod \"e2e-ait-f632bdcd-openshift-default-54bcf7678c-8469t\" (UID: \"81833794-e0f8-4927-91ba-d4567d3c8d11\") " pod="openshift-ingress/e2e-ait-f632bdcd-openshift-default-54bcf7678c-8469t"
Jul 09 20:32:03.548539 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:03.548433    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81833794-e0f8-4927-91ba-d4567d3c8d11-istio-envoy\") pod \"e2e-ait-f632bdcd-openshift-default-54bcf7678c-8469t\" (UID: \"81833794-e0f8-4927-91ba-d4567d3c8d11\") " pod="openshift-ingress/e2e-ait-f632bdcd-openshift-default-54bcf7678c-8469t"
Jul 09 20:32:03.548539 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:03.548475    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4zv9\" (UniqueName: \"kubernetes.io/projected/81833794-e0f8-4927-91ba-d4567d3c8d11-kube-api-access-s4zv9\") pod \"e2e-ait-f632bdcd-openshift-default-54bcf7678c-8469t\" (UID: \"81833794-e0f8-4927-91ba-d4567d3c8d11\") " pod="openshift-ingress/e2e-ait-f632bdcd-openshift-default-54bcf7678c-8469t"
Jul 09 20:32:03.548539 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:03.548507    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81833794-e0f8-4927-91ba-d4567d3c8d11-workload-socket\") pod \"e2e-ait-f632bdcd-openshift-default-54bcf7678c-8469t\" (UID: \"81833794-e0f8-4927-91ba-d4567d3c8d11\") " pod="openshift-ingress/e2e-ait-f632bdcd-openshift-default-54bcf7678c-8469t"
Jul 09 20:32:03.548752 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:03.548562    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81833794-e0f8-4927-91ba-d4567d3c8d11-istio-token\") pod \"e2e-ait-f632bdcd-openshift-default-54bcf7678c-8469t\" (UID: \"81833794-e0f8-4927-91ba-d4567d3c8d11\") " pod="openshift-ingress/e2e-ait-f632bdcd-openshift-default-54bcf7678c-8469t"
Jul 09 20:32:03.548752 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:03.548634    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81833794-e0f8-4927-91ba-d4567d3c8d11-istio-podinfo\") pod \"e2e-ait-f632bdcd-openshift-default-54bcf7678c-8469t\" (UID: \"81833794-e0f8-4927-91ba-d4567d3c8d11\") " pod="openshift-ingress/e2e-ait-f632bdcd-openshift-default-54bcf7678c-8469t"
Jul 09 20:32:03.548752 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:03.548665    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81833794-e0f8-4927-91ba-d4567d3c8d11-credential-socket\") pod \"e2e-ait-f632bdcd-openshift-default-54bcf7678c-8469t\" (UID: \"81833794-e0f8-4927-91ba-d4567d3c8d11\") " pod="openshift-ingress/e2e-ait-f632bdcd-openshift-default-54bcf7678c-8469t"
Jul 09 20:32:03.548752 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:03.548699    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81833794-e0f8-4927-91ba-d4567d3c8d11-workload-certs\") pod \"e2e-ait-f632bdcd-openshift-default-54bcf7678c-8469t\" (UID: \"81833794-e0f8-4927-91ba-d4567d3c8d11\") " pod="openshift-ingress/e2e-ait-f632bdcd-openshift-default-54bcf7678c-8469t"
Jul 09 20:32:03.548929 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:03.548774    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81833794-e0f8-4927-91ba-d4567d3c8d11-istio-data\") pod \"e2e-ait-f632bdcd-openshift-default-54bcf7678c-8469t\" (UID: \"81833794-e0f8-4927-91ba-d4567d3c8d11\") " pod="openshift-ingress/e2e-ait-f632bdcd-openshift-default-54bcf7678c-8469t"
Jul 09 20:32:03.549189 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:03.549160    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81833794-e0f8-4927-91ba-d4567d3c8d11-istiod-ca-cert\") pod \"e2e-ait-f632bdcd-openshift-default-54bcf7678c-8469t\" (UID: \"81833794-e0f8-4927-91ba-d4567d3c8d11\") " pod="openshift-ingress/e2e-ait-f632bdcd-openshift-default-54bcf7678c-8469t"
Jul 09 20:32:03.549433 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:03.549411    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81833794-e0f8-4927-91ba-d4567d3c8d11-workload-socket\") pod \"e2e-ait-f632bdcd-openshift-default-54bcf7678c-8469t\" (UID: \"81833794-e0f8-4927-91ba-d4567d3c8d11\") " pod="openshift-ingress/e2e-ait-f632bdcd-openshift-default-54bcf7678c-8469t"
Jul 09 20:32:03.549515 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:03.549416    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81833794-e0f8-4927-91ba-d4567d3c8d11-workload-certs\") pod \"e2e-ait-f632bdcd-openshift-default-54bcf7678c-8469t\" (UID: \"81833794-e0f8-4927-91ba-d4567d3c8d11\") " pod="openshift-ingress/e2e-ait-f632bdcd-openshift-default-54bcf7678c-8469t"
Jul 09 20:32:03.549515 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:03.549457    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81833794-e0f8-4927-91ba-d4567d3c8d11-credential-socket\") pod \"e2e-ait-f632bdcd-openshift-default-54bcf7678c-8469t\" (UID: \"81833794-e0f8-4927-91ba-d4567d3c8d11\") " pod="openshift-ingress/e2e-ait-f632bdcd-openshift-default-54bcf7678c-8469t"
Jul 09 20:32:03.550738 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:03.550718    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81833794-e0f8-4927-91ba-d4567d3c8d11-istio-envoy\") pod \"e2e-ait-f632bdcd-openshift-default-54bcf7678c-8469t\" (UID: \"81833794-e0f8-4927-91ba-d4567d3c8d11\") " pod="openshift-ingress/e2e-ait-f632bdcd-openshift-default-54bcf7678c-8469t"
Jul 09 20:32:03.551221 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:03.551197    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81833794-e0f8-4927-91ba-d4567d3c8d11-istio-podinfo\") pod \"e2e-ait-f632bdcd-openshift-default-54bcf7678c-8469t\" (UID: \"81833794-e0f8-4927-91ba-d4567d3c8d11\") " pod="openshift-ingress/e2e-ait-f632bdcd-openshift-default-54bcf7678c-8469t"
Jul 09 20:32:03.564931 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:03.564897    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81833794-e0f8-4927-91ba-d4567d3c8d11-istio-token\") pod \"e2e-ait-f632bdcd-openshift-default-54bcf7678c-8469t\" (UID: \"81833794-e0f8-4927-91ba-d4567d3c8d11\") " pod="openshift-ingress/e2e-ait-f632bdcd-openshift-default-54bcf7678c-8469t"
Jul 09 20:32:03.566122 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:03.566099    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4zv9\" (UniqueName: \"kubernetes.io/projected/81833794-e0f8-4927-91ba-d4567d3c8d11-kube-api-access-s4zv9\") pod \"e2e-ait-f632bdcd-openshift-default-54bcf7678c-8469t\" (UID: \"81833794-e0f8-4927-91ba-d4567d3c8d11\") " pod="openshift-ingress/e2e-ait-f632bdcd-openshift-default-54bcf7678c-8469t"
Jul 09 20:32:03.686278 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:03.686183    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f632bdcd-openshift-default-54bcf7678c-8469t"
Jul 09 20:32:03.846353 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:03.846326    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-f632bdcd-openshift-default-54bcf7678c-8469t"]
Jul 09 20:32:03.848975 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:32:03.848946    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81833794_e0f8_4927_91ba_d4567d3c8d11.slice/crio-8d3c7a644c1accf8266d350ccfd43bae0855af30f687cf41573c303a39992a9b WatchSource:0}: Error finding container 8d3c7a644c1accf8266d350ccfd43bae0855af30f687cf41573c303a39992a9b: Status 404 returned error can't find the container with id 8d3c7a644c1accf8266d350ccfd43bae0855af30f687cf41573c303a39992a9b
Jul 09 20:32:03.851029 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:03.850994    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 09 20:32:03.851123 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:03.851088    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 09 20:32:03.851185 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:03.851136    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 09 20:32:04.777554 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:04.777485    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f632bdcd-openshift-default-54bcf7678c-8469t" event={"ID":"81833794-e0f8-4927-91ba-d4567d3c8d11","Type":"ContainerStarted","Data":"4d4be5cfbf2774fab3d3e1324871a05277b5b48ac12eab50d0c397f0346464c8"}
Jul 09 20:32:04.778053 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:04.777564    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f632bdcd-openshift-default-54bcf7678c-8469t" event={"ID":"81833794-e0f8-4927-91ba-d4567d3c8d11","Type":"ContainerStarted","Data":"8d3c7a644c1accf8266d350ccfd43bae0855af30f687cf41573c303a39992a9b"}
Jul 09 20:32:04.784052 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:04.784025    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f632bdcd-797cb97545-6zfdb"]
Jul 09 20:32:04.787795 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:04.787771    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f632bdcd-797cb97545-6zfdb"
Jul 09 20:32:04.790768 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:04.790745    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-f632bdcd-serving-cert\""
Jul 09 20:32:04.801542 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:04.801519    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f632bdcd-797cb97545-6zfdb"]
Jul 09 20:32:04.809719 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:04.809671    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-f632bdcd-openshift-default-54bcf7678c-8469t" podStartSLOduration=1.8096555589999999 podStartE2EDuration="1.809655559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 20:32:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 20:32:04.80747702 +0000 UTC m=+1715.225146825" watchObservedRunningTime="2026-07-09 20:32:04.809655559 +0000 UTC m=+1715.227325367"
Jul 09 20:32:04.858687 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:04.858656    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csz88\" (UniqueName: \"kubernetes.io/projected/167f42f7-8dd2-4309-98e2-7ee2ca0d9d70-kube-api-access-csz88\") pod \"maas-api-e2e-ait-f632bdcd-797cb97545-6zfdb\" (UID: \"167f42f7-8dd2-4309-98e2-7ee2ca0d9d70\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f632bdcd-797cb97545-6zfdb"
Jul 09 20:32:04.858903 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:04.858827    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/167f42f7-8dd2-4309-98e2-7ee2ca0d9d70-maas-api-tls\") pod \"maas-api-e2e-ait-f632bdcd-797cb97545-6zfdb\" (UID: \"167f42f7-8dd2-4309-98e2-7ee2ca0d9d70\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f632bdcd-797cb97545-6zfdb"
Jul 09 20:32:04.859064 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:04.859031    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/167f42f7-8dd2-4309-98e2-7ee2ca0d9d70-openshift-service-ca\") pod \"maas-api-e2e-ait-f632bdcd-797cb97545-6zfdb\" (UID: \"167f42f7-8dd2-4309-98e2-7ee2ca0d9d70\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f632bdcd-797cb97545-6zfdb"
Jul 09 20:32:04.960390 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:04.960349    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/167f42f7-8dd2-4309-98e2-7ee2ca0d9d70-openshift-service-ca\") pod \"maas-api-e2e-ait-f632bdcd-797cb97545-6zfdb\" (UID: \"167f42f7-8dd2-4309-98e2-7ee2ca0d9d70\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f632bdcd-797cb97545-6zfdb"
Jul 09 20:32:04.960568 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:04.960414    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csz88\" (UniqueName: \"kubernetes.io/projected/167f42f7-8dd2-4309-98e2-7ee2ca0d9d70-kube-api-access-csz88\") pod \"maas-api-e2e-ait-f632bdcd-797cb97545-6zfdb\" (UID: \"167f42f7-8dd2-4309-98e2-7ee2ca0d9d70\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f632bdcd-797cb97545-6zfdb"
Jul 09 20:32:04.960568 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:04.960456    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/167f42f7-8dd2-4309-98e2-7ee2ca0d9d70-maas-api-tls\") pod \"maas-api-e2e-ait-f632bdcd-797cb97545-6zfdb\" (UID: \"167f42f7-8dd2-4309-98e2-7ee2ca0d9d70\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f632bdcd-797cb97545-6zfdb"
Jul 09 20:32:04.960643 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:32:04.960583    2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-f632bdcd-serving-cert: secret "maas-api-e2e-ait-f632bdcd-serving-cert" not found
Jul 09 20:32:04.960678 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:32:04.960645    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/167f42f7-8dd2-4309-98e2-7ee2ca0d9d70-maas-api-tls podName:167f42f7-8dd2-4309-98e2-7ee2ca0d9d70 nodeName:}" failed. No retries permitted until 2026-07-09 20:32:05.460628525 +0000 UTC m=+1715.878298312 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/167f42f7-8dd2-4309-98e2-7ee2ca0d9d70-maas-api-tls") pod "maas-api-e2e-ait-f632bdcd-797cb97545-6zfdb" (UID: "167f42f7-8dd2-4309-98e2-7ee2ca0d9d70") : secret "maas-api-e2e-ait-f632bdcd-serving-cert" not found
Jul 09 20:32:04.961185 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:04.961162    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/167f42f7-8dd2-4309-98e2-7ee2ca0d9d70-openshift-service-ca\") pod \"maas-api-e2e-ait-f632bdcd-797cb97545-6zfdb\" (UID: \"167f42f7-8dd2-4309-98e2-7ee2ca0d9d70\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f632bdcd-797cb97545-6zfdb"
Jul 09 20:32:04.974642 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:04.974610    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-csz88\" (UniqueName: \"kubernetes.io/projected/167f42f7-8dd2-4309-98e2-7ee2ca0d9d70-kube-api-access-csz88\") pod \"maas-api-e2e-ait-f632bdcd-797cb97545-6zfdb\" (UID: \"167f42f7-8dd2-4309-98e2-7ee2ca0d9d70\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f632bdcd-797cb97545-6zfdb"
Jul 09 20:32:05.464794 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:05.464755    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/167f42f7-8dd2-4309-98e2-7ee2ca0d9d70-maas-api-tls\") pod \"maas-api-e2e-ait-f632bdcd-797cb97545-6zfdb\" (UID: \"167f42f7-8dd2-4309-98e2-7ee2ca0d9d70\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f632bdcd-797cb97545-6zfdb"
Jul 09 20:32:05.467292 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:05.467268    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/167f42f7-8dd2-4309-98e2-7ee2ca0d9d70-maas-api-tls\") pod \"maas-api-e2e-ait-f632bdcd-797cb97545-6zfdb\" (UID: \"167f42f7-8dd2-4309-98e2-7ee2ca0d9d70\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f632bdcd-797cb97545-6zfdb"
Jul 09 20:32:05.686756 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:05.686723    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-f632bdcd-openshift-default-54bcf7678c-8469t"
Jul 09 20:32:05.700266 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:05.700229    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-f632bdcd-openshift-default-54bcf7678c-8469t"
Jul 09 20:32:05.700480 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:05.700426    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f632bdcd-797cb97545-6zfdb"
Jul 09 20:32:05.784078 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:05.784047    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-f632bdcd-openshift-default-54bcf7678c-8469t"
Jul 09 20:32:05.785008 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:05.784988    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-f632bdcd-openshift-default-54bcf7678c-8469t"
Jul 09 20:32:05.871216 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:05.871166    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f632bdcd-797cb97545-6zfdb"]
Jul 09 20:32:05.874236 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:32:05.874207    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod167f42f7_8dd2_4309_98e2_7ee2ca0d9d70.slice/crio-61b392f9fc9639596e9de7f7320999f76bdb6f557896ef70bd00a0b058d22663 WatchSource:0}: Error finding container 61b392f9fc9639596e9de7f7320999f76bdb6f557896ef70bd00a0b058d22663: Status 404 returned error can't find the container with id 61b392f9fc9639596e9de7f7320999f76bdb6f557896ef70bd00a0b058d22663
Jul 09 20:32:06.791504 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:06.791469    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f632bdcd-797cb97545-6zfdb" event={"ID":"167f42f7-8dd2-4309-98e2-7ee2ca0d9d70","Type":"ContainerStarted","Data":"1c333e23ea3b082c087b21f47b8e9f1775526f86e45295bd686beb6ebcd531ef"}
Jul 09 20:32:06.791886 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:06.791524    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f632bdcd-797cb97545-6zfdb" event={"ID":"167f42f7-8dd2-4309-98e2-7ee2ca0d9d70","Type":"ContainerStarted","Data":"61b392f9fc9639596e9de7f7320999f76bdb6f557896ef70bd00a0b058d22663"}
Jul 09 20:32:06.791886 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:06.791779    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f632bdcd-797cb97545-6zfdb"
Jul 09 20:32:06.810731 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:06.810679    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f632bdcd-797cb97545-6zfdb" podStartSLOduration=2.089708112 podStartE2EDuration="2.810662469s" totalImagesPullingTime="720.954357ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 20:32:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 20:32:06.809201259 +0000 UTC m=+1717.226871070" watchObservedRunningTime="2026-07-09 20:32:06.810662469 +0000 UTC m=+1717.228332277"
Jul 09 20:32:09.882649 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:09.882616    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-f632bdcd-openshift-default-54bcf7678c-8469t"]
Jul 09 20:32:09.883162 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:09.882805    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-f632bdcd-openshift-default-54bcf7678c-8469t" podUID="81833794-e0f8-4927-91ba-d4567d3c8d11" containerName="istio-proxy" containerID="cri-o://4d4be5cfbf2774fab3d3e1324871a05277b5b48ac12eab50d0c397f0346464c8" gracePeriod=30
Jul 09 20:32:10.094024 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:10.093982    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f632bdcd-797cb97545-6zfdb"]
Jul 09 20:32:10.094526 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:10.094172    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f632bdcd-797cb97545-6zfdb" podUID="167f42f7-8dd2-4309-98e2-7ee2ca0d9d70" containerName="maas-api" containerID="cri-o://1c333e23ea3b082c087b21f47b8e9f1775526f86e45295bd686beb6ebcd531ef" gracePeriod=30
Jul 09 20:32:10.350131 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:10.350105    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f632bdcd-797cb97545-6zfdb"
Jul 09 20:32:10.410590 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:10.410558    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-csz88\" (UniqueName: \"kubernetes.io/projected/167f42f7-8dd2-4309-98e2-7ee2ca0d9d70-kube-api-access-csz88\") pod \"167f42f7-8dd2-4309-98e2-7ee2ca0d9d70\" (UID: \"167f42f7-8dd2-4309-98e2-7ee2ca0d9d70\") "
Jul 09 20:32:10.410785 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:10.410620    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/167f42f7-8dd2-4309-98e2-7ee2ca0d9d70-maas-api-tls\") pod \"167f42f7-8dd2-4309-98e2-7ee2ca0d9d70\" (UID: \"167f42f7-8dd2-4309-98e2-7ee2ca0d9d70\") "
Jul 09 20:32:10.410785 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:10.410644    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/167f42f7-8dd2-4309-98e2-7ee2ca0d9d70-openshift-service-ca\") pod \"167f42f7-8dd2-4309-98e2-7ee2ca0d9d70\" (UID: \"167f42f7-8dd2-4309-98e2-7ee2ca0d9d70\") "
Jul 09 20:32:10.411067 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:10.411039    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/167f42f7-8dd2-4309-98e2-7ee2ca0d9d70-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "167f42f7-8dd2-4309-98e2-7ee2ca0d9d70" (UID: "167f42f7-8dd2-4309-98e2-7ee2ca0d9d70"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 20:32:10.412695 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:10.412673    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/167f42f7-8dd2-4309-98e2-7ee2ca0d9d70-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "167f42f7-8dd2-4309-98e2-7ee2ca0d9d70" (UID: "167f42f7-8dd2-4309-98e2-7ee2ca0d9d70"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 20:32:10.412840 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:10.412825    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/167f42f7-8dd2-4309-98e2-7ee2ca0d9d70-kube-api-access-csz88" (OuterVolumeSpecName: "kube-api-access-csz88") pod "167f42f7-8dd2-4309-98e2-7ee2ca0d9d70" (UID: "167f42f7-8dd2-4309-98e2-7ee2ca0d9d70"). InnerVolumeSpecName "kube-api-access-csz88". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 20:32:10.511403 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:10.511292    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-csz88\" (UniqueName: \"kubernetes.io/projected/167f42f7-8dd2-4309-98e2-7ee2ca0d9d70-kube-api-access-csz88\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:32:10.511403 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:10.511345    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/167f42f7-8dd2-4309-98e2-7ee2ca0d9d70-maas-api-tls\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:32:10.511403 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:10.511359    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/167f42f7-8dd2-4309-98e2-7ee2ca0d9d70-openshift-service-ca\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:32:10.811395 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:10.811282    2562 generic.go:356] "Generic (PLEG): container finished" podID="167f42f7-8dd2-4309-98e2-7ee2ca0d9d70" containerID="1c333e23ea3b082c087b21f47b8e9f1775526f86e45295bd686beb6ebcd531ef" exitCode=0
Jul 09 20:32:10.811395 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:10.811339    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f632bdcd-797cb97545-6zfdb"
Jul 09 20:32:10.811395 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:10.811369    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f632bdcd-797cb97545-6zfdb" event={"ID":"167f42f7-8dd2-4309-98e2-7ee2ca0d9d70","Type":"ContainerDied","Data":"1c333e23ea3b082c087b21f47b8e9f1775526f86e45295bd686beb6ebcd531ef"}
Jul 09 20:32:10.811630 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:10.811427    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f632bdcd-797cb97545-6zfdb" event={"ID":"167f42f7-8dd2-4309-98e2-7ee2ca0d9d70","Type":"ContainerDied","Data":"61b392f9fc9639596e9de7f7320999f76bdb6f557896ef70bd00a0b058d22663"}
Jul 09 20:32:10.812155 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:10.811449    2562 scope.go:117] "RemoveContainer" containerID="1c333e23ea3b082c087b21f47b8e9f1775526f86e45295bd686beb6ebcd531ef"
Jul 09 20:32:10.821232 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:10.821216    2562 scope.go:117] "RemoveContainer" containerID="1c333e23ea3b082c087b21f47b8e9f1775526f86e45295bd686beb6ebcd531ef"
Jul 09 20:32:10.821499 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:32:10.821482    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c333e23ea3b082c087b21f47b8e9f1775526f86e45295bd686beb6ebcd531ef\": container with ID starting with 1c333e23ea3b082c087b21f47b8e9f1775526f86e45295bd686beb6ebcd531ef not found: ID does not exist" containerID="1c333e23ea3b082c087b21f47b8e9f1775526f86e45295bd686beb6ebcd531ef"
Jul 09 20:32:10.821556 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:10.821509    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c333e23ea3b082c087b21f47b8e9f1775526f86e45295bd686beb6ebcd531ef"} err="failed to get container status \"1c333e23ea3b082c087b21f47b8e9f1775526f86e45295bd686beb6ebcd531ef\": rpc error: code = NotFound desc = could not find container \"1c333e23ea3b082c087b21f47b8e9f1775526f86e45295bd686beb6ebcd531ef\": container with ID starting with 1c333e23ea3b082c087b21f47b8e9f1775526f86e45295bd686beb6ebcd531ef not found: ID does not exist"
Jul 09 20:32:10.833280 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:10.833260    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f632bdcd-797cb97545-6zfdb"]
Jul 09 20:32:10.836654 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:10.836634    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f632bdcd-797cb97545-6zfdb"]
Jul 09 20:32:12.242141 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:12.242107    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="167f42f7-8dd2-4309-98e2-7ee2ca0d9d70" path="/var/lib/kubelet/pods/167f42f7-8dd2-4309-98e2-7ee2ca0d9d70/volumes"
Jul 09 20:32:15.129308 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:15.129269    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f632bdcd-openshift-default-54bcf7678c-8469t"
Jul 09 20:32:15.251465 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:15.251430    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81833794-e0f8-4927-91ba-d4567d3c8d11-workload-certs\") pod \"81833794-e0f8-4927-91ba-d4567d3c8d11\" (UID: \"81833794-e0f8-4927-91ba-d4567d3c8d11\") "
Jul 09 20:32:15.251674 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:15.251475    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s4zv9\" (UniqueName: \"kubernetes.io/projected/81833794-e0f8-4927-91ba-d4567d3c8d11-kube-api-access-s4zv9\") pod \"81833794-e0f8-4927-91ba-d4567d3c8d11\" (UID: \"81833794-e0f8-4927-91ba-d4567d3c8d11\") "
Jul 09 20:32:15.251674 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:15.251508    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81833794-e0f8-4927-91ba-d4567d3c8d11-credential-socket\") pod \"81833794-e0f8-4927-91ba-d4567d3c8d11\" (UID: \"81833794-e0f8-4927-91ba-d4567d3c8d11\") "
Jul 09 20:32:15.251674 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:15.251536    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81833794-e0f8-4927-91ba-d4567d3c8d11-istiod-ca-cert\") pod \"81833794-e0f8-4927-91ba-d4567d3c8d11\" (UID: \"81833794-e0f8-4927-91ba-d4567d3c8d11\") "
Jul 09 20:32:15.251674 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:15.251569    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81833794-e0f8-4927-91ba-d4567d3c8d11-istio-data\") pod \"81833794-e0f8-4927-91ba-d4567d3c8d11\" (UID: \"81833794-e0f8-4927-91ba-d4567d3c8d11\") "
Jul 09 20:32:15.251674 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:15.251639    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81833794-e0f8-4927-91ba-d4567d3c8d11-istio-podinfo\") pod \"81833794-e0f8-4927-91ba-d4567d3c8d11\" (UID: \"81833794-e0f8-4927-91ba-d4567d3c8d11\") "
Jul 09 20:32:15.251938 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:15.251749    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81833794-e0f8-4927-91ba-d4567d3c8d11-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "81833794-e0f8-4927-91ba-d4567d3c8d11" (UID: "81833794-e0f8-4927-91ba-d4567d3c8d11"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 20:32:15.251938 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:15.251871    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81833794-e0f8-4927-91ba-d4567d3c8d11-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "81833794-e0f8-4927-91ba-d4567d3c8d11" (UID: "81833794-e0f8-4927-91ba-d4567d3c8d11"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 20:32:15.252051 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:15.251963    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81833794-e0f8-4927-91ba-d4567d3c8d11-istio-data" (OuterVolumeSpecName: "istio-data") pod "81833794-e0f8-4927-91ba-d4567d3c8d11" (UID: "81833794-e0f8-4927-91ba-d4567d3c8d11"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 20:32:15.252051 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:15.252035    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81833794-e0f8-4927-91ba-d4567d3c8d11-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "81833794-e0f8-4927-91ba-d4567d3c8d11" (UID: "81833794-e0f8-4927-91ba-d4567d3c8d11"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 20:32:15.252163 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:15.251980    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81833794-e0f8-4927-91ba-d4567d3c8d11-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "81833794-e0f8-4927-91ba-d4567d3c8d11" (UID: "81833794-e0f8-4927-91ba-d4567d3c8d11"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 20:32:15.252163 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:15.252101    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81833794-e0f8-4927-91ba-d4567d3c8d11-workload-socket\") pod \"81833794-e0f8-4927-91ba-d4567d3c8d11\" (UID: \"81833794-e0f8-4927-91ba-d4567d3c8d11\") "
Jul 09 20:32:15.252163 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:15.252145    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81833794-e0f8-4927-91ba-d4567d3c8d11-istio-token\") pod \"81833794-e0f8-4927-91ba-d4567d3c8d11\" (UID: \"81833794-e0f8-4927-91ba-d4567d3c8d11\") "
Jul 09 20:32:15.252327 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:15.252191    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81833794-e0f8-4927-91ba-d4567d3c8d11-istio-envoy\") pod \"81833794-e0f8-4927-91ba-d4567d3c8d11\" (UID: \"81833794-e0f8-4927-91ba-d4567d3c8d11\") "
Jul 09 20:32:15.252523 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:15.252498    2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81833794-e0f8-4927-91ba-d4567d3c8d11-istiod-ca-cert\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:32:15.252523 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:15.252523    2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81833794-e0f8-4927-91ba-d4567d3c8d11-istio-data\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:32:15.252716 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:15.252536    2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81833794-e0f8-4927-91ba-d4567d3c8d11-workload-socket\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:32:15.252716 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:15.252547    2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81833794-e0f8-4927-91ba-d4567d3c8d11-workload-certs\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:32:15.252716 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:15.252560    2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81833794-e0f8-4927-91ba-d4567d3c8d11-credential-socket\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:32:15.254083 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:15.254048    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/81833794-e0f8-4927-91ba-d4567d3c8d11-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "81833794-e0f8-4927-91ba-d4567d3c8d11" (UID: "81833794-e0f8-4927-91ba-d4567d3c8d11"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 09 20:32:15.254188 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:15.254077    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81833794-e0f8-4927-91ba-d4567d3c8d11-istio-token" (OuterVolumeSpecName: "istio-token") pod "81833794-e0f8-4927-91ba-d4567d3c8d11" (UID: "81833794-e0f8-4927-91ba-d4567d3c8d11"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 20:32:15.254263 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:15.254244    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81833794-e0f8-4927-91ba-d4567d3c8d11-kube-api-access-s4zv9" (OuterVolumeSpecName: "kube-api-access-s4zv9") pod "81833794-e0f8-4927-91ba-d4567d3c8d11" (UID: "81833794-e0f8-4927-91ba-d4567d3c8d11"). InnerVolumeSpecName "kube-api-access-s4zv9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 20:32:15.254368 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:15.254351    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81833794-e0f8-4927-91ba-d4567d3c8d11-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "81833794-e0f8-4927-91ba-d4567d3c8d11" (UID: "81833794-e0f8-4927-91ba-d4567d3c8d11"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 20:32:15.353638 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:15.353591    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4zv9\" (UniqueName: \"kubernetes.io/projected/81833794-e0f8-4927-91ba-d4567d3c8d11-kube-api-access-s4zv9\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:32:15.353638 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:15.353632    2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81833794-e0f8-4927-91ba-d4567d3c8d11-istio-podinfo\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:32:15.353638 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:15.353642    2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81833794-e0f8-4927-91ba-d4567d3c8d11-istio-token\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:32:15.353638 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:15.353650    2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81833794-e0f8-4927-91ba-d4567d3c8d11-istio-envoy\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:32:15.839356 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:15.839288    2562 generic.go:356] "Generic (PLEG): container finished" podID="81833794-e0f8-4927-91ba-d4567d3c8d11" containerID="4d4be5cfbf2774fab3d3e1324871a05277b5b48ac12eab50d0c397f0346464c8" exitCode=0
Jul 09 20:32:15.839520 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:15.839383    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f632bdcd-openshift-default-54bcf7678c-8469t"
Jul 09 20:32:15.839520 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:15.839378    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f632bdcd-openshift-default-54bcf7678c-8469t" event={"ID":"81833794-e0f8-4927-91ba-d4567d3c8d11","Type":"ContainerDied","Data":"4d4be5cfbf2774fab3d3e1324871a05277b5b48ac12eab50d0c397f0346464c8"}
Jul 09 20:32:15.839520 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:15.839497    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f632bdcd-openshift-default-54bcf7678c-8469t" event={"ID":"81833794-e0f8-4927-91ba-d4567d3c8d11","Type":"ContainerDied","Data":"8d3c7a644c1accf8266d350ccfd43bae0855af30f687cf41573c303a39992a9b"}
Jul 09 20:32:15.839520 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:15.839520    2562 scope.go:117] "RemoveContainer" containerID="4d4be5cfbf2774fab3d3e1324871a05277b5b48ac12eab50d0c397f0346464c8"
Jul 09 20:32:15.850206 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:15.850190    2562 scope.go:117] "RemoveContainer" containerID="4d4be5cfbf2774fab3d3e1324871a05277b5b48ac12eab50d0c397f0346464c8"
Jul 09 20:32:15.850513 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:32:15.850491    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d4be5cfbf2774fab3d3e1324871a05277b5b48ac12eab50d0c397f0346464c8\": container with ID starting with 4d4be5cfbf2774fab3d3e1324871a05277b5b48ac12eab50d0c397f0346464c8 not found: ID does not exist" containerID="4d4be5cfbf2774fab3d3e1324871a05277b5b48ac12eab50d0c397f0346464c8"
Jul 09 20:32:15.850617 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:15.850519    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d4be5cfbf2774fab3d3e1324871a05277b5b48ac12eab50d0c397f0346464c8"} err="failed to get container status \"4d4be5cfbf2774fab3d3e1324871a05277b5b48ac12eab50d0c397f0346464c8\": rpc error: code = NotFound desc = could not find container \"4d4be5cfbf2774fab3d3e1324871a05277b5b48ac12eab50d0c397f0346464c8\": container with ID starting with 4d4be5cfbf2774fab3d3e1324871a05277b5b48ac12eab50d0c397f0346464c8 not found: ID does not exist"
Jul 09 20:32:15.867509 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:15.867486    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-f632bdcd-openshift-default-54bcf7678c-8469t"]
Jul 09 20:32:15.872994 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:15.872974    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-f632bdcd-openshift-default-54bcf7678c-8469t"]
Jul 09 20:32:15.912430 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:15.912389    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-66f6228f-openshift-default-645cfb4867-h49j2"]
Jul 09 20:32:15.912992 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:15.912962    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81833794-e0f8-4927-91ba-d4567d3c8d11" containerName="istio-proxy"
Jul 09 20:32:15.912992 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:15.912995    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="81833794-e0f8-4927-91ba-d4567d3c8d11" containerName="istio-proxy"
Jul 09 20:32:15.913216 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:15.913027    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="167f42f7-8dd2-4309-98e2-7ee2ca0d9d70" containerName="maas-api"
Jul 09 20:32:15.913216 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:15.913036    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="167f42f7-8dd2-4309-98e2-7ee2ca0d9d70" containerName="maas-api"
Jul 09 20:32:15.913216 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:15.913148    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="81833794-e0f8-4927-91ba-d4567d3c8d11" containerName="istio-proxy"
Jul 09 20:32:15.913216 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:15.913166    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="167f42f7-8dd2-4309-98e2-7ee2ca0d9d70" containerName="maas-api"
Jul 09 20:32:15.918040 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:15.918010    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-66f6228f-openshift-default-645cfb4867-h49j2"
Jul 09 20:32:15.921928 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:15.921904    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-66f6228f-openshift-default-dockercfg-5qk9p\""
Jul 09 20:32:15.929835 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:15.929807    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-66f6228f-openshift-default-645cfb4867-h49j2"]
Jul 09 20:32:15.958141 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:15.958114    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79bc34f4-8d40-4899-a7a0-4f0a52ddb7f7-istio-envoy\") pod \"e2e-ait-66f6228f-openshift-default-645cfb4867-h49j2\" (UID: \"79bc34f4-8d40-4899-a7a0-4f0a52ddb7f7\") " pod="openshift-ingress/e2e-ait-66f6228f-openshift-default-645cfb4867-h49j2"
Jul 09 20:32:15.958324 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:15.958172    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79bc34f4-8d40-4899-a7a0-4f0a52ddb7f7-credential-socket\") pod \"e2e-ait-66f6228f-openshift-default-645cfb4867-h49j2\" (UID: \"79bc34f4-8d40-4899-a7a0-4f0a52ddb7f7\") " pod="openshift-ingress/e2e-ait-66f6228f-openshift-default-645cfb4867-h49j2"
Jul 09 20:32:15.958324 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:15.958230    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79bc34f4-8d40-4899-a7a0-4f0a52ddb7f7-istio-data\") pod \"e2e-ait-66f6228f-openshift-default-645cfb4867-h49j2\" (UID: \"79bc34f4-8d40-4899-a7a0-4f0a52ddb7f7\") " pod="openshift-ingress/e2e-ait-66f6228f-openshift-default-645cfb4867-h49j2"
Jul 09 20:32:15.958445 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:15.958324    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79bc34f4-8d40-4899-a7a0-4f0a52ddb7f7-istio-podinfo\") pod \"e2e-ait-66f6228f-openshift-default-645cfb4867-h49j2\" (UID: \"79bc34f4-8d40-4899-a7a0-4f0a52ddb7f7\") " pod="openshift-ingress/e2e-ait-66f6228f-openshift-default-645cfb4867-h49j2"
Jul 09 20:32:15.958445 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:15.958355    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79bc34f4-8d40-4899-a7a0-4f0a52ddb7f7-istio-token\") pod \"e2e-ait-66f6228f-openshift-default-645cfb4867-h49j2\" (UID: \"79bc34f4-8d40-4899-a7a0-4f0a52ddb7f7\") " pod="openshift-ingress/e2e-ait-66f6228f-openshift-default-645cfb4867-h49j2"
Jul 09 20:32:15.958445 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:15.958377    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79bc34f4-8d40-4899-a7a0-4f0a52ddb7f7-workload-socket\") pod \"e2e-ait-66f6228f-openshift-default-645cfb4867-h49j2\" (UID: \"79bc34f4-8d40-4899-a7a0-4f0a52ddb7f7\") " pod="openshift-ingress/e2e-ait-66f6228f-openshift-default-645cfb4867-h49j2"
Jul 09 20:32:15.958613 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:15.958450    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79bc34f4-8d40-4899-a7a0-4f0a52ddb7f7-workload-certs\") pod \"e2e-ait-66f6228f-openshift-default-645cfb4867-h49j2\" (UID: \"79bc34f4-8d40-4899-a7a0-4f0a52ddb7f7\") " pod="openshift-ingress/e2e-ait-66f6228f-openshift-default-645cfb4867-h49j2"
Jul 09 20:32:15.958613 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:15.958493    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-759wm\" (UniqueName: \"kubernetes.io/projected/79bc34f4-8d40-4899-a7a0-4f0a52ddb7f7-kube-api-access-759wm\") pod \"e2e-ait-66f6228f-openshift-default-645cfb4867-h49j2\" (UID: \"79bc34f4-8d40-4899-a7a0-4f0a52ddb7f7\") " pod="openshift-ingress/e2e-ait-66f6228f-openshift-default-645cfb4867-h49j2"
Jul 09 20:32:15.958613 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:15.958556    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79bc34f4-8d40-4899-a7a0-4f0a52ddb7f7-istiod-ca-cert\") pod \"e2e-ait-66f6228f-openshift-default-645cfb4867-h49j2\" (UID: \"79bc34f4-8d40-4899-a7a0-4f0a52ddb7f7\") " pod="openshift-ingress/e2e-ait-66f6228f-openshift-default-645cfb4867-h49j2"
Jul 09 20:32:16.059742 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:16.059709    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79bc34f4-8d40-4899-a7a0-4f0a52ddb7f7-istio-token\") pod \"e2e-ait-66f6228f-openshift-default-645cfb4867-h49j2\" (UID: \"79bc34f4-8d40-4899-a7a0-4f0a52ddb7f7\") " pod="openshift-ingress/e2e-ait-66f6228f-openshift-default-645cfb4867-h49j2"
Jul 09 20:32:16.059742 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:16.059747    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79bc34f4-8d40-4899-a7a0-4f0a52ddb7f7-workload-socket\") pod \"e2e-ait-66f6228f-openshift-default-645cfb4867-h49j2\" (UID: \"79bc34f4-8d40-4899-a7a0-4f0a52ddb7f7\") " pod="openshift-ingress/e2e-ait-66f6228f-openshift-default-645cfb4867-h49j2"
Jul 09 20:32:16.060003 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:16.059766    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79bc34f4-8d40-4899-a7a0-4f0a52ddb7f7-workload-certs\") pod \"e2e-ait-66f6228f-openshift-default-645cfb4867-h49j2\" (UID: \"79bc34f4-8d40-4899-a7a0-4f0a52ddb7f7\") " pod="openshift-ingress/e2e-ait-66f6228f-openshift-default-645cfb4867-h49j2"
Jul 09 20:32:16.060003 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:16.059797    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-759wm\" (UniqueName: \"kubernetes.io/projected/79bc34f4-8d40-4899-a7a0-4f0a52ddb7f7-kube-api-access-759wm\") pod \"e2e-ait-66f6228f-openshift-default-645cfb4867-h49j2\" (UID: \"79bc34f4-8d40-4899-a7a0-4f0a52ddb7f7\") " pod="openshift-ingress/e2e-ait-66f6228f-openshift-default-645cfb4867-h49j2"
Jul 09 20:32:16.060003 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:16.059833    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79bc34f4-8d40-4899-a7a0-4f0a52ddb7f7-istiod-ca-cert\") pod \"e2e-ait-66f6228f-openshift-default-645cfb4867-h49j2\" (UID: \"79bc34f4-8d40-4899-a7a0-4f0a52ddb7f7\") " pod="openshift-ingress/e2e-ait-66f6228f-openshift-default-645cfb4867-h49j2"
Jul 09 20:32:16.060003 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:16.059883    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79bc34f4-8d40-4899-a7a0-4f0a52ddb7f7-istio-envoy\") pod \"e2e-ait-66f6228f-openshift-default-645cfb4867-h49j2\" (UID: \"79bc34f4-8d40-4899-a7a0-4f0a52ddb7f7\") " pod="openshift-ingress/e2e-ait-66f6228f-openshift-default-645cfb4867-h49j2"
Jul 09 20:32:16.060003 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:16.059967    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79bc34f4-8d40-4899-a7a0-4f0a52ddb7f7-credential-socket\") pod \"e2e-ait-66f6228f-openshift-default-645cfb4867-h49j2\" (UID: \"79bc34f4-8d40-4899-a7a0-4f0a52ddb7f7\") " pod="openshift-ingress/e2e-ait-66f6228f-openshift-default-645cfb4867-h49j2"
Jul 09 20:32:16.060003 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:16.059997    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79bc34f4-8d40-4899-a7a0-4f0a52ddb7f7-istio-data\") pod \"e2e-ait-66f6228f-openshift-default-645cfb4867-h49j2\" (UID: \"79bc34f4-8d40-4899-a7a0-4f0a52ddb7f7\") " pod="openshift-ingress/e2e-ait-66f6228f-openshift-default-645cfb4867-h49j2"
Jul 09 20:32:16.060294 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:16.060033    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79bc34f4-8d40-4899-a7a0-4f0a52ddb7f7-istio-podinfo\") pod \"e2e-ait-66f6228f-openshift-default-645cfb4867-h49j2\" (UID: \"79bc34f4-8d40-4899-a7a0-4f0a52ddb7f7\") " pod="openshift-ingress/e2e-ait-66f6228f-openshift-default-645cfb4867-h49j2"
Jul 09 20:32:16.060294 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:16.060265    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79bc34f4-8d40-4899-a7a0-4f0a52ddb7f7-workload-socket\") pod \"e2e-ait-66f6228f-openshift-default-645cfb4867-h49j2\" (UID: \"79bc34f4-8d40-4899-a7a0-4f0a52ddb7f7\") " pod="openshift-ingress/e2e-ait-66f6228f-openshift-default-645cfb4867-h49j2"
Jul 09 20:32:16.060421 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:16.060348    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79bc34f4-8d40-4899-a7a0-4f0a52ddb7f7-workload-certs\") pod \"e2e-ait-66f6228f-openshift-default-645cfb4867-h49j2\" (UID: \"79bc34f4-8d40-4899-a7a0-4f0a52ddb7f7\") " pod="openshift-ingress/e2e-ait-66f6228f-openshift-default-645cfb4867-h49j2"
Jul 09 20:32:16.060421 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:16.060388    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79bc34f4-8d40-4899-a7a0-4f0a52ddb7f7-istio-data\") pod \"e2e-ait-66f6228f-openshift-default-645cfb4867-h49j2\" (UID: \"79bc34f4-8d40-4899-a7a0-4f0a52ddb7f7\") " pod="openshift-ingress/e2e-ait-66f6228f-openshift-default-645cfb4867-h49j2"
Jul 09 20:32:16.060527 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:16.060441    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79bc34f4-8d40-4899-a7a0-4f0a52ddb7f7-credential-socket\") pod \"e2e-ait-66f6228f-openshift-default-645cfb4867-h49j2\" (UID: \"79bc34f4-8d40-4899-a7a0-4f0a52ddb7f7\") " pod="openshift-ingress/e2e-ait-66f6228f-openshift-default-645cfb4867-h49j2"
Jul 09 20:32:16.060634 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:16.060616    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79bc34f4-8d40-4899-a7a0-4f0a52ddb7f7-istiod-ca-cert\") pod \"e2e-ait-66f6228f-openshift-default-645cfb4867-h49j2\" (UID: \"79bc34f4-8d40-4899-a7a0-4f0a52ddb7f7\") " pod="openshift-ingress/e2e-ait-66f6228f-openshift-default-645cfb4867-h49j2"
Jul 09 20:32:16.062449 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:16.062427    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79bc34f4-8d40-4899-a7a0-4f0a52ddb7f7-istio-envoy\") pod \"e2e-ait-66f6228f-openshift-default-645cfb4867-h49j2\" (UID: \"79bc34f4-8d40-4899-a7a0-4f0a52ddb7f7\") " pod="openshift-ingress/e2e-ait-66f6228f-openshift-default-645cfb4867-h49j2"
Jul 09 20:32:16.062702 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:16.062676    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79bc34f4-8d40-4899-a7a0-4f0a52ddb7f7-istio-podinfo\") pod \"e2e-ait-66f6228f-openshift-default-645cfb4867-h49j2\" (UID: \"79bc34f4-8d40-4899-a7a0-4f0a52ddb7f7\") " pod="openshift-ingress/e2e-ait-66f6228f-openshift-default-645cfb4867-h49j2"
Jul 09 20:32:16.070845 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:16.070823    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79bc34f4-8d40-4899-a7a0-4f0a52ddb7f7-istio-token\") pod \"e2e-ait-66f6228f-openshift-default-645cfb4867-h49j2\" (UID: \"79bc34f4-8d40-4899-a7a0-4f0a52ddb7f7\") " pod="openshift-ingress/e2e-ait-66f6228f-openshift-default-645cfb4867-h49j2"
Jul 09 20:32:16.070956 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:16.070853    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-759wm\" (UniqueName: \"kubernetes.io/projected/79bc34f4-8d40-4899-a7a0-4f0a52ddb7f7-kube-api-access-759wm\") pod \"e2e-ait-66f6228f-openshift-default-645cfb4867-h49j2\" (UID: \"79bc34f4-8d40-4899-a7a0-4f0a52ddb7f7\") " pod="openshift-ingress/e2e-ait-66f6228f-openshift-default-645cfb4867-h49j2"
Jul 09 20:32:16.230249 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:16.230214    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-66f6228f-openshift-default-645cfb4867-h49j2"
Jul 09 20:32:16.242827 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:16.242797    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81833794-e0f8-4927-91ba-d4567d3c8d11" path="/var/lib/kubelet/pods/81833794-e0f8-4927-91ba-d4567d3c8d11/volumes"
Jul 09 20:32:16.370805 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:16.370777    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-66f6228f-openshift-default-645cfb4867-h49j2"]
Jul 09 20:32:16.372841 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:32:16.372807    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79bc34f4_8d40_4899_a7a0_4f0a52ddb7f7.slice/crio-e1250a56f81998f10eccd325d04022470dce388f3d4b3bbf38d81cebac59179a WatchSource:0}: Error finding container e1250a56f81998f10eccd325d04022470dce388f3d4b3bbf38d81cebac59179a: Status 404 returned error can't find the container with id e1250a56f81998f10eccd325d04022470dce388f3d4b3bbf38d81cebac59179a
Jul 09 20:32:16.374926 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:16.374894    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 09 20:32:16.375017 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:16.374963    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 09 20:32:16.375059 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:16.375014    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 09 20:32:16.846627 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:16.846579    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-66f6228f-openshift-default-645cfb4867-h49j2" event={"ID":"79bc34f4-8d40-4899-a7a0-4f0a52ddb7f7","Type":"ContainerStarted","Data":"bb65e7d67fb7ee209c4b1f757210b51c54f384ecd5ed3278b847fc6fc00c6dbf"}
Jul 09 20:32:16.846810 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:16.846683    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-66f6228f-openshift-default-645cfb4867-h49j2" event={"ID":"79bc34f4-8d40-4899-a7a0-4f0a52ddb7f7","Type":"ContainerStarted","Data":"e1250a56f81998f10eccd325d04022470dce388f3d4b3bbf38d81cebac59179a"}
Jul 09 20:32:16.870081 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:16.870028    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-66f6228f-openshift-default-645cfb4867-h49j2" podStartSLOduration=1.870007894 podStartE2EDuration="1.870007894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 20:32:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 20:32:16.867553108 +0000 UTC m=+1727.285222922" watchObservedRunningTime="2026-07-09 20:32:16.870007894 +0000 UTC m=+1727.287677702"
Jul 09 20:32:17.186636 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:17.186552    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-66f6228f-85c6958b5f-hzzg6"]
Jul 09 20:32:17.190434 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:17.190412    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-66f6228f-85c6958b5f-hzzg6"
Jul 09 20:32:17.193485 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:17.193462    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-66f6228f-serving-cert\""
Jul 09 20:32:17.207334 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:17.207290    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-66f6228f-85c6958b5f-hzzg6"]
Jul 09 20:32:17.230492 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:17.230453    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-66f6228f-openshift-default-645cfb4867-h49j2"
Jul 09 20:32:17.235250 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:17.235225    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-66f6228f-openshift-default-645cfb4867-h49j2"
Jul 09 20:32:17.273429 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:17.273398    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dfeacb01-d716-4d46-8f49-4479313c170e-maas-api-tls\") pod \"maas-api-e2e-ait-66f6228f-85c6958b5f-hzzg6\" (UID: \"dfeacb01-d716-4d46-8f49-4479313c170e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-66f6228f-85c6958b5f-hzzg6"
Jul 09 20:32:17.273634 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:17.273516    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2cfj\" (UniqueName: \"kubernetes.io/projected/dfeacb01-d716-4d46-8f49-4479313c170e-kube-api-access-t2cfj\") pod \"maas-api-e2e-ait-66f6228f-85c6958b5f-hzzg6\" (UID: \"dfeacb01-d716-4d46-8f49-4479313c170e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-66f6228f-85c6958b5f-hzzg6"
Jul 09 20:32:17.273634 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:17.273573    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfeacb01-d716-4d46-8f49-4479313c170e-openshift-service-ca\") pod \"maas-api-e2e-ait-66f6228f-85c6958b5f-hzzg6\" (UID: \"dfeacb01-d716-4d46-8f49-4479313c170e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-66f6228f-85c6958b5f-hzzg6"
Jul 09 20:32:17.374508 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:17.374476    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2cfj\" (UniqueName: \"kubernetes.io/projected/dfeacb01-d716-4d46-8f49-4479313c170e-kube-api-access-t2cfj\") pod \"maas-api-e2e-ait-66f6228f-85c6958b5f-hzzg6\" (UID: \"dfeacb01-d716-4d46-8f49-4479313c170e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-66f6228f-85c6958b5f-hzzg6"
Jul 09 20:32:17.374683 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:17.374524    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfeacb01-d716-4d46-8f49-4479313c170e-openshift-service-ca\") pod \"maas-api-e2e-ait-66f6228f-85c6958b5f-hzzg6\" (UID: \"dfeacb01-d716-4d46-8f49-4479313c170e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-66f6228f-85c6958b5f-hzzg6"
Jul 09 20:32:17.374683 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:17.374585    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dfeacb01-d716-4d46-8f49-4479313c170e-maas-api-tls\") pod \"maas-api-e2e-ait-66f6228f-85c6958b5f-hzzg6\" (UID: \"dfeacb01-d716-4d46-8f49-4479313c170e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-66f6228f-85c6958b5f-hzzg6"
Jul 09 20:32:17.374831 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:32:17.374695    2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-66f6228f-serving-cert: secret "maas-api-e2e-ait-66f6228f-serving-cert" not found
Jul 09 20:32:17.374831 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:32:17.374759    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dfeacb01-d716-4d46-8f49-4479313c170e-maas-api-tls podName:dfeacb01-d716-4d46-8f49-4479313c170e nodeName:}" failed. No retries permitted until 2026-07-09 20:32:17.874740804 +0000 UTC m=+1728.292410592 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/dfeacb01-d716-4d46-8f49-4479313c170e-maas-api-tls") pod "maas-api-e2e-ait-66f6228f-85c6958b5f-hzzg6" (UID: "dfeacb01-d716-4d46-8f49-4479313c170e") : secret "maas-api-e2e-ait-66f6228f-serving-cert" not found
Jul 09 20:32:17.375256 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:17.375231    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfeacb01-d716-4d46-8f49-4479313c170e-openshift-service-ca\") pod \"maas-api-e2e-ait-66f6228f-85c6958b5f-hzzg6\" (UID: \"dfeacb01-d716-4d46-8f49-4479313c170e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-66f6228f-85c6958b5f-hzzg6"
Jul 09 20:32:17.410586 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:17.410553    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2cfj\" (UniqueName: \"kubernetes.io/projected/dfeacb01-d716-4d46-8f49-4479313c170e-kube-api-access-t2cfj\") pod \"maas-api-e2e-ait-66f6228f-85c6958b5f-hzzg6\" (UID: \"dfeacb01-d716-4d46-8f49-4479313c170e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-66f6228f-85c6958b5f-hzzg6"
Jul 09 20:32:17.852730 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:17.852700    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-66f6228f-openshift-default-645cfb4867-h49j2"
Jul 09 20:32:17.853629 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:17.853604    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-66f6228f-openshift-default-645cfb4867-h49j2"
Jul 09 20:32:17.880434 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:17.880401    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dfeacb01-d716-4d46-8f49-4479313c170e-maas-api-tls\") pod \"maas-api-e2e-ait-66f6228f-85c6958b5f-hzzg6\" (UID: \"dfeacb01-d716-4d46-8f49-4479313c170e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-66f6228f-85c6958b5f-hzzg6"
Jul 09 20:32:17.883228 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:17.883194    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dfeacb01-d716-4d46-8f49-4479313c170e-maas-api-tls\") pod \"maas-api-e2e-ait-66f6228f-85c6958b5f-hzzg6\" (UID: \"dfeacb01-d716-4d46-8f49-4479313c170e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-66f6228f-85c6958b5f-hzzg6"
Jul 09 20:32:18.103730 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:18.103627    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-66f6228f-85c6958b5f-hzzg6"
Jul 09 20:32:18.244793 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:18.244767    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-66f6228f-85c6958b5f-hzzg6"]
Jul 09 20:32:18.247111 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:32:18.247083    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfeacb01_d716_4d46_8f49_4479313c170e.slice/crio-fc1f1b0158d6916f0da0bf3025a480be71583842ab4fb23f66841caf2a86b5ea WatchSource:0}: Error finding container fc1f1b0158d6916f0da0bf3025a480be71583842ab4fb23f66841caf2a86b5ea: Status 404 returned error can't find the container with id fc1f1b0158d6916f0da0bf3025a480be71583842ab4fb23f66841caf2a86b5ea
Jul 09 20:32:18.862292 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:18.862256    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-66f6228f-85c6958b5f-hzzg6" event={"ID":"dfeacb01-d716-4d46-8f49-4479313c170e","Type":"ContainerStarted","Data":"441650ea20d19233bb511d7470a715fdc0b9e5d414f51b733d9a3c4198fa205c"}
Jul 09 20:32:18.862504 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:18.862330    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-66f6228f-85c6958b5f-hzzg6" event={"ID":"dfeacb01-d716-4d46-8f49-4479313c170e","Type":"ContainerStarted","Data":"fc1f1b0158d6916f0da0bf3025a480be71583842ab4fb23f66841caf2a86b5ea"}
Jul 09 20:32:18.879018 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:18.878909    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-66f6228f-85c6958b5f-hzzg6" podStartSLOduration=1.5068495689999999 podStartE2EDuration="1.87889432s" totalImagesPullingTime="372.044751ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 20:32:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 20:32:18.878430289 +0000 UTC m=+1729.296100098" watchObservedRunningTime="2026-07-09 20:32:18.87889432 +0000 UTC m=+1729.296564128"
Jul 09 20:32:19.867454 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:19.867420    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-66f6228f-85c6958b5f-hzzg6"
Jul 09 20:32:22.647999 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:22.647959    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-66f6228f-85c6958b5f-hzzg6"]
Jul 09 20:32:22.648555 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:22.648224    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-66f6228f-85c6958b5f-hzzg6" podUID="dfeacb01-d716-4d46-8f49-4479313c170e" containerName="maas-api" containerID="cri-o://441650ea20d19233bb511d7470a715fdc0b9e5d414f51b733d9a3c4198fa205c" gracePeriod=30
Jul 09 20:32:22.884143 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:22.884114    2562 generic.go:356] "Generic (PLEG): container finished" podID="dfeacb01-d716-4d46-8f49-4479313c170e" containerID="441650ea20d19233bb511d7470a715fdc0b9e5d414f51b733d9a3c4198fa205c" exitCode=0
Jul 09 20:32:22.884254 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:22.884191    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-66f6228f-85c6958b5f-hzzg6" event={"ID":"dfeacb01-d716-4d46-8f49-4479313c170e","Type":"ContainerDied","Data":"441650ea20d19233bb511d7470a715fdc0b9e5d414f51b733d9a3c4198fa205c"}
Jul 09 20:32:22.884292 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:22.884255    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-66f6228f-85c6958b5f-hzzg6" event={"ID":"dfeacb01-d716-4d46-8f49-4479313c170e","Type":"ContainerDied","Data":"fc1f1b0158d6916f0da0bf3025a480be71583842ab4fb23f66841caf2a86b5ea"}
Jul 09 20:32:22.884292 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:22.884280    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc1f1b0158d6916f0da0bf3025a480be71583842ab4fb23f66841caf2a86b5ea"
Jul 09 20:32:22.896653 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:22.896633    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-66f6228f-85c6958b5f-hzzg6"
Jul 09 20:32:23.027191 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:23.027161    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfeacb01-d716-4d46-8f49-4479313c170e-openshift-service-ca\") pod \"dfeacb01-d716-4d46-8f49-4479313c170e\" (UID: \"dfeacb01-d716-4d46-8f49-4479313c170e\") "
Jul 09 20:32:23.027415 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:23.027230    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dfeacb01-d716-4d46-8f49-4479313c170e-maas-api-tls\") pod \"dfeacb01-d716-4d46-8f49-4479313c170e\" (UID: \"dfeacb01-d716-4d46-8f49-4479313c170e\") "
Jul 09 20:32:23.027415 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:23.027262    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t2cfj\" (UniqueName: \"kubernetes.io/projected/dfeacb01-d716-4d46-8f49-4479313c170e-kube-api-access-t2cfj\") pod \"dfeacb01-d716-4d46-8f49-4479313c170e\" (UID: \"dfeacb01-d716-4d46-8f49-4479313c170e\") "
Jul 09 20:32:23.027584 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:23.027556    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfeacb01-d716-4d46-8f49-4479313c170e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "dfeacb01-d716-4d46-8f49-4479313c170e" (UID: "dfeacb01-d716-4d46-8f49-4479313c170e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 20:32:23.029349 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:23.029312    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dfeacb01-d716-4d46-8f49-4479313c170e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "dfeacb01-d716-4d46-8f49-4479313c170e" (UID: "dfeacb01-d716-4d46-8f49-4479313c170e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 20:32:23.029460 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:23.029397    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfeacb01-d716-4d46-8f49-4479313c170e-kube-api-access-t2cfj" (OuterVolumeSpecName: "kube-api-access-t2cfj") pod "dfeacb01-d716-4d46-8f49-4479313c170e" (UID: "dfeacb01-d716-4d46-8f49-4479313c170e"). InnerVolumeSpecName "kube-api-access-t2cfj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 20:32:23.128635 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:23.128597    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dfeacb01-d716-4d46-8f49-4479313c170e-maas-api-tls\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:32:23.128635 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:23.128627    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2cfj\" (UniqueName: \"kubernetes.io/projected/dfeacb01-d716-4d46-8f49-4479313c170e-kube-api-access-t2cfj\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:32:23.128635 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:23.128637    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfeacb01-d716-4d46-8f49-4479313c170e-openshift-service-ca\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:32:23.889327 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:23.889275    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-66f6228f-85c6958b5f-hzzg6"
Jul 09 20:32:23.909895 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:23.909864    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-66f6228f-85c6958b5f-hzzg6"]
Jul 09 20:32:23.913714 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:23.913691    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-66f6228f-85c6958b5f-hzzg6"]
Jul 09 20:32:24.242218 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:24.242186    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dfeacb01-d716-4d46-8f49-4479313c170e" path="/var/lib/kubelet/pods/dfeacb01-d716-4d46-8f49-4479313c170e/volumes"
Jul 09 20:32:28.430181 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:28.430145    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-66f6228f-openshift-default-645cfb4867-h49j2"]
Jul 09 20:32:28.430628 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:28.430349    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-66f6228f-openshift-default-645cfb4867-h49j2" podUID="79bc34f4-8d40-4899-a7a0-4f0a52ddb7f7" containerName="istio-proxy" containerID="cri-o://bb65e7d67fb7ee209c4b1f757210b51c54f384ecd5ed3278b847fc6fc00c6dbf" gracePeriod=30
Jul 09 20:32:32.855142 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:32.855103    2562 patch_prober.go:28] interesting pod/e2e-ait-66f6228f-openshift-default-645cfb4867-h49j2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 09 20:32:32.855534 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:32.855172    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-66f6228f-openshift-default-645cfb4867-h49j2" podUID="79bc34f4-8d40-4899-a7a0-4f0a52ddb7f7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 20:32:33.679708 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:33.679685    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-66f6228f-openshift-default-645cfb4867-h49j2"
Jul 09 20:32:33.829633 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:33.829596    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79bc34f4-8d40-4899-a7a0-4f0a52ddb7f7-workload-socket\") pod \"79bc34f4-8d40-4899-a7a0-4f0a52ddb7f7\" (UID: \"79bc34f4-8d40-4899-a7a0-4f0a52ddb7f7\") "
Jul 09 20:32:33.829841 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:33.829643    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79bc34f4-8d40-4899-a7a0-4f0a52ddb7f7-istio-token\") pod \"79bc34f4-8d40-4899-a7a0-4f0a52ddb7f7\" (UID: \"79bc34f4-8d40-4899-a7a0-4f0a52ddb7f7\") "
Jul 09 20:32:33.829841 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:33.829680    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79bc34f4-8d40-4899-a7a0-4f0a52ddb7f7-istio-podinfo\") pod \"79bc34f4-8d40-4899-a7a0-4f0a52ddb7f7\" (UID: \"79bc34f4-8d40-4899-a7a0-4f0a52ddb7f7\") "
Jul 09 20:32:33.829841 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:33.829719    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79bc34f4-8d40-4899-a7a0-4f0a52ddb7f7-workload-certs\") pod \"79bc34f4-8d40-4899-a7a0-4f0a52ddb7f7\" (UID: \"79bc34f4-8d40-4899-a7a0-4f0a52ddb7f7\") "
Jul 09 20:32:33.829841 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:33.829735    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-759wm\" (UniqueName: \"kubernetes.io/projected/79bc34f4-8d40-4899-a7a0-4f0a52ddb7f7-kube-api-access-759wm\") pod \"79bc34f4-8d40-4899-a7a0-4f0a52ddb7f7\" (UID: \"79bc34f4-8d40-4899-a7a0-4f0a52ddb7f7\") "
Jul 09 20:32:33.829841 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:33.829777    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79bc34f4-8d40-4899-a7a0-4f0a52ddb7f7-istio-data\") pod \"79bc34f4-8d40-4899-a7a0-4f0a52ddb7f7\" (UID: \"79bc34f4-8d40-4899-a7a0-4f0a52ddb7f7\") "
Jul 09 20:32:33.829841 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:33.829835    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79bc34f4-8d40-4899-a7a0-4f0a52ddb7f7-credential-socket\") pod \"79bc34f4-8d40-4899-a7a0-4f0a52ddb7f7\" (UID: \"79bc34f4-8d40-4899-a7a0-4f0a52ddb7f7\") "
Jul 09 20:32:33.830224 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:33.829898    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79bc34f4-8d40-4899-a7a0-4f0a52ddb7f7-istiod-ca-cert\") pod \"79bc34f4-8d40-4899-a7a0-4f0a52ddb7f7\" (UID: \"79bc34f4-8d40-4899-a7a0-4f0a52ddb7f7\") "
Jul 09 20:32:33.830224 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:33.829956    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79bc34f4-8d40-4899-a7a0-4f0a52ddb7f7-istio-envoy\") pod \"79bc34f4-8d40-4899-a7a0-4f0a52ddb7f7\" (UID: \"79bc34f4-8d40-4899-a7a0-4f0a52ddb7f7\") "
Jul 09 20:32:33.830224 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:33.829892    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79bc34f4-8d40-4899-a7a0-4f0a52ddb7f7-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "79bc34f4-8d40-4899-a7a0-4f0a52ddb7f7" (UID: "79bc34f4-8d40-4899-a7a0-4f0a52ddb7f7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 20:32:33.830224 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:33.830099    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79bc34f4-8d40-4899-a7a0-4f0a52ddb7f7-istio-data" (OuterVolumeSpecName: "istio-data") pod "79bc34f4-8d40-4899-a7a0-4f0a52ddb7f7" (UID: "79bc34f4-8d40-4899-a7a0-4f0a52ddb7f7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 20:32:33.830224 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:33.830043    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79bc34f4-8d40-4899-a7a0-4f0a52ddb7f7-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "79bc34f4-8d40-4899-a7a0-4f0a52ddb7f7" (UID: "79bc34f4-8d40-4899-a7a0-4f0a52ddb7f7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 20:32:33.830224 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:33.830188    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79bc34f4-8d40-4899-a7a0-4f0a52ddb7f7-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "79bc34f4-8d40-4899-a7a0-4f0a52ddb7f7" (UID: "79bc34f4-8d40-4899-a7a0-4f0a52ddb7f7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 20:32:33.830224 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:33.830209    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79bc34f4-8d40-4899-a7a0-4f0a52ddb7f7-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "79bc34f4-8d40-4899-a7a0-4f0a52ddb7f7" (UID: "79bc34f4-8d40-4899-a7a0-4f0a52ddb7f7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 20:32:33.830567 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:33.830336    2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79bc34f4-8d40-4899-a7a0-4f0a52ddb7f7-workload-socket\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:32:33.830567 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:33.830354    2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79bc34f4-8d40-4899-a7a0-4f0a52ddb7f7-workload-certs\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:32:33.830567 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:33.830363    2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79bc34f4-8d40-4899-a7a0-4f0a52ddb7f7-istio-data\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:32:33.830567 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:33.830371    2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79bc34f4-8d40-4899-a7a0-4f0a52ddb7f7-credential-socket\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:32:33.830567 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:33.830380    2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79bc34f4-8d40-4899-a7a0-4f0a52ddb7f7-istiod-ca-cert\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:32:33.832039 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:33.832014    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79bc34f4-8d40-4899-a7a0-4f0a52ddb7f7-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "79bc34f4-8d40-4899-a7a0-4f0a52ddb7f7" (UID: "79bc34f4-8d40-4899-a7a0-4f0a52ddb7f7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 20:32:33.832152 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:33.832094    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/79bc34f4-8d40-4899-a7a0-4f0a52ddb7f7-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "79bc34f4-8d40-4899-a7a0-4f0a52ddb7f7" (UID: "79bc34f4-8d40-4899-a7a0-4f0a52ddb7f7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 09 20:32:33.832342 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:33.832322    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79bc34f4-8d40-4899-a7a0-4f0a52ddb7f7-kube-api-access-759wm" (OuterVolumeSpecName: "kube-api-access-759wm") pod "79bc34f4-8d40-4899-a7a0-4f0a52ddb7f7" (UID: "79bc34f4-8d40-4899-a7a0-4f0a52ddb7f7"). InnerVolumeSpecName "kube-api-access-759wm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 20:32:33.832420 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:33.832370    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79bc34f4-8d40-4899-a7a0-4f0a52ddb7f7-istio-token" (OuterVolumeSpecName: "istio-token") pod "79bc34f4-8d40-4899-a7a0-4f0a52ddb7f7" (UID: "79bc34f4-8d40-4899-a7a0-4f0a52ddb7f7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 20:32:33.931212 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:33.931176    2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79bc34f4-8d40-4899-a7a0-4f0a52ddb7f7-istio-envoy\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:32:33.931212 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:33.931205    2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79bc34f4-8d40-4899-a7a0-4f0a52ddb7f7-istio-token\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:32:33.931212 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:33.931214    2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79bc34f4-8d40-4899-a7a0-4f0a52ddb7f7-istio-podinfo\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:32:33.931662 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:33.931224    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-759wm\" (UniqueName: \"kubernetes.io/projected/79bc34f4-8d40-4899-a7a0-4f0a52ddb7f7-kube-api-access-759wm\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:32:33.939177 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:33.939142    2562 generic.go:356] "Generic (PLEG): container finished" podID="79bc34f4-8d40-4899-a7a0-4f0a52ddb7f7" containerID="bb65e7d67fb7ee209c4b1f757210b51c54f384ecd5ed3278b847fc6fc00c6dbf" exitCode=0
Jul 09 20:32:33.939330 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:33.939193    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-66f6228f-openshift-default-645cfb4867-h49j2"
Jul 09 20:32:33.939330 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:33.939197    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-66f6228f-openshift-default-645cfb4867-h49j2" event={"ID":"79bc34f4-8d40-4899-a7a0-4f0a52ddb7f7","Type":"ContainerDied","Data":"bb65e7d67fb7ee209c4b1f757210b51c54f384ecd5ed3278b847fc6fc00c6dbf"}
Jul 09 20:32:33.939330 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:33.939251    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-66f6228f-openshift-default-645cfb4867-h49j2" event={"ID":"79bc34f4-8d40-4899-a7a0-4f0a52ddb7f7","Type":"ContainerDied","Data":"e1250a56f81998f10eccd325d04022470dce388f3d4b3bbf38d81cebac59179a"}
Jul 09 20:32:33.939330 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:33.939282    2562 scope.go:117] "RemoveContainer" containerID="bb65e7d67fb7ee209c4b1f757210b51c54f384ecd5ed3278b847fc6fc00c6dbf"
Jul 09 20:32:33.948889 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:33.948871    2562 scope.go:117] "RemoveContainer" containerID="bb65e7d67fb7ee209c4b1f757210b51c54f384ecd5ed3278b847fc6fc00c6dbf"
Jul 09 20:32:33.949129 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:32:33.949111    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb65e7d67fb7ee209c4b1f757210b51c54f384ecd5ed3278b847fc6fc00c6dbf\": container with ID starting with bb65e7d67fb7ee209c4b1f757210b51c54f384ecd5ed3278b847fc6fc00c6dbf not found: ID does not exist" containerID="bb65e7d67fb7ee209c4b1f757210b51c54f384ecd5ed3278b847fc6fc00c6dbf"
Jul 09 20:32:33.949169 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:33.949139    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb65e7d67fb7ee209c4b1f757210b51c54f384ecd5ed3278b847fc6fc00c6dbf"} err="failed to get container status \"bb65e7d67fb7ee209c4b1f757210b51c54f384ecd5ed3278b847fc6fc00c6dbf\": rpc error: code = NotFound desc = could not find container \"bb65e7d67fb7ee209c4b1f757210b51c54f384ecd5ed3278b847fc6fc00c6dbf\": container with ID starting with bb65e7d67fb7ee209c4b1f757210b51c54f384ecd5ed3278b847fc6fc00c6dbf not found: ID does not exist"
Jul 09 20:32:33.959842 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:33.959807    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-66f6228f-openshift-default-645cfb4867-h49j2"]
Jul 09 20:32:33.964420 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:33.964398    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-66f6228f-openshift-default-645cfb4867-h49j2"]
Jul 09 20:32:34.242322 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:34.242277    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79bc34f4-8d40-4899-a7a0-4f0a52ddb7f7" path="/var/lib/kubelet/pods/79bc34f4-8d40-4899-a7a0-4f0a52ddb7f7/volumes"
Jul 09 20:32:34.642110 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:34.642012    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-ba8485ab-openshift-default-8664fd9c67-l25fn"]
Jul 09 20:32:34.642599 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:34.642579    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dfeacb01-d716-4d46-8f49-4479313c170e" containerName="maas-api"
Jul 09 20:32:34.642599 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:34.642599    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="dfeacb01-d716-4d46-8f49-4479313c170e" containerName="maas-api"
Jul 09 20:32:34.642748 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:34.642615    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79bc34f4-8d40-4899-a7a0-4f0a52ddb7f7" containerName="istio-proxy"
Jul 09 20:32:34.642748 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:34.642621    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="79bc34f4-8d40-4899-a7a0-4f0a52ddb7f7" containerName="istio-proxy"
Jul 09 20:32:34.642748 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:34.642692    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="dfeacb01-d716-4d46-8f49-4479313c170e" containerName="maas-api"
Jul 09 20:32:34.642748 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:34.642701    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="79bc34f4-8d40-4899-a7a0-4f0a52ddb7f7" containerName="istio-proxy"
Jul 09 20:32:34.647541 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:34.647514    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-ba8485ab-openshift-default-8664fd9c67-l25fn"
Jul 09 20:32:34.650616 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:34.650592    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-ba8485ab-openshift-default-dockercfg-slsbf\""
Jul 09 20:32:34.655733 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:34.655432    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-ba8485ab-openshift-default-8664fd9c67-l25fn"]
Jul 09 20:32:34.738775 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:34.738743    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cc86fdb-15fd-401f-b3a8-0eb31f43da02-istio-podinfo\") pod \"e2e-derive-ba8485ab-openshift-default-8664fd9c67-l25fn\" (UID: \"1cc86fdb-15fd-401f-b3a8-0eb31f43da02\") " pod="openshift-ingress/e2e-derive-ba8485ab-openshift-default-8664fd9c67-l25fn"
Jul 09 20:32:34.738775 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:34.738778    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cc86fdb-15fd-401f-b3a8-0eb31f43da02-istiod-ca-cert\") pod \"e2e-derive-ba8485ab-openshift-default-8664fd9c67-l25fn\" (UID: \"1cc86fdb-15fd-401f-b3a8-0eb31f43da02\") " pod="openshift-ingress/e2e-derive-ba8485ab-openshift-default-8664fd9c67-l25fn"
Jul 09 20:32:34.738982 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:34.738805    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cc86fdb-15fd-401f-b3a8-0eb31f43da02-workload-certs\") pod \"e2e-derive-ba8485ab-openshift-default-8664fd9c67-l25fn\" (UID: \"1cc86fdb-15fd-401f-b3a8-0eb31f43da02\") " pod="openshift-ingress/e2e-derive-ba8485ab-openshift-default-8664fd9c67-l25fn"
Jul 09 20:32:34.738982 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:34.738916    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cc86fdb-15fd-401f-b3a8-0eb31f43da02-istio-envoy\") pod \"e2e-derive-ba8485ab-openshift-default-8664fd9c67-l25fn\" (UID: \"1cc86fdb-15fd-401f-b3a8-0eb31f43da02\") " pod="openshift-ingress/e2e-derive-ba8485ab-openshift-default-8664fd9c67-l25fn"
Jul 09 20:32:34.738982 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:34.738945    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cc86fdb-15fd-401f-b3a8-0eb31f43da02-istio-data\") pod \"e2e-derive-ba8485ab-openshift-default-8664fd9c67-l25fn\" (UID: \"1cc86fdb-15fd-401f-b3a8-0eb31f43da02\") " pod="openshift-ingress/e2e-derive-ba8485ab-openshift-default-8664fd9c67-l25fn"
Jul 09 20:32:34.738982 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:34.738967    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cc86fdb-15fd-401f-b3a8-0eb31f43da02-istio-token\") pod \"e2e-derive-ba8485ab-openshift-default-8664fd9c67-l25fn\" (UID: \"1cc86fdb-15fd-401f-b3a8-0eb31f43da02\") " pod="openshift-ingress/e2e-derive-ba8485ab-openshift-default-8664fd9c67-l25fn"
Jul 09 20:32:34.739154 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:34.739003    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cc86fdb-15fd-401f-b3a8-0eb31f43da02-credential-socket\") pod \"e2e-derive-ba8485ab-openshift-default-8664fd9c67-l25fn\" (UID: \"1cc86fdb-15fd-401f-b3a8-0eb31f43da02\") " pod="openshift-ingress/e2e-derive-ba8485ab-openshift-default-8664fd9c67-l25fn"
Jul 09 20:32:34.739154 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:34.739028    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4qbj\" (UniqueName: \"kubernetes.io/projected/1cc86fdb-15fd-401f-b3a8-0eb31f43da02-kube-api-access-k4qbj\") pod \"e2e-derive-ba8485ab-openshift-default-8664fd9c67-l25fn\" (UID: \"1cc86fdb-15fd-401f-b3a8-0eb31f43da02\") " pod="openshift-ingress/e2e-derive-ba8485ab-openshift-default-8664fd9c67-l25fn"
Jul 09 20:32:34.739154 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:34.739053    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cc86fdb-15fd-401f-b3a8-0eb31f43da02-workload-socket\") pod \"e2e-derive-ba8485ab-openshift-default-8664fd9c67-l25fn\" (UID: \"1cc86fdb-15fd-401f-b3a8-0eb31f43da02\") " pod="openshift-ingress/e2e-derive-ba8485ab-openshift-default-8664fd9c67-l25fn"
Jul 09 20:32:34.840070 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:34.840011    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cc86fdb-15fd-401f-b3a8-0eb31f43da02-istio-token\") pod \"e2e-derive-ba8485ab-openshift-default-8664fd9c67-l25fn\" (UID: \"1cc86fdb-15fd-401f-b3a8-0eb31f43da02\") " pod="openshift-ingress/e2e-derive-ba8485ab-openshift-default-8664fd9c67-l25fn"
Jul 09 20:32:34.840070 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:34.840076    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cc86fdb-15fd-401f-b3a8-0eb31f43da02-credential-socket\") pod \"e2e-derive-ba8485ab-openshift-default-8664fd9c67-l25fn\" (UID: \"1cc86fdb-15fd-401f-b3a8-0eb31f43da02\") " pod="openshift-ingress/e2e-derive-ba8485ab-openshift-default-8664fd9c67-l25fn"
Jul 09 20:32:34.840374 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:34.840095    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4qbj\" (UniqueName: \"kubernetes.io/projected/1cc86fdb-15fd-401f-b3a8-0eb31f43da02-kube-api-access-k4qbj\") pod \"e2e-derive-ba8485ab-openshift-default-8664fd9c67-l25fn\" (UID: \"1cc86fdb-15fd-401f-b3a8-0eb31f43da02\") " pod="openshift-ingress/e2e-derive-ba8485ab-openshift-default-8664fd9c67-l25fn"
Jul 09 20:32:34.840374 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:34.840121    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cc86fdb-15fd-401f-b3a8-0eb31f43da02-workload-socket\") pod \"e2e-derive-ba8485ab-openshift-default-8664fd9c67-l25fn\" (UID: \"1cc86fdb-15fd-401f-b3a8-0eb31f43da02\") " pod="openshift-ingress/e2e-derive-ba8485ab-openshift-default-8664fd9c67-l25fn"
Jul 09 20:32:34.840374 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:34.840157    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cc86fdb-15fd-401f-b3a8-0eb31f43da02-istio-podinfo\") pod \"e2e-derive-ba8485ab-openshift-default-8664fd9c67-l25fn\" (UID: \"1cc86fdb-15fd-401f-b3a8-0eb31f43da02\") " pod="openshift-ingress/e2e-derive-ba8485ab-openshift-default-8664fd9c67-l25fn"
Jul 09 20:32:34.840374 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:34.840173    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cc86fdb-15fd-401f-b3a8-0eb31f43da02-istiod-ca-cert\") pod \"e2e-derive-ba8485ab-openshift-default-8664fd9c67-l25fn\" (UID: \"1cc86fdb-15fd-401f-b3a8-0eb31f43da02\") " pod="openshift-ingress/e2e-derive-ba8485ab-openshift-default-8664fd9c67-l25fn"
Jul 09 20:32:34.840374 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:34.840201    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cc86fdb-15fd-401f-b3a8-0eb31f43da02-workload-certs\") pod \"e2e-derive-ba8485ab-openshift-default-8664fd9c67-l25fn\" (UID: \"1cc86fdb-15fd-401f-b3a8-0eb31f43da02\") " pod="openshift-ingress/e2e-derive-ba8485ab-openshift-default-8664fd9c67-l25fn"
Jul 09 20:32:34.840374 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:34.840282    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cc86fdb-15fd-401f-b3a8-0eb31f43da02-istio-envoy\") pod \"e2e-derive-ba8485ab-openshift-default-8664fd9c67-l25fn\" (UID: \"1cc86fdb-15fd-401f-b3a8-0eb31f43da02\") " pod="openshift-ingress/e2e-derive-ba8485ab-openshift-default-8664fd9c67-l25fn"
Jul 09 20:32:34.840374 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:34.840342    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cc86fdb-15fd-401f-b3a8-0eb31f43da02-istio-data\") pod \"e2e-derive-ba8485ab-openshift-default-8664fd9c67-l25fn\" (UID: \"1cc86fdb-15fd-401f-b3a8-0eb31f43da02\") " pod="openshift-ingress/e2e-derive-ba8485ab-openshift-default-8664fd9c67-l25fn"
Jul 09 20:32:34.840724 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:34.840588    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cc86fdb-15fd-401f-b3a8-0eb31f43da02-credential-socket\") pod \"e2e-derive-ba8485ab-openshift-default-8664fd9c67-l25fn\" (UID: \"1cc86fdb-15fd-401f-b3a8-0eb31f43da02\") " pod="openshift-ingress/e2e-derive-ba8485ab-openshift-default-8664fd9c67-l25fn"
Jul 09 20:32:34.840724 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:34.840630    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cc86fdb-15fd-401f-b3a8-0eb31f43da02-workload-certs\") pod \"e2e-derive-ba8485ab-openshift-default-8664fd9c67-l25fn\" (UID: \"1cc86fdb-15fd-401f-b3a8-0eb31f43da02\") " pod="openshift-ingress/e2e-derive-ba8485ab-openshift-default-8664fd9c67-l25fn"
Jul 09 20:32:34.840724 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:34.840710    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cc86fdb-15fd-401f-b3a8-0eb31f43da02-istio-data\") pod \"e2e-derive-ba8485ab-openshift-default-8664fd9c67-l25fn\" (UID: \"1cc86fdb-15fd-401f-b3a8-0eb31f43da02\") " pod="openshift-ingress/e2e-derive-ba8485ab-openshift-default-8664fd9c67-l25fn"
Jul 09 20:32:34.840946 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:34.840925    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cc86fdb-15fd-401f-b3a8-0eb31f43da02-istiod-ca-cert\") pod \"e2e-derive-ba8485ab-openshift-default-8664fd9c67-l25fn\" (UID: \"1cc86fdb-15fd-401f-b3a8-0eb31f43da02\") " pod="openshift-ingress/e2e-derive-ba8485ab-openshift-default-8664fd9c67-l25fn"
Jul 09 20:32:34.841004 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:34.840939    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cc86fdb-15fd-401f-b3a8-0eb31f43da02-workload-socket\") pod \"e2e-derive-ba8485ab-openshift-default-8664fd9c67-l25fn\" (UID: \"1cc86fdb-15fd-401f-b3a8-0eb31f43da02\") " pod="openshift-ingress/e2e-derive-ba8485ab-openshift-default-8664fd9c67-l25fn"
Jul 09 20:32:34.842656 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:34.842629    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cc86fdb-15fd-401f-b3a8-0eb31f43da02-istio-envoy\") pod \"e2e-derive-ba8485ab-openshift-default-8664fd9c67-l25fn\" (UID: \"1cc86fdb-15fd-401f-b3a8-0eb31f43da02\") " pod="openshift-ingress/e2e-derive-ba8485ab-openshift-default-8664fd9c67-l25fn"
Jul 09 20:32:34.842888 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:34.842865    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cc86fdb-15fd-401f-b3a8-0eb31f43da02-istio-podinfo\") pod \"e2e-derive-ba8485ab-openshift-default-8664fd9c67-l25fn\" (UID: \"1cc86fdb-15fd-401f-b3a8-0eb31f43da02\") " pod="openshift-ingress/e2e-derive-ba8485ab-openshift-default-8664fd9c67-l25fn"
Jul 09 20:32:34.848912 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:34.848881    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cc86fdb-15fd-401f-b3a8-0eb31f43da02-istio-token\") pod \"e2e-derive-ba8485ab-openshift-default-8664fd9c67-l25fn\" (UID: \"1cc86fdb-15fd-401f-b3a8-0eb31f43da02\") " pod="openshift-ingress/e2e-derive-ba8485ab-openshift-default-8664fd9c67-l25fn"
Jul 09 20:32:34.849039 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:34.848974    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4qbj\" (UniqueName: \"kubernetes.io/projected/1cc86fdb-15fd-401f-b3a8-0eb31f43da02-kube-api-access-k4qbj\") pod \"e2e-derive-ba8485ab-openshift-default-8664fd9c67-l25fn\" (UID: \"1cc86fdb-15fd-401f-b3a8-0eb31f43da02\") " pod="openshift-ingress/e2e-derive-ba8485ab-openshift-default-8664fd9c67-l25fn"
Jul 09 20:32:34.959465 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:34.959439    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-ba8485ab-openshift-default-8664fd9c67-l25fn"
Jul 09 20:32:35.097800 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:35.097772    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-ba8485ab-openshift-default-8664fd9c67-l25fn"]
Jul 09 20:32:35.099897 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:32:35.099862    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cc86fdb_15fd_401f_b3a8_0eb31f43da02.slice/crio-9cc7cdb1110965357430ff94a01928d6dc3e97618de3fe081c26b423dc558095 WatchSource:0}: Error finding container 9cc7cdb1110965357430ff94a01928d6dc3e97618de3fe081c26b423dc558095: Status 404 returned error can't find the container with id 9cc7cdb1110965357430ff94a01928d6dc3e97618de3fe081c26b423dc558095
Jul 09 20:32:35.102142 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:35.102109    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 09 20:32:35.102241 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:35.102168    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 09 20:32:35.102241 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:35.102199    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 09 20:32:35.857019 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:35.856981    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-ba8485ab-865b8d4cdc-sznsn"]
Jul 09 20:32:35.860675 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:35.860654    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ba8485ab-865b8d4cdc-sznsn"
Jul 09 20:32:35.863867 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:35.863841    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-ba8485ab-serving-cert\""
Jul 09 20:32:35.869866 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:35.869843    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-ba8485ab-865b8d4cdc-sznsn"]
Jul 09 20:32:35.949801 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:35.949761    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be4ea940-308a-4e20-9ba0-7f9842dced2d-openshift-service-ca\") pod \"maas-api-e2e-derive-ba8485ab-865b8d4cdc-sznsn\" (UID: \"be4ea940-308a-4e20-9ba0-7f9842dced2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ba8485ab-865b8d4cdc-sznsn"
Jul 09 20:32:35.949988 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:35.949838    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rn6zm\" (UniqueName: \"kubernetes.io/projected/be4ea940-308a-4e20-9ba0-7f9842dced2d-kube-api-access-rn6zm\") pod \"maas-api-e2e-derive-ba8485ab-865b8d4cdc-sznsn\" (UID: \"be4ea940-308a-4e20-9ba0-7f9842dced2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ba8485ab-865b8d4cdc-sznsn"
Jul 09 20:32:35.949988 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:35.949925    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be4ea940-308a-4e20-9ba0-7f9842dced2d-maas-api-tls\") pod \"maas-api-e2e-derive-ba8485ab-865b8d4cdc-sznsn\" (UID: \"be4ea940-308a-4e20-9ba0-7f9842dced2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ba8485ab-865b8d4cdc-sznsn"
Jul 09 20:32:35.954016 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:35.953983    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-ba8485ab-openshift-default-8664fd9c67-l25fn" event={"ID":"1cc86fdb-15fd-401f-b3a8-0eb31f43da02","Type":"ContainerStarted","Data":"6c51fe956270bf94f7a0621cda4af312450cd3f4f28024087bb89a3e66bbba83"}
Jul 09 20:32:35.954196 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:35.954044    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-ba8485ab-openshift-default-8664fd9c67-l25fn" event={"ID":"1cc86fdb-15fd-401f-b3a8-0eb31f43da02","Type":"ContainerStarted","Data":"9cc7cdb1110965357430ff94a01928d6dc3e97618de3fe081c26b423dc558095"}
Jul 09 20:32:35.959534 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:35.959504    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-ba8485ab-openshift-default-8664fd9c67-l25fn"
Jul 09 20:32:35.984334 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:35.984268    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-ba8485ab-openshift-default-8664fd9c67-l25fn" podStartSLOduration=1.9842530329999999 podStartE2EDuration="1.984253033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 20:32:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 20:32:35.98149906 +0000 UTC m=+1746.399168869" watchObservedRunningTime="2026-07-09 20:32:35.984253033 +0000 UTC m=+1746.401922843"
Jul 09 20:32:36.050740 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:36.050711    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be4ea940-308a-4e20-9ba0-7f9842dced2d-openshift-service-ca\") pod \"maas-api-e2e-derive-ba8485ab-865b8d4cdc-sznsn\" (UID: \"be4ea940-308a-4e20-9ba0-7f9842dced2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ba8485ab-865b8d4cdc-sznsn"
Jul 09 20:32:36.050909 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:36.050752    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rn6zm\" (UniqueName: \"kubernetes.io/projected/be4ea940-308a-4e20-9ba0-7f9842dced2d-kube-api-access-rn6zm\") pod \"maas-api-e2e-derive-ba8485ab-865b8d4cdc-sznsn\" (UID: \"be4ea940-308a-4e20-9ba0-7f9842dced2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ba8485ab-865b8d4cdc-sznsn"
Jul 09 20:32:36.050909 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:36.050832    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be4ea940-308a-4e20-9ba0-7f9842dced2d-maas-api-tls\") pod \"maas-api-e2e-derive-ba8485ab-865b8d4cdc-sznsn\" (UID: \"be4ea940-308a-4e20-9ba0-7f9842dced2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ba8485ab-865b8d4cdc-sznsn"
Jul 09 20:32:36.051458 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:36.051435    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be4ea940-308a-4e20-9ba0-7f9842dced2d-openshift-service-ca\") pod \"maas-api-e2e-derive-ba8485ab-865b8d4cdc-sznsn\" (UID: \"be4ea940-308a-4e20-9ba0-7f9842dced2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ba8485ab-865b8d4cdc-sznsn"
Jul 09 20:32:36.053312 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:36.053273    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be4ea940-308a-4e20-9ba0-7f9842dced2d-maas-api-tls\") pod \"maas-api-e2e-derive-ba8485ab-865b8d4cdc-sznsn\" (UID: \"be4ea940-308a-4e20-9ba0-7f9842dced2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ba8485ab-865b8d4cdc-sznsn"
Jul 09 20:32:36.059120 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:36.059090    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rn6zm\" (UniqueName: \"kubernetes.io/projected/be4ea940-308a-4e20-9ba0-7f9842dced2d-kube-api-access-rn6zm\") pod \"maas-api-e2e-derive-ba8485ab-865b8d4cdc-sznsn\" (UID: \"be4ea940-308a-4e20-9ba0-7f9842dced2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ba8485ab-865b8d4cdc-sznsn"
Jul 09 20:32:36.174275 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:36.174175    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ba8485ab-865b8d4cdc-sznsn"
Jul 09 20:32:36.309562 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:36.309532    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-ba8485ab-865b8d4cdc-sznsn"]
Jul 09 20:32:36.311338 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:32:36.311292    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe4ea940_308a_4e20_9ba0_7f9842dced2d.slice/crio-7c6daef23b25069c319e02dbdf09752192098a8582f96352d2b180d7453eb822 WatchSource:0}: Error finding container 7c6daef23b25069c319e02dbdf09752192098a8582f96352d2b180d7453eb822: Status 404 returned error can't find the container with id 7c6daef23b25069c319e02dbdf09752192098a8582f96352d2b180d7453eb822
Jul 09 20:32:36.959915 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:36.959875    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ba8485ab-865b8d4cdc-sznsn" event={"ID":"be4ea940-308a-4e20-9ba0-7f9842dced2d","Type":"ContainerStarted","Data":"9a1bdc3e9493cc26690122e4ec02f757516a0e59be9cfcff9e63d49d63d1f0a0"}
Jul 09 20:32:36.960324 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:36.959937    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ba8485ab-865b8d4cdc-sznsn" event={"ID":"be4ea940-308a-4e20-9ba0-7f9842dced2d","Type":"ContainerStarted","Data":"7c6daef23b25069c319e02dbdf09752192098a8582f96352d2b180d7453eb822"}
Jul 09 20:32:36.964480 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:36.964454    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-ba8485ab-openshift-default-8664fd9c67-l25fn"
Jul 09 20:32:36.978714 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:36.978666    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ba8485ab-865b8d4cdc-sznsn" podStartSLOduration=1.511366496 podStartE2EDuration="1.978650238s" totalImagesPullingTime="467.283742ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 20:32:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 20:32:36.976806131 +0000 UTC m=+1747.394475941" watchObservedRunningTime="2026-07-09 20:32:36.978650238 +0000 UTC m=+1747.396320046"
Jul 09 20:32:37.964849 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:37.964810    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ba8485ab-865b8d4cdc-sznsn"
Jul 09 20:32:37.964849 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:37.964859    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-ba8485ab-openshift-default-8664fd9c67-l25fn"
Jul 09 20:32:37.965861 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:37.965841    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-ba8485ab-openshift-default-8664fd9c67-l25fn"
Jul 09 20:32:40.575876 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:40.575846    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-ba8485ab-openshift-default-8664fd9c67-l25fn"]
Jul 09 20:32:40.821452 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:40.821404    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-ba8485ab-865b8d4cdc-sznsn"]
Jul 09 20:32:40.821968 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:40.821905    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ba8485ab-865b8d4cdc-sznsn" podUID="be4ea940-308a-4e20-9ba0-7f9842dced2d" containerName="maas-api" containerID="cri-o://9a1bdc3e9493cc26690122e4ec02f757516a0e59be9cfcff9e63d49d63d1f0a0" gracePeriod=30
Jul 09 20:32:40.981867 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:40.981836    2562 generic.go:356] "Generic (PLEG): container finished" podID="be4ea940-308a-4e20-9ba0-7f9842dced2d" containerID="9a1bdc3e9493cc26690122e4ec02f757516a0e59be9cfcff9e63d49d63d1f0a0" exitCode=0
Jul 09 20:32:40.982034 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:40.981909    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ba8485ab-865b8d4cdc-sznsn" event={"ID":"be4ea940-308a-4e20-9ba0-7f9842dced2d","Type":"ContainerDied","Data":"9a1bdc3e9493cc26690122e4ec02f757516a0e59be9cfcff9e63d49d63d1f0a0"}
Jul 09 20:32:40.982100 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:40.982049    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-ba8485ab-openshift-default-8664fd9c67-l25fn" podUID="1cc86fdb-15fd-401f-b3a8-0eb31f43da02" containerName="istio-proxy" containerID="cri-o://6c51fe956270bf94f7a0621cda4af312450cd3f4f28024087bb89a3e66bbba83" gracePeriod=30
Jul 09 20:32:41.140867 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:41.140842    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ba8485ab-865b8d4cdc-sznsn"
Jul 09 20:32:41.301619 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:41.301583    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be4ea940-308a-4e20-9ba0-7f9842dced2d-openshift-service-ca\") pod \"be4ea940-308a-4e20-9ba0-7f9842dced2d\" (UID: \"be4ea940-308a-4e20-9ba0-7f9842dced2d\") "
Jul 09 20:32:41.301794 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:41.301642    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rn6zm\" (UniqueName: \"kubernetes.io/projected/be4ea940-308a-4e20-9ba0-7f9842dced2d-kube-api-access-rn6zm\") pod \"be4ea940-308a-4e20-9ba0-7f9842dced2d\" (UID: \"be4ea940-308a-4e20-9ba0-7f9842dced2d\") "
Jul 09 20:32:41.301794 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:41.301753    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be4ea940-308a-4e20-9ba0-7f9842dced2d-maas-api-tls\") pod \"be4ea940-308a-4e20-9ba0-7f9842dced2d\" (UID: \"be4ea940-308a-4e20-9ba0-7f9842dced2d\") "
Jul 09 20:32:41.301985 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:41.301960    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be4ea940-308a-4e20-9ba0-7f9842dced2d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "be4ea940-308a-4e20-9ba0-7f9842dced2d" (UID: "be4ea940-308a-4e20-9ba0-7f9842dced2d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 20:32:41.302084 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:41.302068    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be4ea940-308a-4e20-9ba0-7f9842dced2d-openshift-service-ca\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:32:41.303787 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:41.303763    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be4ea940-308a-4e20-9ba0-7f9842dced2d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "be4ea940-308a-4e20-9ba0-7f9842dced2d" (UID: "be4ea940-308a-4e20-9ba0-7f9842dced2d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 20:32:41.303907 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:41.303833    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be4ea940-308a-4e20-9ba0-7f9842dced2d-kube-api-access-rn6zm" (OuterVolumeSpecName: "kube-api-access-rn6zm") pod "be4ea940-308a-4e20-9ba0-7f9842dced2d" (UID: "be4ea940-308a-4e20-9ba0-7f9842dced2d"). InnerVolumeSpecName "kube-api-access-rn6zm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 20:32:41.403298 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:41.403206    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rn6zm\" (UniqueName: \"kubernetes.io/projected/be4ea940-308a-4e20-9ba0-7f9842dced2d-kube-api-access-rn6zm\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:32:41.403298 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:41.403236    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be4ea940-308a-4e20-9ba0-7f9842dced2d-maas-api-tls\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:32:41.988272 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:41.988243    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ba8485ab-865b8d4cdc-sznsn"
Jul 09 20:32:41.988715 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:41.988274    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ba8485ab-865b8d4cdc-sznsn" event={"ID":"be4ea940-308a-4e20-9ba0-7f9842dced2d","Type":"ContainerDied","Data":"7c6daef23b25069c319e02dbdf09752192098a8582f96352d2b180d7453eb822"}
Jul 09 20:32:41.988715 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:41.988357    2562 scope.go:117] "RemoveContainer" containerID="9a1bdc3e9493cc26690122e4ec02f757516a0e59be9cfcff9e63d49d63d1f0a0"
Jul 09 20:32:42.010624 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:42.010598    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-ba8485ab-865b8d4cdc-sznsn"]
Jul 09 20:32:42.015613 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:42.015589    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-ba8485ab-865b8d4cdc-sznsn"]
Jul 09 20:32:42.244390 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:42.244284    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be4ea940-308a-4e20-9ba0-7f9842dced2d" path="/var/lib/kubelet/pods/be4ea940-308a-4e20-9ba0-7f9842dced2d/volumes"
Jul 09 20:32:46.955780 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:46.955751    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-ba8485ab-openshift-default-8664fd9c67-l25fn"
Jul 09 20:32:47.015889 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:47.015801    2562 generic.go:356] "Generic (PLEG): container finished" podID="1cc86fdb-15fd-401f-b3a8-0eb31f43da02" containerID="6c51fe956270bf94f7a0621cda4af312450cd3f4f28024087bb89a3e66bbba83" exitCode=0
Jul 09 20:32:47.015889 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:47.015840    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-ba8485ab-openshift-default-8664fd9c67-l25fn" event={"ID":"1cc86fdb-15fd-401f-b3a8-0eb31f43da02","Type":"ContainerDied","Data":"6c51fe956270bf94f7a0621cda4af312450cd3f4f28024087bb89a3e66bbba83"}
Jul 09 20:32:47.015889 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:47.015850    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-ba8485ab-openshift-default-8664fd9c67-l25fn"
Jul 09 20:32:47.015889 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:47.015882    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-ba8485ab-openshift-default-8664fd9c67-l25fn" event={"ID":"1cc86fdb-15fd-401f-b3a8-0eb31f43da02","Type":"ContainerDied","Data":"9cc7cdb1110965357430ff94a01928d6dc3e97618de3fe081c26b423dc558095"}
Jul 09 20:32:47.016165 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:47.015904    2562 scope.go:117] "RemoveContainer" containerID="6c51fe956270bf94f7a0621cda4af312450cd3f4f28024087bb89a3e66bbba83"
Jul 09 20:32:47.025022 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:47.025006    2562 scope.go:117] "RemoveContainer" containerID="6c51fe956270bf94f7a0621cda4af312450cd3f4f28024087bb89a3e66bbba83"
Jul 09 20:32:47.025326 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:32:47.025285    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c51fe956270bf94f7a0621cda4af312450cd3f4f28024087bb89a3e66bbba83\": container with ID starting with 6c51fe956270bf94f7a0621cda4af312450cd3f4f28024087bb89a3e66bbba83 not found: ID does not exist" containerID="6c51fe956270bf94f7a0621cda4af312450cd3f4f28024087bb89a3e66bbba83"
Jul 09 20:32:47.025392 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:47.025336    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c51fe956270bf94f7a0621cda4af312450cd3f4f28024087bb89a3e66bbba83"} err="failed to get container status \"6c51fe956270bf94f7a0621cda4af312450cd3f4f28024087bb89a3e66bbba83\": rpc error: code = NotFound desc = could not find container \"6c51fe956270bf94f7a0621cda4af312450cd3f4f28024087bb89a3e66bbba83\": container with ID starting with 6c51fe956270bf94f7a0621cda4af312450cd3f4f28024087bb89a3e66bbba83 not found: ID does not exist"
Jul 09 20:32:47.059329 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:47.059273    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k4qbj\" (UniqueName: \"kubernetes.io/projected/1cc86fdb-15fd-401f-b3a8-0eb31f43da02-kube-api-access-k4qbj\") pod \"1cc86fdb-15fd-401f-b3a8-0eb31f43da02\" (UID: \"1cc86fdb-15fd-401f-b3a8-0eb31f43da02\") "
Jul 09 20:32:47.059496 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:47.059346    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cc86fdb-15fd-401f-b3a8-0eb31f43da02-workload-certs\") pod \"1cc86fdb-15fd-401f-b3a8-0eb31f43da02\" (UID: \"1cc86fdb-15fd-401f-b3a8-0eb31f43da02\") "
Jul 09 20:32:47.059496 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:47.059366    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cc86fdb-15fd-401f-b3a8-0eb31f43da02-istio-envoy\") pod \"1cc86fdb-15fd-401f-b3a8-0eb31f43da02\" (UID: \"1cc86fdb-15fd-401f-b3a8-0eb31f43da02\") "
Jul 09 20:32:47.059496 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:47.059414    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cc86fdb-15fd-401f-b3a8-0eb31f43da02-istio-podinfo\") pod \"1cc86fdb-15fd-401f-b3a8-0eb31f43da02\" (UID: \"1cc86fdb-15fd-401f-b3a8-0eb31f43da02\") "
Jul 09 20:32:47.059496 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:47.059445    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cc86fdb-15fd-401f-b3a8-0eb31f43da02-istio-data\") pod \"1cc86fdb-15fd-401f-b3a8-0eb31f43da02\" (UID: \"1cc86fdb-15fd-401f-b3a8-0eb31f43da02\") "
Jul 09 20:32:47.059711 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:47.059509    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cc86fdb-15fd-401f-b3a8-0eb31f43da02-istio-token\") pod \"1cc86fdb-15fd-401f-b3a8-0eb31f43da02\" (UID: \"1cc86fdb-15fd-401f-b3a8-0eb31f43da02\") "
Jul 09 20:32:47.059711 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:47.059555    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cc86fdb-15fd-401f-b3a8-0eb31f43da02-workload-socket\") pod \"1cc86fdb-15fd-401f-b3a8-0eb31f43da02\" (UID: \"1cc86fdb-15fd-401f-b3a8-0eb31f43da02\") "
Jul 09 20:32:47.059711 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:47.059588    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cc86fdb-15fd-401f-b3a8-0eb31f43da02-credential-socket\") pod \"1cc86fdb-15fd-401f-b3a8-0eb31f43da02\" (UID: \"1cc86fdb-15fd-401f-b3a8-0eb31f43da02\") "
Jul 09 20:32:47.059711 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:47.059627    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cc86fdb-15fd-401f-b3a8-0eb31f43da02-istiod-ca-cert\") pod \"1cc86fdb-15fd-401f-b3a8-0eb31f43da02\" (UID: \"1cc86fdb-15fd-401f-b3a8-0eb31f43da02\") "
Jul 09 20:32:47.059711 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:47.059687    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cc86fdb-15fd-401f-b3a8-0eb31f43da02-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1cc86fdb-15fd-401f-b3a8-0eb31f43da02" (UID: "1cc86fdb-15fd-401f-b3a8-0eb31f43da02"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 20:32:47.060223 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:47.059949    2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cc86fdb-15fd-401f-b3a8-0eb31f43da02-workload-certs\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:32:47.060223 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:47.059971    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cc86fdb-15fd-401f-b3a8-0eb31f43da02-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1cc86fdb-15fd-401f-b3a8-0eb31f43da02" (UID: "1cc86fdb-15fd-401f-b3a8-0eb31f43da02"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 20:32:47.060223 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:47.060128    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cc86fdb-15fd-401f-b3a8-0eb31f43da02-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1cc86fdb-15fd-401f-b3a8-0eb31f43da02" (UID: "1cc86fdb-15fd-401f-b3a8-0eb31f43da02"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 20:32:47.060223 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:47.060172    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cc86fdb-15fd-401f-b3a8-0eb31f43da02-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1cc86fdb-15fd-401f-b3a8-0eb31f43da02" (UID: "1cc86fdb-15fd-401f-b3a8-0eb31f43da02"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 20:32:47.060527 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:47.060401    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cc86fdb-15fd-401f-b3a8-0eb31f43da02-istio-data" (OuterVolumeSpecName: "istio-data") pod "1cc86fdb-15fd-401f-b3a8-0eb31f43da02" (UID: "1cc86fdb-15fd-401f-b3a8-0eb31f43da02"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 20:32:47.061700 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:47.061669    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cc86fdb-15fd-401f-b3a8-0eb31f43da02-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1cc86fdb-15fd-401f-b3a8-0eb31f43da02" (UID: "1cc86fdb-15fd-401f-b3a8-0eb31f43da02"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 20:32:47.061700 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:47.061669    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cc86fdb-15fd-401f-b3a8-0eb31f43da02-kube-api-access-k4qbj" (OuterVolumeSpecName: "kube-api-access-k4qbj") pod "1cc86fdb-15fd-401f-b3a8-0eb31f43da02" (UID: "1cc86fdb-15fd-401f-b3a8-0eb31f43da02"). InnerVolumeSpecName "kube-api-access-k4qbj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 20:32:47.061979 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:47.061957    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cc86fdb-15fd-401f-b3a8-0eb31f43da02-istio-token" (OuterVolumeSpecName: "istio-token") pod "1cc86fdb-15fd-401f-b3a8-0eb31f43da02" (UID: "1cc86fdb-15fd-401f-b3a8-0eb31f43da02"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 20:32:47.062042 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:47.062018    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1cc86fdb-15fd-401f-b3a8-0eb31f43da02-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1cc86fdb-15fd-401f-b3a8-0eb31f43da02" (UID: "1cc86fdb-15fd-401f-b3a8-0eb31f43da02"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 09 20:32:47.161163 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:47.161123    2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cc86fdb-15fd-401f-b3a8-0eb31f43da02-istiod-ca-cert\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:32:47.161163 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:47.161154    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4qbj\" (UniqueName: \"kubernetes.io/projected/1cc86fdb-15fd-401f-b3a8-0eb31f43da02-kube-api-access-k4qbj\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:32:47.161163 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:47.161167    2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cc86fdb-15fd-401f-b3a8-0eb31f43da02-istio-envoy\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:32:47.161444 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:47.161175    2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cc86fdb-15fd-401f-b3a8-0eb31f43da02-istio-podinfo\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:32:47.161444 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:47.161183    2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cc86fdb-15fd-401f-b3a8-0eb31f43da02-istio-data\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:32:47.161444 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:47.161190    2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cc86fdb-15fd-401f-b3a8-0eb31f43da02-istio-token\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:32:47.161444 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:47.161198    2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cc86fdb-15fd-401f-b3a8-0eb31f43da02-workload-socket\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:32:47.161444 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:47.161206    2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cc86fdb-15fd-401f-b3a8-0eb31f43da02-credential-socket\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:32:47.339490 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:47.339459    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-ba8485ab-openshift-default-8664fd9c67-l25fn"]
Jul 09 20:32:47.343612 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:47.343588    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-ba8485ab-openshift-default-8664fd9c67-l25fn"]
Jul 09 20:32:48.242677 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:32:48.242642    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1cc86fdb-15fd-401f-b3a8-0eb31f43da02" path="/var/lib/kubelet/pods/1cc86fdb-15fd-401f-b3a8-0eb31f43da02/volumes"
Jul 09 20:33:30.179281 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:33:30.179246    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-cqs6d_578878d0-e6b3-4725-a698-796cfc169fa6/console-operator/1.log"
Jul 09 20:33:30.184489 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:33:30.184461    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-cqs6d_578878d0-e6b3-4725-a698-796cfc169fa6/console-operator/1.log"
Jul 09 20:33:30.185425 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:33:30.185406    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hds4g_4b4468fd-1708-487b-8d69-bb7d817e25de/ovn-acl-logging/0.log"
Jul 09 20:33:30.190076 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:33:30.190057    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hds4g_4b4468fd-1708-487b-8d69-bb7d817e25de/ovn-acl-logging/0.log"
Jul 09 20:34:22.006853 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:22.006818    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-f1ef3751-openshift-default-ff56b44bc-524qh"]
Jul 09 20:34:22.007255 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:22.007189    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be4ea940-308a-4e20-9ba0-7f9842dced2d" containerName="maas-api"
Jul 09 20:34:22.007255 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:22.007200    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="be4ea940-308a-4e20-9ba0-7f9842dced2d" containerName="maas-api"
Jul 09 20:34:22.007255 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:22.007225    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1cc86fdb-15fd-401f-b3a8-0eb31f43da02" containerName="istio-proxy"
Jul 09 20:34:22.007255 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:22.007230    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="1cc86fdb-15fd-401f-b3a8-0eb31f43da02" containerName="istio-proxy"
Jul 09 20:34:22.007429 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:22.007285    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="1cc86fdb-15fd-401f-b3a8-0eb31f43da02" containerName="istio-proxy"
Jul 09 20:34:22.007429 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:22.007294    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="be4ea940-308a-4e20-9ba0-7f9842dced2d" containerName="maas-api"
Jul 09 20:34:22.009310 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:22.009279    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f1ef3751-openshift-default-ff56b44bc-524qh"
Jul 09 20:34:22.012684 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:22.012666    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-f1ef3751-openshift-default-dockercfg-76xdz\""
Jul 09 20:34:22.033286 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:22.033256    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-f1ef3751-openshift-default-ff56b44bc-524qh"]
Jul 09 20:34:22.070026 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:22.070001    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6083094a-8f65-4b46-8055-9d4878d634d4-istio-data\") pod \"e2e-mt-f1ef3751-openshift-default-ff56b44bc-524qh\" (UID: \"6083094a-8f65-4b46-8055-9d4878d634d4\") " pod="openshift-ingress/e2e-mt-f1ef3751-openshift-default-ff56b44bc-524qh"
Jul 09 20:34:22.070138 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:22.070032    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6083094a-8f65-4b46-8055-9d4878d634d4-istio-podinfo\") pod \"e2e-mt-f1ef3751-openshift-default-ff56b44bc-524qh\" (UID: \"6083094a-8f65-4b46-8055-9d4878d634d4\") " pod="openshift-ingress/e2e-mt-f1ef3751-openshift-default-ff56b44bc-524qh"
Jul 09 20:34:22.070138 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:22.070073    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6083094a-8f65-4b46-8055-9d4878d634d4-istio-token\") pod \"e2e-mt-f1ef3751-openshift-default-ff56b44bc-524qh\" (UID: \"6083094a-8f65-4b46-8055-9d4878d634d4\") " pod="openshift-ingress/e2e-mt-f1ef3751-openshift-default-ff56b44bc-524qh"
Jul 09 20:34:22.070235 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:22.070158    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6083094a-8f65-4b46-8055-9d4878d634d4-workload-socket\") pod \"e2e-mt-f1ef3751-openshift-default-ff56b44bc-524qh\" (UID: \"6083094a-8f65-4b46-8055-9d4878d634d4\") " pod="openshift-ingress/e2e-mt-f1ef3751-openshift-default-ff56b44bc-524qh"
Jul 09 20:34:22.070235 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:22.070208    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6083094a-8f65-4b46-8055-9d4878d634d4-credential-socket\") pod \"e2e-mt-f1ef3751-openshift-default-ff56b44bc-524qh\" (UID: \"6083094a-8f65-4b46-8055-9d4878d634d4\") " pod="openshift-ingress/e2e-mt-f1ef3751-openshift-default-ff56b44bc-524qh"
Jul 09 20:34:22.070359 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:22.070255    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbz6v\" (UniqueName: \"kubernetes.io/projected/6083094a-8f65-4b46-8055-9d4878d634d4-kube-api-access-rbz6v\") pod \"e2e-mt-f1ef3751-openshift-default-ff56b44bc-524qh\" (UID: \"6083094a-8f65-4b46-8055-9d4878d634d4\") " pod="openshift-ingress/e2e-mt-f1ef3751-openshift-default-ff56b44bc-524qh"
Jul 09 20:34:22.070359 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:22.070334    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6083094a-8f65-4b46-8055-9d4878d634d4-workload-certs\") pod \"e2e-mt-f1ef3751-openshift-default-ff56b44bc-524qh\" (UID: \"6083094a-8f65-4b46-8055-9d4878d634d4\") " pod="openshift-ingress/e2e-mt-f1ef3751-openshift-default-ff56b44bc-524qh"
Jul 09 20:34:22.070444 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:22.070366    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6083094a-8f65-4b46-8055-9d4878d634d4-istio-envoy\") pod \"e2e-mt-f1ef3751-openshift-default-ff56b44bc-524qh\" (UID: \"6083094a-8f65-4b46-8055-9d4878d634d4\") " pod="openshift-ingress/e2e-mt-f1ef3751-openshift-default-ff56b44bc-524qh"
Jul 09 20:34:22.070444 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:22.070381    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6083094a-8f65-4b46-8055-9d4878d634d4-istiod-ca-cert\") pod \"e2e-mt-f1ef3751-openshift-default-ff56b44bc-524qh\" (UID: \"6083094a-8f65-4b46-8055-9d4878d634d4\") " pod="openshift-ingress/e2e-mt-f1ef3751-openshift-default-ff56b44bc-524qh"
Jul 09 20:34:22.170771 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:22.170735    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbz6v\" (UniqueName: \"kubernetes.io/projected/6083094a-8f65-4b46-8055-9d4878d634d4-kube-api-access-rbz6v\") pod \"e2e-mt-f1ef3751-openshift-default-ff56b44bc-524qh\" (UID: \"6083094a-8f65-4b46-8055-9d4878d634d4\") " pod="openshift-ingress/e2e-mt-f1ef3751-openshift-default-ff56b44bc-524qh"
Jul 09 20:34:22.170933 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:22.170787    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6083094a-8f65-4b46-8055-9d4878d634d4-workload-certs\") pod \"e2e-mt-f1ef3751-openshift-default-ff56b44bc-524qh\" (UID: \"6083094a-8f65-4b46-8055-9d4878d634d4\") " pod="openshift-ingress/e2e-mt-f1ef3751-openshift-default-ff56b44bc-524qh"
Jul 09 20:34:22.170933 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:22.170816    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6083094a-8f65-4b46-8055-9d4878d634d4-istio-envoy\") pod \"e2e-mt-f1ef3751-openshift-default-ff56b44bc-524qh\" (UID: \"6083094a-8f65-4b46-8055-9d4878d634d4\") " pod="openshift-ingress/e2e-mt-f1ef3751-openshift-default-ff56b44bc-524qh"
Jul 09 20:34:22.170933 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:22.170832    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6083094a-8f65-4b46-8055-9d4878d634d4-istiod-ca-cert\") pod \"e2e-mt-f1ef3751-openshift-default-ff56b44bc-524qh\" (UID: \"6083094a-8f65-4b46-8055-9d4878d634d4\") " pod="openshift-ingress/e2e-mt-f1ef3751-openshift-default-ff56b44bc-524qh"
Jul 09 20:34:22.170933 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:22.170857    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6083094a-8f65-4b46-8055-9d4878d634d4-istio-data\") pod \"e2e-mt-f1ef3751-openshift-default-ff56b44bc-524qh\" (UID: \"6083094a-8f65-4b46-8055-9d4878d634d4\") " pod="openshift-ingress/e2e-mt-f1ef3751-openshift-default-ff56b44bc-524qh"
Jul 09 20:34:22.170933 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:22.170872    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6083094a-8f65-4b46-8055-9d4878d634d4-istio-podinfo\") pod \"e2e-mt-f1ef3751-openshift-default-ff56b44bc-524qh\" (UID: \"6083094a-8f65-4b46-8055-9d4878d634d4\") " pod="openshift-ingress/e2e-mt-f1ef3751-openshift-default-ff56b44bc-524qh"
Jul 09 20:34:22.170933 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:22.170896    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6083094a-8f65-4b46-8055-9d4878d634d4-istio-token\") pod \"e2e-mt-f1ef3751-openshift-default-ff56b44bc-524qh\" (UID: \"6083094a-8f65-4b46-8055-9d4878d634d4\") " pod="openshift-ingress/e2e-mt-f1ef3751-openshift-default-ff56b44bc-524qh"
Jul 09 20:34:22.170933 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:22.170924    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6083094a-8f65-4b46-8055-9d4878d634d4-workload-socket\") pod \"e2e-mt-f1ef3751-openshift-default-ff56b44bc-524qh\" (UID: \"6083094a-8f65-4b46-8055-9d4878d634d4\") " pod="openshift-ingress/e2e-mt-f1ef3751-openshift-default-ff56b44bc-524qh"
Jul 09 20:34:22.171331 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:22.170963    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6083094a-8f65-4b46-8055-9d4878d634d4-credential-socket\") pod \"e2e-mt-f1ef3751-openshift-default-ff56b44bc-524qh\" (UID: \"6083094a-8f65-4b46-8055-9d4878d634d4\") " pod="openshift-ingress/e2e-mt-f1ef3751-openshift-default-ff56b44bc-524qh"
Jul 09 20:34:22.171414 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:22.171363    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6083094a-8f65-4b46-8055-9d4878d634d4-istio-data\") pod \"e2e-mt-f1ef3751-openshift-default-ff56b44bc-524qh\" (UID: \"6083094a-8f65-4b46-8055-9d4878d634d4\") " pod="openshift-ingress/e2e-mt-f1ef3751-openshift-default-ff56b44bc-524qh"
Jul 09 20:34:22.171414 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:22.171395    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6083094a-8f65-4b46-8055-9d4878d634d4-workload-certs\") pod \"e2e-mt-f1ef3751-openshift-default-ff56b44bc-524qh\" (UID: \"6083094a-8f65-4b46-8055-9d4878d634d4\") " pod="openshift-ingress/e2e-mt-f1ef3751-openshift-default-ff56b44bc-524qh"
Jul 09 20:34:22.171717 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:22.171696    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6083094a-8f65-4b46-8055-9d4878d634d4-credential-socket\") pod \"e2e-mt-f1ef3751-openshift-default-ff56b44bc-524qh\" (UID: \"6083094a-8f65-4b46-8055-9d4878d634d4\") " pod="openshift-ingress/e2e-mt-f1ef3751-openshift-default-ff56b44bc-524qh"
Jul 09 20:34:22.171820 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:22.171729    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6083094a-8f65-4b46-8055-9d4878d634d4-workload-socket\") pod \"e2e-mt-f1ef3751-openshift-default-ff56b44bc-524qh\" (UID: \"6083094a-8f65-4b46-8055-9d4878d634d4\") " pod="openshift-ingress/e2e-mt-f1ef3751-openshift-default-ff56b44bc-524qh"
Jul 09 20:34:22.171873 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:22.171817    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6083094a-8f65-4b46-8055-9d4878d634d4-istiod-ca-cert\") pod \"e2e-mt-f1ef3751-openshift-default-ff56b44bc-524qh\" (UID: \"6083094a-8f65-4b46-8055-9d4878d634d4\") " pod="openshift-ingress/e2e-mt-f1ef3751-openshift-default-ff56b44bc-524qh"
Jul 09 20:34:22.173111 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:22.173092    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6083094a-8f65-4b46-8055-9d4878d634d4-istio-envoy\") pod \"e2e-mt-f1ef3751-openshift-default-ff56b44bc-524qh\" (UID: \"6083094a-8f65-4b46-8055-9d4878d634d4\") " pod="openshift-ingress/e2e-mt-f1ef3751-openshift-default-ff56b44bc-524qh"
Jul 09 20:34:22.173714 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:22.173696    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6083094a-8f65-4b46-8055-9d4878d634d4-istio-podinfo\") pod \"e2e-mt-f1ef3751-openshift-default-ff56b44bc-524qh\" (UID: \"6083094a-8f65-4b46-8055-9d4878d634d4\") " pod="openshift-ingress/e2e-mt-f1ef3751-openshift-default-ff56b44bc-524qh"
Jul 09 20:34:22.178625 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:22.178601    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6083094a-8f65-4b46-8055-9d4878d634d4-istio-token\") pod \"e2e-mt-f1ef3751-openshift-default-ff56b44bc-524qh\" (UID: \"6083094a-8f65-4b46-8055-9d4878d634d4\") " pod="openshift-ingress/e2e-mt-f1ef3751-openshift-default-ff56b44bc-524qh"
Jul 09 20:34:22.179003 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:22.178978    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbz6v\" (UniqueName: \"kubernetes.io/projected/6083094a-8f65-4b46-8055-9d4878d634d4-kube-api-access-rbz6v\") pod \"e2e-mt-f1ef3751-openshift-default-ff56b44bc-524qh\" (UID: \"6083094a-8f65-4b46-8055-9d4878d634d4\") " pod="openshift-ingress/e2e-mt-f1ef3751-openshift-default-ff56b44bc-524qh"
Jul 09 20:34:22.320265 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:22.320204    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f1ef3751-openshift-default-ff56b44bc-524qh"
Jul 09 20:34:22.455832 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:22.455806    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-f1ef3751-openshift-default-ff56b44bc-524qh"]
Jul 09 20:34:22.457574 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:34:22.457548    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6083094a_8f65_4b46_8055_9d4878d634d4.slice/crio-5b7a86e75b39e78d99fdb79fcbb1a83238fbfdf3a91a6e953b88ab343c656c9f WatchSource:0}: Error finding container 5b7a86e75b39e78d99fdb79fcbb1a83238fbfdf3a91a6e953b88ab343c656c9f: Status 404 returned error can't find the container with id 5b7a86e75b39e78d99fdb79fcbb1a83238fbfdf3a91a6e953b88ab343c656c9f
Jul 09 20:34:22.460170 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:22.460137    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 09 20:34:22.460230 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:22.460200    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 09 20:34:22.460270 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:22.460231    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 09 20:34:22.476367 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:22.476333    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f1ef3751-openshift-default-ff56b44bc-524qh" event={"ID":"6083094a-8f65-4b46-8055-9d4878d634d4","Type":"ContainerStarted","Data":"5b7a86e75b39e78d99fdb79fcbb1a83238fbfdf3a91a6e953b88ab343c656c9f"}
Jul 09 20:34:23.482913 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:23.482878    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f1ef3751-openshift-default-ff56b44bc-524qh" event={"ID":"6083094a-8f65-4b46-8055-9d4878d634d4","Type":"ContainerStarted","Data":"0b9f35aed24e907a60e94bc26cf4439376b395aac304afa6c21d8a2c36e715b0"}
Jul 09 20:34:23.505470 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:23.505423    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-f1ef3751-openshift-default-ff56b44bc-524qh" podStartSLOduration=2.505408047 podStartE2EDuration="2.505408047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 20:34:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 20:34:23.503653042 +0000 UTC m=+1853.921322851" watchObservedRunningTime="2026-07-09 20:34:23.505408047 +0000 UTC m=+1853.923077856"
Jul 09 20:34:24.321094 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:24.321060    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-f1ef3751-openshift-default-ff56b44bc-524qh"
Jul 09 20:34:24.325720 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:24.325697    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-f1ef3751-openshift-default-ff56b44bc-524qh"
Jul 09 20:34:24.487776 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:24.487743    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-f1ef3751-openshift-default-ff56b44bc-524qh"
Jul 09 20:34:24.488554 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:24.488533    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-f1ef3751-openshift-default-ff56b44bc-524qh"
Jul 09 20:34:25.492932 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:25.492900    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f1ef3751-888bd85d4-7fctx"]
Jul 09 20:34:25.495928 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:25.495907    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f1ef3751-888bd85d4-7fctx"
Jul 09 20:34:25.498778 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:25.498755    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-f1ef3751-serving-cert\""
Jul 09 20:34:25.504326 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:25.503969    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f1ef3751-888bd85d4-7fctx"]
Jul 09 20:34:25.598227 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:25.598198    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ccfr\" (UniqueName: \"kubernetes.io/projected/d3944024-ab2d-4ce7-8571-dff2b5cab2b6-kube-api-access-6ccfr\") pod \"maas-api-e2e-mt-f1ef3751-888bd85d4-7fctx\" (UID: \"d3944024-ab2d-4ce7-8571-dff2b5cab2b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f1ef3751-888bd85d4-7fctx"
Jul 09 20:34:25.598394 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:25.598312    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3944024-ab2d-4ce7-8571-dff2b5cab2b6-openshift-service-ca\") pod \"maas-api-e2e-mt-f1ef3751-888bd85d4-7fctx\" (UID: \"d3944024-ab2d-4ce7-8571-dff2b5cab2b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f1ef3751-888bd85d4-7fctx"
Jul 09 20:34:25.598647 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:25.598472    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3944024-ab2d-4ce7-8571-dff2b5cab2b6-maas-api-tls\") pod \"maas-api-e2e-mt-f1ef3751-888bd85d4-7fctx\" (UID: \"d3944024-ab2d-4ce7-8571-dff2b5cab2b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f1ef3751-888bd85d4-7fctx"
Jul 09 20:34:25.699315 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:25.699274    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3944024-ab2d-4ce7-8571-dff2b5cab2b6-openshift-service-ca\") pod \"maas-api-e2e-mt-f1ef3751-888bd85d4-7fctx\" (UID: \"d3944024-ab2d-4ce7-8571-dff2b5cab2b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f1ef3751-888bd85d4-7fctx"
Jul 09 20:34:25.699473 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:25.699366    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3944024-ab2d-4ce7-8571-dff2b5cab2b6-maas-api-tls\") pod \"maas-api-e2e-mt-f1ef3751-888bd85d4-7fctx\" (UID: \"d3944024-ab2d-4ce7-8571-dff2b5cab2b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f1ef3751-888bd85d4-7fctx"
Jul 09 20:34:25.699473 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:25.699384    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ccfr\" (UniqueName: \"kubernetes.io/projected/d3944024-ab2d-4ce7-8571-dff2b5cab2b6-kube-api-access-6ccfr\") pod \"maas-api-e2e-mt-f1ef3751-888bd85d4-7fctx\" (UID: \"d3944024-ab2d-4ce7-8571-dff2b5cab2b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f1ef3751-888bd85d4-7fctx"
Jul 09 20:34:25.699588 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:34:25.699500    2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-f1ef3751-serving-cert: secret "maas-api-e2e-mt-f1ef3751-serving-cert" not found
Jul 09 20:34:25.699588 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:34:25.699561    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3944024-ab2d-4ce7-8571-dff2b5cab2b6-maas-api-tls podName:d3944024-ab2d-4ce7-8571-dff2b5cab2b6 nodeName:}" failed. No retries permitted until 2026-07-09 20:34:26.199545 +0000 UTC m=+1856.617214808 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d3944024-ab2d-4ce7-8571-dff2b5cab2b6-maas-api-tls") pod "maas-api-e2e-mt-f1ef3751-888bd85d4-7fctx" (UID: "d3944024-ab2d-4ce7-8571-dff2b5cab2b6") : secret "maas-api-e2e-mt-f1ef3751-serving-cert" not found
Jul 09 20:34:25.699982 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:25.699959    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3944024-ab2d-4ce7-8571-dff2b5cab2b6-openshift-service-ca\") pod \"maas-api-e2e-mt-f1ef3751-888bd85d4-7fctx\" (UID: \"d3944024-ab2d-4ce7-8571-dff2b5cab2b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f1ef3751-888bd85d4-7fctx"
Jul 09 20:34:25.714556 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:25.714529    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ccfr\" (UniqueName: \"kubernetes.io/projected/d3944024-ab2d-4ce7-8571-dff2b5cab2b6-kube-api-access-6ccfr\") pod \"maas-api-e2e-mt-f1ef3751-888bd85d4-7fctx\" (UID: \"d3944024-ab2d-4ce7-8571-dff2b5cab2b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f1ef3751-888bd85d4-7fctx"
Jul 09 20:34:26.204223 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:26.204190    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3944024-ab2d-4ce7-8571-dff2b5cab2b6-maas-api-tls\") pod \"maas-api-e2e-mt-f1ef3751-888bd85d4-7fctx\" (UID: \"d3944024-ab2d-4ce7-8571-dff2b5cab2b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f1ef3751-888bd85d4-7fctx"
Jul 09 20:34:26.206741 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:26.206711    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3944024-ab2d-4ce7-8571-dff2b5cab2b6-maas-api-tls\") pod \"maas-api-e2e-mt-f1ef3751-888bd85d4-7fctx\" (UID: \"d3944024-ab2d-4ce7-8571-dff2b5cab2b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f1ef3751-888bd85d4-7fctx"
Jul 09 20:34:26.408365 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:26.408332    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f1ef3751-888bd85d4-7fctx"
Jul 09 20:34:26.543976 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:26.543948    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f1ef3751-888bd85d4-7fctx"]
Jul 09 20:34:26.546043 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:34:26.546007    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3944024_ab2d_4ce7_8571_dff2b5cab2b6.slice/crio-0d10589839aff17c2dffcb1c7cbe4dfcf826f88d409dfdba12b1ffc275afd920 WatchSource:0}: Error finding container 0d10589839aff17c2dffcb1c7cbe4dfcf826f88d409dfdba12b1ffc275afd920: Status 404 returned error can't find the container with id 0d10589839aff17c2dffcb1c7cbe4dfcf826f88d409dfdba12b1ffc275afd920
Jul 09 20:34:27.506506 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:27.506473    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f1ef3751-888bd85d4-7fctx" event={"ID":"d3944024-ab2d-4ce7-8571-dff2b5cab2b6","Type":"ContainerStarted","Data":"c7d9200ba39522ccb1828b19eacd77002122cfbd4affecf47720dcd42526dc1e"}
Jul 09 20:34:27.506607 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:27.506524    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f1ef3751-888bd85d4-7fctx" event={"ID":"d3944024-ab2d-4ce7-8571-dff2b5cab2b6","Type":"ContainerStarted","Data":"0d10589839aff17c2dffcb1c7cbe4dfcf826f88d409dfdba12b1ffc275afd920"}
Jul 09 20:34:27.506607 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:27.506583    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f1ef3751-888bd85d4-7fctx"
Jul 09 20:34:27.540887 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:27.540831    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f1ef3751-888bd85d4-7fctx" podStartSLOduration=1.6837730359999998 podStartE2EDuration="2.540812092s" totalImagesPullingTime="857.039056ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 20:34:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 20:34:27.540594131 +0000 UTC m=+1857.958263942" watchObservedRunningTime="2026-07-09 20:34:27.540812092 +0000 UTC m=+1857.958481901"
Jul 09 20:34:33.529759 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:33.529730    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f1ef3751-888bd85d4-7fctx"
Jul 09 20:34:37.705017 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:37.704982    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f1ef3751-888bd85d4-7fctx"]
Jul 09 20:34:37.705505 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:37.705205    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f1ef3751-888bd85d4-7fctx" podUID="d3944024-ab2d-4ce7-8571-dff2b5cab2b6" containerName="maas-api" containerID="cri-o://c7d9200ba39522ccb1828b19eacd77002122cfbd4affecf47720dcd42526dc1e" gracePeriod=30
Jul 09 20:34:37.956500 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:37.956437    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f1ef3751-888bd85d4-7fctx"
Jul 09 20:34:38.003339 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:38.003283    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3944024-ab2d-4ce7-8571-dff2b5cab2b6-openshift-service-ca\") pod \"d3944024-ab2d-4ce7-8571-dff2b5cab2b6\" (UID: \"d3944024-ab2d-4ce7-8571-dff2b5cab2b6\") "
Jul 09 20:34:38.003339 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:38.003331    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3944024-ab2d-4ce7-8571-dff2b5cab2b6-maas-api-tls\") pod \"d3944024-ab2d-4ce7-8571-dff2b5cab2b6\" (UID: \"d3944024-ab2d-4ce7-8571-dff2b5cab2b6\") "
Jul 09 20:34:38.003571 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:38.003420    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6ccfr\" (UniqueName: \"kubernetes.io/projected/d3944024-ab2d-4ce7-8571-dff2b5cab2b6-kube-api-access-6ccfr\") pod \"d3944024-ab2d-4ce7-8571-dff2b5cab2b6\" (UID: \"d3944024-ab2d-4ce7-8571-dff2b5cab2b6\") "
Jul 09 20:34:38.003917 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:38.003892    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3944024-ab2d-4ce7-8571-dff2b5cab2b6-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d3944024-ab2d-4ce7-8571-dff2b5cab2b6" (UID: "d3944024-ab2d-4ce7-8571-dff2b5cab2b6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 20:34:38.006097 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:38.006062    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3944024-ab2d-4ce7-8571-dff2b5cab2b6-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d3944024-ab2d-4ce7-8571-dff2b5cab2b6" (UID: "d3944024-ab2d-4ce7-8571-dff2b5cab2b6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 20:34:38.006097 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:38.006091    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3944024-ab2d-4ce7-8571-dff2b5cab2b6-kube-api-access-6ccfr" (OuterVolumeSpecName: "kube-api-access-6ccfr") pod "d3944024-ab2d-4ce7-8571-dff2b5cab2b6" (UID: "d3944024-ab2d-4ce7-8571-dff2b5cab2b6"). InnerVolumeSpecName "kube-api-access-6ccfr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 20:34:38.104146 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:38.104111    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ccfr\" (UniqueName: \"kubernetes.io/projected/d3944024-ab2d-4ce7-8571-dff2b5cab2b6-kube-api-access-6ccfr\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:34:38.104146 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:38.104140    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3944024-ab2d-4ce7-8571-dff2b5cab2b6-openshift-service-ca\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:34:38.104146 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:38.104151    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3944024-ab2d-4ce7-8571-dff2b5cab2b6-maas-api-tls\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:34:38.561542 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:38.561446    2562 generic.go:356] "Generic (PLEG): container finished" podID="d3944024-ab2d-4ce7-8571-dff2b5cab2b6" containerID="c7d9200ba39522ccb1828b19eacd77002122cfbd4affecf47720dcd42526dc1e" exitCode=0
Jul 09 20:34:38.561542 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:38.561501    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f1ef3751-888bd85d4-7fctx" event={"ID":"d3944024-ab2d-4ce7-8571-dff2b5cab2b6","Type":"ContainerDied","Data":"c7d9200ba39522ccb1828b19eacd77002122cfbd4affecf47720dcd42526dc1e"}
Jul 09 20:34:38.561542 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:38.561513    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f1ef3751-888bd85d4-7fctx"
Jul 09 20:34:38.561816 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:38.561581    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f1ef3751-888bd85d4-7fctx" event={"ID":"d3944024-ab2d-4ce7-8571-dff2b5cab2b6","Type":"ContainerDied","Data":"0d10589839aff17c2dffcb1c7cbe4dfcf826f88d409dfdba12b1ffc275afd920"}
Jul 09 20:34:38.561816 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:38.561607    2562 scope.go:117] "RemoveContainer" containerID="c7d9200ba39522ccb1828b19eacd77002122cfbd4affecf47720dcd42526dc1e"
Jul 09 20:34:38.569930 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:38.569906    2562 scope.go:117] "RemoveContainer" containerID="c7d9200ba39522ccb1828b19eacd77002122cfbd4affecf47720dcd42526dc1e"
Jul 09 20:34:38.570157 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:34:38.570137    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7d9200ba39522ccb1828b19eacd77002122cfbd4affecf47720dcd42526dc1e\": container with ID starting with c7d9200ba39522ccb1828b19eacd77002122cfbd4affecf47720dcd42526dc1e not found: ID does not exist" containerID="c7d9200ba39522ccb1828b19eacd77002122cfbd4affecf47720dcd42526dc1e"
Jul 09 20:34:38.570215 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:38.570164    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7d9200ba39522ccb1828b19eacd77002122cfbd4affecf47720dcd42526dc1e"} err="failed to get container status \"c7d9200ba39522ccb1828b19eacd77002122cfbd4affecf47720dcd42526dc1e\": rpc error: code = NotFound desc = could not find container \"c7d9200ba39522ccb1828b19eacd77002122cfbd4affecf47720dcd42526dc1e\": container with ID starting with c7d9200ba39522ccb1828b19eacd77002122cfbd4affecf47720dcd42526dc1e not found: ID does not exist"
Jul 09 20:34:38.598971 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:38.598951    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f1ef3751-888bd85d4-7fctx"]
Jul 09 20:34:38.613241 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:38.613221    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f1ef3751-888bd85d4-7fctx"]
Jul 09 20:34:39.967713 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:39.966753    2562 patch_prober.go:28] interesting pod/e2e-mt-f1ef3751-openshift-default-ff56b44bc-524qh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": read tcp 10.134.0.2:44902->10.134.0.53:15021: read: connection reset by peer" start-of-body=
Jul 09 20:34:39.967713 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:39.966823    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-f1ef3751-openshift-default-ff56b44bc-524qh" podUID="6083094a-8f65-4b46-8055-9d4878d634d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": read tcp 10.134.0.2:44902->10.134.0.53:15021: read: connection reset by peer"
Jul 09 20:34:40.242435 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:40.242356    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d3944024-ab2d-4ce7-8571-dff2b5cab2b6" path="/var/lib/kubelet/pods/d3944024-ab2d-4ce7-8571-dff2b5cab2b6/volumes"
Jul 09 20:34:40.572710 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:40.572644    2562 generic.go:356] "Generic (PLEG): container finished" podID="6083094a-8f65-4b46-8055-9d4878d634d4" containerID="0b9f35aed24e907a60e94bc26cf4439376b395aac304afa6c21d8a2c36e715b0" exitCode=0
Jul 09 20:34:40.572846 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:40.572716    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f1ef3751-openshift-default-ff56b44bc-524qh" event={"ID":"6083094a-8f65-4b46-8055-9d4878d634d4","Type":"ContainerDied","Data":"0b9f35aed24e907a60e94bc26cf4439376b395aac304afa6c21d8a2c36e715b0"}
Jul 09 20:34:40.573090 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:40.573075    2562 scope.go:117] "RemoveContainer" containerID="0b9f35aed24e907a60e94bc26cf4439376b395aac304afa6c21d8a2c36e715b0"
Jul 09 20:34:40.574044 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:40.574013    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 09 20:34:40.574124 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:40.574104    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 09 20:34:40.574172 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:40.574158    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 09 20:34:41.579344 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:41.579282    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f1ef3751-openshift-default-ff56b44bc-524qh" event={"ID":"6083094a-8f65-4b46-8055-9d4878d634d4","Type":"ContainerStarted","Data":"bf3ea3da3ed47f2d082893f2c1fc980ea8ef3cd3b1bf42e9f11fbc9612b37659"}
Jul 09 20:34:42.321375 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:42.321345    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-f1ef3751-openshift-default-ff56b44bc-524qh"
Jul 09 20:34:42.325073 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:42.325045    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-f1ef3751-openshift-default-ff56b44bc-524qh"
Jul 09 20:34:42.585241 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:42.585167    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-f1ef3751-openshift-default-ff56b44bc-524qh"
Jul 09 20:34:42.586134 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:42.586115    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-f1ef3751-openshift-default-ff56b44bc-524qh"
Jul 09 20:34:43.657471 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:43.657434    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-f1ef3751-openshift-default-ff56b44bc-524qh"]
Jul 09 20:34:44.595393 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:44.595354    2562 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-mt-f1ef3751-openshift-default-ff56b44bc-524qh" secret="" err="secret \"e2e-mt-f1ef3751-openshift-default-dockercfg-76xdz\" not found"
Jul 09 20:34:45.600353 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:45.600263    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-f1ef3751-openshift-default-ff56b44bc-524qh" podUID="6083094a-8f65-4b46-8055-9d4878d634d4" containerName="istio-proxy" containerID="cri-o://bf3ea3da3ed47f2d082893f2c1fc980ea8ef3cd3b1bf42e9f11fbc9612b37659" gracePeriod=30
Jul 09 20:34:50.859939 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:50.859916    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f1ef3751-openshift-default-ff56b44bc-524qh"
Jul 09 20:34:51.005663 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:51.005637    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6083094a-8f65-4b46-8055-9d4878d634d4-credential-socket\") pod \"6083094a-8f65-4b46-8055-9d4878d634d4\" (UID: \"6083094a-8f65-4b46-8055-9d4878d634d4\") "
Jul 09 20:34:51.005812 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:51.005686    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6083094a-8f65-4b46-8055-9d4878d634d4-istio-data\") pod \"6083094a-8f65-4b46-8055-9d4878d634d4\" (UID: \"6083094a-8f65-4b46-8055-9d4878d634d4\") "
Jul 09 20:34:51.005812 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:51.005729    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6083094a-8f65-4b46-8055-9d4878d634d4-istio-envoy\") pod \"6083094a-8f65-4b46-8055-9d4878d634d4\" (UID: \"6083094a-8f65-4b46-8055-9d4878d634d4\") "
Jul 09 20:34:51.005812 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:51.005753    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6083094a-8f65-4b46-8055-9d4878d634d4-istiod-ca-cert\") pod \"6083094a-8f65-4b46-8055-9d4878d634d4\" (UID: \"6083094a-8f65-4b46-8055-9d4878d634d4\") "
Jul 09 20:34:51.005812 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:51.005791    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rbz6v\" (UniqueName: \"kubernetes.io/projected/6083094a-8f65-4b46-8055-9d4878d634d4-kube-api-access-rbz6v\") pod \"6083094a-8f65-4b46-8055-9d4878d634d4\" (UID: \"6083094a-8f65-4b46-8055-9d4878d634d4\") "
Jul 09 20:34:51.006021 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:51.005814    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6083094a-8f65-4b46-8055-9d4878d634d4-workload-certs\") pod \"6083094a-8f65-4b46-8055-9d4878d634d4\" (UID: \"6083094a-8f65-4b46-8055-9d4878d634d4\") "
Jul 09 20:34:51.006021 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:51.005849    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6083094a-8f65-4b46-8055-9d4878d634d4-workload-socket\") pod \"6083094a-8f65-4b46-8055-9d4878d634d4\" (UID: \"6083094a-8f65-4b46-8055-9d4878d634d4\") "
Jul 09 20:34:51.006021 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:51.005916    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6083094a-8f65-4b46-8055-9d4878d634d4-istio-token\") pod \"6083094a-8f65-4b46-8055-9d4878d634d4\" (UID: \"6083094a-8f65-4b46-8055-9d4878d634d4\") "
Jul 09 20:34:51.006021 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:51.005925    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6083094a-8f65-4b46-8055-9d4878d634d4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6083094a-8f65-4b46-8055-9d4878d634d4" (UID: "6083094a-8f65-4b46-8055-9d4878d634d4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 20:34:51.006021 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:51.005946    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6083094a-8f65-4b46-8055-9d4878d634d4-istio-podinfo\") pod \"6083094a-8f65-4b46-8055-9d4878d634d4\" (UID: \"6083094a-8f65-4b46-8055-9d4878d634d4\") "
Jul 09 20:34:51.006281 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:51.006118    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6083094a-8f65-4b46-8055-9d4878d634d4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6083094a-8f65-4b46-8055-9d4878d634d4" (UID: "6083094a-8f65-4b46-8055-9d4878d634d4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 20:34:51.006281 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:51.006131    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6083094a-8f65-4b46-8055-9d4878d634d4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6083094a-8f65-4b46-8055-9d4878d634d4" (UID: "6083094a-8f65-4b46-8055-9d4878d634d4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 20:34:51.006281 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:51.006225    2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6083094a-8f65-4b46-8055-9d4878d634d4-istiod-ca-cert\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:34:51.006281 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:51.006240    2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6083094a-8f65-4b46-8055-9d4878d634d4-workload-certs\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:34:51.006281 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:51.006253    2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6083094a-8f65-4b46-8055-9d4878d634d4-credential-socket\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:34:51.006587 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:51.006419    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6083094a-8f65-4b46-8055-9d4878d634d4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6083094a-8f65-4b46-8055-9d4878d634d4" (UID: "6083094a-8f65-4b46-8055-9d4878d634d4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 20:34:51.006806 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:51.006749    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6083094a-8f65-4b46-8055-9d4878d634d4-istio-data" (OuterVolumeSpecName: "istio-data") pod "6083094a-8f65-4b46-8055-9d4878d634d4" (UID: "6083094a-8f65-4b46-8055-9d4878d634d4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 20:34:51.008372 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:51.008335    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6083094a-8f65-4b46-8055-9d4878d634d4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6083094a-8f65-4b46-8055-9d4878d634d4" (UID: "6083094a-8f65-4b46-8055-9d4878d634d4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 20:34:51.008758 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:51.008732    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6083094a-8f65-4b46-8055-9d4878d634d4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6083094a-8f65-4b46-8055-9d4878d634d4" (UID: "6083094a-8f65-4b46-8055-9d4878d634d4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 09 20:34:51.008758 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:51.008741    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6083094a-8f65-4b46-8055-9d4878d634d4-istio-token" (OuterVolumeSpecName: "istio-token") pod "6083094a-8f65-4b46-8055-9d4878d634d4" (UID: "6083094a-8f65-4b46-8055-9d4878d634d4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 20:34:51.008895 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:51.008783    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6083094a-8f65-4b46-8055-9d4878d634d4-kube-api-access-rbz6v" (OuterVolumeSpecName: "kube-api-access-rbz6v") pod "6083094a-8f65-4b46-8055-9d4878d634d4" (UID: "6083094a-8f65-4b46-8055-9d4878d634d4"). InnerVolumeSpecName "kube-api-access-rbz6v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 20:34:51.107539 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:51.107514    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbz6v\" (UniqueName: \"kubernetes.io/projected/6083094a-8f65-4b46-8055-9d4878d634d4-kube-api-access-rbz6v\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:34:51.107539 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:51.107537    2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6083094a-8f65-4b46-8055-9d4878d634d4-workload-socket\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:34:51.107663 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:51.107547    2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6083094a-8f65-4b46-8055-9d4878d634d4-istio-token\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:34:51.107663 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:51.107556    2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6083094a-8f65-4b46-8055-9d4878d634d4-istio-podinfo\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:34:51.107663 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:51.107564    2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6083094a-8f65-4b46-8055-9d4878d634d4-istio-data\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:34:51.107663 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:51.107571    2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6083094a-8f65-4b46-8055-9d4878d634d4-istio-envoy\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:34:51.630979 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:51.630943    2562 generic.go:356] "Generic (PLEG): container finished" podID="6083094a-8f65-4b46-8055-9d4878d634d4" containerID="bf3ea3da3ed47f2d082893f2c1fc980ea8ef3cd3b1bf42e9f11fbc9612b37659" exitCode=0
Jul 09 20:34:51.631118 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:51.631002    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f1ef3751-openshift-default-ff56b44bc-524qh"
Jul 09 20:34:51.631118 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:51.631036    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f1ef3751-openshift-default-ff56b44bc-524qh" event={"ID":"6083094a-8f65-4b46-8055-9d4878d634d4","Type":"ContainerDied","Data":"bf3ea3da3ed47f2d082893f2c1fc980ea8ef3cd3b1bf42e9f11fbc9612b37659"}
Jul 09 20:34:51.631118 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:51.631098    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f1ef3751-openshift-default-ff56b44bc-524qh" event={"ID":"6083094a-8f65-4b46-8055-9d4878d634d4","Type":"ContainerDied","Data":"5b7a86e75b39e78d99fdb79fcbb1a83238fbfdf3a91a6e953b88ab343c656c9f"}
Jul 09 20:34:51.631227 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:51.631125    2562 scope.go:117] "RemoveContainer" containerID="bf3ea3da3ed47f2d082893f2c1fc980ea8ef3cd3b1bf42e9f11fbc9612b37659"
Jul 09 20:34:51.640357 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:51.640336    2562 scope.go:117] "RemoveContainer" containerID="0b9f35aed24e907a60e94bc26cf4439376b395aac304afa6c21d8a2c36e715b0"
Jul 09 20:34:51.651517 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:51.649752    2562 scope.go:117] "RemoveContainer" containerID="bf3ea3da3ed47f2d082893f2c1fc980ea8ef3cd3b1bf42e9f11fbc9612b37659"
Jul 09 20:34:51.651916 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:34:51.651886    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf3ea3da3ed47f2d082893f2c1fc980ea8ef3cd3b1bf42e9f11fbc9612b37659\": container with ID starting with bf3ea3da3ed47f2d082893f2c1fc980ea8ef3cd3b1bf42e9f11fbc9612b37659 not found: ID does not exist" containerID="bf3ea3da3ed47f2d082893f2c1fc980ea8ef3cd3b1bf42e9f11fbc9612b37659"
Jul 09 20:34:51.652038 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:51.651943    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf3ea3da3ed47f2d082893f2c1fc980ea8ef3cd3b1bf42e9f11fbc9612b37659"} err="failed to get container status \"bf3ea3da3ed47f2d082893f2c1fc980ea8ef3cd3b1bf42e9f11fbc9612b37659\": rpc error: code = NotFound desc = could not find container \"bf3ea3da3ed47f2d082893f2c1fc980ea8ef3cd3b1bf42e9f11fbc9612b37659\": container with ID starting with bf3ea3da3ed47f2d082893f2c1fc980ea8ef3cd3b1bf42e9f11fbc9612b37659 not found: ID does not exist"
Jul 09 20:34:51.652038 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:51.651970    2562 scope.go:117] "RemoveContainer" containerID="0b9f35aed24e907a60e94bc26cf4439376b395aac304afa6c21d8a2c36e715b0"
Jul 09 20:34:51.652660 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:34:51.652632    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b9f35aed24e907a60e94bc26cf4439376b395aac304afa6c21d8a2c36e715b0\": container with ID starting with 0b9f35aed24e907a60e94bc26cf4439376b395aac304afa6c21d8a2c36e715b0 not found: ID does not exist" containerID="0b9f35aed24e907a60e94bc26cf4439376b395aac304afa6c21d8a2c36e715b0"
Jul 09 20:34:51.652788 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:51.652668    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b9f35aed24e907a60e94bc26cf4439376b395aac304afa6c21d8a2c36e715b0"} err="failed to get container status \"0b9f35aed24e907a60e94bc26cf4439376b395aac304afa6c21d8a2c36e715b0\": rpc error: code = NotFound desc = could not find container \"0b9f35aed24e907a60e94bc26cf4439376b395aac304afa6c21d8a2c36e715b0\": container with ID starting with 0b9f35aed24e907a60e94bc26cf4439376b395aac304afa6c21d8a2c36e715b0 not found: ID does not exist"
Jul 09 20:34:51.654590 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:51.654566    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-f1ef3751-openshift-default-ff56b44bc-524qh"]
Jul 09 20:34:51.658629 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:51.658606    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-f1ef3751-openshift-default-ff56b44bc-524qh"]
Jul 09 20:34:52.241983 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:34:52.241951    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6083094a-8f65-4b46-8055-9d4878d634d4" path="/var/lib/kubelet/pods/6083094a-8f65-4b46-8055-9d4878d634d4/volumes"
Jul 09 20:35:03.114684 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:35:03.114650    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-92a49739-openshift-default-544dd4597f-92wsm"]
Jul 09 20:35:03.115158 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:35:03.115023    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6083094a-8f65-4b46-8055-9d4878d634d4" containerName="istio-proxy"
Jul 09 20:35:03.115158 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:35:03.115034    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="6083094a-8f65-4b46-8055-9d4878d634d4" containerName="istio-proxy"
Jul 09 20:35:03.115158 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:35:03.115047    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6083094a-8f65-4b46-8055-9d4878d634d4" containerName="istio-proxy"
Jul 09 20:35:03.115158 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:35:03.115053    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="6083094a-8f65-4b46-8055-9d4878d634d4" containerName="istio-proxy"
Jul 09 20:35:03.115158 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:35:03.115072    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3944024-ab2d-4ce7-8571-dff2b5cab2b6" containerName="maas-api"
Jul 09 20:35:03.115158 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:35:03.115077    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3944024-ab2d-4ce7-8571-dff2b5cab2b6" containerName="maas-api"
Jul 09 20:35:03.115158 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:35:03.115133    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3944024-ab2d-4ce7-8571-dff2b5cab2b6" containerName="maas-api"
Jul 09 20:35:03.115158 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:35:03.115142    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="6083094a-8f65-4b46-8055-9d4878d634d4" containerName="istio-proxy"
Jul 09 20:35:03.115158 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:35:03.115148    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="6083094a-8f65-4b46-8055-9d4878d634d4" containerName="istio-proxy"
Jul 09 20:35:03.118151 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:35:03.118133    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-92a49739-openshift-default-544dd4597f-92wsm"
Jul 09 20:35:03.121113 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:35:03.121092    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-92a49739-openshift-default-dockercfg-2tmzx\""
Jul 09 20:35:03.129240 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:35:03.129217    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-92a49739-openshift-default-544dd4597f-92wsm"]
Jul 09 20:35:03.209646 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:35:03.209618    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1044447-019a-4db2-8503-5ad39e127582-istio-data\") pod \"e2e-mt-92a49739-openshift-default-544dd4597f-92wsm\" (UID: \"f1044447-019a-4db2-8503-5ad39e127582\") " pod="openshift-ingress/e2e-mt-92a49739-openshift-default-544dd4597f-92wsm"
Jul 09 20:35:03.209766 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:35:03.209695    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1044447-019a-4db2-8503-5ad39e127582-workload-socket\") pod \"e2e-mt-92a49739-openshift-default-544dd4597f-92wsm\" (UID: \"f1044447-019a-4db2-8503-5ad39e127582\") " pod="openshift-ingress/e2e-mt-92a49739-openshift-default-544dd4597f-92wsm"
Jul 09 20:35:03.209766 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:35:03.209747    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1044447-019a-4db2-8503-5ad39e127582-istio-podinfo\") pod \"e2e-mt-92a49739-openshift-default-544dd4597f-92wsm\" (UID: \"f1044447-019a-4db2-8503-5ad39e127582\") " pod="openshift-ingress/e2e-mt-92a49739-openshift-default-544dd4597f-92wsm"
Jul 09 20:35:03.209868 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:35:03.209819    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1044447-019a-4db2-8503-5ad39e127582-istio-token\") pod \"e2e-mt-92a49739-openshift-default-544dd4597f-92wsm\" (UID: \"f1044447-019a-4db2-8503-5ad39e127582\") " pod="openshift-ingress/e2e-mt-92a49739-openshift-default-544dd4597f-92wsm"
Jul 09 20:35:03.209868 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:35:03.209859    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1044447-019a-4db2-8503-5ad39e127582-workload-certs\") pod \"e2e-mt-92a49739-openshift-default-544dd4597f-92wsm\" (UID: \"f1044447-019a-4db2-8503-5ad39e127582\") " pod="openshift-ingress/e2e-mt-92a49739-openshift-default-544dd4597f-92wsm"
Jul 09 20:35:03.209953 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:35:03.209883    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1044447-019a-4db2-8503-5ad39e127582-istio-envoy\") pod \"e2e-mt-92a49739-openshift-default-544dd4597f-92wsm\" (UID: \"f1044447-019a-4db2-8503-5ad39e127582\") " pod="openshift-ingress/e2e-mt-92a49739-openshift-default-544dd4597f-92wsm"
Jul 09 20:35:03.209953 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:35:03.209907    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dr9rx\" (UniqueName: \"kubernetes.io/projected/f1044447-019a-4db2-8503-5ad39e127582-kube-api-access-dr9rx\") pod \"e2e-mt-92a49739-openshift-default-544dd4597f-92wsm\" (UID: \"f1044447-019a-4db2-8503-5ad39e127582\") " pod="openshift-ingress/e2e-mt-92a49739-openshift-default-544dd4597f-92wsm"
Jul 09 20:35:03.209953 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:35:03.209937    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1044447-019a-4db2-8503-5ad39e127582-credential-socket\") pod \"e2e-mt-92a49739-openshift-default-544dd4597f-92wsm\" (UID: \"f1044447-019a-4db2-8503-5ad39e127582\") " pod="openshift-ingress/e2e-mt-92a49739-openshift-default-544dd4597f-92wsm"
Jul 09 20:35:03.210051 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:35:03.209968    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1044447-019a-4db2-8503-5ad39e127582-istiod-ca-cert\") pod \"e2e-mt-92a49739-openshift-default-544dd4597f-92wsm\" (UID: \"f1044447-019a-4db2-8503-5ad39e127582\") " pod="openshift-ingress/e2e-mt-92a49739-openshift-default-544dd4597f-92wsm"
Jul 09 20:35:03.311122 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:35:03.311096    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1044447-019a-4db2-8503-5ad39e127582-workload-socket\") pod \"e2e-mt-92a49739-openshift-default-544dd4597f-92wsm\" (UID: \"f1044447-019a-4db2-8503-5ad39e127582\") " pod="openshift-ingress/e2e-mt-92a49739-openshift-default-544dd4597f-92wsm"
Jul 09 20:35:03.311258 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:35:03.311137    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1044447-019a-4db2-8503-5ad39e127582-istio-podinfo\") pod \"e2e-mt-92a49739-openshift-default-544dd4597f-92wsm\" (UID: \"f1044447-019a-4db2-8503-5ad39e127582\") " pod="openshift-ingress/e2e-mt-92a49739-openshift-default-544dd4597f-92wsm"
Jul 09 20:35:03.311258 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:35:03.311170    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1044447-019a-4db2-8503-5ad39e127582-istio-token\") pod \"e2e-mt-92a49739-openshift-default-544dd4597f-92wsm\" (UID: \"f1044447-019a-4db2-8503-5ad39e127582\") " pod="openshift-ingress/e2e-mt-92a49739-openshift-default-544dd4597f-92wsm"
Jul 09 20:35:03.311258 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:35:03.311188    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1044447-019a-4db2-8503-5ad39e127582-workload-certs\") pod \"e2e-mt-92a49739-openshift-default-544dd4597f-92wsm\" (UID: \"f1044447-019a-4db2-8503-5ad39e127582\") " pod="openshift-ingress/e2e-mt-92a49739-openshift-default-544dd4597f-92wsm"
Jul 09 20:35:03.311258 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:35:03.311204    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1044447-019a-4db2-8503-5ad39e127582-istio-envoy\") pod \"e2e-mt-92a49739-openshift-default-544dd4597f-92wsm\" (UID: \"f1044447-019a-4db2-8503-5ad39e127582\") " pod="openshift-ingress/e2e-mt-92a49739-openshift-default-544dd4597f-92wsm"
Jul 09 20:35:03.311258 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:35:03.311219    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dr9rx\" (UniqueName: \"kubernetes.io/projected/f1044447-019a-4db2-8503-5ad39e127582-kube-api-access-dr9rx\") pod \"e2e-mt-92a49739-openshift-default-544dd4597f-92wsm\" (UID: \"f1044447-019a-4db2-8503-5ad39e127582\") " pod="openshift-ingress/e2e-mt-92a49739-openshift-default-544dd4597f-92wsm"
Jul 09 20:35:03.311258 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:35:03.311237    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1044447-019a-4db2-8503-5ad39e127582-credential-socket\") pod \"e2e-mt-92a49739-openshift-default-544dd4597f-92wsm\" (UID: \"f1044447-019a-4db2-8503-5ad39e127582\") " pod="openshift-ingress/e2e-mt-92a49739-openshift-default-544dd4597f-92wsm"
Jul 09 20:35:03.311621 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:35:03.311265    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1044447-019a-4db2-8503-5ad39e127582-istiod-ca-cert\") pod \"e2e-mt-92a49739-openshift-default-544dd4597f-92wsm\" (UID: \"f1044447-019a-4db2-8503-5ad39e127582\") " pod="openshift-ingress/e2e-mt-92a49739-openshift-default-544dd4597f-92wsm"
Jul 09 20:35:03.311621 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:35:03.311334    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1044447-019a-4db2-8503-5ad39e127582-istio-data\") pod \"e2e-mt-92a49739-openshift-default-544dd4597f-92wsm\" (UID: \"f1044447-019a-4db2-8503-5ad39e127582\") " pod="openshift-ingress/e2e-mt-92a49739-openshift-default-544dd4597f-92wsm"
Jul 09 20:35:03.311621 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:35:03.311495    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1044447-019a-4db2-8503-5ad39e127582-workload-socket\") pod \"e2e-mt-92a49739-openshift-default-544dd4597f-92wsm\" (UID: \"f1044447-019a-4db2-8503-5ad39e127582\") " pod="openshift-ingress/e2e-mt-92a49739-openshift-default-544dd4597f-92wsm"
Jul 09 20:35:03.311765 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:35:03.311617    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1044447-019a-4db2-8503-5ad39e127582-workload-certs\") pod \"e2e-mt-92a49739-openshift-default-544dd4597f-92wsm\" (UID: \"f1044447-019a-4db2-8503-5ad39e127582\") " pod="openshift-ingress/e2e-mt-92a49739-openshift-default-544dd4597f-92wsm"
Jul 09 20:35:03.311826 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:35:03.311802    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1044447-019a-4db2-8503-5ad39e127582-istio-data\") pod \"e2e-mt-92a49739-openshift-default-544dd4597f-92wsm\" (UID: \"f1044447-019a-4db2-8503-5ad39e127582\") " pod="openshift-ingress/e2e-mt-92a49739-openshift-default-544dd4597f-92wsm"
Jul 09 20:35:03.311880 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:35:03.311859    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1044447-019a-4db2-8503-5ad39e127582-credential-socket\") pod \"e2e-mt-92a49739-openshift-default-544dd4597f-92wsm\" (UID: \"f1044447-019a-4db2-8503-5ad39e127582\") " pod="openshift-ingress/e2e-mt-92a49739-openshift-default-544dd4597f-92wsm"
Jul 09 20:35:03.312149 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:35:03.312123    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1044447-019a-4db2-8503-5ad39e127582-istiod-ca-cert\") pod \"e2e-mt-92a49739-openshift-default-544dd4597f-92wsm\" (UID: \"f1044447-019a-4db2-8503-5ad39e127582\") " pod="openshift-ingress/e2e-mt-92a49739-openshift-default-544dd4597f-92wsm"
Jul 09 20:35:03.313499 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:35:03.313479    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1044447-019a-4db2-8503-5ad39e127582-istio-envoy\") pod \"e2e-mt-92a49739-openshift-default-544dd4597f-92wsm\" (UID: \"f1044447-019a-4db2-8503-5ad39e127582\") " pod="openshift-ingress/e2e-mt-92a49739-openshift-default-544dd4597f-92wsm"
Jul 09 20:35:03.313701 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:35:03.313684    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1044447-019a-4db2-8503-5ad39e127582-istio-podinfo\") pod \"e2e-mt-92a49739-openshift-default-544dd4597f-92wsm\" (UID: \"f1044447-019a-4db2-8503-5ad39e127582\") " pod="openshift-ingress/e2e-mt-92a49739-openshift-default-544dd4597f-92wsm"
Jul 09 20:35:03.319433 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:35:03.319411    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1044447-019a-4db2-8503-5ad39e127582-istio-token\") pod \"e2e-mt-92a49739-openshift-default-544dd4597f-92wsm\" (UID: \"f1044447-019a-4db2-8503-5ad39e127582\") " pod="openshift-ingress/e2e-mt-92a49739-openshift-default-544dd4597f-92wsm"
Jul 09 20:35:03.319620 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:35:03.319599    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dr9rx\" (UniqueName: \"kubernetes.io/projected/f1044447-019a-4db2-8503-5ad39e127582-kube-api-access-dr9rx\") pod \"e2e-mt-92a49739-openshift-default-544dd4597f-92wsm\" (UID: \"f1044447-019a-4db2-8503-5ad39e127582\") " pod="openshift-ingress/e2e-mt-92a49739-openshift-default-544dd4597f-92wsm"
Jul 09 20:35:03.429997 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:35:03.429941    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-92a49739-openshift-default-544dd4597f-92wsm"
Jul 09 20:35:03.567620 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:35:03.567595    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-92a49739-openshift-default-544dd4597f-92wsm"]
Jul 09 20:35:03.569287 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:35:03.569261    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1044447_019a_4db2_8503_5ad39e127582.slice/crio-94444cb665182aa46dd17e648d8742ddbe6a613d950c5f9063ccc98d9b716fef WatchSource:0}: Error finding container 94444cb665182aa46dd17e648d8742ddbe6a613d950c5f9063ccc98d9b716fef: Status 404 returned error can't find the container with id 94444cb665182aa46dd17e648d8742ddbe6a613d950c5f9063ccc98d9b716fef
Jul 09 20:35:03.571536 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:35:03.571499    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 09 20:35:03.571612 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:35:03.571586    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 09 20:35:03.571651 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:35:03.571636    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 09 20:35:03.691896 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:35:03.691817    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-92a49739-openshift-default-544dd4597f-92wsm" event={"ID":"f1044447-019a-4db2-8503-5ad39e127582","Type":"ContainerStarted","Data":"a0a3e49fad127d7dc983b521185224d1c568b8eccc326b64ab862d05737b5e06"}
Jul 09 20:35:03.691896 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:35:03.691882    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-92a49739-openshift-default-544dd4597f-92wsm" event={"ID":"f1044447-019a-4db2-8503-5ad39e127582","Type":"ContainerStarted","Data":"94444cb665182aa46dd17e648d8742ddbe6a613d950c5f9063ccc98d9b716fef"}
Jul 09 20:35:03.711840 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:35:03.711789    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-92a49739-openshift-default-544dd4597f-92wsm" podStartSLOduration=0.711773503 podStartE2EDuration="711.773503ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 20:35:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 20:35:03.70967971 +0000 UTC m=+1894.127349520" watchObservedRunningTime="2026-07-09 20:35:03.711773503 +0000 UTC m=+1894.129443311"
Jul 09 20:35:04.430337 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:35:04.430292    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-92a49739-openshift-default-544dd4597f-92wsm"
Jul 09 20:35:04.435758 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:35:04.435734    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-92a49739-openshift-default-544dd4597f-92wsm"
Jul 09 20:35:04.696755 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:35:04.696668    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-92a49739-openshift-default-544dd4597f-92wsm"
Jul 09 20:35:04.697604 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:35:04.697572    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-92a49739-openshift-default-544dd4597f-92wsm"
Jul 09 20:35:06.007882 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:35:06.007852    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-92a49739-55c56bdd56-d4kp2"]
Jul 09 20:35:06.016213 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:35:06.016183    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-92a49739-55c56bdd56-d4kp2"
Jul 09 20:35:06.019806 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:35:06.019785    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-92a49739-serving-cert\""
Jul 09 20:35:06.030468 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:35:06.030439    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-92a49739-55c56bdd56-d4kp2"]
Jul 09 20:35:06.135974 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:35:06.135936    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/096b5d9d-99d2-41d0-b766-8be17a07e84c-openshift-service-ca\") pod \"maas-api-e2e-mt-92a49739-55c56bdd56-d4kp2\" (UID: \"096b5d9d-99d2-41d0-b766-8be17a07e84c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-92a49739-55c56bdd56-d4kp2"
Jul 09 20:35:06.135974 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:35:06.135970    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqjwb\" (UniqueName: \"kubernetes.io/projected/096b5d9d-99d2-41d0-b766-8be17a07e84c-kube-api-access-tqjwb\") pod \"maas-api-e2e-mt-92a49739-55c56bdd56-d4kp2\" (UID: \"096b5d9d-99d2-41d0-b766-8be17a07e84c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-92a49739-55c56bdd56-d4kp2"
Jul 09 20:35:06.136199 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:35:06.136006    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/096b5d9d-99d2-41d0-b766-8be17a07e84c-maas-api-tls\") pod \"maas-api-e2e-mt-92a49739-55c56bdd56-d4kp2\" (UID: \"096b5d9d-99d2-41d0-b766-8be17a07e84c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-92a49739-55c56bdd56-d4kp2"
Jul 09 20:35:06.236585 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:35:06.236555    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/096b5d9d-99d2-41d0-b766-8be17a07e84c-openshift-service-ca\") pod \"maas-api-e2e-mt-92a49739-55c56bdd56-d4kp2\" (UID: \"096b5d9d-99d2-41d0-b766-8be17a07e84c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-92a49739-55c56bdd56-d4kp2"
Jul 09 20:35:06.236767 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:35:06.236585    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqjwb\" (UniqueName: \"kubernetes.io/projected/096b5d9d-99d2-41d0-b766-8be17a07e84c-kube-api-access-tqjwb\") pod \"maas-api-e2e-mt-92a49739-55c56bdd56-d4kp2\" (UID: \"096b5d9d-99d2-41d0-b766-8be17a07e84c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-92a49739-55c56bdd56-d4kp2"
Jul 09 20:35:06.236767 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:35:06.236642    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/096b5d9d-99d2-41d0-b766-8be17a07e84c-maas-api-tls\") pod \"maas-api-e2e-mt-92a49739-55c56bdd56-d4kp2\" (UID: \"096b5d9d-99d2-41d0-b766-8be17a07e84c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-92a49739-55c56bdd56-d4kp2"
Jul 09 20:35:06.237241 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:35:06.237218    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/096b5d9d-99d2-41d0-b766-8be17a07e84c-openshift-service-ca\") pod \"maas-api-e2e-mt-92a49739-55c56bdd56-d4kp2\" (UID: \"096b5d9d-99d2-41d0-b766-8be17a07e84c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-92a49739-55c56bdd56-d4kp2"
Jul 09 20:35:06.239229 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:35:06.239204    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/096b5d9d-99d2-41d0-b766-8be17a07e84c-maas-api-tls\") pod \"maas-api-e2e-mt-92a49739-55c56bdd56-d4kp2\" (UID: \"096b5d9d-99d2-41d0-b766-8be17a07e84c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-92a49739-55c56bdd56-d4kp2"
Jul 09 20:35:06.246584 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:35:06.246563    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqjwb\" (UniqueName: \"kubernetes.io/projected/096b5d9d-99d2-41d0-b766-8be17a07e84c-kube-api-access-tqjwb\") pod \"maas-api-e2e-mt-92a49739-55c56bdd56-d4kp2\" (UID: \"096b5d9d-99d2-41d0-b766-8be17a07e84c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-92a49739-55c56bdd56-d4kp2"
Jul 09 20:35:06.327547 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:35:06.327443    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-92a49739-55c56bdd56-d4kp2"
Jul 09 20:35:06.469080 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:35:06.469056    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-92a49739-55c56bdd56-d4kp2"]
Jul 09 20:35:06.471328 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:35:06.471287    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod096b5d9d_99d2_41d0_b766_8be17a07e84c.slice/crio-47a1e0eb35d6a10e865cacd67bc5c11349c8d3abb51ded541b09a1b7314de0b4 WatchSource:0}: Error finding container 47a1e0eb35d6a10e865cacd67bc5c11349c8d3abb51ded541b09a1b7314de0b4: Status 404 returned error can't find the container with id 47a1e0eb35d6a10e865cacd67bc5c11349c8d3abb51ded541b09a1b7314de0b4
Jul 09 20:35:06.472960 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:35:06.472939    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 09 20:35:06.709359 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:35:06.709321    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-92a49739-55c56bdd56-d4kp2" event={"ID":"096b5d9d-99d2-41d0-b766-8be17a07e84c","Type":"ContainerStarted","Data":"47a1e0eb35d6a10e865cacd67bc5c11349c8d3abb51ded541b09a1b7314de0b4"}
Jul 09 20:35:07.715923 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:35:07.715880    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-92a49739-55c56bdd56-d4kp2" event={"ID":"096b5d9d-99d2-41d0-b766-8be17a07e84c","Type":"ContainerStarted","Data":"1498df8e4c822cc8b90d9a7c9e569f21ac8ad8fd019c4cc2533b77ceaef4440a"}
Jul 09 20:35:07.716316 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:35:07.715947    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-92a49739-55c56bdd56-d4kp2"
Jul 09 20:35:07.735883 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:35:07.735833    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-92a49739-55c56bdd56-d4kp2" podStartSLOduration=2.413128655 podStartE2EDuration="2.735817743s" totalImagesPullingTime="322.689088ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 20:35:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 20:35:07.734337294 +0000 UTC m=+1898.152007107" watchObservedRunningTime="2026-07-09 20:35:07.735817743 +0000 UTC m=+1898.153487552"
Jul 09 20:35:13.734118 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:35:13.734080    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-92a49739-55c56bdd56-d4kp2"
Jul 09 20:35:20.139919 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:35:20.138943    2562 patch_prober.go:28] interesting pod/e2e-mt-92a49739-openshift-default-544dd4597f-92wsm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": read tcp 10.134.0.2:33884->10.134.0.55:15021: read: connection reset by peer" start-of-body=
Jul 09 20:35:20.139919 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:35:20.139003    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-92a49739-openshift-default-544dd4597f-92wsm" podUID="f1044447-019a-4db2-8503-5ad39e127582" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": read tcp 10.134.0.2:33884->10.134.0.55:15021: read: connection reset by peer"
Jul 09 20:35:20.775601 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:35:20.775567    2562 generic.go:356] "Generic (PLEG): container finished" podID="f1044447-019a-4db2-8503-5ad39e127582" containerID="a0a3e49fad127d7dc983b521185224d1c568b8eccc326b64ab862d05737b5e06" exitCode=0
Jul 09 20:35:20.775765 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:35:20.775645    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-92a49739-openshift-default-544dd4597f-92wsm" event={"ID":"f1044447-019a-4db2-8503-5ad39e127582","Type":"ContainerDied","Data":"a0a3e49fad127d7dc983b521185224d1c568b8eccc326b64ab862d05737b5e06"}
Jul 09 20:35:20.776033 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:35:20.776019    2562 scope.go:117] "RemoveContainer" containerID="a0a3e49fad127d7dc983b521185224d1c568b8eccc326b64ab862d05737b5e06"
Jul 09 20:35:20.776959 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:35:20.776929    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 09 20:35:20.777054 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:35:20.776993    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 09 20:35:20.777054 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:35:20.777024    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 09 20:35:21.782535 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:35:21.782507    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-92a49739-openshift-default-544dd4597f-92wsm" event={"ID":"f1044447-019a-4db2-8503-5ad39e127582","Type":"ContainerStarted","Data":"f85e0f5a4c2d5a79fa4e71f4f79d61809174ac80cf137a5802a2ed2b040ef875"}
Jul 09 20:35:22.430891 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:35:22.430849    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-92a49739-openshift-default-544dd4597f-92wsm"
Jul 09 20:35:22.431374 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:35:22.431347    2562 patch_prober.go:28] interesting pod/e2e-mt-92a49739-openshift-default-544dd4597f-92wsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Jul 09 20:35:22.431580 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:35:22.431552    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-92a49739-openshift-default-544dd4597f-92wsm" podUID="f1044447-019a-4db2-8503-5ad39e127582" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Jul 09 20:35:23.431129 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:35:23.431091    2562 patch_prober.go:28] interesting pod/e2e-mt-92a49739-openshift-default-544dd4597f-92wsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Jul 09 20:35:23.431728 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:35:23.431696    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-92a49739-openshift-default-544dd4597f-92wsm" podUID="f1044447-019a-4db2-8503-5ad39e127582" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Jul 09 20:35:24.432205 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:35:24.431763    2562 patch_prober.go:28] interesting pod/e2e-mt-92a49739-openshift-default-544dd4597f-92wsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Jul 09 20:35:24.432205 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:35:24.431828    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-92a49739-openshift-default-544dd4597f-92wsm" podUID="f1044447-019a-4db2-8503-5ad39e127582" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Jul 09 20:35:25.430286 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:35:25.430253    2562 patch_prober.go:28] interesting pod/e2e-mt-92a49739-openshift-default-544dd4597f-92wsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Jul 09 20:35:25.430489 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:35:25.430334    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-92a49739-openshift-default-544dd4597f-92wsm" podUID="f1044447-019a-4db2-8503-5ad39e127582" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Jul 09 20:35:26.434319 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:35:26.434272    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-92a49739-openshift-default-544dd4597f-92wsm"
Jul 09 20:35:26.434769 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:35:26.434752    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-92a49739-openshift-default-544dd4597f-92wsm"
Jul 09 20:35:26.435459 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:35:26.435442    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-92a49739-openshift-default-544dd4597f-92wsm"
Jul 09 20:35:53.155323 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:35:53.155281    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-92a49739-55c56bdd56-d4kp2"]
Jul 09 20:35:53.155872 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:35:53.155505    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-92a49739-55c56bdd56-d4kp2" podUID="096b5d9d-99d2-41d0-b766-8be17a07e84c" containerName="maas-api" containerID="cri-o://1498df8e4c822cc8b90d9a7c9e569f21ac8ad8fd019c4cc2533b77ceaef4440a" gracePeriod=30
Jul 09 20:35:53.403483 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:35:53.403454    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-92a49739-55c56bdd56-d4kp2"
Jul 09 20:35:53.535217 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:35:53.535194    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/096b5d9d-99d2-41d0-b766-8be17a07e84c-maas-api-tls\") pod \"096b5d9d-99d2-41d0-b766-8be17a07e84c\" (UID: \"096b5d9d-99d2-41d0-b766-8be17a07e84c\") "
Jul 09 20:35:53.535362 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:35:53.535236    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/096b5d9d-99d2-41d0-b766-8be17a07e84c-openshift-service-ca\") pod \"096b5d9d-99d2-41d0-b766-8be17a07e84c\" (UID: \"096b5d9d-99d2-41d0-b766-8be17a07e84c\") "
Jul 09 20:35:53.535362 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:35:53.535265    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tqjwb\" (UniqueName: \"kubernetes.io/projected/096b5d9d-99d2-41d0-b766-8be17a07e84c-kube-api-access-tqjwb\") pod \"096b5d9d-99d2-41d0-b766-8be17a07e84c\" (UID: \"096b5d9d-99d2-41d0-b766-8be17a07e84c\") "
Jul 09 20:35:53.535593 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:35:53.535568    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/096b5d9d-99d2-41d0-b766-8be17a07e84c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "096b5d9d-99d2-41d0-b766-8be17a07e84c" (UID: "096b5d9d-99d2-41d0-b766-8be17a07e84c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 20:35:53.537239 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:35:53.537217    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/096b5d9d-99d2-41d0-b766-8be17a07e84c-kube-api-access-tqjwb" (OuterVolumeSpecName: "kube-api-access-tqjwb") pod "096b5d9d-99d2-41d0-b766-8be17a07e84c" (UID: "096b5d9d-99d2-41d0-b766-8be17a07e84c"). InnerVolumeSpecName "kube-api-access-tqjwb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 20:35:53.537239 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:35:53.537230    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/096b5d9d-99d2-41d0-b766-8be17a07e84c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "096b5d9d-99d2-41d0-b766-8be17a07e84c" (UID: "096b5d9d-99d2-41d0-b766-8be17a07e84c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 20:35:53.636820 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:35:53.636795    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/096b5d9d-99d2-41d0-b766-8be17a07e84c-maas-api-tls\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:35:53.636820 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:35:53.636818    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/096b5d9d-99d2-41d0-b766-8be17a07e84c-openshift-service-ca\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:35:53.636949 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:35:53.636832    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqjwb\" (UniqueName: \"kubernetes.io/projected/096b5d9d-99d2-41d0-b766-8be17a07e84c-kube-api-access-tqjwb\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:35:53.930528 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:35:53.930469    2562 generic.go:356] "Generic (PLEG): container finished" podID="096b5d9d-99d2-41d0-b766-8be17a07e84c" containerID="1498df8e4c822cc8b90d9a7c9e569f21ac8ad8fd019c4cc2533b77ceaef4440a" exitCode=0
Jul 09 20:35:53.930528 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:35:53.930510    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-92a49739-55c56bdd56-d4kp2"
Jul 09 20:35:53.930662 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:35:53.930548    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-92a49739-55c56bdd56-d4kp2" event={"ID":"096b5d9d-99d2-41d0-b766-8be17a07e84c","Type":"ContainerDied","Data":"1498df8e4c822cc8b90d9a7c9e569f21ac8ad8fd019c4cc2533b77ceaef4440a"}
Jul 09 20:35:53.930662 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:35:53.930600    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-92a49739-55c56bdd56-d4kp2" event={"ID":"096b5d9d-99d2-41d0-b766-8be17a07e84c","Type":"ContainerDied","Data":"47a1e0eb35d6a10e865cacd67bc5c11349c8d3abb51ded541b09a1b7314de0b4"}
Jul 09 20:35:53.930662 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:35:53.930626    2562 scope.go:117] "RemoveContainer" containerID="1498df8e4c822cc8b90d9a7c9e569f21ac8ad8fd019c4cc2533b77ceaef4440a"
Jul 09 20:35:53.939396 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:35:53.939378    2562 scope.go:117] "RemoveContainer" containerID="1498df8e4c822cc8b90d9a7c9e569f21ac8ad8fd019c4cc2533b77ceaef4440a"
Jul 09 20:35:53.939627 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:35:53.939610    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1498df8e4c822cc8b90d9a7c9e569f21ac8ad8fd019c4cc2533b77ceaef4440a\": container with ID starting with 1498df8e4c822cc8b90d9a7c9e569f21ac8ad8fd019c4cc2533b77ceaef4440a not found: ID does not exist" containerID="1498df8e4c822cc8b90d9a7c9e569f21ac8ad8fd019c4cc2533b77ceaef4440a"
Jul 09 20:35:53.939677 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:35:53.939637    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1498df8e4c822cc8b90d9a7c9e569f21ac8ad8fd019c4cc2533b77ceaef4440a"} err="failed to get container status \"1498df8e4c822cc8b90d9a7c9e569f21ac8ad8fd019c4cc2533b77ceaef4440a\": rpc error: code = NotFound desc = could not find container \"1498df8e4c822cc8b90d9a7c9e569f21ac8ad8fd019c4cc2533b77ceaef4440a\": container with ID starting with 1498df8e4c822cc8b90d9a7c9e569f21ac8ad8fd019c4cc2533b77ceaef4440a not found: ID does not exist"
Jul 09 20:35:53.952391 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:35:53.952370    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-92a49739-55c56bdd56-d4kp2"]
Jul 09 20:35:53.961833 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:35:53.961813    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-92a49739-55c56bdd56-d4kp2"]
Jul 09 20:35:54.241780 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:35:54.241758    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="096b5d9d-99d2-41d0-b766-8be17a07e84c" path="/var/lib/kubelet/pods/096b5d9d-99d2-41d0-b766-8be17a07e84c/volumes"
Jul 09 20:36:46.817978 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:36:46.817944    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-92a49739-openshift-default-544dd4597f-92wsm"]
Jul 09 20:36:46.818374 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:36:46.818204    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-92a49739-openshift-default-544dd4597f-92wsm" podUID="f1044447-019a-4db2-8503-5ad39e127582" containerName="istio-proxy" containerID="cri-o://f85e0f5a4c2d5a79fa4e71f4f79d61809174ac80cf137a5802a2ed2b040ef875" gracePeriod=30
Jul 09 20:36:52.071833 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:36:52.071809    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-92a49739-openshift-default-544dd4597f-92wsm"
Jul 09 20:36:52.169132 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:36:52.169074    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1044447-019a-4db2-8503-5ad39e127582-istio-podinfo\") pod \"f1044447-019a-4db2-8503-5ad39e127582\" (UID: \"f1044447-019a-4db2-8503-5ad39e127582\") "
Jul 09 20:36:52.169132 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:36:52.169109    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1044447-019a-4db2-8503-5ad39e127582-istio-envoy\") pod \"f1044447-019a-4db2-8503-5ad39e127582\" (UID: \"f1044447-019a-4db2-8503-5ad39e127582\") "
Jul 09 20:36:52.169279 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:36:52.169150    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1044447-019a-4db2-8503-5ad39e127582-istiod-ca-cert\") pod \"f1044447-019a-4db2-8503-5ad39e127582\" (UID: \"f1044447-019a-4db2-8503-5ad39e127582\") "
Jul 09 20:36:52.169279 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:36:52.169180    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1044447-019a-4db2-8503-5ad39e127582-istio-data\") pod \"f1044447-019a-4db2-8503-5ad39e127582\" (UID: \"f1044447-019a-4db2-8503-5ad39e127582\") "
Jul 09 20:36:52.169279 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:36:52.169250    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1044447-019a-4db2-8503-5ad39e127582-workload-certs\") pod \"f1044447-019a-4db2-8503-5ad39e127582\" (UID: \"f1044447-019a-4db2-8503-5ad39e127582\") "
Jul 09 20:36:52.169477 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:36:52.169279    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1044447-019a-4db2-8503-5ad39e127582-credential-socket\") pod \"f1044447-019a-4db2-8503-5ad39e127582\" (UID: \"f1044447-019a-4db2-8503-5ad39e127582\") "
Jul 09 20:36:52.169477 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:36:52.169330    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1044447-019a-4db2-8503-5ad39e127582-workload-socket\") pod \"f1044447-019a-4db2-8503-5ad39e127582\" (UID: \"f1044447-019a-4db2-8503-5ad39e127582\") "
Jul 09 20:36:52.169477 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:36:52.169355    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1044447-019a-4db2-8503-5ad39e127582-istio-token\") pod \"f1044447-019a-4db2-8503-5ad39e127582\" (UID: \"f1044447-019a-4db2-8503-5ad39e127582\") "
Jul 09 20:36:52.169477 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:36:52.169400    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dr9rx\" (UniqueName: \"kubernetes.io/projected/f1044447-019a-4db2-8503-5ad39e127582-kube-api-access-dr9rx\") pod \"f1044447-019a-4db2-8503-5ad39e127582\" (UID: \"f1044447-019a-4db2-8503-5ad39e127582\") "
Jul 09 20:36:52.169686 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:36:52.169636    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1044447-019a-4db2-8503-5ad39e127582-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f1044447-019a-4db2-8503-5ad39e127582" (UID: "f1044447-019a-4db2-8503-5ad39e127582"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 20:36:52.169742 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:36:52.169673    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1044447-019a-4db2-8503-5ad39e127582-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f1044447-019a-4db2-8503-5ad39e127582" (UID: "f1044447-019a-4db2-8503-5ad39e127582"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 20:36:52.169742 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:36:52.169691    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1044447-019a-4db2-8503-5ad39e127582-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f1044447-019a-4db2-8503-5ad39e127582" (UID: "f1044447-019a-4db2-8503-5ad39e127582"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 20:36:52.169742 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:36:52.169695    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1044447-019a-4db2-8503-5ad39e127582-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f1044447-019a-4db2-8503-5ad39e127582" (UID: "f1044447-019a-4db2-8503-5ad39e127582"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 20:36:52.170121 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:36:52.170088    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1044447-019a-4db2-8503-5ad39e127582-istio-data" (OuterVolumeSpecName: "istio-data") pod "f1044447-019a-4db2-8503-5ad39e127582" (UID: "f1044447-019a-4db2-8503-5ad39e127582"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 20:36:52.171213 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:36:52.171190    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f1044447-019a-4db2-8503-5ad39e127582-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f1044447-019a-4db2-8503-5ad39e127582" (UID: "f1044447-019a-4db2-8503-5ad39e127582"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 09 20:36:52.171563 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:36:52.171544    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1044447-019a-4db2-8503-5ad39e127582-kube-api-access-dr9rx" (OuterVolumeSpecName: "kube-api-access-dr9rx") pod "f1044447-019a-4db2-8503-5ad39e127582" (UID: "f1044447-019a-4db2-8503-5ad39e127582"). InnerVolumeSpecName "kube-api-access-dr9rx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 20:36:52.171645 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:36:52.171588    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1044447-019a-4db2-8503-5ad39e127582-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f1044447-019a-4db2-8503-5ad39e127582" (UID: "f1044447-019a-4db2-8503-5ad39e127582"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 20:36:52.171645 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:36:52.171588    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1044447-019a-4db2-8503-5ad39e127582-istio-token" (OuterVolumeSpecName: "istio-token") pod "f1044447-019a-4db2-8503-5ad39e127582" (UID: "f1044447-019a-4db2-8503-5ad39e127582"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 20:36:52.184158 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:36:52.184137    2562 generic.go:356] "Generic (PLEG): container finished" podID="f1044447-019a-4db2-8503-5ad39e127582" containerID="f85e0f5a4c2d5a79fa4e71f4f79d61809174ac80cf137a5802a2ed2b040ef875" exitCode=0
Jul 09 20:36:52.184239 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:36:52.184175    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-92a49739-openshift-default-544dd4597f-92wsm" event={"ID":"f1044447-019a-4db2-8503-5ad39e127582","Type":"ContainerDied","Data":"f85e0f5a4c2d5a79fa4e71f4f79d61809174ac80cf137a5802a2ed2b040ef875"}
Jul 09 20:36:52.184239 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:36:52.184227    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-92a49739-openshift-default-544dd4597f-92wsm" event={"ID":"f1044447-019a-4db2-8503-5ad39e127582","Type":"ContainerDied","Data":"94444cb665182aa46dd17e648d8742ddbe6a613d950c5f9063ccc98d9b716fef"}
Jul 09 20:36:52.184394 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:36:52.184252    2562 scope.go:117] "RemoveContainer" containerID="f85e0f5a4c2d5a79fa4e71f4f79d61809174ac80cf137a5802a2ed2b040ef875"
Jul 09 20:36:52.184394 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:36:52.184188    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-92a49739-openshift-default-544dd4597f-92wsm"
Jul 09 20:36:52.192797 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:36:52.192781    2562 scope.go:117] "RemoveContainer" containerID="a0a3e49fad127d7dc983b521185224d1c568b8eccc326b64ab862d05737b5e06"
Jul 09 20:36:52.200697 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:36:52.200681    2562 scope.go:117] "RemoveContainer" containerID="f85e0f5a4c2d5a79fa4e71f4f79d61809174ac80cf137a5802a2ed2b040ef875"
Jul 09 20:36:52.200968 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:36:52.200949    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f85e0f5a4c2d5a79fa4e71f4f79d61809174ac80cf137a5802a2ed2b040ef875\": container with ID starting with f85e0f5a4c2d5a79fa4e71f4f79d61809174ac80cf137a5802a2ed2b040ef875 not found: ID does not exist" containerID="f85e0f5a4c2d5a79fa4e71f4f79d61809174ac80cf137a5802a2ed2b040ef875"
Jul 09 20:36:52.201031 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:36:52.200975    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f85e0f5a4c2d5a79fa4e71f4f79d61809174ac80cf137a5802a2ed2b040ef875"} err="failed to get container status \"f85e0f5a4c2d5a79fa4e71f4f79d61809174ac80cf137a5802a2ed2b040ef875\": rpc error: code = NotFound desc = could not find container \"f85e0f5a4c2d5a79fa4e71f4f79d61809174ac80cf137a5802a2ed2b040ef875\": container with ID starting with f85e0f5a4c2d5a79fa4e71f4f79d61809174ac80cf137a5802a2ed2b040ef875 not found: ID does not exist"
Jul 09 20:36:52.201031 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:36:52.200992    2562 scope.go:117] "RemoveContainer" containerID="a0a3e49fad127d7dc983b521185224d1c568b8eccc326b64ab862d05737b5e06"
Jul 09 20:36:52.201250 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:36:52.201228    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0a3e49fad127d7dc983b521185224d1c568b8eccc326b64ab862d05737b5e06\": container with ID starting with a0a3e49fad127d7dc983b521185224d1c568b8eccc326b64ab862d05737b5e06 not found: ID does not exist" containerID="a0a3e49fad127d7dc983b521185224d1c568b8eccc326b64ab862d05737b5e06"
Jul 09 20:36:52.201333 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:36:52.201257    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0a3e49fad127d7dc983b521185224d1c568b8eccc326b64ab862d05737b5e06"} err="failed to get container status \"a0a3e49fad127d7dc983b521185224d1c568b8eccc326b64ab862d05737b5e06\": rpc error: code = NotFound desc = could not find container \"a0a3e49fad127d7dc983b521185224d1c568b8eccc326b64ab862d05737b5e06\": container with ID starting with a0a3e49fad127d7dc983b521185224d1c568b8eccc326b64ab862d05737b5e06 not found: ID does not exist"
Jul 09 20:36:52.206915 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:36:52.206896    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-92a49739-openshift-default-544dd4597f-92wsm"]
Jul 09 20:36:52.210857 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:36:52.210836    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-92a49739-openshift-default-544dd4597f-92wsm"]
Jul 09 20:36:52.247535 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:36:52.242878    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f1044447-019a-4db2-8503-5ad39e127582" path="/var/lib/kubelet/pods/f1044447-019a-4db2-8503-5ad39e127582/volumes"
Jul 09 20:36:52.270126 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:36:52.270105    2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1044447-019a-4db2-8503-5ad39e127582-istio-envoy\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:36:52.270126 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:36:52.270123    2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1044447-019a-4db2-8503-5ad39e127582-istiod-ca-cert\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:36:52.270228 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:36:52.270131    2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1044447-019a-4db2-8503-5ad39e127582-istio-data\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:36:52.270228 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:36:52.270139    2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1044447-019a-4db2-8503-5ad39e127582-workload-certs\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:36:52.270228 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:36:52.270147    2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1044447-019a-4db2-8503-5ad39e127582-credential-socket\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:36:52.270228 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:36:52.270156    2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1044447-019a-4db2-8503-5ad39e127582-workload-socket\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:36:52.270228 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:36:52.270165    2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1044447-019a-4db2-8503-5ad39e127582-istio-token\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:36:52.270228 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:36:52.270173    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dr9rx\" (UniqueName: \"kubernetes.io/projected/f1044447-019a-4db2-8503-5ad39e127582-kube-api-access-dr9rx\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:36:52.270228 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:36:52.270181    2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1044447-019a-4db2-8503-5ad39e127582-istio-podinfo\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:37:42.784182 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:42.784153    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-c0edf2-openshift-default-54f964d9f7-c85nc"]
Jul 09 20:37:42.784650 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:42.784565    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1044447-019a-4db2-8503-5ad39e127582" containerName="istio-proxy"
Jul 09 20:37:42.784650 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:42.784578    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1044447-019a-4db2-8503-5ad39e127582" containerName="istio-proxy"
Jul 09 20:37:42.784650 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:42.784590    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1044447-019a-4db2-8503-5ad39e127582" containerName="istio-proxy"
Jul 09 20:37:42.784650 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:42.784596    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1044447-019a-4db2-8503-5ad39e127582" containerName="istio-proxy"
Jul 09 20:37:42.784650 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:42.784601    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="096b5d9d-99d2-41d0-b766-8be17a07e84c" containerName="maas-api"
Jul 09 20:37:42.784650 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:42.784607    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="096b5d9d-99d2-41d0-b766-8be17a07e84c" containerName="maas-api"
Jul 09 20:37:42.784861 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:42.784678    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1044447-019a-4db2-8503-5ad39e127582" containerName="istio-proxy"
Jul 09 20:37:42.784861 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:42.784689    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="096b5d9d-99d2-41d0-b766-8be17a07e84c" containerName="maas-api"
Jul 09 20:37:42.784861 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:42.784702    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1044447-019a-4db2-8503-5ad39e127582" containerName="istio-proxy"
Jul 09 20:37:42.787648 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:42.787632    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c0edf2-openshift-default-54f964d9f7-c85nc"
Jul 09 20:37:42.790548 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:42.790529    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-c0edf2-openshift-default-dockercfg-n8sss\""
Jul 09 20:37:42.800020 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:42.799997    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-c0edf2-openshift-default-54f964d9f7-c85nc"]
Jul 09 20:37:42.833655 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:42.833635    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/799ba2bb-8a49-46f2-b985-1e4a1f06e52b-istio-token\") pod \"e2e-inf-a-c0edf2-openshift-default-54f964d9f7-c85nc\" (UID: \"799ba2bb-8a49-46f2-b985-1e4a1f06e52b\") " pod="openshift-ingress/e2e-inf-a-c0edf2-openshift-default-54f964d9f7-c85nc"
Jul 09 20:37:42.833751 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:42.833664    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/799ba2bb-8a49-46f2-b985-1e4a1f06e52b-istiod-ca-cert\") pod \"e2e-inf-a-c0edf2-openshift-default-54f964d9f7-c85nc\" (UID: \"799ba2bb-8a49-46f2-b985-1e4a1f06e52b\") " pod="openshift-ingress/e2e-inf-a-c0edf2-openshift-default-54f964d9f7-c85nc"
Jul 09 20:37:42.833751 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:42.833696    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/799ba2bb-8a49-46f2-b985-1e4a1f06e52b-workload-certs\") pod \"e2e-inf-a-c0edf2-openshift-default-54f964d9f7-c85nc\" (UID: \"799ba2bb-8a49-46f2-b985-1e4a1f06e52b\") " pod="openshift-ingress/e2e-inf-a-c0edf2-openshift-default-54f964d9f7-c85nc"
Jul 09 20:37:42.833824 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:42.833755    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/799ba2bb-8a49-46f2-b985-1e4a1f06e52b-workload-socket\") pod \"e2e-inf-a-c0edf2-openshift-default-54f964d9f7-c85nc\" (UID: \"799ba2bb-8a49-46f2-b985-1e4a1f06e52b\") " pod="openshift-ingress/e2e-inf-a-c0edf2-openshift-default-54f964d9f7-c85nc"
Jul 09 20:37:42.833824 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:42.833799    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/799ba2bb-8a49-46f2-b985-1e4a1f06e52b-istio-podinfo\") pod \"e2e-inf-a-c0edf2-openshift-default-54f964d9f7-c85nc\" (UID: \"799ba2bb-8a49-46f2-b985-1e4a1f06e52b\") " pod="openshift-ingress/e2e-inf-a-c0edf2-openshift-default-54f964d9f7-c85nc"
Jul 09 20:37:42.833824 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:42.833822    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/799ba2bb-8a49-46f2-b985-1e4a1f06e52b-istio-data\") pod \"e2e-inf-a-c0edf2-openshift-default-54f964d9f7-c85nc\" (UID: \"799ba2bb-8a49-46f2-b985-1e4a1f06e52b\") " pod="openshift-ingress/e2e-inf-a-c0edf2-openshift-default-54f964d9f7-c85nc"
Jul 09 20:37:42.833939 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:42.833843    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/799ba2bb-8a49-46f2-b985-1e4a1f06e52b-istio-envoy\") pod \"e2e-inf-a-c0edf2-openshift-default-54f964d9f7-c85nc\" (UID: \"799ba2bb-8a49-46f2-b985-1e4a1f06e52b\") " pod="openshift-ingress/e2e-inf-a-c0edf2-openshift-default-54f964d9f7-c85nc"
Jul 09 20:37:42.833939 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:42.833873    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r56bp\" (UniqueName: \"kubernetes.io/projected/799ba2bb-8a49-46f2-b985-1e4a1f06e52b-kube-api-access-r56bp\") pod \"e2e-inf-a-c0edf2-openshift-default-54f964d9f7-c85nc\" (UID: \"799ba2bb-8a49-46f2-b985-1e4a1f06e52b\") " pod="openshift-ingress/e2e-inf-a-c0edf2-openshift-default-54f964d9f7-c85nc"
Jul 09 20:37:42.833939 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:42.833896    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/799ba2bb-8a49-46f2-b985-1e4a1f06e52b-credential-socket\") pod \"e2e-inf-a-c0edf2-openshift-default-54f964d9f7-c85nc\" (UID: \"799ba2bb-8a49-46f2-b985-1e4a1f06e52b\") " pod="openshift-ingress/e2e-inf-a-c0edf2-openshift-default-54f964d9f7-c85nc"
Jul 09 20:37:42.935013 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:42.934979    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/799ba2bb-8a49-46f2-b985-1e4a1f06e52b-workload-certs\") pod \"e2e-inf-a-c0edf2-openshift-default-54f964d9f7-c85nc\" (UID: \"799ba2bb-8a49-46f2-b985-1e4a1f06e52b\") " pod="openshift-ingress/e2e-inf-a-c0edf2-openshift-default-54f964d9f7-c85nc"
Jul 09 20:37:42.935157 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:42.935020    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/799ba2bb-8a49-46f2-b985-1e4a1f06e52b-workload-socket\") pod \"e2e-inf-a-c0edf2-openshift-default-54f964d9f7-c85nc\" (UID: \"799ba2bb-8a49-46f2-b985-1e4a1f06e52b\") " pod="openshift-ingress/e2e-inf-a-c0edf2-openshift-default-54f964d9f7-c85nc"
Jul 09 20:37:42.935157 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:42.935066    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/799ba2bb-8a49-46f2-b985-1e4a1f06e52b-istio-podinfo\") pod \"e2e-inf-a-c0edf2-openshift-default-54f964d9f7-c85nc\" (UID: \"799ba2bb-8a49-46f2-b985-1e4a1f06e52b\") " pod="openshift-ingress/e2e-inf-a-c0edf2-openshift-default-54f964d9f7-c85nc"
Jul 09 20:37:42.935157 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:42.935096    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/799ba2bb-8a49-46f2-b985-1e4a1f06e52b-istio-data\") pod \"e2e-inf-a-c0edf2-openshift-default-54f964d9f7-c85nc\" (UID: \"799ba2bb-8a49-46f2-b985-1e4a1f06e52b\") " pod="openshift-ingress/e2e-inf-a-c0edf2-openshift-default-54f964d9f7-c85nc"
Jul 09 20:37:42.935157 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:42.935127    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/799ba2bb-8a49-46f2-b985-1e4a1f06e52b-istio-envoy\") pod \"e2e-inf-a-c0edf2-openshift-default-54f964d9f7-c85nc\" (UID: \"799ba2bb-8a49-46f2-b985-1e4a1f06e52b\") " pod="openshift-ingress/e2e-inf-a-c0edf2-openshift-default-54f964d9f7-c85nc"
Jul 09 20:37:42.935157 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:42.935152    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r56bp\" (UniqueName: \"kubernetes.io/projected/799ba2bb-8a49-46f2-b985-1e4a1f06e52b-kube-api-access-r56bp\") pod \"e2e-inf-a-c0edf2-openshift-default-54f964d9f7-c85nc\" (UID: \"799ba2bb-8a49-46f2-b985-1e4a1f06e52b\") " pod="openshift-ingress/e2e-inf-a-c0edf2-openshift-default-54f964d9f7-c85nc"
Jul 09 20:37:42.935445 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:42.935183    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/799ba2bb-8a49-46f2-b985-1e4a1f06e52b-credential-socket\") pod \"e2e-inf-a-c0edf2-openshift-default-54f964d9f7-c85nc\" (UID: \"799ba2bb-8a49-46f2-b985-1e4a1f06e52b\") " pod="openshift-ingress/e2e-inf-a-c0edf2-openshift-default-54f964d9f7-c85nc"
Jul 09 20:37:42.935445 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:42.935246    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/799ba2bb-8a49-46f2-b985-1e4a1f06e52b-istio-token\") pod \"e2e-inf-a-c0edf2-openshift-default-54f964d9f7-c85nc\" (UID: \"799ba2bb-8a49-46f2-b985-1e4a1f06e52b\") " pod="openshift-ingress/e2e-inf-a-c0edf2-openshift-default-54f964d9f7-c85nc"
Jul 09 20:37:42.935445 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:42.935380    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/799ba2bb-8a49-46f2-b985-1e4a1f06e52b-istiod-ca-cert\") pod \"e2e-inf-a-c0edf2-openshift-default-54f964d9f7-c85nc\" (UID: \"799ba2bb-8a49-46f2-b985-1e4a1f06e52b\") " pod="openshift-ingress/e2e-inf-a-c0edf2-openshift-default-54f964d9f7-c85nc"
Jul 09 20:37:42.935445 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:42.935409    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/799ba2bb-8a49-46f2-b985-1e4a1f06e52b-workload-certs\") pod \"e2e-inf-a-c0edf2-openshift-default-54f964d9f7-c85nc\" (UID: \"799ba2bb-8a49-46f2-b985-1e4a1f06e52b\") " pod="openshift-ingress/e2e-inf-a-c0edf2-openshift-default-54f964d9f7-c85nc"
Jul 09 20:37:42.935677 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:42.935461    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/799ba2bb-8a49-46f2-b985-1e4a1f06e52b-workload-socket\") pod \"e2e-inf-a-c0edf2-openshift-default-54f964d9f7-c85nc\" (UID: \"799ba2bb-8a49-46f2-b985-1e4a1f06e52b\") " pod="openshift-ingress/e2e-inf-a-c0edf2-openshift-default-54f964d9f7-c85nc"
Jul 09 20:37:42.935677 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:42.935629    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/799ba2bb-8a49-46f2-b985-1e4a1f06e52b-credential-socket\") pod \"e2e-inf-a-c0edf2-openshift-default-54f964d9f7-c85nc\" (UID: \"799ba2bb-8a49-46f2-b985-1e4a1f06e52b\") " pod="openshift-ingress/e2e-inf-a-c0edf2-openshift-default-54f964d9f7-c85nc"
Jul 09 20:37:42.935849 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:42.935828    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/799ba2bb-8a49-46f2-b985-1e4a1f06e52b-istio-data\") pod \"e2e-inf-a-c0edf2-openshift-default-54f964d9f7-c85nc\" (UID: \"799ba2bb-8a49-46f2-b985-1e4a1f06e52b\") " pod="openshift-ingress/e2e-inf-a-c0edf2-openshift-default-54f964d9f7-c85nc"
Jul 09 20:37:42.936374 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:42.936349    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/799ba2bb-8a49-46f2-b985-1e4a1f06e52b-istiod-ca-cert\") pod \"e2e-inf-a-c0edf2-openshift-default-54f964d9f7-c85nc\" (UID: \"799ba2bb-8a49-46f2-b985-1e4a1f06e52b\") " pod="openshift-ingress/e2e-inf-a-c0edf2-openshift-default-54f964d9f7-c85nc"
Jul 09 20:37:42.937437 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:42.937415    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/799ba2bb-8a49-46f2-b985-1e4a1f06e52b-istio-envoy\") pod \"e2e-inf-a-c0edf2-openshift-default-54f964d9f7-c85nc\" (UID: \"799ba2bb-8a49-46f2-b985-1e4a1f06e52b\") " pod="openshift-ingress/e2e-inf-a-c0edf2-openshift-default-54f964d9f7-c85nc"
Jul 09 20:37:42.937655 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:42.937636    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/799ba2bb-8a49-46f2-b985-1e4a1f06e52b-istio-podinfo\") pod \"e2e-inf-a-c0edf2-openshift-default-54f964d9f7-c85nc\" (UID: \"799ba2bb-8a49-46f2-b985-1e4a1f06e52b\") " pod="openshift-ingress/e2e-inf-a-c0edf2-openshift-default-54f964d9f7-c85nc"
Jul 09 20:37:42.942973 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:42.942949    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/799ba2bb-8a49-46f2-b985-1e4a1f06e52b-istio-token\") pod \"e2e-inf-a-c0edf2-openshift-default-54f964d9f7-c85nc\" (UID: \"799ba2bb-8a49-46f2-b985-1e4a1f06e52b\") " pod="openshift-ingress/e2e-inf-a-c0edf2-openshift-default-54f964d9f7-c85nc"
Jul 09 20:37:42.943206 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:42.943186    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r56bp\" (UniqueName: \"kubernetes.io/projected/799ba2bb-8a49-46f2-b985-1e4a1f06e52b-kube-api-access-r56bp\") pod \"e2e-inf-a-c0edf2-openshift-default-54f964d9f7-c85nc\" (UID: \"799ba2bb-8a49-46f2-b985-1e4a1f06e52b\") " pod="openshift-ingress/e2e-inf-a-c0edf2-openshift-default-54f964d9f7-c85nc"
Jul 09 20:37:43.097536 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:43.097484    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c0edf2-openshift-default-54f964d9f7-c85nc"
Jul 09 20:37:43.230431 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:43.230405    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-c0edf2-openshift-default-54f964d9f7-c85nc"]
Jul 09 20:37:43.232230 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:37:43.232201    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod799ba2bb_8a49_46f2_b985_1e4a1f06e52b.slice/crio-9f6ff43d6537e7f30d91e408302101356a457b7ee825cefab4e5da6f5ee10e23 WatchSource:0}: Error finding container 9f6ff43d6537e7f30d91e408302101356a457b7ee825cefab4e5da6f5ee10e23: Status 404 returned error can't find the container with id 9f6ff43d6537e7f30d91e408302101356a457b7ee825cefab4e5da6f5ee10e23
Jul 09 20:37:43.234633 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:43.234592    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 09 20:37:43.234724 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:43.234677    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 09 20:37:43.234764 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:43.234728    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 09 20:37:43.394469 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:43.394390    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c0edf2-openshift-default-54f964d9f7-c85nc" event={"ID":"799ba2bb-8a49-46f2-b985-1e4a1f06e52b","Type":"ContainerStarted","Data":"42a2699848e09d5eb1dc6674c2a7073d2a96f2325fbd243c3269e4105d3a408d"}
Jul 09 20:37:43.394469 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:43.394445    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c0edf2-openshift-default-54f964d9f7-c85nc" event={"ID":"799ba2bb-8a49-46f2-b985-1e4a1f06e52b","Type":"ContainerStarted","Data":"9f6ff43d6537e7f30d91e408302101356a457b7ee825cefab4e5da6f5ee10e23"}
Jul 09 20:37:43.413199 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:43.413142    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-c0edf2-openshift-default-54f964d9f7-c85nc" podStartSLOduration=1.413128561 podStartE2EDuration="1.413128561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 20:37:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 20:37:43.41229624 +0000 UTC m=+2053.829966047" watchObservedRunningTime="2026-07-09 20:37:43.413128561 +0000 UTC m=+2053.830798449"
Jul 09 20:37:44.097862 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:44.097830    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-c0edf2-openshift-default-54f964d9f7-c85nc"
Jul 09 20:37:44.102600 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:44.102572    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-c0edf2-openshift-default-54f964d9f7-c85nc"
Jul 09 20:37:44.398949 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:44.398887    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-c0edf2-openshift-default-54f964d9f7-c85nc"
Jul 09 20:37:44.399687 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:44.399667    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-c0edf2-openshift-default-54f964d9f7-c85nc"
Jul 09 20:37:45.432280 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:45.432246    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c0edf2-b584777f6-t8j25"]
Jul 09 20:37:45.436261 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:45.436240    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c0edf2-b584777f6-t8j25"
Jul 09 20:37:45.439107 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:45.439087    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-c0edf2-serving-cert\""
Jul 09 20:37:45.443686 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:45.443662    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c0edf2-b584777f6-t8j25"]
Jul 09 20:37:45.456535 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:45.456512    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2crs\" (UniqueName: \"kubernetes.io/projected/a6f3daa6-460b-4b55-87df-7a9eb7436c9f-kube-api-access-z2crs\") pod \"maas-api-e2e-inf-a-c0edf2-b584777f6-t8j25\" (UID: \"a6f3daa6-460b-4b55-87df-7a9eb7436c9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c0edf2-b584777f6-t8j25"
Jul 09 20:37:45.456717 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:45.456620    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6f3daa6-460b-4b55-87df-7a9eb7436c9f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c0edf2-b584777f6-t8j25\" (UID: \"a6f3daa6-460b-4b55-87df-7a9eb7436c9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c0edf2-b584777f6-t8j25"
Jul 09 20:37:45.456787 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:45.456727    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6f3daa6-460b-4b55-87df-7a9eb7436c9f-maas-api-tls\") pod \"maas-api-e2e-inf-a-c0edf2-b584777f6-t8j25\" (UID: \"a6f3daa6-460b-4b55-87df-7a9eb7436c9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c0edf2-b584777f6-t8j25"
Jul 09 20:37:45.558322 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:45.558271    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2crs\" (UniqueName: \"kubernetes.io/projected/a6f3daa6-460b-4b55-87df-7a9eb7436c9f-kube-api-access-z2crs\") pod \"maas-api-e2e-inf-a-c0edf2-b584777f6-t8j25\" (UID: \"a6f3daa6-460b-4b55-87df-7a9eb7436c9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c0edf2-b584777f6-t8j25"
Jul 09 20:37:45.558497 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:45.558333    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6f3daa6-460b-4b55-87df-7a9eb7436c9f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c0edf2-b584777f6-t8j25\" (UID: \"a6f3daa6-460b-4b55-87df-7a9eb7436c9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c0edf2-b584777f6-t8j25"
Jul 09 20:37:45.558497 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:45.558376    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6f3daa6-460b-4b55-87df-7a9eb7436c9f-maas-api-tls\") pod \"maas-api-e2e-inf-a-c0edf2-b584777f6-t8j25\" (UID: \"a6f3daa6-460b-4b55-87df-7a9eb7436c9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c0edf2-b584777f6-t8j25"
Jul 09 20:37:45.558497 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:37:45.558495    2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-c0edf2-serving-cert: secret "maas-api-e2e-inf-a-c0edf2-serving-cert" not found
Jul 09 20:37:45.558672 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:37:45.558551    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6f3daa6-460b-4b55-87df-7a9eb7436c9f-maas-api-tls podName:a6f3daa6-460b-4b55-87df-7a9eb7436c9f nodeName:}" failed. No retries permitted until 2026-07-09 20:37:46.058534328 +0000 UTC m=+2056.476204115 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a6f3daa6-460b-4b55-87df-7a9eb7436c9f-maas-api-tls") pod "maas-api-e2e-inf-a-c0edf2-b584777f6-t8j25" (UID: "a6f3daa6-460b-4b55-87df-7a9eb7436c9f") : secret "maas-api-e2e-inf-a-c0edf2-serving-cert" not found
Jul 09 20:37:45.559072 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:45.559049    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6f3daa6-460b-4b55-87df-7a9eb7436c9f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c0edf2-b584777f6-t8j25\" (UID: \"a6f3daa6-460b-4b55-87df-7a9eb7436c9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c0edf2-b584777f6-t8j25"
Jul 09 20:37:45.570809 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:45.570781    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2crs\" (UniqueName: \"kubernetes.io/projected/a6f3daa6-460b-4b55-87df-7a9eb7436c9f-kube-api-access-z2crs\") pod \"maas-api-e2e-inf-a-c0edf2-b584777f6-t8j25\" (UID: \"a6f3daa6-460b-4b55-87df-7a9eb7436c9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c0edf2-b584777f6-t8j25"
Jul 09 20:37:46.063158 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:46.063131    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6f3daa6-460b-4b55-87df-7a9eb7436c9f-maas-api-tls\") pod \"maas-api-e2e-inf-a-c0edf2-b584777f6-t8j25\" (UID: \"a6f3daa6-460b-4b55-87df-7a9eb7436c9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c0edf2-b584777f6-t8j25"
Jul 09 20:37:46.065412 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:46.065391    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6f3daa6-460b-4b55-87df-7a9eb7436c9f-maas-api-tls\") pod \"maas-api-e2e-inf-a-c0edf2-b584777f6-t8j25\" (UID: \"a6f3daa6-460b-4b55-87df-7a9eb7436c9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c0edf2-b584777f6-t8j25"
Jul 09 20:37:46.350975 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:46.350892    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c0edf2-b584777f6-t8j25"
Jul 09 20:37:46.484926 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:46.484903    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c0edf2-b584777f6-t8j25"]
Jul 09 20:37:46.486630 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:37:46.486600    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6f3daa6_460b_4b55_87df_7a9eb7436c9f.slice/crio-0c9efb2e655eb1d3e57f15f9db853069604eb26d3d8eaf83a5d5a65faf814d30 WatchSource:0}: Error finding container 0c9efb2e655eb1d3e57f15f9db853069604eb26d3d8eaf83a5d5a65faf814d30: Status 404 returned error can't find the container with id 0c9efb2e655eb1d3e57f15f9db853069604eb26d3d8eaf83a5d5a65faf814d30
Jul 09 20:37:47.413620 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:47.413552    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c0edf2-b584777f6-t8j25" event={"ID":"a6f3daa6-460b-4b55-87df-7a9eb7436c9f","Type":"ContainerStarted","Data":"ac54d8a6829e18f10cdf658285fd897ee065b4abf782593c4586f5a73ae2ed26"}
Jul 09 20:37:47.413620 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:47.413596    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c0edf2-b584777f6-t8j25" event={"ID":"a6f3daa6-460b-4b55-87df-7a9eb7436c9f","Type":"ContainerStarted","Data":"0c9efb2e655eb1d3e57f15f9db853069604eb26d3d8eaf83a5d5a65faf814d30"}
Jul 09 20:37:47.413841 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:47.413625    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c0edf2-b584777f6-t8j25"
Jul 09 20:37:47.436048 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:47.436008    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c0edf2-b584777f6-t8j25" podStartSLOduration=1.923664147 podStartE2EDuration="2.435992713s" totalImagesPullingTime="512.328566ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 20:37:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 20:37:47.434345703 +0000 UTC m=+2057.852015512" watchObservedRunningTime="2026-07-09 20:37:47.435992713 +0000 UTC m=+2057.853662519"
Jul 09 20:37:50.462953 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:50.462912    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-00edb0-openshift-default-57cdc9785-6zxgq"]
Jul 09 20:37:50.466936 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:50.466913    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-00edb0-openshift-default-57cdc9785-6zxgq"
Jul 09 20:37:50.469745 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:50.469726    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-00edb0-openshift-default-dockercfg-c9d4c\""
Jul 09 20:37:50.475986 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:50.475965    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-00edb0-openshift-default-57cdc9785-6zxgq"]
Jul 09 20:37:50.496472 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:50.496452    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d80fc5f7-0dfb-49e6-b168-5da6b11a12f4-workload-socket\") pod \"e2e-inf-b-00edb0-openshift-default-57cdc9785-6zxgq\" (UID: \"d80fc5f7-0dfb-49e6-b168-5da6b11a12f4\") " pod="openshift-ingress/e2e-inf-b-00edb0-openshift-default-57cdc9785-6zxgq"
Jul 09 20:37:50.496567 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:50.496492    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d80fc5f7-0dfb-49e6-b168-5da6b11a12f4-istio-envoy\") pod \"e2e-inf-b-00edb0-openshift-default-57cdc9785-6zxgq\" (UID: \"d80fc5f7-0dfb-49e6-b168-5da6b11a12f4\") " pod="openshift-ingress/e2e-inf-b-00edb0-openshift-default-57cdc9785-6zxgq"
Jul 09 20:37:50.496567 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:50.496512    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d80fc5f7-0dfb-49e6-b168-5da6b11a12f4-workload-certs\") pod \"e2e-inf-b-00edb0-openshift-default-57cdc9785-6zxgq\" (UID: \"d80fc5f7-0dfb-49e6-b168-5da6b11a12f4\") " pod="openshift-ingress/e2e-inf-b-00edb0-openshift-default-57cdc9785-6zxgq"
Jul 09 20:37:50.496567 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:50.496536    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d80fc5f7-0dfb-49e6-b168-5da6b11a12f4-credential-socket\") pod \"e2e-inf-b-00edb0-openshift-default-57cdc9785-6zxgq\" (UID: \"d80fc5f7-0dfb-49e6-b168-5da6b11a12f4\") " pod="openshift-ingress/e2e-inf-b-00edb0-openshift-default-57cdc9785-6zxgq"
Jul 09 20:37:50.496702 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:50.496588    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d80fc5f7-0dfb-49e6-b168-5da6b11a12f4-istiod-ca-cert\") pod \"e2e-inf-b-00edb0-openshift-default-57cdc9785-6zxgq\" (UID: \"d80fc5f7-0dfb-49e6-b168-5da6b11a12f4\") " pod="openshift-ingress/e2e-inf-b-00edb0-openshift-default-57cdc9785-6zxgq"
Jul 09 20:37:50.496752 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:50.496717    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d80fc5f7-0dfb-49e6-b168-5da6b11a12f4-istio-token\") pod \"e2e-inf-b-00edb0-openshift-default-57cdc9785-6zxgq\" (UID: \"d80fc5f7-0dfb-49e6-b168-5da6b11a12f4\") " pod="openshift-ingress/e2e-inf-b-00edb0-openshift-default-57cdc9785-6zxgq"
Jul 09 20:37:50.496806 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:50.496760    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8b59j\" (UniqueName: \"kubernetes.io/projected/d80fc5f7-0dfb-49e6-b168-5da6b11a12f4-kube-api-access-8b59j\") pod \"e2e-inf-b-00edb0-openshift-default-57cdc9785-6zxgq\" (UID: \"d80fc5f7-0dfb-49e6-b168-5da6b11a12f4\") " pod="openshift-ingress/e2e-inf-b-00edb0-openshift-default-57cdc9785-6zxgq"
Jul 09 20:37:50.496806 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:50.496789    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d80fc5f7-0dfb-49e6-b168-5da6b11a12f4-istio-data\") pod \"e2e-inf-b-00edb0-openshift-default-57cdc9785-6zxgq\" (UID: \"d80fc5f7-0dfb-49e6-b168-5da6b11a12f4\") " pod="openshift-ingress/e2e-inf-b-00edb0-openshift-default-57cdc9785-6zxgq"
Jul 09 20:37:50.496890 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:50.496831    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d80fc5f7-0dfb-49e6-b168-5da6b11a12f4-istio-podinfo\") pod \"e2e-inf-b-00edb0-openshift-default-57cdc9785-6zxgq\" (UID: \"d80fc5f7-0dfb-49e6-b168-5da6b11a12f4\") " pod="openshift-ingress/e2e-inf-b-00edb0-openshift-default-57cdc9785-6zxgq"
Jul 09 20:37:50.597883 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:50.597846    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d80fc5f7-0dfb-49e6-b168-5da6b11a12f4-istio-envoy\") pod \"e2e-inf-b-00edb0-openshift-default-57cdc9785-6zxgq\" (UID: \"d80fc5f7-0dfb-49e6-b168-5da6b11a12f4\") " pod="openshift-ingress/e2e-inf-b-00edb0-openshift-default-57cdc9785-6zxgq"
Jul 09 20:37:50.598060 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:50.597890    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d80fc5f7-0dfb-49e6-b168-5da6b11a12f4-workload-certs\") pod \"e2e-inf-b-00edb0-openshift-default-57cdc9785-6zxgq\" (UID: \"d80fc5f7-0dfb-49e6-b168-5da6b11a12f4\") " pod="openshift-ingress/e2e-inf-b-00edb0-openshift-default-57cdc9785-6zxgq"
Jul 09 20:37:50.598060 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:50.597915    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d80fc5f7-0dfb-49e6-b168-5da6b11a12f4-credential-socket\") pod \"e2e-inf-b-00edb0-openshift-default-57cdc9785-6zxgq\" (UID: \"d80fc5f7-0dfb-49e6-b168-5da6b11a12f4\") " pod="openshift-ingress/e2e-inf-b-00edb0-openshift-default-57cdc9785-6zxgq"
Jul 09 20:37:50.598060 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:50.597937    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d80fc5f7-0dfb-49e6-b168-5da6b11a12f4-istiod-ca-cert\") pod \"e2e-inf-b-00edb0-openshift-default-57cdc9785-6zxgq\" (UID: \"d80fc5f7-0dfb-49e6-b168-5da6b11a12f4\") " pod="openshift-ingress/e2e-inf-b-00edb0-openshift-default-57cdc9785-6zxgq"
Jul 09 20:37:50.598060 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:50.597995    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d80fc5f7-0dfb-49e6-b168-5da6b11a12f4-istio-token\") pod \"e2e-inf-b-00edb0-openshift-default-57cdc9785-6zxgq\" (UID: \"d80fc5f7-0dfb-49e6-b168-5da6b11a12f4\") " pod="openshift-ingress/e2e-inf-b-00edb0-openshift-default-57cdc9785-6zxgq"
Jul 09 20:37:50.598060 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:50.598025    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8b59j\" (UniqueName: \"kubernetes.io/projected/d80fc5f7-0dfb-49e6-b168-5da6b11a12f4-kube-api-access-8b59j\") pod \"e2e-inf-b-00edb0-openshift-default-57cdc9785-6zxgq\" (UID: \"d80fc5f7-0dfb-49e6-b168-5da6b11a12f4\") " pod="openshift-ingress/e2e-inf-b-00edb0-openshift-default-57cdc9785-6zxgq"
Jul 09 20:37:50.598060 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:50.598050    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d80fc5f7-0dfb-49e6-b168-5da6b11a12f4-istio-data\") pod \"e2e-inf-b-00edb0-openshift-default-57cdc9785-6zxgq\" (UID: \"d80fc5f7-0dfb-49e6-b168-5da6b11a12f4\") " pod="openshift-ingress/e2e-inf-b-00edb0-openshift-default-57cdc9785-6zxgq"
Jul 09 20:37:50.598413 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:50.598101    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d80fc5f7-0dfb-49e6-b168-5da6b11a12f4-istio-podinfo\") pod \"e2e-inf-b-00edb0-openshift-default-57cdc9785-6zxgq\" (UID: \"d80fc5f7-0dfb-49e6-b168-5da6b11a12f4\") " pod="openshift-ingress/e2e-inf-b-00edb0-openshift-default-57cdc9785-6zxgq"
Jul 09 20:37:50.598413 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:50.598172    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d80fc5f7-0dfb-49e6-b168-5da6b11a12f4-workload-socket\") pod \"e2e-inf-b-00edb0-openshift-default-57cdc9785-6zxgq\" (UID: \"d80fc5f7-0dfb-49e6-b168-5da6b11a12f4\") " pod="openshift-ingress/e2e-inf-b-00edb0-openshift-default-57cdc9785-6zxgq"
Jul 09 20:37:50.598413 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:50.598340    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d80fc5f7-0dfb-49e6-b168-5da6b11a12f4-workload-certs\") pod \"e2e-inf-b-00edb0-openshift-default-57cdc9785-6zxgq\" (UID: \"d80fc5f7-0dfb-49e6-b168-5da6b11a12f4\") " pod="openshift-ingress/e2e-inf-b-00edb0-openshift-default-57cdc9785-6zxgq"
Jul 09 20:37:50.598413 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:50.598372    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d80fc5f7-0dfb-49e6-b168-5da6b11a12f4-credential-socket\") pod \"e2e-inf-b-00edb0-openshift-default-57cdc9785-6zxgq\" (UID: \"d80fc5f7-0dfb-49e6-b168-5da6b11a12f4\") " pod="openshift-ingress/e2e-inf-b-00edb0-openshift-default-57cdc9785-6zxgq"
Jul 09 20:37:50.598816 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:50.598793    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d80fc5f7-0dfb-49e6-b168-5da6b11a12f4-istiod-ca-cert\") pod \"e2e-inf-b-00edb0-openshift-default-57cdc9785-6zxgq\" (UID: \"d80fc5f7-0dfb-49e6-b168-5da6b11a12f4\") " pod="openshift-ingress/e2e-inf-b-00edb0-openshift-default-57cdc9785-6zxgq"
Jul 09 20:37:50.598897 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:50.598797    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d80fc5f7-0dfb-49e6-b168-5da6b11a12f4-workload-socket\") pod \"e2e-inf-b-00edb0-openshift-default-57cdc9785-6zxgq\" (UID: \"d80fc5f7-0dfb-49e6-b168-5da6b11a12f4\") " pod="openshift-ingress/e2e-inf-b-00edb0-openshift-default-57cdc9785-6zxgq"
Jul 09 20:37:50.598897 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:50.598865    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d80fc5f7-0dfb-49e6-b168-5da6b11a12f4-istio-data\") pod \"e2e-inf-b-00edb0-openshift-default-57cdc9785-6zxgq\" (UID: \"d80fc5f7-0dfb-49e6-b168-5da6b11a12f4\") " pod="openshift-ingress/e2e-inf-b-00edb0-openshift-default-57cdc9785-6zxgq"
Jul 09 20:37:50.600205 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:50.600186    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d80fc5f7-0dfb-49e6-b168-5da6b11a12f4-istio-envoy\") pod \"e2e-inf-b-00edb0-openshift-default-57cdc9785-6zxgq\" (UID: \"d80fc5f7-0dfb-49e6-b168-5da6b11a12f4\") " pod="openshift-ingress/e2e-inf-b-00edb0-openshift-default-57cdc9785-6zxgq"
Jul 09 20:37:50.600782 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:50.600762    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d80fc5f7-0dfb-49e6-b168-5da6b11a12f4-istio-podinfo\") pod \"e2e-inf-b-00edb0-openshift-default-57cdc9785-6zxgq\" (UID: \"d80fc5f7-0dfb-49e6-b168-5da6b11a12f4\") " pod="openshift-ingress/e2e-inf-b-00edb0-openshift-default-57cdc9785-6zxgq"
Jul 09 20:37:50.606941 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:50.606920    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d80fc5f7-0dfb-49e6-b168-5da6b11a12f4-istio-token\") pod \"e2e-inf-b-00edb0-openshift-default-57cdc9785-6zxgq\" (UID: \"d80fc5f7-0dfb-49e6-b168-5da6b11a12f4\") " pod="openshift-ingress/e2e-inf-b-00edb0-openshift-default-57cdc9785-6zxgq"
Jul 09 20:37:50.607024 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:50.606955    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8b59j\" (UniqueName: \"kubernetes.io/projected/d80fc5f7-0dfb-49e6-b168-5da6b11a12f4-kube-api-access-8b59j\") pod \"e2e-inf-b-00edb0-openshift-default-57cdc9785-6zxgq\" (UID: \"d80fc5f7-0dfb-49e6-b168-5da6b11a12f4\") " pod="openshift-ingress/e2e-inf-b-00edb0-openshift-default-57cdc9785-6zxgq"
Jul 09 20:37:50.778809 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:50.778751    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-00edb0-openshift-default-57cdc9785-6zxgq"
Jul 09 20:37:50.905602 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:50.905573    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-00edb0-openshift-default-57cdc9785-6zxgq"]
Jul 09 20:37:50.907281 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:37:50.907253    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd80fc5f7_0dfb_49e6_b168_5da6b11a12f4.slice/crio-8876a773f07322d9ed3b8827144d4b07def1a22303cdb2a05cac00b9e2220d55 WatchSource:0}: Error finding container 8876a773f07322d9ed3b8827144d4b07def1a22303cdb2a05cac00b9e2220d55: Status 404 returned error can't find the container with id 8876a773f07322d9ed3b8827144d4b07def1a22303cdb2a05cac00b9e2220d55
Jul 09 20:37:50.909562 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:50.909515    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 09 20:37:50.909624 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:50.909600    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 09 20:37:50.909663 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:50.909647    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 09 20:37:51.436840 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:51.436762    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-00edb0-openshift-default-57cdc9785-6zxgq" event={"ID":"d80fc5f7-0dfb-49e6-b168-5da6b11a12f4","Type":"ContainerStarted","Data":"9c99c8f437341f56d4507a61dbfa553363dbe8c8e49dd6af984bff22471825be"}
Jul 09 20:37:51.436840 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:51.436828    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-00edb0-openshift-default-57cdc9785-6zxgq" event={"ID":"d80fc5f7-0dfb-49e6-b168-5da6b11a12f4","Type":"ContainerStarted","Data":"8876a773f07322d9ed3b8827144d4b07def1a22303cdb2a05cac00b9e2220d55"}
Jul 09 20:37:51.459442 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:51.459399    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-00edb0-openshift-default-57cdc9785-6zxgq" podStartSLOduration=1.459386008 podStartE2EDuration="1.459386008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 20:37:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 20:37:51.45621716 +0000 UTC m=+2061.873886981" watchObservedRunningTime="2026-07-09 20:37:51.459386008 +0000 UTC m=+2061.877055814"
Jul 09 20:37:51.779604 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:51.779576    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-00edb0-openshift-default-57cdc9785-6zxgq"
Jul 09 20:37:51.784029 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:51.784009    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-00edb0-openshift-default-57cdc9785-6zxgq"
Jul 09 20:37:52.441142 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:52.441121    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-00edb0-openshift-default-57cdc9785-6zxgq"
Jul 09 20:37:52.441904 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:52.441887    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-00edb0-openshift-default-57cdc9785-6zxgq"
Jul 09 20:37:53.427904 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:53.427879    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c0edf2-b584777f6-t8j25"
Jul 09 20:37:53.482742 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:53.482716    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-00edb0-8547fc74bb-zg4qw"]
Jul 09 20:37:53.486571 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:53.486548    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-00edb0-8547fc74bb-zg4qw"
Jul 09 20:37:53.490397 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:53.490335    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-00edb0-serving-cert\""
Jul 09 20:37:53.502574 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:53.502554    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-00edb0-8547fc74bb-zg4qw"]
Jul 09 20:37:53.522242 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:53.522185    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07d89f9a-124e-4255-af4a-4d7e6c3e3e6e-maas-api-tls\") pod \"maas-api-e2e-inf-b-00edb0-8547fc74bb-zg4qw\" (UID: \"07d89f9a-124e-4255-af4a-4d7e6c3e3e6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-00edb0-8547fc74bb-zg4qw"
Jul 09 20:37:53.522498 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:53.522261    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07d89f9a-124e-4255-af4a-4d7e6c3e3e6e-openshift-service-ca\") pod \"maas-api-e2e-inf-b-00edb0-8547fc74bb-zg4qw\" (UID: \"07d89f9a-124e-4255-af4a-4d7e6c3e3e6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-00edb0-8547fc74bb-zg4qw"
Jul 09 20:37:53.522751 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:53.522726    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxmlf\" (UniqueName: \"kubernetes.io/projected/07d89f9a-124e-4255-af4a-4d7e6c3e3e6e-kube-api-access-lxmlf\") pod \"maas-api-e2e-inf-b-00edb0-8547fc74bb-zg4qw\" (UID: \"07d89f9a-124e-4255-af4a-4d7e6c3e3e6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-00edb0-8547fc74bb-zg4qw"
Jul 09 20:37:53.623771 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:53.623727    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxmlf\" (UniqueName: \"kubernetes.io/projected/07d89f9a-124e-4255-af4a-4d7e6c3e3e6e-kube-api-access-lxmlf\") pod \"maas-api-e2e-inf-b-00edb0-8547fc74bb-zg4qw\" (UID: \"07d89f9a-124e-4255-af4a-4d7e6c3e3e6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-00edb0-8547fc74bb-zg4qw"
Jul 09 20:37:53.623935 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:53.623844    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07d89f9a-124e-4255-af4a-4d7e6c3e3e6e-maas-api-tls\") pod \"maas-api-e2e-inf-b-00edb0-8547fc74bb-zg4qw\" (UID: \"07d89f9a-124e-4255-af4a-4d7e6c3e3e6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-00edb0-8547fc74bb-zg4qw"
Jul 09 20:37:53.623935 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:53.623895    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07d89f9a-124e-4255-af4a-4d7e6c3e3e6e-openshift-service-ca\") pod \"maas-api-e2e-inf-b-00edb0-8547fc74bb-zg4qw\" (UID: \"07d89f9a-124e-4255-af4a-4d7e6c3e3e6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-00edb0-8547fc74bb-zg4qw"
Jul 09 20:37:53.624051 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:37:53.623938    2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-00edb0-serving-cert: secret "maas-api-e2e-inf-b-00edb0-serving-cert" not found
Jul 09 20:37:53.624051 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:37:53.624023    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07d89f9a-124e-4255-af4a-4d7e6c3e3e6e-maas-api-tls podName:07d89f9a-124e-4255-af4a-4d7e6c3e3e6e nodeName:}" failed. No retries permitted until 2026-07-09 20:37:54.124001038 +0000 UTC m=+2064.541670824 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/07d89f9a-124e-4255-af4a-4d7e6c3e3e6e-maas-api-tls") pod "maas-api-e2e-inf-b-00edb0-8547fc74bb-zg4qw" (UID: "07d89f9a-124e-4255-af4a-4d7e6c3e3e6e") : secret "maas-api-e2e-inf-b-00edb0-serving-cert" not found
Jul 09 20:37:53.624542 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:53.624526    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07d89f9a-124e-4255-af4a-4d7e6c3e3e6e-openshift-service-ca\") pod \"maas-api-e2e-inf-b-00edb0-8547fc74bb-zg4qw\" (UID: \"07d89f9a-124e-4255-af4a-4d7e6c3e3e6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-00edb0-8547fc74bb-zg4qw"
Jul 09 20:37:53.640181 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:53.640165    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxmlf\" (UniqueName: \"kubernetes.io/projected/07d89f9a-124e-4255-af4a-4d7e6c3e3e6e-kube-api-access-lxmlf\") pod \"maas-api-e2e-inf-b-00edb0-8547fc74bb-zg4qw\" (UID: \"07d89f9a-124e-4255-af4a-4d7e6c3e3e6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-00edb0-8547fc74bb-zg4qw"
Jul 09 20:37:54.128605 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:54.128572    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07d89f9a-124e-4255-af4a-4d7e6c3e3e6e-maas-api-tls\") pod \"maas-api-e2e-inf-b-00edb0-8547fc74bb-zg4qw\" (UID: \"07d89f9a-124e-4255-af4a-4d7e6c3e3e6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-00edb0-8547fc74bb-zg4qw"
Jul 09 20:37:54.130801 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:54.130777    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07d89f9a-124e-4255-af4a-4d7e6c3e3e6e-maas-api-tls\") pod \"maas-api-e2e-inf-b-00edb0-8547fc74bb-zg4qw\" (UID: \"07d89f9a-124e-4255-af4a-4d7e6c3e3e6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-00edb0-8547fc74bb-zg4qw"
Jul 09 20:37:54.398972 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:54.398876    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-00edb0-8547fc74bb-zg4qw"
Jul 09 20:37:54.592186 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:54.592156    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-00edb0-8547fc74bb-zg4qw"]
Jul 09 20:37:54.594110 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:37:54.594083    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07d89f9a_124e_4255_af4a_4d7e6c3e3e6e.slice/crio-fafa3353d809e92cbebf73ca189849d99c997359b78af0dbe2b486fb292e4f72 WatchSource:0}: Error finding container fafa3353d809e92cbebf73ca189849d99c997359b78af0dbe2b486fb292e4f72: Status 404 returned error can't find the container with id fafa3353d809e92cbebf73ca189849d99c997359b78af0dbe2b486fb292e4f72
Jul 09 20:37:55.458386 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:55.458350    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-00edb0-8547fc74bb-zg4qw" event={"ID":"07d89f9a-124e-4255-af4a-4d7e6c3e3e6e","Type":"ContainerStarted","Data":"7eadee10ebbe9467b5c12d3aeab25390a4c339b64069d219f8db737ce849e6e2"}
Jul 09 20:37:55.458574 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:55.458401    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-00edb0-8547fc74bb-zg4qw" event={"ID":"07d89f9a-124e-4255-af4a-4d7e6c3e3e6e","Type":"ContainerStarted","Data":"fafa3353d809e92cbebf73ca189849d99c997359b78af0dbe2b486fb292e4f72"}
Jul 09 20:37:55.458574 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:55.458439    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-00edb0-8547fc74bb-zg4qw"
Jul 09 20:37:55.480741 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:37:55.480699    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-00edb0-8547fc74bb-zg4qw" podStartSLOduration=2.051848873 podStartE2EDuration="2.480685424s" totalImagesPullingTime="428.836551ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 20:37:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 20:37:55.479203916 +0000 UTC m=+2065.896873730" watchObservedRunningTime="2026-07-09 20:37:55.480685424 +0000 UTC m=+2065.898355271"
Jul 09 20:38:00.440427 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:38:00.440393    2562 patch_prober.go:28] interesting pod/e2e-inf-a-c0edf2-openshift-default-54f964d9f7-c85nc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 09 20:38:00.440782 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:38:00.440440    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-c0edf2-openshift-default-54f964d9f7-c85nc" podUID="799ba2bb-8a49-46f2-b985-1e4a1f06e52b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 09 20:38:01.468682 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:38:01.468654    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-00edb0-8547fc74bb-zg4qw"
Jul 09 20:38:01.487390 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:38:01.487360    2562 generic.go:356] "Generic (PLEG): container finished" podID="799ba2bb-8a49-46f2-b985-1e4a1f06e52b" containerID="42a2699848e09d5eb1dc6674c2a7073d2a96f2325fbd243c3269e4105d3a408d" exitCode=0
Jul 09 20:38:01.487543 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:38:01.487435    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c0edf2-openshift-default-54f964d9f7-c85nc" event={"ID":"799ba2bb-8a49-46f2-b985-1e4a1f06e52b","Type":"ContainerDied","Data":"42a2699848e09d5eb1dc6674c2a7073d2a96f2325fbd243c3269e4105d3a408d"}
Jul 09 20:38:01.487826 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:38:01.487810    2562 scope.go:117] "RemoveContainer" containerID="42a2699848e09d5eb1dc6674c2a7073d2a96f2325fbd243c3269e4105d3a408d"
Jul 09 20:38:01.488798 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:38:01.488770    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 09 20:38:01.488891 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:38:01.488842    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 09 20:38:01.488891 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:38:01.488885    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 09 20:38:02.497981 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:38:02.497946    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c0edf2-openshift-default-54f964d9f7-c85nc" event={"ID":"799ba2bb-8a49-46f2-b985-1e4a1f06e52b","Type":"ContainerStarted","Data":"e976ba56408640f14fff0986be2964d1f2bdff56f8b4f1f2e451fb7abe92078b"}
Jul 09 20:38:03.098466 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:38:03.098437    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-c0edf2-openshift-default-54f964d9f7-c85nc"
Jul 09 20:38:03.098699 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:38:03.098549    2562 patch_prober.go:28] interesting pod/e2e-inf-a-c0edf2-openshift-default-54f964d9f7-c85nc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Jul 09 20:38:03.098820 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:38:03.098803    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c0edf2-openshift-default-54f964d9f7-c85nc" podUID="799ba2bb-8a49-46f2-b985-1e4a1f06e52b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Jul 09 20:38:04.098317 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:38:04.098279    2562 patch_prober.go:28] interesting pod/e2e-inf-a-c0edf2-openshift-default-54f964d9f7-c85nc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Jul 09 20:38:04.098699 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:38:04.098380    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c0edf2-openshift-default-54f964d9f7-c85nc" podUID="799ba2bb-8a49-46f2-b985-1e4a1f06e52b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Jul 09 20:38:05.098289 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:38:05.098261    2562 patch_prober.go:28] interesting pod/e2e-inf-a-c0edf2-openshift-default-54f964d9f7-c85nc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Jul 09 20:38:05.098445 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:38:05.098330    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c0edf2-openshift-default-54f964d9f7-c85nc" podUID="799ba2bb-8a49-46f2-b985-1e4a1f06e52b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Jul 09 20:38:06.098100 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:38:06.098062    2562 patch_prober.go:28] interesting pod/e2e-inf-a-c0edf2-openshift-default-54f964d9f7-c85nc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Jul 09 20:38:06.098323 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:38:06.098121    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c0edf2-openshift-default-54f964d9f7-c85nc" podUID="799ba2bb-8a49-46f2-b985-1e4a1f06e52b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Jul 09 20:38:07.101201 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:38:07.101166    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-c0edf2-openshift-default-54f964d9f7-c85nc"
Jul 09 20:38:07.101677 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:38:07.101585    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-c0edf2-openshift-default-54f964d9f7-c85nc"
Jul 09 20:38:07.102273 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:38:07.102256    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-c0edf2-openshift-default-54f964d9f7-c85nc"
Jul 09 20:38:07.973425 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:38:07.972698    2562 patch_prober.go:28] interesting pod/e2e-inf-b-00edb0-openshift-default-57cdc9785-6zxgq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": read tcp 10.134.0.2:50928->10.134.0.59:15021: read: connection reset by peer" start-of-body=
Jul 09 20:38:07.973425 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:38:07.972758    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-00edb0-openshift-default-57cdc9785-6zxgq" podUID="d80fc5f7-0dfb-49e6-b168-5da6b11a12f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": read tcp 10.134.0.2:50928->10.134.0.59:15021: read: connection reset by peer"
Jul 09 20:38:08.536706 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:38:08.536677    2562 generic.go:356] "Generic (PLEG): container finished" podID="d80fc5f7-0dfb-49e6-b168-5da6b11a12f4" containerID="9c99c8f437341f56d4507a61dbfa553363dbe8c8e49dd6af984bff22471825be" exitCode=0
Jul 09 20:38:08.537026 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:38:08.536748    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-00edb0-openshift-default-57cdc9785-6zxgq" event={"ID":"d80fc5f7-0dfb-49e6-b168-5da6b11a12f4","Type":"ContainerDied","Data":"9c99c8f437341f56d4507a61dbfa553363dbe8c8e49dd6af984bff22471825be"}
Jul 09 20:38:08.537193 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:38:08.537179    2562 scope.go:117] "RemoveContainer" containerID="9c99c8f437341f56d4507a61dbfa553363dbe8c8e49dd6af984bff22471825be"
Jul 09 20:38:08.538191 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:38:08.538164    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 09 20:38:08.538252 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:38:08.538227    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 09 20:38:08.538287 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:38:08.538260    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 09 20:38:09.543465 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:38:09.543432    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-00edb0-openshift-default-57cdc9785-6zxgq" event={"ID":"d80fc5f7-0dfb-49e6-b168-5da6b11a12f4","Type":"ContainerStarted","Data":"291ca0108331e7e9d17360346a6b076f40bb30663ce5dfa4d5571a2e5e73339f"}
Jul 09 20:38:09.779382 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:38:09.779350    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-00edb0-openshift-default-57cdc9785-6zxgq"
Jul 09 20:38:09.779582 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:38:09.779564    2562 patch_prober.go:28] interesting pod/e2e-inf-b-00edb0-openshift-default-57cdc9785-6zxgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Jul 09 20:38:09.779663 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:38:09.779611    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-00edb0-openshift-default-57cdc9785-6zxgq" podUID="d80fc5f7-0dfb-49e6-b168-5da6b11a12f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Jul 09 20:38:10.779876 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:38:10.779846    2562 patch_prober.go:28] interesting pod/e2e-inf-b-00edb0-openshift-default-57cdc9785-6zxgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Jul 09 20:38:10.780335 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:38:10.779899    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-00edb0-openshift-default-57cdc9785-6zxgq" podUID="d80fc5f7-0dfb-49e6-b168-5da6b11a12f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Jul 09 20:38:11.779776 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:38:11.779747    2562 patch_prober.go:28] interesting pod/e2e-inf-b-00edb0-openshift-default-57cdc9785-6zxgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Jul 09 20:38:11.779944 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:38:11.779793    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-00edb0-openshift-default-57cdc9785-6zxgq" podUID="d80fc5f7-0dfb-49e6-b168-5da6b11a12f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Jul 09 20:38:12.779477 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:38:12.779446    2562 patch_prober.go:28] interesting pod/e2e-inf-b-00edb0-openshift-default-57cdc9785-6zxgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Jul 09 20:38:12.779640 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:38:12.779501    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-00edb0-openshift-default-57cdc9785-6zxgq" podUID="d80fc5f7-0dfb-49e6-b168-5da6b11a12f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Jul 09 20:38:13.782506 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:38:13.782476    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-00edb0-openshift-default-57cdc9785-6zxgq"
Jul 09 20:38:13.783005 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:38:13.782761    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-00edb0-openshift-default-57cdc9785-6zxgq"
Jul 09 20:38:13.783433 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:38:13.783420    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-00edb0-openshift-default-57cdc9785-6zxgq"
Jul 09 20:38:30.211866 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:38:30.211840    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-cqs6d_578878d0-e6b3-4725-a698-796cfc169fa6/console-operator/1.log"
Jul 09 20:38:30.217155 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:38:30.217134    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hds4g_4b4468fd-1708-487b-8d69-bb7d817e25de/ovn-acl-logging/0.log"
Jul 09 20:38:30.217842 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:38:30.217815    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-cqs6d_578878d0-e6b3-4725-a698-796cfc169fa6/console-operator/1.log"
Jul 09 20:38:30.223079 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:38:30.223062    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hds4g_4b4468fd-1708-487b-8d69-bb7d817e25de/ovn-acl-logging/0.log"
Jul 09 20:38:30.332880 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:38:30.332860    2562 scope.go:117] "RemoveContainer" containerID="441650ea20d19233bb511d7470a715fdc0b9e5d414f51b733d9a3c4198fa205c"
Jul 09 20:38:58.128594 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:38:58.128562    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c0edf2-b584777f6-t8j25"]
Jul 09 20:38:58.129211 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:38:58.128787    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c0edf2-b584777f6-t8j25" podUID="a6f3daa6-460b-4b55-87df-7a9eb7436c9f" containerName="maas-api" containerID="cri-o://ac54d8a6829e18f10cdf658285fd897ee065b4abf782593c4586f5a73ae2ed26" gracePeriod=30
Jul 09 20:38:58.375567 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:38:58.375539    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c0edf2-b584777f6-t8j25"
Jul 09 20:38:58.462774 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:38:58.462755    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z2crs\" (UniqueName: \"kubernetes.io/projected/a6f3daa6-460b-4b55-87df-7a9eb7436c9f-kube-api-access-z2crs\") pod \"a6f3daa6-460b-4b55-87df-7a9eb7436c9f\" (UID: \"a6f3daa6-460b-4b55-87df-7a9eb7436c9f\") "
Jul 09 20:38:58.462899 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:38:58.462881    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6f3daa6-460b-4b55-87df-7a9eb7436c9f-maas-api-tls\") pod \"a6f3daa6-460b-4b55-87df-7a9eb7436c9f\" (UID: \"a6f3daa6-460b-4b55-87df-7a9eb7436c9f\") "
Jul 09 20:38:58.462951 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:38:58.462939    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6f3daa6-460b-4b55-87df-7a9eb7436c9f-openshift-service-ca\") pod \"a6f3daa6-460b-4b55-87df-7a9eb7436c9f\" (UID: \"a6f3daa6-460b-4b55-87df-7a9eb7436c9f\") "
Jul 09 20:38:58.463255 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:38:58.463232    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6f3daa6-460b-4b55-87df-7a9eb7436c9f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a6f3daa6-460b-4b55-87df-7a9eb7436c9f" (UID: "a6f3daa6-460b-4b55-87df-7a9eb7436c9f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 20:38:58.464602 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:38:58.464584    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6f3daa6-460b-4b55-87df-7a9eb7436c9f-kube-api-access-z2crs" (OuterVolumeSpecName: "kube-api-access-z2crs") pod "a6f3daa6-460b-4b55-87df-7a9eb7436c9f" (UID: "a6f3daa6-460b-4b55-87df-7a9eb7436c9f"). InnerVolumeSpecName "kube-api-access-z2crs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 20:38:58.464701 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:38:58.464687    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6f3daa6-460b-4b55-87df-7a9eb7436c9f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a6f3daa6-460b-4b55-87df-7a9eb7436c9f" (UID: "a6f3daa6-460b-4b55-87df-7a9eb7436c9f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 20:38:58.564488 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:38:58.564470    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6f3daa6-460b-4b55-87df-7a9eb7436c9f-openshift-service-ca\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:38:58.564488 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:38:58.564490    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2crs\" (UniqueName: \"kubernetes.io/projected/a6f3daa6-460b-4b55-87df-7a9eb7436c9f-kube-api-access-z2crs\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:38:58.564599 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:38:58.564499    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6f3daa6-460b-4b55-87df-7a9eb7436c9f-maas-api-tls\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:38:58.761296 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:38:58.761240    2562 generic.go:356] "Generic (PLEG): container finished" podID="a6f3daa6-460b-4b55-87df-7a9eb7436c9f" containerID="ac54d8a6829e18f10cdf658285fd897ee065b4abf782593c4586f5a73ae2ed26" exitCode=0
Jul 09 20:38:58.761296 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:38:58.761281    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c0edf2-b584777f6-t8j25"
Jul 09 20:38:58.761465 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:38:58.761296    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c0edf2-b584777f6-t8j25" event={"ID":"a6f3daa6-460b-4b55-87df-7a9eb7436c9f","Type":"ContainerDied","Data":"ac54d8a6829e18f10cdf658285fd897ee065b4abf782593c4586f5a73ae2ed26"}
Jul 09 20:38:58.761465 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:38:58.761373    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c0edf2-b584777f6-t8j25" event={"ID":"a6f3daa6-460b-4b55-87df-7a9eb7436c9f","Type":"ContainerDied","Data":"0c9efb2e655eb1d3e57f15f9db853069604eb26d3d8eaf83a5d5a65faf814d30"}
Jul 09 20:38:58.761465 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:38:58.761405    2562 scope.go:117] "RemoveContainer" containerID="ac54d8a6829e18f10cdf658285fd897ee065b4abf782593c4586f5a73ae2ed26"
Jul 09 20:38:58.770211 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:38:58.770196    2562 scope.go:117] "RemoveContainer" containerID="ac54d8a6829e18f10cdf658285fd897ee065b4abf782593c4586f5a73ae2ed26"
Jul 09 20:38:58.770486 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:38:58.770468    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac54d8a6829e18f10cdf658285fd897ee065b4abf782593c4586f5a73ae2ed26\": container with ID starting with ac54d8a6829e18f10cdf658285fd897ee065b4abf782593c4586f5a73ae2ed26 not found: ID does not exist" containerID="ac54d8a6829e18f10cdf658285fd897ee065b4abf782593c4586f5a73ae2ed26"
Jul 09 20:38:58.770542 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:38:58.770495    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac54d8a6829e18f10cdf658285fd897ee065b4abf782593c4586f5a73ae2ed26"} err="failed to get container status \"ac54d8a6829e18f10cdf658285fd897ee065b4abf782593c4586f5a73ae2ed26\": rpc error: code = NotFound desc = could not find container \"ac54d8a6829e18f10cdf658285fd897ee065b4abf782593c4586f5a73ae2ed26\": container with ID starting with ac54d8a6829e18f10cdf658285fd897ee065b4abf782593c4586f5a73ae2ed26 not found: ID does not exist"
Jul 09 20:38:58.783557 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:38:58.783539    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c0edf2-b584777f6-t8j25"]
Jul 09 20:38:58.785357 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:38:58.785338    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c0edf2-b584777f6-t8j25"]
Jul 09 20:39:00.240653 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:00.240632    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a6f3daa6-460b-4b55-87df-7a9eb7436c9f" path="/var/lib/kubelet/pods/a6f3daa6-460b-4b55-87df-7a9eb7436c9f/volumes"
Jul 09 20:39:39.958276 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:39.958243    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-c0edf2-openshift-default-54f964d9f7-c85nc"]
Jul 09 20:39:39.958762 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:39.958503    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-c0edf2-openshift-default-54f964d9f7-c85nc" podUID="799ba2bb-8a49-46f2-b985-1e4a1f06e52b" containerName="istio-proxy" containerID="cri-o://e976ba56408640f14fff0986be2964d1f2bdff56f8b4f1f2e451fb7abe92078b" gracePeriod=30
Jul 09 20:39:41.134244 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:41.134209    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-00edb0-8547fc74bb-zg4qw"]
Jul 09 20:39:41.134653 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:41.134464    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-00edb0-8547fc74bb-zg4qw" podUID="07d89f9a-124e-4255-af4a-4d7e6c3e3e6e" containerName="maas-api" containerID="cri-o://7eadee10ebbe9467b5c12d3aeab25390a4c339b64069d219f8db737ce849e6e2" gracePeriod=30
Jul 09 20:39:41.377099 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:41.377075    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-00edb0-8547fc74bb-zg4qw"
Jul 09 20:39:41.484676 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:41.484654    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07d89f9a-124e-4255-af4a-4d7e6c3e3e6e-maas-api-tls\") pod \"07d89f9a-124e-4255-af4a-4d7e6c3e3e6e\" (UID: \"07d89f9a-124e-4255-af4a-4d7e6c3e3e6e\") "
Jul 09 20:39:41.484787 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:41.484683    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07d89f9a-124e-4255-af4a-4d7e6c3e3e6e-openshift-service-ca\") pod \"07d89f9a-124e-4255-af4a-4d7e6c3e3e6e\" (UID: \"07d89f9a-124e-4255-af4a-4d7e6c3e3e6e\") "
Jul 09 20:39:41.484787 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:41.484710    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lxmlf\" (UniqueName: \"kubernetes.io/projected/07d89f9a-124e-4255-af4a-4d7e6c3e3e6e-kube-api-access-lxmlf\") pod \"07d89f9a-124e-4255-af4a-4d7e6c3e3e6e\" (UID: \"07d89f9a-124e-4255-af4a-4d7e6c3e3e6e\") "
Jul 09 20:39:41.485022 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:41.484999    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07d89f9a-124e-4255-af4a-4d7e6c3e3e6e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "07d89f9a-124e-4255-af4a-4d7e6c3e3e6e" (UID: "07d89f9a-124e-4255-af4a-4d7e6c3e3e6e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 20:39:41.486596 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:41.486573    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07d89f9a-124e-4255-af4a-4d7e6c3e3e6e-kube-api-access-lxmlf" (OuterVolumeSpecName: "kube-api-access-lxmlf") pod "07d89f9a-124e-4255-af4a-4d7e6c3e3e6e" (UID: "07d89f9a-124e-4255-af4a-4d7e6c3e3e6e"). InnerVolumeSpecName "kube-api-access-lxmlf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 20:39:41.486697 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:41.486602    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07d89f9a-124e-4255-af4a-4d7e6c3e3e6e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "07d89f9a-124e-4255-af4a-4d7e6c3e3e6e" (UID: "07d89f9a-124e-4255-af4a-4d7e6c3e3e6e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 20:39:41.585525 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:41.585496    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07d89f9a-124e-4255-af4a-4d7e6c3e3e6e-maas-api-tls\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:39:41.585525 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:41.585524    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07d89f9a-124e-4255-af4a-4d7e6c3e3e6e-openshift-service-ca\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:39:41.585676 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:41.585550    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxmlf\" (UniqueName: \"kubernetes.io/projected/07d89f9a-124e-4255-af4a-4d7e6c3e3e6e-kube-api-access-lxmlf\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:39:41.954461 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:41.954432    2562 generic.go:356] "Generic (PLEG): container finished" podID="07d89f9a-124e-4255-af4a-4d7e6c3e3e6e" containerID="7eadee10ebbe9467b5c12d3aeab25390a4c339b64069d219f8db737ce849e6e2" exitCode=0
Jul 09 20:39:41.954565 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:41.954477    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-00edb0-8547fc74bb-zg4qw"
Jul 09 20:39:41.954565 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:41.954519    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-00edb0-8547fc74bb-zg4qw" event={"ID":"07d89f9a-124e-4255-af4a-4d7e6c3e3e6e","Type":"ContainerDied","Data":"7eadee10ebbe9467b5c12d3aeab25390a4c339b64069d219f8db737ce849e6e2"}
Jul 09 20:39:41.954636 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:41.954571    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-00edb0-8547fc74bb-zg4qw" event={"ID":"07d89f9a-124e-4255-af4a-4d7e6c3e3e6e","Type":"ContainerDied","Data":"fafa3353d809e92cbebf73ca189849d99c997359b78af0dbe2b486fb292e4f72"}
Jul 09 20:39:41.954636 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:41.954592    2562 scope.go:117] "RemoveContainer" containerID="7eadee10ebbe9467b5c12d3aeab25390a4c339b64069d219f8db737ce849e6e2"
Jul 09 20:39:41.962403 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:41.962389    2562 scope.go:117] "RemoveContainer" containerID="7eadee10ebbe9467b5c12d3aeab25390a4c339b64069d219f8db737ce849e6e2"
Jul 09 20:39:41.962629 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:39:41.962609    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7eadee10ebbe9467b5c12d3aeab25390a4c339b64069d219f8db737ce849e6e2\": container with ID starting with 7eadee10ebbe9467b5c12d3aeab25390a4c339b64069d219f8db737ce849e6e2 not found: ID does not exist" containerID="7eadee10ebbe9467b5c12d3aeab25390a4c339b64069d219f8db737ce849e6e2"
Jul 09 20:39:41.962691 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:41.962637    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7eadee10ebbe9467b5c12d3aeab25390a4c339b64069d219f8db737ce849e6e2"} err="failed to get container status \"7eadee10ebbe9467b5c12d3aeab25390a4c339b64069d219f8db737ce849e6e2\": rpc error: code = NotFound desc = could not find container \"7eadee10ebbe9467b5c12d3aeab25390a4c339b64069d219f8db737ce849e6e2\": container with ID starting with 7eadee10ebbe9467b5c12d3aeab25390a4c339b64069d219f8db737ce849e6e2 not found: ID does not exist"
Jul 09 20:39:41.978972 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:41.978950    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-00edb0-8547fc74bb-zg4qw"]
Jul 09 20:39:41.983632 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:41.983613    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-00edb0-8547fc74bb-zg4qw"]
Jul 09 20:39:42.241650 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:42.241600    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="07d89f9a-124e-4255-af4a-4d7e6c3e3e6e" path="/var/lib/kubelet/pods/07d89f9a-124e-4255-af4a-4d7e6c3e3e6e/volumes"
Jul 09 20:39:45.198383 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:45.198361    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c0edf2-openshift-default-54f964d9f7-c85nc"
Jul 09 20:39:45.313459 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:45.313408    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/799ba2bb-8a49-46f2-b985-1e4a1f06e52b-istio-envoy\") pod \"799ba2bb-8a49-46f2-b985-1e4a1f06e52b\" (UID: \"799ba2bb-8a49-46f2-b985-1e4a1f06e52b\") "
Jul 09 20:39:45.313459 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:45.313451    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/799ba2bb-8a49-46f2-b985-1e4a1f06e52b-istio-data\") pod \"799ba2bb-8a49-46f2-b985-1e4a1f06e52b\" (UID: \"799ba2bb-8a49-46f2-b985-1e4a1f06e52b\") "
Jul 09 20:39:45.313687 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:45.313486    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/799ba2bb-8a49-46f2-b985-1e4a1f06e52b-istio-podinfo\") pod \"799ba2bb-8a49-46f2-b985-1e4a1f06e52b\" (UID: \"799ba2bb-8a49-46f2-b985-1e4a1f06e52b\") "
Jul 09 20:39:45.313687 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:45.313511    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r56bp\" (UniqueName: \"kubernetes.io/projected/799ba2bb-8a49-46f2-b985-1e4a1f06e52b-kube-api-access-r56bp\") pod \"799ba2bb-8a49-46f2-b985-1e4a1f06e52b\" (UID: \"799ba2bb-8a49-46f2-b985-1e4a1f06e52b\") "
Jul 09 20:39:45.313687 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:45.313537    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/799ba2bb-8a49-46f2-b985-1e4a1f06e52b-istiod-ca-cert\") pod \"799ba2bb-8a49-46f2-b985-1e4a1f06e52b\" (UID: \"799ba2bb-8a49-46f2-b985-1e4a1f06e52b\") "
Jul 09 20:39:45.313687 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:45.313582    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/799ba2bb-8a49-46f2-b985-1e4a1f06e52b-credential-socket\") pod \"799ba2bb-8a49-46f2-b985-1e4a1f06e52b\" (UID: \"799ba2bb-8a49-46f2-b985-1e4a1f06e52b\") "
Jul 09 20:39:45.313687 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:45.313622    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/799ba2bb-8a49-46f2-b985-1e4a1f06e52b-workload-socket\") pod \"799ba2bb-8a49-46f2-b985-1e4a1f06e52b\" (UID: \"799ba2bb-8a49-46f2-b985-1e4a1f06e52b\") "
Jul 09 20:39:45.313687 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:45.313659    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/799ba2bb-8a49-46f2-b985-1e4a1f06e52b-workload-certs\") pod \"799ba2bb-8a49-46f2-b985-1e4a1f06e52b\" (UID: \"799ba2bb-8a49-46f2-b985-1e4a1f06e52b\") "
Jul 09 20:39:45.313687 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:45.313689    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/799ba2bb-8a49-46f2-b985-1e4a1f06e52b-istio-token\") pod \"799ba2bb-8a49-46f2-b985-1e4a1f06e52b\" (UID: \"799ba2bb-8a49-46f2-b985-1e4a1f06e52b\") "
Jul 09 20:39:45.314145 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:45.314061    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/799ba2bb-8a49-46f2-b985-1e4a1f06e52b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "799ba2bb-8a49-46f2-b985-1e4a1f06e52b" (UID: "799ba2bb-8a49-46f2-b985-1e4a1f06e52b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 20:39:45.314145 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:45.314093    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/799ba2bb-8a49-46f2-b985-1e4a1f06e52b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "799ba2bb-8a49-46f2-b985-1e4a1f06e52b" (UID: "799ba2bb-8a49-46f2-b985-1e4a1f06e52b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 20:39:45.314145 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:45.314026    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/799ba2bb-8a49-46f2-b985-1e4a1f06e52b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "799ba2bb-8a49-46f2-b985-1e4a1f06e52b" (UID: "799ba2bb-8a49-46f2-b985-1e4a1f06e52b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 20:39:45.314412 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:45.314282    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/799ba2bb-8a49-46f2-b985-1e4a1f06e52b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "799ba2bb-8a49-46f2-b985-1e4a1f06e52b" (UID: "799ba2bb-8a49-46f2-b985-1e4a1f06e52b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 20:39:45.314515 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:45.314498    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/799ba2bb-8a49-46f2-b985-1e4a1f06e52b-istio-data" (OuterVolumeSpecName: "istio-data") pod "799ba2bb-8a49-46f2-b985-1e4a1f06e52b" (UID: "799ba2bb-8a49-46f2-b985-1e4a1f06e52b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 20:39:45.315912 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:45.315886    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/799ba2bb-8a49-46f2-b985-1e4a1f06e52b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "799ba2bb-8a49-46f2-b985-1e4a1f06e52b" (UID: "799ba2bb-8a49-46f2-b985-1e4a1f06e52b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 20:39:45.315912 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:45.315897    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/799ba2bb-8a49-46f2-b985-1e4a1f06e52b-kube-api-access-r56bp" (OuterVolumeSpecName: "kube-api-access-r56bp") pod "799ba2bb-8a49-46f2-b985-1e4a1f06e52b" (UID: "799ba2bb-8a49-46f2-b985-1e4a1f06e52b"). InnerVolumeSpecName "kube-api-access-r56bp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 20:39:45.316076 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:45.315993    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/799ba2bb-8a49-46f2-b985-1e4a1f06e52b-istio-token" (OuterVolumeSpecName: "istio-token") pod "799ba2bb-8a49-46f2-b985-1e4a1f06e52b" (UID: "799ba2bb-8a49-46f2-b985-1e4a1f06e52b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 20:39:45.316076 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:45.316052    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/799ba2bb-8a49-46f2-b985-1e4a1f06e52b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "799ba2bb-8a49-46f2-b985-1e4a1f06e52b" (UID: "799ba2bb-8a49-46f2-b985-1e4a1f06e52b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 09 20:39:45.414959 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:45.414936    2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/799ba2bb-8a49-46f2-b985-1e4a1f06e52b-istio-data\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:39:45.414959 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:45.414954    2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/799ba2bb-8a49-46f2-b985-1e4a1f06e52b-istio-podinfo\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:39:45.415127 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:45.414963    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r56bp\" (UniqueName: \"kubernetes.io/projected/799ba2bb-8a49-46f2-b985-1e4a1f06e52b-kube-api-access-r56bp\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:39:45.415127 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:45.414972    2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/799ba2bb-8a49-46f2-b985-1e4a1f06e52b-istiod-ca-cert\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:39:45.415127 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:45.414980    2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/799ba2bb-8a49-46f2-b985-1e4a1f06e52b-credential-socket\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:39:45.415127 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:45.414987    2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/799ba2bb-8a49-46f2-b985-1e4a1f06e52b-workload-socket\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:39:45.415127 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:45.414996    2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/799ba2bb-8a49-46f2-b985-1e4a1f06e52b-workload-certs\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:39:45.415127 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:45.415005    2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/799ba2bb-8a49-46f2-b985-1e4a1f06e52b-istio-token\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:39:45.415127 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:45.415013    2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/799ba2bb-8a49-46f2-b985-1e4a1f06e52b-istio-envoy\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:39:45.973277 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:45.973250    2562 generic.go:356] "Generic (PLEG): container finished" podID="799ba2bb-8a49-46f2-b985-1e4a1f06e52b" containerID="e976ba56408640f14fff0986be2964d1f2bdff56f8b4f1f2e451fb7abe92078b" exitCode=0
Jul 09 20:39:45.973397 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:45.973285    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c0edf2-openshift-default-54f964d9f7-c85nc"
Jul 09 20:39:45.973397 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:45.973339    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c0edf2-openshift-default-54f964d9f7-c85nc" event={"ID":"799ba2bb-8a49-46f2-b985-1e4a1f06e52b","Type":"ContainerDied","Data":"e976ba56408640f14fff0986be2964d1f2bdff56f8b4f1f2e451fb7abe92078b"}
Jul 09 20:39:45.973397 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:45.973388    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c0edf2-openshift-default-54f964d9f7-c85nc" event={"ID":"799ba2bb-8a49-46f2-b985-1e4a1f06e52b","Type":"ContainerDied","Data":"9f6ff43d6537e7f30d91e408302101356a457b7ee825cefab4e5da6f5ee10e23"}
Jul 09 20:39:45.973526 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:45.973410    2562 scope.go:117] "RemoveContainer" containerID="e976ba56408640f14fff0986be2964d1f2bdff56f8b4f1f2e451fb7abe92078b"
Jul 09 20:39:45.982235 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:45.982221    2562 scope.go:117] "RemoveContainer" containerID="42a2699848e09d5eb1dc6674c2a7073d2a96f2325fbd243c3269e4105d3a408d"
Jul 09 20:39:45.989622 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:45.989606    2562 scope.go:117] "RemoveContainer" containerID="e976ba56408640f14fff0986be2964d1f2bdff56f8b4f1f2e451fb7abe92078b"
Jul 09 20:39:45.989913 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:39:45.989896    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e976ba56408640f14fff0986be2964d1f2bdff56f8b4f1f2e451fb7abe92078b\": container with ID starting with e976ba56408640f14fff0986be2964d1f2bdff56f8b4f1f2e451fb7abe92078b not found: ID does not exist" containerID="e976ba56408640f14fff0986be2964d1f2bdff56f8b4f1f2e451fb7abe92078b"
Jul 09 20:39:45.989965 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:45.989924    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e976ba56408640f14fff0986be2964d1f2bdff56f8b4f1f2e451fb7abe92078b"} err="failed to get container status \"e976ba56408640f14fff0986be2964d1f2bdff56f8b4f1f2e451fb7abe92078b\": rpc error: code = NotFound desc = could not find container \"e976ba56408640f14fff0986be2964d1f2bdff56f8b4f1f2e451fb7abe92078b\": container with ID starting with e976ba56408640f14fff0986be2964d1f2bdff56f8b4f1f2e451fb7abe92078b not found: ID does not exist"
Jul 09 20:39:45.989965 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:45.989940    2562 scope.go:117] "RemoveContainer" containerID="42a2699848e09d5eb1dc6674c2a7073d2a96f2325fbd243c3269e4105d3a408d"
Jul 09 20:39:45.990160 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:39:45.990142    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42a2699848e09d5eb1dc6674c2a7073d2a96f2325fbd243c3269e4105d3a408d\": container with ID starting with 42a2699848e09d5eb1dc6674c2a7073d2a96f2325fbd243c3269e4105d3a408d not found: ID does not exist" containerID="42a2699848e09d5eb1dc6674c2a7073d2a96f2325fbd243c3269e4105d3a408d"
Jul 09 20:39:45.990206 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:45.990163    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42a2699848e09d5eb1dc6674c2a7073d2a96f2325fbd243c3269e4105d3a408d"} err="failed to get container status \"42a2699848e09d5eb1dc6674c2a7073d2a96f2325fbd243c3269e4105d3a408d\": rpc error: code = NotFound desc = could not find container \"42a2699848e09d5eb1dc6674c2a7073d2a96f2325fbd243c3269e4105d3a408d\": container with ID starting with 42a2699848e09d5eb1dc6674c2a7073d2a96f2325fbd243c3269e4105d3a408d not found: ID does not exist"
Jul 09 20:39:45.995773 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:45.995752    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-c0edf2-openshift-default-54f964d9f7-c85nc"]
Jul 09 20:39:46.000086 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:46.000059    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-c0edf2-openshift-default-54f964d9f7-c85nc"]
Jul 09 20:39:46.242189 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:46.242126    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="799ba2bb-8a49-46f2-b985-1e4a1f06e52b" path="/var/lib/kubelet/pods/799ba2bb-8a49-46f2-b985-1e4a1f06e52b/volumes"
Jul 09 20:39:46.785172 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:46.785139    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-00edb0-openshift-default-57cdc9785-6zxgq"]
Jul 09 20:39:46.785482 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:46.785430    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-00edb0-openshift-default-57cdc9785-6zxgq" podUID="d80fc5f7-0dfb-49e6-b168-5da6b11a12f4" containerName="istio-proxy" containerID="cri-o://291ca0108331e7e9d17360346a6b076f40bb30663ce5dfa4d5571a2e5e73339f" gracePeriod=30
Jul 09 20:39:47.669771 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:47.669737    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-4c278a-openshift-default-7667669d89-ns249"]
Jul 09 20:39:47.670231 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:47.670222    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07d89f9a-124e-4255-af4a-4d7e6c3e3e6e" containerName="maas-api"
Jul 09 20:39:47.670282 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:47.670235    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="07d89f9a-124e-4255-af4a-4d7e6c3e3e6e" containerName="maas-api"
Jul 09 20:39:47.670282 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:47.670248    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6f3daa6-460b-4b55-87df-7a9eb7436c9f" containerName="maas-api"
Jul 09 20:39:47.670282 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:47.670258    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6f3daa6-460b-4b55-87df-7a9eb7436c9f" containerName="maas-api"
Jul 09 20:39:47.670282 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:47.670275    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="799ba2bb-8a49-46f2-b985-1e4a1f06e52b" containerName="istio-proxy"
Jul 09 20:39:47.670427 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:47.670284    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="799ba2bb-8a49-46f2-b985-1e4a1f06e52b" containerName="istio-proxy"
Jul 09 20:39:47.670427 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:47.670314    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="799ba2bb-8a49-46f2-b985-1e4a1f06e52b" containerName="istio-proxy"
Jul 09 20:39:47.670427 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:47.670323    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="799ba2bb-8a49-46f2-b985-1e4a1f06e52b" containerName="istio-proxy"
Jul 09 20:39:47.670427 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:47.670421    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="799ba2bb-8a49-46f2-b985-1e4a1f06e52b" containerName="istio-proxy"
Jul 09 20:39:47.670541 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:47.670437    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="07d89f9a-124e-4255-af4a-4d7e6c3e3e6e" containerName="maas-api"
Jul 09 20:39:47.670541 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:47.670446    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6f3daa6-460b-4b55-87df-7a9eb7436c9f" containerName="maas-api"
Jul 09 20:39:47.675034 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:47.675014    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-4c278a-openshift-default-7667669d89-ns249"
Jul 09 20:39:47.678463 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:47.678443    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-4c278a-openshift-default-dockercfg-gvxmj\""
Jul 09 20:39:47.685831 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:47.685810    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-4c278a-openshift-default-7667669d89-ns249"]
Jul 09 20:39:47.835996 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:47.835966    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b47ec9db-f060-4486-8599-d35bbcaff3ca-workload-certs\") pod \"e2e-api-a-4c278a-openshift-default-7667669d89-ns249\" (UID: \"b47ec9db-f060-4486-8599-d35bbcaff3ca\") " pod="openshift-ingress/e2e-api-a-4c278a-openshift-default-7667669d89-ns249"
Jul 09 20:39:47.836147 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:47.836001    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b47ec9db-f060-4486-8599-d35bbcaff3ca-istiod-ca-cert\") pod \"e2e-api-a-4c278a-openshift-default-7667669d89-ns249\" (UID: \"b47ec9db-f060-4486-8599-d35bbcaff3ca\") " pod="openshift-ingress/e2e-api-a-4c278a-openshift-default-7667669d89-ns249"
Jul 09 20:39:47.836147 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:47.836031    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b47ec9db-f060-4486-8599-d35bbcaff3ca-workload-socket\") pod \"e2e-api-a-4c278a-openshift-default-7667669d89-ns249\" (UID: \"b47ec9db-f060-4486-8599-d35bbcaff3ca\") " pod="openshift-ingress/e2e-api-a-4c278a-openshift-default-7667669d89-ns249"
Jul 09 20:39:47.836147 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:47.836079    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b47ec9db-f060-4486-8599-d35bbcaff3ca-istio-token\") pod \"e2e-api-a-4c278a-openshift-default-7667669d89-ns249\" (UID: \"b47ec9db-f060-4486-8599-d35bbcaff3ca\") " pod="openshift-ingress/e2e-api-a-4c278a-openshift-default-7667669d89-ns249"
Jul 09 20:39:47.836147 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:47.836098    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b47ec9db-f060-4486-8599-d35bbcaff3ca-istio-podinfo\") pod \"e2e-api-a-4c278a-openshift-default-7667669d89-ns249\" (UID: \"b47ec9db-f060-4486-8599-d35bbcaff3ca\") " pod="openshift-ingress/e2e-api-a-4c278a-openshift-default-7667669d89-ns249"
Jul 09 20:39:47.836147 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:47.836116    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b47ec9db-f060-4486-8599-d35bbcaff3ca-credential-socket\") pod \"e2e-api-a-4c278a-openshift-default-7667669d89-ns249\" (UID: \"b47ec9db-f060-4486-8599-d35bbcaff3ca\") " pod="openshift-ingress/e2e-api-a-4c278a-openshift-default-7667669d89-ns249"
Jul 09 20:39:47.836147 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:47.836141    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b47ec9db-f060-4486-8599-d35bbcaff3ca-istio-data\") pod \"e2e-api-a-4c278a-openshift-default-7667669d89-ns249\" (UID: \"b47ec9db-f060-4486-8599-d35bbcaff3ca\") " pod="openshift-ingress/e2e-api-a-4c278a-openshift-default-7667669d89-ns249"
Jul 09 20:39:47.836397 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:47.836200    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b47ec9db-f060-4486-8599-d35bbcaff3ca-istio-envoy\") pod \"e2e-api-a-4c278a-openshift-default-7667669d89-ns249\" (UID: \"b47ec9db-f060-4486-8599-d35bbcaff3ca\") " pod="openshift-ingress/e2e-api-a-4c278a-openshift-default-7667669d89-ns249"
Jul 09 20:39:47.836397 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:47.836231    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44t2r\" (UniqueName: \"kubernetes.io/projected/b47ec9db-f060-4486-8599-d35bbcaff3ca-kube-api-access-44t2r\") pod \"e2e-api-a-4c278a-openshift-default-7667669d89-ns249\" (UID: \"b47ec9db-f060-4486-8599-d35bbcaff3ca\") " pod="openshift-ingress/e2e-api-a-4c278a-openshift-default-7667669d89-ns249"
Jul 09 20:39:47.937569 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:47.937480    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44t2r\" (UniqueName: \"kubernetes.io/projected/b47ec9db-f060-4486-8599-d35bbcaff3ca-kube-api-access-44t2r\") pod \"e2e-api-a-4c278a-openshift-default-7667669d89-ns249\" (UID: \"b47ec9db-f060-4486-8599-d35bbcaff3ca\") " pod="openshift-ingress/e2e-api-a-4c278a-openshift-default-7667669d89-ns249"
Jul 09 20:39:47.937569 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:47.937546    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b47ec9db-f060-4486-8599-d35bbcaff3ca-workload-certs\") pod \"e2e-api-a-4c278a-openshift-default-7667669d89-ns249\" (UID: \"b47ec9db-f060-4486-8599-d35bbcaff3ca\") " pod="openshift-ingress/e2e-api-a-4c278a-openshift-default-7667669d89-ns249"
Jul 09 20:39:47.937700 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:47.937581    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b47ec9db-f060-4486-8599-d35bbcaff3ca-istiod-ca-cert\") pod \"e2e-api-a-4c278a-openshift-default-7667669d89-ns249\" (UID: \"b47ec9db-f060-4486-8599-d35bbcaff3ca\") " pod="openshift-ingress/e2e-api-a-4c278a-openshift-default-7667669d89-ns249"
Jul 09 20:39:47.937700 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:47.937622    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b47ec9db-f060-4486-8599-d35bbcaff3ca-workload-socket\") pod \"e2e-api-a-4c278a-openshift-default-7667669d89-ns249\" (UID: \"b47ec9db-f060-4486-8599-d35bbcaff3ca\") " pod="openshift-ingress/e2e-api-a-4c278a-openshift-default-7667669d89-ns249"
Jul 09 20:39:47.937700 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:47.937661    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b47ec9db-f060-4486-8599-d35bbcaff3ca-istio-token\") pod \"e2e-api-a-4c278a-openshift-default-7667669d89-ns249\" (UID: \"b47ec9db-f060-4486-8599-d35bbcaff3ca\") " pod="openshift-ingress/e2e-api-a-4c278a-openshift-default-7667669d89-ns249"
Jul 09 20:39:47.937700 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:47.937688    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b47ec9db-f060-4486-8599-d35bbcaff3ca-istio-podinfo\") pod \"e2e-api-a-4c278a-openshift-default-7667669d89-ns249\" (UID: \"b47ec9db-f060-4486-8599-d35bbcaff3ca\") " pod="openshift-ingress/e2e-api-a-4c278a-openshift-default-7667669d89-ns249"
Jul 09 20:39:47.937879 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:47.937714    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b47ec9db-f060-4486-8599-d35bbcaff3ca-credential-socket\") pod \"e2e-api-a-4c278a-openshift-default-7667669d89-ns249\" (UID: \"b47ec9db-f060-4486-8599-d35bbcaff3ca\") " pod="openshift-ingress/e2e-api-a-4c278a-openshift-default-7667669d89-ns249"
Jul 09 20:39:47.937879 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:47.937836    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b47ec9db-f060-4486-8599-d35bbcaff3ca-istio-data\") pod \"e2e-api-a-4c278a-openshift-default-7667669d89-ns249\" (UID: \"b47ec9db-f060-4486-8599-d35bbcaff3ca\") " pod="openshift-ingress/e2e-api-a-4c278a-openshift-default-7667669d89-ns249"
Jul 09 20:39:47.937978 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:47.937929    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b47ec9db-f060-4486-8599-d35bbcaff3ca-istio-envoy\") pod \"e2e-api-a-4c278a-openshift-default-7667669d89-ns249\" (UID: \"b47ec9db-f060-4486-8599-d35bbcaff3ca\") " pod="openshift-ingress/e2e-api-a-4c278a-openshift-default-7667669d89-ns249"
Jul 09 20:39:47.938027 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:47.937992    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b47ec9db-f060-4486-8599-d35bbcaff3ca-workload-certs\") pod \"e2e-api-a-4c278a-openshift-default-7667669d89-ns249\" (UID: \"b47ec9db-f060-4486-8599-d35bbcaff3ca\") " pod="openshift-ingress/e2e-api-a-4c278a-openshift-default-7667669d89-ns249"
Jul 09 20:39:47.938079 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:47.938064    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b47ec9db-f060-4486-8599-d35bbcaff3ca-credential-socket\") pod \"e2e-api-a-4c278a-openshift-default-7667669d89-ns249\" (UID: \"b47ec9db-f060-4486-8599-d35bbcaff3ca\") " pod="openshift-ingress/e2e-api-a-4c278a-openshift-default-7667669d89-ns249"
Jul 09 20:39:47.938210 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:47.938177    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b47ec9db-f060-4486-8599-d35bbcaff3ca-istio-data\") pod \"e2e-api-a-4c278a-openshift-default-7667669d89-ns249\" (UID: \"b47ec9db-f060-4486-8599-d35bbcaff3ca\") " pod="openshift-ingress/e2e-api-a-4c278a-openshift-default-7667669d89-ns249"
Jul 09 20:39:47.938284 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:47.938227    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b47ec9db-f060-4486-8599-d35bbcaff3ca-workload-socket\") pod \"e2e-api-a-4c278a-openshift-default-7667669d89-ns249\" (UID: \"b47ec9db-f060-4486-8599-d35bbcaff3ca\") " pod="openshift-ingress/e2e-api-a-4c278a-openshift-default-7667669d89-ns249"
Jul 09 20:39:47.938393 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:47.938377    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b47ec9db-f060-4486-8599-d35bbcaff3ca-istiod-ca-cert\") pod \"e2e-api-a-4c278a-openshift-default-7667669d89-ns249\" (UID: \"b47ec9db-f060-4486-8599-d35bbcaff3ca\") " pod="openshift-ingress/e2e-api-a-4c278a-openshift-default-7667669d89-ns249"
Jul 09 20:39:47.940087 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:47.940059    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b47ec9db-f060-4486-8599-d35bbcaff3ca-istio-envoy\") pod \"e2e-api-a-4c278a-openshift-default-7667669d89-ns249\" (UID: \"b47ec9db-f060-4486-8599-d35bbcaff3ca\") " pod="openshift-ingress/e2e-api-a-4c278a-openshift-default-7667669d89-ns249"
Jul 09 20:39:47.940411 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:47.940393    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b47ec9db-f060-4486-8599-d35bbcaff3ca-istio-podinfo\") pod \"e2e-api-a-4c278a-openshift-default-7667669d89-ns249\" (UID: \"b47ec9db-f060-4486-8599-d35bbcaff3ca\") " pod="openshift-ingress/e2e-api-a-4c278a-openshift-default-7667669d89-ns249"
Jul 09 20:39:47.946601 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:47.946581    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b47ec9db-f060-4486-8599-d35bbcaff3ca-istio-token\") pod \"e2e-api-a-4c278a-openshift-default-7667669d89-ns249\" (UID: \"b47ec9db-f060-4486-8599-d35bbcaff3ca\") " pod="openshift-ingress/e2e-api-a-4c278a-openshift-default-7667669d89-ns249"
Jul 09 20:39:47.946695 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:47.946609    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-44t2r\" (UniqueName: \"kubernetes.io/projected/b47ec9db-f060-4486-8599-d35bbcaff3ca-kube-api-access-44t2r\") pod \"e2e-api-a-4c278a-openshift-default-7667669d89-ns249\" (UID: \"b47ec9db-f060-4486-8599-d35bbcaff3ca\") " pod="openshift-ingress/e2e-api-a-4c278a-openshift-default-7667669d89-ns249"
Jul 09 20:39:47.986194 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:47.986158    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-4c278a-openshift-default-7667669d89-ns249"
Jul 09 20:39:48.121852 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:48.121823    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-4c278a-openshift-default-7667669d89-ns249"]
Jul 09 20:39:48.122956 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:39:48.122928    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb47ec9db_f060_4486_8599_d35bbcaff3ca.slice/crio-e039272c1bc32698d6959ee12d934e7c7ef63ec22d3e6dcae427c6909160bf45 WatchSource:0}: Error finding container e039272c1bc32698d6959ee12d934e7c7ef63ec22d3e6dcae427c6909160bf45: Status 404 returned error can't find the container with id e039272c1bc32698d6959ee12d934e7c7ef63ec22d3e6dcae427c6909160bf45
Jul 09 20:39:48.125112 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:48.125072    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 09 20:39:48.125185 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:48.125156    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 09 20:39:48.125221 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:48.125196    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 09 20:39:48.990546 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:48.990511    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4c278a-openshift-default-7667669d89-ns249" event={"ID":"b47ec9db-f060-4486-8599-d35bbcaff3ca","Type":"ContainerStarted","Data":"852904c13dd736e352f81abcc29fda1e8f26f805432455d1277a843a938fc390"}
Jul 09 20:39:48.990961 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:48.990566    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4c278a-openshift-default-7667669d89-ns249" event={"ID":"b47ec9db-f060-4486-8599-d35bbcaff3ca","Type":"ContainerStarted","Data":"e039272c1bc32698d6959ee12d934e7c7ef63ec22d3e6dcae427c6909160bf45"}
Jul 09 20:39:49.010565 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:49.010522    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-4c278a-openshift-default-7667669d89-ns249" podStartSLOduration=2.010509331 podStartE2EDuration="2.010509331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 20:39:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 20:39:49.009746886 +0000 UTC m=+2179.427416695" watchObservedRunningTime="2026-07-09 20:39:49.010509331 +0000 UTC m=+2179.428179203"
Jul 09 20:39:49.986699 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:49.986670    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-4c278a-openshift-default-7667669d89-ns249"
Jul 09 20:39:49.991179 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:49.991153    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-4c278a-openshift-default-7667669d89-ns249"
Jul 09 20:39:49.994774 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:49.994750    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-4c278a-openshift-default-7667669d89-ns249"
Jul 09 20:39:49.995282 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:49.995266    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-4c278a-openshift-default-7667669d89-ns249"
Jul 09 20:39:50.964830 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:50.964796    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4c278a-7fd96487c4-2m7pr"]
Jul 09 20:39:50.965338 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:50.965295    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="799ba2bb-8a49-46f2-b985-1e4a1f06e52b" containerName="istio-proxy"
Jul 09 20:39:50.968766 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:50.968743    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4c278a-7fd96487c4-2m7pr"
Jul 09 20:39:50.971797 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:50.971771    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-4c278a-serving-cert\""
Jul 09 20:39:50.977274 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:50.977139    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4c278a-7fd96487c4-2m7pr"]
Jul 09 20:39:51.060931 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:51.060906    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9vgj\" (UniqueName: \"kubernetes.io/projected/0b5a42ab-60d6-42f3-96b6-eaf3b22e8b08-kube-api-access-n9vgj\") pod \"maas-api-e2e-api-a-4c278a-7fd96487c4-2m7pr\" (UID: \"0b5a42ab-60d6-42f3-96b6-eaf3b22e8b08\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4c278a-7fd96487c4-2m7pr"
Jul 09 20:39:51.061231 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:51.060949    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b5a42ab-60d6-42f3-96b6-eaf3b22e8b08-maas-api-tls\") pod \"maas-api-e2e-api-a-4c278a-7fd96487c4-2m7pr\" (UID: \"0b5a42ab-60d6-42f3-96b6-eaf3b22e8b08\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4c278a-7fd96487c4-2m7pr"
Jul 09 20:39:51.061231 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:51.060979    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b5a42ab-60d6-42f3-96b6-eaf3b22e8b08-openshift-service-ca\") pod \"maas-api-e2e-api-a-4c278a-7fd96487c4-2m7pr\" (UID: \"0b5a42ab-60d6-42f3-96b6-eaf3b22e8b08\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4c278a-7fd96487c4-2m7pr"
Jul 09 20:39:51.162206 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:51.162173    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9vgj\" (UniqueName: \"kubernetes.io/projected/0b5a42ab-60d6-42f3-96b6-eaf3b22e8b08-kube-api-access-n9vgj\") pod \"maas-api-e2e-api-a-4c278a-7fd96487c4-2m7pr\" (UID: \"0b5a42ab-60d6-42f3-96b6-eaf3b22e8b08\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4c278a-7fd96487c4-2m7pr"
Jul 09 20:39:51.162400 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:51.162215    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b5a42ab-60d6-42f3-96b6-eaf3b22e8b08-maas-api-tls\") pod \"maas-api-e2e-api-a-4c278a-7fd96487c4-2m7pr\" (UID: \"0b5a42ab-60d6-42f3-96b6-eaf3b22e8b08\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4c278a-7fd96487c4-2m7pr"
Jul 09 20:39:51.162400 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:51.162235    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b5a42ab-60d6-42f3-96b6-eaf3b22e8b08-openshift-service-ca\") pod \"maas-api-e2e-api-a-4c278a-7fd96487c4-2m7pr\" (UID: \"0b5a42ab-60d6-42f3-96b6-eaf3b22e8b08\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4c278a-7fd96487c4-2m7pr"
Jul 09 20:39:51.162400 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:39:51.162330    2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-4c278a-serving-cert: secret "maas-api-e2e-api-a-4c278a-serving-cert" not found
Jul 09 20:39:51.162400 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:39:51.162392    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b5a42ab-60d6-42f3-96b6-eaf3b22e8b08-maas-api-tls podName:0b5a42ab-60d6-42f3-96b6-eaf3b22e8b08 nodeName:}" failed. No retries permitted until 2026-07-09 20:39:51.662377988 +0000 UTC m=+2182.080047774 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0b5a42ab-60d6-42f3-96b6-eaf3b22e8b08-maas-api-tls") pod "maas-api-e2e-api-a-4c278a-7fd96487c4-2m7pr" (UID: "0b5a42ab-60d6-42f3-96b6-eaf3b22e8b08") : secret "maas-api-e2e-api-a-4c278a-serving-cert" not found
Jul 09 20:39:51.162826 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:51.162810    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b5a42ab-60d6-42f3-96b6-eaf3b22e8b08-openshift-service-ca\") pod \"maas-api-e2e-api-a-4c278a-7fd96487c4-2m7pr\" (UID: \"0b5a42ab-60d6-42f3-96b6-eaf3b22e8b08\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4c278a-7fd96487c4-2m7pr"
Jul 09 20:39:51.192400 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:51.192376    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9vgj\" (UniqueName: \"kubernetes.io/projected/0b5a42ab-60d6-42f3-96b6-eaf3b22e8b08-kube-api-access-n9vgj\") pod \"maas-api-e2e-api-a-4c278a-7fd96487c4-2m7pr\" (UID: \"0b5a42ab-60d6-42f3-96b6-eaf3b22e8b08\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4c278a-7fd96487c4-2m7pr"
Jul 09 20:39:51.665990 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:51.665958    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b5a42ab-60d6-42f3-96b6-eaf3b22e8b08-maas-api-tls\") pod \"maas-api-e2e-api-a-4c278a-7fd96487c4-2m7pr\" (UID: \"0b5a42ab-60d6-42f3-96b6-eaf3b22e8b08\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4c278a-7fd96487c4-2m7pr"
Jul 09 20:39:51.668911 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:51.668881    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b5a42ab-60d6-42f3-96b6-eaf3b22e8b08-maas-api-tls\") pod \"maas-api-e2e-api-a-4c278a-7fd96487c4-2m7pr\" (UID: \"0b5a42ab-60d6-42f3-96b6-eaf3b22e8b08\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4c278a-7fd96487c4-2m7pr"
Jul 09 20:39:51.881292 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:51.881263    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4c278a-7fd96487c4-2m7pr"
Jul 09 20:39:52.009744 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:52.009714    2562 generic.go:356] "Generic (PLEG): container finished" podID="d80fc5f7-0dfb-49e6-b168-5da6b11a12f4" containerID="291ca0108331e7e9d17360346a6b076f40bb30663ce5dfa4d5571a2e5e73339f" exitCode=0
Jul 09 20:39:52.009894 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:52.009788    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-00edb0-openshift-default-57cdc9785-6zxgq" event={"ID":"d80fc5f7-0dfb-49e6-b168-5da6b11a12f4","Type":"ContainerDied","Data":"291ca0108331e7e9d17360346a6b076f40bb30663ce5dfa4d5571a2e5e73339f"}
Jul 09 20:39:52.009894 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:52.009865    2562 scope.go:117] "RemoveContainer" containerID="9c99c8f437341f56d4507a61dbfa553363dbe8c8e49dd6af984bff22471825be"
Jul 09 20:39:52.034950 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:52.034920    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4c278a-7fd96487c4-2m7pr"]
Jul 09 20:39:52.037700 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:39:52.037674    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b5a42ab_60d6_42f3_96b6_eaf3b22e8b08.slice/crio-c29089d59589dc4a57ef61e21910a17c0fa00df0a12cb63bbdc9175313e33376 WatchSource:0}: Error finding container c29089d59589dc4a57ef61e21910a17c0fa00df0a12cb63bbdc9175313e33376: Status 404 returned error can't find the container with id c29089d59589dc4a57ef61e21910a17c0fa00df0a12cb63bbdc9175313e33376
Jul 09 20:39:52.043371 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:52.043353    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-00edb0-openshift-default-57cdc9785-6zxgq"
Jul 09 20:39:52.069160 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:52.069141    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d80fc5f7-0dfb-49e6-b168-5da6b11a12f4-istio-data\") pod \"d80fc5f7-0dfb-49e6-b168-5da6b11a12f4\" (UID: \"d80fc5f7-0dfb-49e6-b168-5da6b11a12f4\") "
Jul 09 20:39:52.069449 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:52.069176    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d80fc5f7-0dfb-49e6-b168-5da6b11a12f4-istiod-ca-cert\") pod \"d80fc5f7-0dfb-49e6-b168-5da6b11a12f4\" (UID: \"d80fc5f7-0dfb-49e6-b168-5da6b11a12f4\") "
Jul 09 20:39:52.069664 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:52.069637    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d80fc5f7-0dfb-49e6-b168-5da6b11a12f4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d80fc5f7-0dfb-49e6-b168-5da6b11a12f4" (UID: "d80fc5f7-0dfb-49e6-b168-5da6b11a12f4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 20:39:52.069949 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:52.069930    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d80fc5f7-0dfb-49e6-b168-5da6b11a12f4-istio-data" (OuterVolumeSpecName: "istio-data") pod "d80fc5f7-0dfb-49e6-b168-5da6b11a12f4" (UID: "d80fc5f7-0dfb-49e6-b168-5da6b11a12f4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 20:39:52.169879 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:52.169855    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8b59j\" (UniqueName: \"kubernetes.io/projected/d80fc5f7-0dfb-49e6-b168-5da6b11a12f4-kube-api-access-8b59j\") pod \"d80fc5f7-0dfb-49e6-b168-5da6b11a12f4\" (UID: \"d80fc5f7-0dfb-49e6-b168-5da6b11a12f4\") "
Jul 09 20:39:52.169980 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:52.169904    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d80fc5f7-0dfb-49e6-b168-5da6b11a12f4-istio-envoy\") pod \"d80fc5f7-0dfb-49e6-b168-5da6b11a12f4\" (UID: \"d80fc5f7-0dfb-49e6-b168-5da6b11a12f4\") "
Jul 09 20:39:52.169980 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:52.169927    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d80fc5f7-0dfb-49e6-b168-5da6b11a12f4-credential-socket\") pod \"d80fc5f7-0dfb-49e6-b168-5da6b11a12f4\" (UID: \"d80fc5f7-0dfb-49e6-b168-5da6b11a12f4\") "
Jul 09 20:39:52.169980 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:52.169958    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d80fc5f7-0dfb-49e6-b168-5da6b11a12f4-istio-podinfo\") pod \"d80fc5f7-0dfb-49e6-b168-5da6b11a12f4\" (UID: \"d80fc5f7-0dfb-49e6-b168-5da6b11a12f4\") "
Jul 09 20:39:52.170132 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:52.169983    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d80fc5f7-0dfb-49e6-b168-5da6b11a12f4-workload-socket\") pod \"d80fc5f7-0dfb-49e6-b168-5da6b11a12f4\" (UID: \"d80fc5f7-0dfb-49e6-b168-5da6b11a12f4\") "
Jul 09 20:39:52.170132 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:52.170005    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d80fc5f7-0dfb-49e6-b168-5da6b11a12f4-istio-token\") pod \"d80fc5f7-0dfb-49e6-b168-5da6b11a12f4\" (UID: \"d80fc5f7-0dfb-49e6-b168-5da6b11a12f4\") "
Jul 09 20:39:52.170132 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:52.170050    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d80fc5f7-0dfb-49e6-b168-5da6b11a12f4-workload-certs\") pod \"d80fc5f7-0dfb-49e6-b168-5da6b11a12f4\" (UID: \"d80fc5f7-0dfb-49e6-b168-5da6b11a12f4\") "
Jul 09 20:39:52.170278 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:52.170156    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d80fc5f7-0dfb-49e6-b168-5da6b11a12f4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d80fc5f7-0dfb-49e6-b168-5da6b11a12f4" (UID: "d80fc5f7-0dfb-49e6-b168-5da6b11a12f4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 20:39:52.170278 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:52.170249    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d80fc5f7-0dfb-49e6-b168-5da6b11a12f4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d80fc5f7-0dfb-49e6-b168-5da6b11a12f4" (UID: "d80fc5f7-0dfb-49e6-b168-5da6b11a12f4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 20:39:52.170418 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:52.170399    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d80fc5f7-0dfb-49e6-b168-5da6b11a12f4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d80fc5f7-0dfb-49e6-b168-5da6b11a12f4" (UID: "d80fc5f7-0dfb-49e6-b168-5da6b11a12f4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 20:39:52.170481 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:52.170463    2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d80fc5f7-0dfb-49e6-b168-5da6b11a12f4-credential-socket\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:39:52.170532 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:52.170487    2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d80fc5f7-0dfb-49e6-b168-5da6b11a12f4-workload-socket\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:39:52.170532 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:52.170501    2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d80fc5f7-0dfb-49e6-b168-5da6b11a12f4-istio-data\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:39:52.170532 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:52.170513    2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d80fc5f7-0dfb-49e6-b168-5da6b11a12f4-istiod-ca-cert\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:39:52.171862 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:52.171843    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d80fc5f7-0dfb-49e6-b168-5da6b11a12f4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d80fc5f7-0dfb-49e6-b168-5da6b11a12f4" (UID: "d80fc5f7-0dfb-49e6-b168-5da6b11a12f4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 20:39:52.171918 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:52.171897    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d80fc5f7-0dfb-49e6-b168-5da6b11a12f4-kube-api-access-8b59j" (OuterVolumeSpecName: "kube-api-access-8b59j") pod "d80fc5f7-0dfb-49e6-b168-5da6b11a12f4" (UID: "d80fc5f7-0dfb-49e6-b168-5da6b11a12f4"). InnerVolumeSpecName "kube-api-access-8b59j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 20:39:52.172276 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:52.172253    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d80fc5f7-0dfb-49e6-b168-5da6b11a12f4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d80fc5f7-0dfb-49e6-b168-5da6b11a12f4" (UID: "d80fc5f7-0dfb-49e6-b168-5da6b11a12f4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 09 20:39:52.172380 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:52.172256    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d80fc5f7-0dfb-49e6-b168-5da6b11a12f4-istio-token" (OuterVolumeSpecName: "istio-token") pod "d80fc5f7-0dfb-49e6-b168-5da6b11a12f4" (UID: "d80fc5f7-0dfb-49e6-b168-5da6b11a12f4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 20:39:52.270870 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:52.270850    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8b59j\" (UniqueName: \"kubernetes.io/projected/d80fc5f7-0dfb-49e6-b168-5da6b11a12f4-kube-api-access-8b59j\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:39:52.270870 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:52.270872    2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d80fc5f7-0dfb-49e6-b168-5da6b11a12f4-istio-envoy\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:39:52.270983 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:52.270880    2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d80fc5f7-0dfb-49e6-b168-5da6b11a12f4-istio-podinfo\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:39:52.270983 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:52.270888    2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d80fc5f7-0dfb-49e6-b168-5da6b11a12f4-istio-token\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:39:52.270983 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:52.270896    2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d80fc5f7-0dfb-49e6-b168-5da6b11a12f4-workload-certs\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:39:53.015575 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:53.015547    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4c278a-7fd96487c4-2m7pr" event={"ID":"0b5a42ab-60d6-42f3-96b6-eaf3b22e8b08","Type":"ContainerStarted","Data":"c4df296fc8137ea7aa4106c39ce4b36e2076be772a2fdfb6bb321f3694850107"}
Jul 09 20:39:53.015745 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:53.015591    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4c278a-7fd96487c4-2m7pr" event={"ID":"0b5a42ab-60d6-42f3-96b6-eaf3b22e8b08","Type":"ContainerStarted","Data":"c29089d59589dc4a57ef61e21910a17c0fa00df0a12cb63bbdc9175313e33376"}
Jul 09 20:39:53.015745 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:53.015635    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4c278a-7fd96487c4-2m7pr"
Jul 09 20:39:53.016866 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:53.016847    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-00edb0-openshift-default-57cdc9785-6zxgq"
Jul 09 20:39:53.016866 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:53.016856    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-00edb0-openshift-default-57cdc9785-6zxgq" event={"ID":"d80fc5f7-0dfb-49e6-b168-5da6b11a12f4","Type":"ContainerDied","Data":"8876a773f07322d9ed3b8827144d4b07def1a22303cdb2a05cac00b9e2220d55"}
Jul 09 20:39:53.017019 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:53.016902    2562 scope.go:117] "RemoveContainer" containerID="291ca0108331e7e9d17360346a6b076f40bb30663ce5dfa4d5571a2e5e73339f"
Jul 09 20:39:53.033213 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:53.033173    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4c278a-7fd96487c4-2m7pr" podStartSLOduration=2.596308907 podStartE2EDuration="3.033160519s" totalImagesPullingTime="436.851612ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 20:39:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 20:39:53.031780725 +0000 UTC m=+2183.449450528" watchObservedRunningTime="2026-07-09 20:39:53.033160519 +0000 UTC m=+2183.450830359"
Jul 09 20:39:53.046165 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:53.046141    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-00edb0-openshift-default-57cdc9785-6zxgq"]
Jul 09 20:39:53.050541 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:53.050520    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-00edb0-openshift-default-57cdc9785-6zxgq"]
Jul 09 20:39:54.241166 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:54.241141    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d80fc5f7-0dfb-49e6-b168-5da6b11a12f4" path="/var/lib/kubelet/pods/d80fc5f7-0dfb-49e6-b168-5da6b11a12f4/volumes"
Jul 09 20:39:56.192590 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:56.192549    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-9df7ce-openshift-default-b7fbc5db5-9j65l"]
Jul 09 20:39:56.193190 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:56.193170    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d80fc5f7-0dfb-49e6-b168-5da6b11a12f4" containerName="istio-proxy"
Jul 09 20:39:56.193283 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:56.193194    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d80fc5f7-0dfb-49e6-b168-5da6b11a12f4" containerName="istio-proxy"
Jul 09 20:39:56.193283 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:56.193218    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d80fc5f7-0dfb-49e6-b168-5da6b11a12f4" containerName="istio-proxy"
Jul 09 20:39:56.193283 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:56.193227    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d80fc5f7-0dfb-49e6-b168-5da6b11a12f4" containerName="istio-proxy"
Jul 09 20:39:56.193490 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:56.193329    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="d80fc5f7-0dfb-49e6-b168-5da6b11a12f4" containerName="istio-proxy"
Jul 09 20:39:56.193490 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:56.193343    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="d80fc5f7-0dfb-49e6-b168-5da6b11a12f4" containerName="istio-proxy"
Jul 09 20:39:56.198092 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:56.198072    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-9df7ce-openshift-default-b7fbc5db5-9j65l"
Jul 09 20:39:56.199737 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:56.199713    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/366ce8c4-6f65-4d55-849e-6fd02a8526a0-istio-token\") pod \"e2e-api-b-9df7ce-openshift-default-b7fbc5db5-9j65l\" (UID: \"366ce8c4-6f65-4d55-849e-6fd02a8526a0\") " pod="openshift-ingress/e2e-api-b-9df7ce-openshift-default-b7fbc5db5-9j65l"
Jul 09 20:39:56.199840 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:56.199751    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/366ce8c4-6f65-4d55-849e-6fd02a8526a0-istio-data\") pod \"e2e-api-b-9df7ce-openshift-default-b7fbc5db5-9j65l\" (UID: \"366ce8c4-6f65-4d55-849e-6fd02a8526a0\") " pod="openshift-ingress/e2e-api-b-9df7ce-openshift-default-b7fbc5db5-9j65l"
Jul 09 20:39:56.199840 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:56.199812    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/366ce8c4-6f65-4d55-849e-6fd02a8526a0-istiod-ca-cert\") pod \"e2e-api-b-9df7ce-openshift-default-b7fbc5db5-9j65l\" (UID: \"366ce8c4-6f65-4d55-849e-6fd02a8526a0\") " pod="openshift-ingress/e2e-api-b-9df7ce-openshift-default-b7fbc5db5-9j65l"
Jul 09 20:39:56.199949 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:56.199842    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54d77\" (UniqueName: \"kubernetes.io/projected/366ce8c4-6f65-4d55-849e-6fd02a8526a0-kube-api-access-54d77\") pod \"e2e-api-b-9df7ce-openshift-default-b7fbc5db5-9j65l\" (UID: \"366ce8c4-6f65-4d55-849e-6fd02a8526a0\") " pod="openshift-ingress/e2e-api-b-9df7ce-openshift-default-b7fbc5db5-9j65l"
Jul 09 20:39:56.199989 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:56.199938    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/366ce8c4-6f65-4d55-849e-6fd02a8526a0-credential-socket\") pod \"e2e-api-b-9df7ce-openshift-default-b7fbc5db5-9j65l\" (UID: \"366ce8c4-6f65-4d55-849e-6fd02a8526a0\") " pod="openshift-ingress/e2e-api-b-9df7ce-openshift-default-b7fbc5db5-9j65l"
Jul 09 20:39:56.199989 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:56.199974    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/366ce8c4-6f65-4d55-849e-6fd02a8526a0-istio-podinfo\") pod \"e2e-api-b-9df7ce-openshift-default-b7fbc5db5-9j65l\" (UID: \"366ce8c4-6f65-4d55-849e-6fd02a8526a0\") " pod="openshift-ingress/e2e-api-b-9df7ce-openshift-default-b7fbc5db5-9j65l"
Jul 09 20:39:56.200084 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:56.200063    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/366ce8c4-6f65-4d55-849e-6fd02a8526a0-workload-socket\") pod \"e2e-api-b-9df7ce-openshift-default-b7fbc5db5-9j65l\" (UID: \"366ce8c4-6f65-4d55-849e-6fd02a8526a0\") " pod="openshift-ingress/e2e-api-b-9df7ce-openshift-default-b7fbc5db5-9j65l"
Jul 09 20:39:56.200203 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:56.200097    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/366ce8c4-6f65-4d55-849e-6fd02a8526a0-istio-envoy\") pod \"e2e-api-b-9df7ce-openshift-default-b7fbc5db5-9j65l\" (UID: \"366ce8c4-6f65-4d55-849e-6fd02a8526a0\") " pod="openshift-ingress/e2e-api-b-9df7ce-openshift-default-b7fbc5db5-9j65l"
Jul 09 20:39:56.200203 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:56.200132    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/366ce8c4-6f65-4d55-849e-6fd02a8526a0-workload-certs\") pod \"e2e-api-b-9df7ce-openshift-default-b7fbc5db5-9j65l\" (UID: \"366ce8c4-6f65-4d55-849e-6fd02a8526a0\") " pod="openshift-ingress/e2e-api-b-9df7ce-openshift-default-b7fbc5db5-9j65l"
Jul 09 20:39:56.201257 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:56.201237    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-9df7ce-openshift-default-dockercfg-4wnkt\""
Jul 09 20:39:56.207653 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:56.206771    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-9df7ce-openshift-default-b7fbc5db5-9j65l"]
Jul 09 20:39:56.301206 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:56.301179    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/366ce8c4-6f65-4d55-849e-6fd02a8526a0-istio-token\") pod \"e2e-api-b-9df7ce-openshift-default-b7fbc5db5-9j65l\" (UID: \"366ce8c4-6f65-4d55-849e-6fd02a8526a0\") " pod="openshift-ingress/e2e-api-b-9df7ce-openshift-default-b7fbc5db5-9j65l"
Jul 09 20:39:56.301206 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:56.301206    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/366ce8c4-6f65-4d55-849e-6fd02a8526a0-istio-data\") pod \"e2e-api-b-9df7ce-openshift-default-b7fbc5db5-9j65l\" (UID: \"366ce8c4-6f65-4d55-849e-6fd02a8526a0\") " pod="openshift-ingress/e2e-api-b-9df7ce-openshift-default-b7fbc5db5-9j65l"
Jul 09 20:39:56.301464 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:56.301244    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/366ce8c4-6f65-4d55-849e-6fd02a8526a0-istiod-ca-cert\") pod \"e2e-api-b-9df7ce-openshift-default-b7fbc5db5-9j65l\" (UID: \"366ce8c4-6f65-4d55-849e-6fd02a8526a0\") " pod="openshift-ingress/e2e-api-b-9df7ce-openshift-default-b7fbc5db5-9j65l"
Jul 09 20:39:56.301464 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:56.301263    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54d77\" (UniqueName: \"kubernetes.io/projected/366ce8c4-6f65-4d55-849e-6fd02a8526a0-kube-api-access-54d77\") pod \"e2e-api-b-9df7ce-openshift-default-b7fbc5db5-9j65l\" (UID: \"366ce8c4-6f65-4d55-849e-6fd02a8526a0\") " pod="openshift-ingress/e2e-api-b-9df7ce-openshift-default-b7fbc5db5-9j65l"
Jul 09 20:39:56.301464 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:56.301289    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/366ce8c4-6f65-4d55-849e-6fd02a8526a0-credential-socket\") pod \"e2e-api-b-9df7ce-openshift-default-b7fbc5db5-9j65l\" (UID: \"366ce8c4-6f65-4d55-849e-6fd02a8526a0\") " pod="openshift-ingress/e2e-api-b-9df7ce-openshift-default-b7fbc5db5-9j65l"
Jul 09 20:39:56.301464 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:56.301333    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/366ce8c4-6f65-4d55-849e-6fd02a8526a0-istio-podinfo\") pod \"e2e-api-b-9df7ce-openshift-default-b7fbc5db5-9j65l\" (UID: \"366ce8c4-6f65-4d55-849e-6fd02a8526a0\") " pod="openshift-ingress/e2e-api-b-9df7ce-openshift-default-b7fbc5db5-9j65l"
Jul 09 20:39:56.301464 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:56.301461    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/366ce8c4-6f65-4d55-849e-6fd02a8526a0-workload-socket\") pod \"e2e-api-b-9df7ce-openshift-default-b7fbc5db5-9j65l\" (UID: \"366ce8c4-6f65-4d55-849e-6fd02a8526a0\") " pod="openshift-ingress/e2e-api-b-9df7ce-openshift-default-b7fbc5db5-9j65l"
Jul 09 20:39:56.301792 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:56.301520    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/366ce8c4-6f65-4d55-849e-6fd02a8526a0-istio-envoy\") pod \"e2e-api-b-9df7ce-openshift-default-b7fbc5db5-9j65l\" (UID: \"366ce8c4-6f65-4d55-849e-6fd02a8526a0\") " pod="openshift-ingress/e2e-api-b-9df7ce-openshift-default-b7fbc5db5-9j65l"
Jul 09 20:39:56.301792 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:56.301568    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/366ce8c4-6f65-4d55-849e-6fd02a8526a0-workload-certs\") pod \"e2e-api-b-9df7ce-openshift-default-b7fbc5db5-9j65l\" (UID: \"366ce8c4-6f65-4d55-849e-6fd02a8526a0\") " pod="openshift-ingress/e2e-api-b-9df7ce-openshift-default-b7fbc5db5-9j65l"
Jul 09 20:39:56.301792 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:56.301772    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/366ce8c4-6f65-4d55-849e-6fd02a8526a0-istio-data\") pod \"e2e-api-b-9df7ce-openshift-default-b7fbc5db5-9j65l\" (UID: \"366ce8c4-6f65-4d55-849e-6fd02a8526a0\") " pod="openshift-ingress/e2e-api-b-9df7ce-openshift-default-b7fbc5db5-9j65l"
Jul 09 20:39:56.301962 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:56.301822    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/366ce8c4-6f65-4d55-849e-6fd02a8526a0-credential-socket\") pod \"e2e-api-b-9df7ce-openshift-default-b7fbc5db5-9j65l\" (UID: \"366ce8c4-6f65-4d55-849e-6fd02a8526a0\") " pod="openshift-ingress/e2e-api-b-9df7ce-openshift-default-b7fbc5db5-9j65l"
Jul 09 20:39:56.302068 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:56.301988    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/366ce8c4-6f65-4d55-849e-6fd02a8526a0-workload-certs\") pod \"e2e-api-b-9df7ce-openshift-default-b7fbc5db5-9j65l\" (UID: \"366ce8c4-6f65-4d55-849e-6fd02a8526a0\") " pod="openshift-ingress/e2e-api-b-9df7ce-openshift-default-b7fbc5db5-9j65l"
Jul 09 20:39:56.302068 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:56.302031    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/366ce8c4-6f65-4d55-849e-6fd02a8526a0-workload-socket\") pod \"e2e-api-b-9df7ce-openshift-default-b7fbc5db5-9j65l\" (UID: \"366ce8c4-6f65-4d55-849e-6fd02a8526a0\") " pod="openshift-ingress/e2e-api-b-9df7ce-openshift-default-b7fbc5db5-9j65l"
Jul 09 20:39:56.302494 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:56.302442    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/366ce8c4-6f65-4d55-849e-6fd02a8526a0-istiod-ca-cert\") pod \"e2e-api-b-9df7ce-openshift-default-b7fbc5db5-9j65l\" (UID: \"366ce8c4-6f65-4d55-849e-6fd02a8526a0\") " pod="openshift-ingress/e2e-api-b-9df7ce-openshift-default-b7fbc5db5-9j65l"
Jul 09 20:39:56.304113 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:56.304092    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/366ce8c4-6f65-4d55-849e-6fd02a8526a0-istio-envoy\") pod \"e2e-api-b-9df7ce-openshift-default-b7fbc5db5-9j65l\" (UID: \"366ce8c4-6f65-4d55-849e-6fd02a8526a0\") " pod="openshift-ingress/e2e-api-b-9df7ce-openshift-default-b7fbc5db5-9j65l"
Jul 09 20:39:56.304673 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:56.304653    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/366ce8c4-6f65-4d55-849e-6fd02a8526a0-istio-podinfo\") pod \"e2e-api-b-9df7ce-openshift-default-b7fbc5db5-9j65l\" (UID: \"366ce8c4-6f65-4d55-849e-6fd02a8526a0\") " pod="openshift-ingress/e2e-api-b-9df7ce-openshift-default-b7fbc5db5-9j65l"
Jul 09 20:39:56.308929 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:56.308899    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/366ce8c4-6f65-4d55-849e-6fd02a8526a0-istio-token\") pod \"e2e-api-b-9df7ce-openshift-default-b7fbc5db5-9j65l\" (UID: \"366ce8c4-6f65-4d55-849e-6fd02a8526a0\") " pod="openshift-ingress/e2e-api-b-9df7ce-openshift-default-b7fbc5db5-9j65l"
Jul 09 20:39:56.309236 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:56.309220    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-54d77\" (UniqueName: \"kubernetes.io/projected/366ce8c4-6f65-4d55-849e-6fd02a8526a0-kube-api-access-54d77\") pod \"e2e-api-b-9df7ce-openshift-default-b7fbc5db5-9j65l\" (UID: \"366ce8c4-6f65-4d55-849e-6fd02a8526a0\") " pod="openshift-ingress/e2e-api-b-9df7ce-openshift-default-b7fbc5db5-9j65l"
Jul 09 20:39:56.511678 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:56.511654    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-9df7ce-openshift-default-b7fbc5db5-9j65l"
Jul 09 20:39:56.638763 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:56.638735    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-9df7ce-openshift-default-b7fbc5db5-9j65l"]
Jul 09 20:39:56.639070 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:39:56.639050    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod366ce8c4_6f65_4d55_849e_6fd02a8526a0.slice/crio-326a4dd93e54ae5ab5b6c4bbcc06c4e2cabfae93ccea096e287dcf58a979beb6 WatchSource:0}: Error finding container 326a4dd93e54ae5ab5b6c4bbcc06c4e2cabfae93ccea096e287dcf58a979beb6: Status 404 returned error can't find the container with id 326a4dd93e54ae5ab5b6c4bbcc06c4e2cabfae93ccea096e287dcf58a979beb6
Jul 09 20:39:56.641219 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:56.641156    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 09 20:39:56.641219 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:56.641218    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 09 20:39:56.641453 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:56.641245    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 09 20:39:57.037906 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:57.037855    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9df7ce-openshift-default-b7fbc5db5-9j65l" event={"ID":"366ce8c4-6f65-4d55-849e-6fd02a8526a0","Type":"ContainerStarted","Data":"3c5939cbb7801d52270847df51b7d95b3b5e8ff567e77ee82420e8744be03dd3"}
Jul 09 20:39:57.038042 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:57.037925    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9df7ce-openshift-default-b7fbc5db5-9j65l" event={"ID":"366ce8c4-6f65-4d55-849e-6fd02a8526a0","Type":"ContainerStarted","Data":"326a4dd93e54ae5ab5b6c4bbcc06c4e2cabfae93ccea096e287dcf58a979beb6"}
Jul 09 20:39:57.058089 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:57.058047    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-9df7ce-openshift-default-b7fbc5db5-9j65l" podStartSLOduration=1.058033882 podStartE2EDuration="1.058033882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 20:39:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 20:39:57.055632627 +0000 UTC m=+2187.473302433" watchObservedRunningTime="2026-07-09 20:39:57.058033882 +0000 UTC m=+2187.475703748"
Jul 09 20:39:57.512345 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:57.512324    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-9df7ce-openshift-default-b7fbc5db5-9j65l"
Jul 09 20:39:57.516808 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:57.516786    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-9df7ce-openshift-default-b7fbc5db5-9j65l"
Jul 09 20:39:58.043356 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:58.043326    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-9df7ce-openshift-default-b7fbc5db5-9j65l"
Jul 09 20:39:58.044102 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:58.044085    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-9df7ce-openshift-default-b7fbc5db5-9j65l"
Jul 09 20:39:59.028089 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:59.028059    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4c278a-7fd96487c4-2m7pr"
Jul 09 20:39:59.036490 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:59.036459    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-9df7ce-74566d9c74-b5lvb"]
Jul 09 20:39:59.047625 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:59.047583    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9df7ce-74566d9c74-b5lvb"
Jul 09 20:39:59.049784 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:59.049649    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-9df7ce-74566d9c74-b5lvb"]
Jul 09 20:39:59.052257 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:59.052231    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-9df7ce-serving-cert\""
Jul 09 20:39:59.122734 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:59.122697    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70a603a6-f438-4ed1-b185-f7cc2802b3d2-openshift-service-ca\") pod \"maas-api-e2e-api-b-9df7ce-74566d9c74-b5lvb\" (UID: \"70a603a6-f438-4ed1-b185-f7cc2802b3d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9df7ce-74566d9c74-b5lvb"
Jul 09 20:39:59.122859 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:59.122747    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qkt4\" (UniqueName: \"kubernetes.io/projected/70a603a6-f438-4ed1-b185-f7cc2802b3d2-kube-api-access-6qkt4\") pod \"maas-api-e2e-api-b-9df7ce-74566d9c74-b5lvb\" (UID: \"70a603a6-f438-4ed1-b185-f7cc2802b3d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9df7ce-74566d9c74-b5lvb"
Jul 09 20:39:59.122859 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:59.122773    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70a603a6-f438-4ed1-b185-f7cc2802b3d2-maas-api-tls\") pod \"maas-api-e2e-api-b-9df7ce-74566d9c74-b5lvb\" (UID: \"70a603a6-f438-4ed1-b185-f7cc2802b3d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9df7ce-74566d9c74-b5lvb"
Jul 09 20:39:59.223353 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:59.223327    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70a603a6-f438-4ed1-b185-f7cc2802b3d2-openshift-service-ca\") pod \"maas-api-e2e-api-b-9df7ce-74566d9c74-b5lvb\" (UID: \"70a603a6-f438-4ed1-b185-f7cc2802b3d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9df7ce-74566d9c74-b5lvb"
Jul 09 20:39:59.223471 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:59.223364    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qkt4\" (UniqueName: \"kubernetes.io/projected/70a603a6-f438-4ed1-b185-f7cc2802b3d2-kube-api-access-6qkt4\") pod \"maas-api-e2e-api-b-9df7ce-74566d9c74-b5lvb\" (UID: \"70a603a6-f438-4ed1-b185-f7cc2802b3d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9df7ce-74566d9c74-b5lvb"
Jul 09 20:39:59.223471 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:59.223396    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70a603a6-f438-4ed1-b185-f7cc2802b3d2-maas-api-tls\") pod \"maas-api-e2e-api-b-9df7ce-74566d9c74-b5lvb\" (UID: \"70a603a6-f438-4ed1-b185-f7cc2802b3d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9df7ce-74566d9c74-b5lvb"
Jul 09 20:39:59.224002 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:59.223985    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70a603a6-f438-4ed1-b185-f7cc2802b3d2-openshift-service-ca\") pod \"maas-api-e2e-api-b-9df7ce-74566d9c74-b5lvb\" (UID: \"70a603a6-f438-4ed1-b185-f7cc2802b3d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9df7ce-74566d9c74-b5lvb"
Jul 09 20:39:59.225578 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:59.225559    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70a603a6-f438-4ed1-b185-f7cc2802b3d2-maas-api-tls\") pod \"maas-api-e2e-api-b-9df7ce-74566d9c74-b5lvb\" (UID: \"70a603a6-f438-4ed1-b185-f7cc2802b3d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9df7ce-74566d9c74-b5lvb"
Jul 09 20:39:59.239639 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:59.239619    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qkt4\" (UniqueName: \"kubernetes.io/projected/70a603a6-f438-4ed1-b185-f7cc2802b3d2-kube-api-access-6qkt4\") pod \"maas-api-e2e-api-b-9df7ce-74566d9c74-b5lvb\" (UID: \"70a603a6-f438-4ed1-b185-f7cc2802b3d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9df7ce-74566d9c74-b5lvb"
Jul 09 20:39:59.360913 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:59.360830    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9df7ce-74566d9c74-b5lvb"
Jul 09 20:39:59.498027 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:39:59.498002    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-9df7ce-74566d9c74-b5lvb"]
Jul 09 20:39:59.499934 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:39:59.499908    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70a603a6_f438_4ed1_b185_f7cc2802b3d2.slice/crio-38aed225f0d2f86490f3d530a1caba3844527cd1de4a8f42407cf062c1157afa WatchSource:0}: Error finding container 38aed225f0d2f86490f3d530a1caba3844527cd1de4a8f42407cf062c1157afa: Status 404 returned error can't find the container with id 38aed225f0d2f86490f3d530a1caba3844527cd1de4a8f42407cf062c1157afa
Jul 09 20:40:00.062453 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:00.062422    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9df7ce-74566d9c74-b5lvb" event={"ID":"70a603a6-f438-4ed1-b185-f7cc2802b3d2","Type":"ContainerStarted","Data":"74dfde2835a2607b9718d2b746582337e8961bcd1dd4e45d2e0a49bfb8e9a1ec"}
Jul 09 20:40:00.062855 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:00.062473    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9df7ce-74566d9c74-b5lvb" event={"ID":"70a603a6-f438-4ed1-b185-f7cc2802b3d2","Type":"ContainerStarted","Data":"38aed225f0d2f86490f3d530a1caba3844527cd1de4a8f42407cf062c1157afa"}
Jul 09 20:40:00.062855 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:00.062514    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9df7ce-74566d9c74-b5lvb"
Jul 09 20:40:00.103314 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:00.103265    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9df7ce-74566d9c74-b5lvb" podStartSLOduration=0.70444219 podStartE2EDuration="1.103251063s" totalImagesPullingTime="398.808873ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 20:39:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 20:40:00.100601213 +0000 UTC m=+2190.518271019" watchObservedRunningTime="2026-07-09 20:40:00.103251063 +0000 UTC m=+2190.520920869"
Jul 09 20:40:06.036449 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:06.036409    2562 patch_prober.go:28] interesting pod/e2e-api-a-4c278a-openshift-default-7667669d89-ns249 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 09 20:40:06.036832 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:06.036478    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-4c278a-openshift-default-7667669d89-ns249" podUID="b47ec9db-f060-4486-8599-d35bbcaff3ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 09 20:40:06.072440 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:06.072416    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9df7ce-74566d9c74-b5lvb"
Jul 09 20:40:07.098114 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:07.098087    2562 generic.go:356] "Generic (PLEG): container finished" podID="b47ec9db-f060-4486-8599-d35bbcaff3ca" containerID="852904c13dd736e352f81abcc29fda1e8f26f805432455d1277a843a938fc390" exitCode=0
Jul 09 20:40:07.098498 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:07.098162    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4c278a-openshift-default-7667669d89-ns249" event={"ID":"b47ec9db-f060-4486-8599-d35bbcaff3ca","Type":"ContainerDied","Data":"852904c13dd736e352f81abcc29fda1e8f26f805432455d1277a843a938fc390"}
Jul 09 20:40:07.098498 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:07.098495    2562 scope.go:117] "RemoveContainer" containerID="852904c13dd736e352f81abcc29fda1e8f26f805432455d1277a843a938fc390"
Jul 09 20:40:07.099449 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:07.099421    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 09 20:40:07.099534 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:07.099500    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 09 20:40:07.099598 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:07.099548    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 09 20:40:08.105619 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:08.105588    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4c278a-openshift-default-7667669d89-ns249" event={"ID":"b47ec9db-f060-4486-8599-d35bbcaff3ca","Type":"ContainerStarted","Data":"9cad3ad4ced1e600cee8c247b4ebf56d92d065b18d9cdfa88ddac335dab00f5c"}
Jul 09 20:40:08.986975 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:08.986936    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-4c278a-openshift-default-7667669d89-ns249"
Jul 09 20:40:08.987193 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:08.987173    2562 patch_prober.go:28] interesting pod/e2e-api-a-4c278a-openshift-default-7667669d89-ns249 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Jul 09 20:40:08.987239 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:08.987215    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4c278a-openshift-default-7667669d89-ns249" podUID="b47ec9db-f060-4486-8599-d35bbcaff3ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Jul 09 20:40:09.986659 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:09.986623    2562 patch_prober.go:28] interesting pod/e2e-api-a-4c278a-openshift-default-7667669d89-ns249 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Jul 09 20:40:09.987057 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:09.986673    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4c278a-openshift-default-7667669d89-ns249" podUID="b47ec9db-f060-4486-8599-d35bbcaff3ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Jul 09 20:40:10.987356 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:10.987292    2562 patch_prober.go:28] interesting pod/e2e-api-a-4c278a-openshift-default-7667669d89-ns249 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Jul 09 20:40:10.987731 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:10.987380    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4c278a-openshift-default-7667669d89-ns249" podUID="b47ec9db-f060-4486-8599-d35bbcaff3ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Jul 09 20:40:11.989529 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:11.989495    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-4c278a-openshift-default-7667669d89-ns249"
Jul 09 20:40:11.989972 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:11.989784    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-4c278a-openshift-default-7667669d89-ns249"
Jul 09 20:40:11.990382 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:11.990367    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-4c278a-openshift-default-7667669d89-ns249"
Jul 09 20:40:12.534272 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:12.534243    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4c278a-7fd96487c4-2m7pr"]
Jul 09 20:40:12.534501 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:12.534476    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4c278a-7fd96487c4-2m7pr" podUID="0b5a42ab-60d6-42f3-96b6-eaf3b22e8b08" containerName="maas-api" containerID="cri-o://c4df296fc8137ea7aa4106c39ce4b36e2076be772a2fdfb6bb321f3694850107" gracePeriod=30
Jul 09 20:40:12.777737 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:12.777715    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4c278a-7fd96487c4-2m7pr"
Jul 09 20:40:12.863828 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:12.863774    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n9vgj\" (UniqueName: \"kubernetes.io/projected/0b5a42ab-60d6-42f3-96b6-eaf3b22e8b08-kube-api-access-n9vgj\") pod \"0b5a42ab-60d6-42f3-96b6-eaf3b22e8b08\" (UID: \"0b5a42ab-60d6-42f3-96b6-eaf3b22e8b08\") "
Jul 09 20:40:12.863929 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:12.863871    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b5a42ab-60d6-42f3-96b6-eaf3b22e8b08-openshift-service-ca\") pod \"0b5a42ab-60d6-42f3-96b6-eaf3b22e8b08\" (UID: \"0b5a42ab-60d6-42f3-96b6-eaf3b22e8b08\") "
Jul 09 20:40:12.863929 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:12.863898    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b5a42ab-60d6-42f3-96b6-eaf3b22e8b08-maas-api-tls\") pod \"0b5a42ab-60d6-42f3-96b6-eaf3b22e8b08\" (UID: \"0b5a42ab-60d6-42f3-96b6-eaf3b22e8b08\") "
Jul 09 20:40:12.864277 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:12.864245    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b5a42ab-60d6-42f3-96b6-eaf3b22e8b08-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0b5a42ab-60d6-42f3-96b6-eaf3b22e8b08" (UID: "0b5a42ab-60d6-42f3-96b6-eaf3b22e8b08"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 20:40:12.865787 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:12.865756    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b5a42ab-60d6-42f3-96b6-eaf3b22e8b08-kube-api-access-n9vgj" (OuterVolumeSpecName: "kube-api-access-n9vgj") pod "0b5a42ab-60d6-42f3-96b6-eaf3b22e8b08" (UID: "0b5a42ab-60d6-42f3-96b6-eaf3b22e8b08"). InnerVolumeSpecName "kube-api-access-n9vgj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 20:40:12.865885 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:12.865761    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b5a42ab-60d6-42f3-96b6-eaf3b22e8b08-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0b5a42ab-60d6-42f3-96b6-eaf3b22e8b08" (UID: "0b5a42ab-60d6-42f3-96b6-eaf3b22e8b08"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 20:40:12.964454 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:12.964430    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9vgj\" (UniqueName: \"kubernetes.io/projected/0b5a42ab-60d6-42f3-96b6-eaf3b22e8b08-kube-api-access-n9vgj\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:40:12.964454 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:12.964451    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b5a42ab-60d6-42f3-96b6-eaf3b22e8b08-openshift-service-ca\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:40:12.964580 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:12.964465    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b5a42ab-60d6-42f3-96b6-eaf3b22e8b08-maas-api-tls\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:40:13.144468 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:13.144391    2562 generic.go:356] "Generic (PLEG): container finished" podID="0b5a42ab-60d6-42f3-96b6-eaf3b22e8b08" containerID="c4df296fc8137ea7aa4106c39ce4b36e2076be772a2fdfb6bb321f3694850107" exitCode=0
Jul 09 20:40:13.144468 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:13.144449    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4c278a-7fd96487c4-2m7pr"
Jul 09 20:40:13.144888 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:13.144486    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4c278a-7fd96487c4-2m7pr" event={"ID":"0b5a42ab-60d6-42f3-96b6-eaf3b22e8b08","Type":"ContainerDied","Data":"c4df296fc8137ea7aa4106c39ce4b36e2076be772a2fdfb6bb321f3694850107"}
Jul 09 20:40:13.144888 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:13.144550    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4c278a-7fd96487c4-2m7pr" event={"ID":"0b5a42ab-60d6-42f3-96b6-eaf3b22e8b08","Type":"ContainerDied","Data":"c29089d59589dc4a57ef61e21910a17c0fa00df0a12cb63bbdc9175313e33376"}
Jul 09 20:40:13.144888 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:13.144583    2562 scope.go:117] "RemoveContainer" containerID="c4df296fc8137ea7aa4106c39ce4b36e2076be772a2fdfb6bb321f3694850107"
Jul 09 20:40:13.153842 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:13.153696    2562 scope.go:117] "RemoveContainer" containerID="c4df296fc8137ea7aa4106c39ce4b36e2076be772a2fdfb6bb321f3694850107"
Jul 09 20:40:13.154105 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:40:13.154015    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4df296fc8137ea7aa4106c39ce4b36e2076be772a2fdfb6bb321f3694850107\": container with ID starting with c4df296fc8137ea7aa4106c39ce4b36e2076be772a2fdfb6bb321f3694850107 not found: ID does not exist" containerID="c4df296fc8137ea7aa4106c39ce4b36e2076be772a2fdfb6bb321f3694850107"
Jul 09 20:40:13.154105 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:13.154054    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4df296fc8137ea7aa4106c39ce4b36e2076be772a2fdfb6bb321f3694850107"} err="failed to get container status \"c4df296fc8137ea7aa4106c39ce4b36e2076be772a2fdfb6bb321f3694850107\": rpc error: code = NotFound desc = could not find container \"c4df296fc8137ea7aa4106c39ce4b36e2076be772a2fdfb6bb321f3694850107\": container with ID starting with c4df296fc8137ea7aa4106c39ce4b36e2076be772a2fdfb6bb321f3694850107 not found: ID does not exist"
Jul 09 20:40:13.170222 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:13.170093    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4c278a-7fd96487c4-2m7pr"]
Jul 09 20:40:13.172350 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:13.172331    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4c278a-7fd96487c4-2m7pr"]
Jul 09 20:40:14.085439 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:14.085406    2562 patch_prober.go:28] interesting pod/e2e-api-b-9df7ce-openshift-default-b7fbc5db5-9j65l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 09 20:40:14.085628 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:14.085465    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-9df7ce-openshift-default-b7fbc5db5-9j65l" podUID="366ce8c4-6f65-4d55-849e-6fd02a8526a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 09 20:40:14.241897 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:14.241861    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b5a42ab-60d6-42f3-96b6-eaf3b22e8b08" path="/var/lib/kubelet/pods/0b5a42ab-60d6-42f3-96b6-eaf3b22e8b08/volumes"
Jul 09 20:40:15.155512 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:15.155486    2562 generic.go:356] "Generic (PLEG): container finished" podID="366ce8c4-6f65-4d55-849e-6fd02a8526a0" containerID="3c5939cbb7801d52270847df51b7d95b3b5e8ff567e77ee82420e8744be03dd3" exitCode=0
Jul 09 20:40:15.155662 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:15.155555    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9df7ce-openshift-default-b7fbc5db5-9j65l" event={"ID":"366ce8c4-6f65-4d55-849e-6fd02a8526a0","Type":"ContainerDied","Data":"3c5939cbb7801d52270847df51b7d95b3b5e8ff567e77ee82420e8744be03dd3"}
Jul 09 20:40:15.155910 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:15.155897    2562 scope.go:117] "RemoveContainer" containerID="3c5939cbb7801d52270847df51b7d95b3b5e8ff567e77ee82420e8744be03dd3"
Jul 09 20:40:15.156919 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:15.156889    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 09 20:40:15.157015 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:15.156997    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 09 20:40:15.157060 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:15.157047    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 09 20:40:16.162817 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:16.162784    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9df7ce-openshift-default-b7fbc5db5-9j65l" event={"ID":"366ce8c4-6f65-4d55-849e-6fd02a8526a0","Type":"ContainerStarted","Data":"6c5acf6cc5a133207a35c7a0d85445e8fab95bf49b1ff7d27e94861465fca8e5"}
Jul 09 20:40:16.512939 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:16.512904    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-9df7ce-openshift-default-b7fbc5db5-9j65l"
Jul 09 20:40:16.512939 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:16.512930    2562 patch_prober.go:28] interesting pod/e2e-api-b-9df7ce-openshift-default-b7fbc5db5-9j65l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Jul 09 20:40:16.513198 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:16.512993    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-9df7ce-openshift-default-b7fbc5db5-9j65l" podUID="366ce8c4-6f65-4d55-849e-6fd02a8526a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Jul 09 20:40:17.513373 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:17.512915    2562 patch_prober.go:28] interesting pod/e2e-api-b-9df7ce-openshift-default-b7fbc5db5-9j65l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Jul 09 20:40:17.513373 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:17.513004    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-9df7ce-openshift-default-b7fbc5db5-9j65l" podUID="366ce8c4-6f65-4d55-849e-6fd02a8526a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Jul 09 20:40:18.037324 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:18.036279    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-4c278a-openshift-default-7667669d89-ns249"]
Jul 09 20:40:18.037324 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:18.036579    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-4c278a-openshift-default-7667669d89-ns249" podUID="b47ec9db-f060-4486-8599-d35bbcaff3ca" containerName="istio-proxy" containerID="cri-o://9cad3ad4ced1e600cee8c247b4ebf56d92d065b18d9cdfa88ddac335dab00f5c" gracePeriod=30
Jul 09 20:40:18.512035 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:18.512000    2562 patch_prober.go:28] interesting pod/e2e-api-b-9df7ce-openshift-default-b7fbc5db5-9j65l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Jul 09 20:40:18.512885 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:18.512851    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-9df7ce-openshift-default-b7fbc5db5-9j65l" podUID="366ce8c4-6f65-4d55-849e-6fd02a8526a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Jul 09 20:40:19.245931 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:19.245111    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-9df7ce-74566d9c74-b5lvb"]
Jul 09 20:40:19.245931 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:19.245420    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9df7ce-74566d9c74-b5lvb" podUID="70a603a6-f438-4ed1-b185-f7cc2802b3d2" containerName="maas-api" containerID="cri-o://74dfde2835a2607b9718d2b746582337e8961bcd1dd4e45d2e0a49bfb8e9a1ec" gracePeriod=30
Jul 09 20:40:19.498532 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:19.498471    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9df7ce-74566d9c74-b5lvb"
Jul 09 20:40:19.534368 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:19.533895    2562 patch_prober.go:28] interesting pod/e2e-api-b-9df7ce-openshift-default-b7fbc5db5-9j65l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Jul 09 20:40:19.534368 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:19.533949    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-9df7ce-openshift-default-b7fbc5db5-9j65l" podUID="366ce8c4-6f65-4d55-849e-6fd02a8526a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Jul 09 20:40:19.636585 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:19.636559    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70a603a6-f438-4ed1-b185-f7cc2802b3d2-openshift-service-ca\") pod \"70a603a6-f438-4ed1-b185-f7cc2802b3d2\" (UID: \"70a603a6-f438-4ed1-b185-f7cc2802b3d2\") "
Jul 09 20:40:19.636707 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:19.636596    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70a603a6-f438-4ed1-b185-f7cc2802b3d2-maas-api-tls\") pod \"70a603a6-f438-4ed1-b185-f7cc2802b3d2\" (UID: \"70a603a6-f438-4ed1-b185-f7cc2802b3d2\") "
Jul 09 20:40:19.636707 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:19.636660    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6qkt4\" (UniqueName: \"kubernetes.io/projected/70a603a6-f438-4ed1-b185-f7cc2802b3d2-kube-api-access-6qkt4\") pod \"70a603a6-f438-4ed1-b185-f7cc2802b3d2\" (UID: \"70a603a6-f438-4ed1-b185-f7cc2802b3d2\") "
Jul 09 20:40:19.636894 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:19.636868    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70a603a6-f438-4ed1-b185-f7cc2802b3d2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "70a603a6-f438-4ed1-b185-f7cc2802b3d2" (UID: "70a603a6-f438-4ed1-b185-f7cc2802b3d2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 20:40:19.638671 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:19.638643    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70a603a6-f438-4ed1-b185-f7cc2802b3d2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "70a603a6-f438-4ed1-b185-f7cc2802b3d2" (UID: "70a603a6-f438-4ed1-b185-f7cc2802b3d2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 20:40:19.638769 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:19.638673    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70a603a6-f438-4ed1-b185-f7cc2802b3d2-kube-api-access-6qkt4" (OuterVolumeSpecName: "kube-api-access-6qkt4") pod "70a603a6-f438-4ed1-b185-f7cc2802b3d2" (UID: "70a603a6-f438-4ed1-b185-f7cc2802b3d2"). InnerVolumeSpecName "kube-api-access-6qkt4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 20:40:19.737381 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:19.737361    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qkt4\" (UniqueName: \"kubernetes.io/projected/70a603a6-f438-4ed1-b185-f7cc2802b3d2-kube-api-access-6qkt4\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:40:19.737381 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:19.737380    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70a603a6-f438-4ed1-b185-f7cc2802b3d2-openshift-service-ca\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:40:19.737507 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:19.737390    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70a603a6-f438-4ed1-b185-f7cc2802b3d2-maas-api-tls\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:40:20.183292 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:20.183263    2562 generic.go:356] "Generic (PLEG): container finished" podID="70a603a6-f438-4ed1-b185-f7cc2802b3d2" containerID="74dfde2835a2607b9718d2b746582337e8961bcd1dd4e45d2e0a49bfb8e9a1ec" exitCode=0
Jul 09 20:40:20.183471 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:20.183314    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9df7ce-74566d9c74-b5lvb"
Jul 09 20:40:20.183471 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:20.183331    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9df7ce-74566d9c74-b5lvb" event={"ID":"70a603a6-f438-4ed1-b185-f7cc2802b3d2","Type":"ContainerDied","Data":"74dfde2835a2607b9718d2b746582337e8961bcd1dd4e45d2e0a49bfb8e9a1ec"}
Jul 09 20:40:20.183471 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:20.183377    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9df7ce-74566d9c74-b5lvb" event={"ID":"70a603a6-f438-4ed1-b185-f7cc2802b3d2","Type":"ContainerDied","Data":"38aed225f0d2f86490f3d530a1caba3844527cd1de4a8f42407cf062c1157afa"}
Jul 09 20:40:20.183471 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:20.183421    2562 scope.go:117] "RemoveContainer" containerID="74dfde2835a2607b9718d2b746582337e8961bcd1dd4e45d2e0a49bfb8e9a1ec"
Jul 09 20:40:20.191717 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:20.191699    2562 scope.go:117] "RemoveContainer" containerID="74dfde2835a2607b9718d2b746582337e8961bcd1dd4e45d2e0a49bfb8e9a1ec"
Jul 09 20:40:20.191977 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:40:20.191960    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74dfde2835a2607b9718d2b746582337e8961bcd1dd4e45d2e0a49bfb8e9a1ec\": container with ID starting with 74dfde2835a2607b9718d2b746582337e8961bcd1dd4e45d2e0a49bfb8e9a1ec not found: ID does not exist" containerID="74dfde2835a2607b9718d2b746582337e8961bcd1dd4e45d2e0a49bfb8e9a1ec"
Jul 09 20:40:20.192029 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:20.191985    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74dfde2835a2607b9718d2b746582337e8961bcd1dd4e45d2e0a49bfb8e9a1ec"} err="failed to get container status \"74dfde2835a2607b9718d2b746582337e8961bcd1dd4e45d2e0a49bfb8e9a1ec\": rpc error: code = NotFound desc = could not find container \"74dfde2835a2607b9718d2b746582337e8961bcd1dd4e45d2e0a49bfb8e9a1ec\": container with ID starting with 74dfde2835a2607b9718d2b746582337e8961bcd1dd4e45d2e0a49bfb8e9a1ec not found: ID does not exist"
Jul 09 20:40:20.205109 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:20.205090    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-9df7ce-74566d9c74-b5lvb"]
Jul 09 20:40:20.208388 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:20.208370    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-9df7ce-74566d9c74-b5lvb"]
Jul 09 20:40:20.241584 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:20.241558    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70a603a6-f438-4ed1-b185-f7cc2802b3d2" path="/var/lib/kubelet/pods/70a603a6-f438-4ed1-b185-f7cc2802b3d2/volumes"
Jul 09 20:40:20.515151 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:20.515136    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-9df7ce-openshift-default-b7fbc5db5-9j65l"
Jul 09 20:40:20.515488 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:20.515475    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-9df7ce-openshift-default-b7fbc5db5-9j65l"
Jul 09 20:40:20.516033 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:20.516019    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-9df7ce-openshift-default-b7fbc5db5-9j65l"
Jul 09 20:40:23.200920 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:23.200890    2562 generic.go:356] "Generic (PLEG): container finished" podID="b47ec9db-f060-4486-8599-d35bbcaff3ca" containerID="9cad3ad4ced1e600cee8c247b4ebf56d92d065b18d9cdfa88ddac335dab00f5c" exitCode=0
Jul 09 20:40:23.201285 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:23.200964    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4c278a-openshift-default-7667669d89-ns249" event={"ID":"b47ec9db-f060-4486-8599-d35bbcaff3ca","Type":"ContainerDied","Data":"9cad3ad4ced1e600cee8c247b4ebf56d92d065b18d9cdfa88ddac335dab00f5c"}
Jul 09 20:40:23.201285 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:23.201047    2562 scope.go:117] "RemoveContainer" containerID="852904c13dd736e352f81abcc29fda1e8f26f805432455d1277a843a938fc390"
Jul 09 20:40:23.289274 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:23.289254    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-4c278a-openshift-default-7667669d89-ns249"
Jul 09 20:40:23.467833 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:23.467816    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b47ec9db-f060-4486-8599-d35bbcaff3ca-credential-socket\") pod \"b47ec9db-f060-4486-8599-d35bbcaff3ca\" (UID: \"b47ec9db-f060-4486-8599-d35bbcaff3ca\") "
Jul 09 20:40:23.467970 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:23.467867    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b47ec9db-f060-4486-8599-d35bbcaff3ca-workload-socket\") pod \"b47ec9db-f060-4486-8599-d35bbcaff3ca\" (UID: \"b47ec9db-f060-4486-8599-d35bbcaff3ca\") "
Jul 09 20:40:23.467970 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:23.467899    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b47ec9db-f060-4486-8599-d35bbcaff3ca-istiod-ca-cert\") pod \"b47ec9db-f060-4486-8599-d35bbcaff3ca\" (UID: \"b47ec9db-f060-4486-8599-d35bbcaff3ca\") "
Jul 09 20:40:23.467970 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:23.467920    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b47ec9db-f060-4486-8599-d35bbcaff3ca-istio-envoy\") pod \"b47ec9db-f060-4486-8599-d35bbcaff3ca\" (UID: \"b47ec9db-f060-4486-8599-d35bbcaff3ca\") "
Jul 09 20:40:23.467970 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:23.467943    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b47ec9db-f060-4486-8599-d35bbcaff3ca-istio-podinfo\") pod \"b47ec9db-f060-4486-8599-d35bbcaff3ca\" (UID: \"b47ec9db-f060-4486-8599-d35bbcaff3ca\") "
Jul 09 20:40:23.467970 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:23.467966    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b47ec9db-f060-4486-8599-d35bbcaff3ca-workload-certs\") pod \"b47ec9db-f060-4486-8599-d35bbcaff3ca\" (UID: \"b47ec9db-f060-4486-8599-d35bbcaff3ca\") "
Jul 09 20:40:23.468224 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:23.468054    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b47ec9db-f060-4486-8599-d35bbcaff3ca-istio-token\") pod \"b47ec9db-f060-4486-8599-d35bbcaff3ca\" (UID: \"b47ec9db-f060-4486-8599-d35bbcaff3ca\") "
Jul 09 20:40:23.468224 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:23.468078    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-44t2r\" (UniqueName: \"kubernetes.io/projected/b47ec9db-f060-4486-8599-d35bbcaff3ca-kube-api-access-44t2r\") pod \"b47ec9db-f060-4486-8599-d35bbcaff3ca\" (UID: \"b47ec9db-f060-4486-8599-d35bbcaff3ca\") "
Jul 09 20:40:23.468224 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:23.468109    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b47ec9db-f060-4486-8599-d35bbcaff3ca-istio-data\") pod \"b47ec9db-f060-4486-8599-d35bbcaff3ca\" (UID: \"b47ec9db-f060-4486-8599-d35bbcaff3ca\") "
Jul 09 20:40:23.468224 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:23.468096    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b47ec9db-f060-4486-8599-d35bbcaff3ca-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b47ec9db-f060-4486-8599-d35bbcaff3ca" (UID: "b47ec9db-f060-4486-8599-d35bbcaff3ca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 20:40:23.468224 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:23.468164    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b47ec9db-f060-4486-8599-d35bbcaff3ca-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b47ec9db-f060-4486-8599-d35bbcaff3ca" (UID: "b47ec9db-f060-4486-8599-d35bbcaff3ca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 20:40:23.468522 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:23.468333    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b47ec9db-f060-4486-8599-d35bbcaff3ca-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b47ec9db-f060-4486-8599-d35bbcaff3ca" (UID: "b47ec9db-f060-4486-8599-d35bbcaff3ca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 20:40:23.468635 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:23.468610    2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b47ec9db-f060-4486-8599-d35bbcaff3ca-credential-socket\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:40:23.468709 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:23.468641    2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b47ec9db-f060-4486-8599-d35bbcaff3ca-workload-socket\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:40:23.468709 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:23.468655    2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b47ec9db-f060-4486-8599-d35bbcaff3ca-workload-certs\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:40:23.468709 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:23.468684    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b47ec9db-f060-4486-8599-d35bbcaff3ca-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b47ec9db-f060-4486-8599-d35bbcaff3ca" (UID: "b47ec9db-f060-4486-8599-d35bbcaff3ca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 20:40:23.469223 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:23.469197    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b47ec9db-f060-4486-8599-d35bbcaff3ca-istio-data" (OuterVolumeSpecName: "istio-data") pod "b47ec9db-f060-4486-8599-d35bbcaff3ca" (UID: "b47ec9db-f060-4486-8599-d35bbcaff3ca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 20:40:23.470316 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:23.470276    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b47ec9db-f060-4486-8599-d35bbcaff3ca-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b47ec9db-f060-4486-8599-d35bbcaff3ca" (UID: "b47ec9db-f060-4486-8599-d35bbcaff3ca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 09 20:40:23.470421 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:23.470326    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b47ec9db-f060-4486-8599-d35bbcaff3ca-kube-api-access-44t2r" (OuterVolumeSpecName: "kube-api-access-44t2r") pod "b47ec9db-f060-4486-8599-d35bbcaff3ca" (UID: "b47ec9db-f060-4486-8599-d35bbcaff3ca"). InnerVolumeSpecName "kube-api-access-44t2r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 20:40:23.470523 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:23.470501    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b47ec9db-f060-4486-8599-d35bbcaff3ca-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b47ec9db-f060-4486-8599-d35bbcaff3ca" (UID: "b47ec9db-f060-4486-8599-d35bbcaff3ca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 20:40:23.470767 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:23.470743    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b47ec9db-f060-4486-8599-d35bbcaff3ca-istio-token" (OuterVolumeSpecName: "istio-token") pod "b47ec9db-f060-4486-8599-d35bbcaff3ca" (UID: "b47ec9db-f060-4486-8599-d35bbcaff3ca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 20:40:23.569835 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:23.569810    2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b47ec9db-f060-4486-8599-d35bbcaff3ca-istiod-ca-cert\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:40:23.569920 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:23.569837    2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b47ec9db-f060-4486-8599-d35bbcaff3ca-istio-envoy\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:40:23.569920 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:23.569851    2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b47ec9db-f060-4486-8599-d35bbcaff3ca-istio-podinfo\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:40:23.569920 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:23.569863    2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b47ec9db-f060-4486-8599-d35bbcaff3ca-istio-token\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:40:23.569920 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:23.569873    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44t2r\" (UniqueName: \"kubernetes.io/projected/b47ec9db-f060-4486-8599-d35bbcaff3ca-kube-api-access-44t2r\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:40:23.569920 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:23.569882    2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b47ec9db-f060-4486-8599-d35bbcaff3ca-istio-data\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:40:24.207290 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:24.207267    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-4c278a-openshift-default-7667669d89-ns249"
Jul 09 20:40:24.207687 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:24.207268    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4c278a-openshift-default-7667669d89-ns249" event={"ID":"b47ec9db-f060-4486-8599-d35bbcaff3ca","Type":"ContainerDied","Data":"e039272c1bc32698d6959ee12d934e7c7ef63ec22d3e6dcae427c6909160bf45"}
Jul 09 20:40:24.207687 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:24.207392    2562 scope.go:117] "RemoveContainer" containerID="9cad3ad4ced1e600cee8c247b4ebf56d92d065b18d9cdfa88ddac335dab00f5c"
Jul 09 20:40:24.233006 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:24.232985    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-4c278a-openshift-default-7667669d89-ns249"]
Jul 09 20:40:24.241798 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:24.241782    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-4c278a-openshift-default-7667669d89-ns249"]
Jul 09 20:40:25.276863 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:25.276832    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-9df7ce-openshift-default-b7fbc5db5-9j65l"]
Jul 09 20:40:25.277319 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:25.277001    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-9df7ce-openshift-default-b7fbc5db5-9j65l" podUID="366ce8c4-6f65-4d55-849e-6fd02a8526a0" containerName="istio-proxy" containerID="cri-o://6c5acf6cc5a133207a35c7a0d85445e8fab95bf49b1ff7d27e94861465fca8e5" gracePeriod=30
Jul 09 20:40:26.179403 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:26.179374    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-a56db7-openshift-default-76dfdd4bf5-5rqvp"]
Jul 09 20:40:26.179751 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:26.179738    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b5a42ab-60d6-42f3-96b6-eaf3b22e8b08" containerName="maas-api"
Jul 09 20:40:26.179797 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:26.179753    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b5a42ab-60d6-42f3-96b6-eaf3b22e8b08" containerName="maas-api"
Jul 09 20:40:26.179797 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:26.179764    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b47ec9db-f060-4486-8599-d35bbcaff3ca" containerName="istio-proxy"
Jul 09 20:40:26.179797 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:26.179769    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b47ec9db-f060-4486-8599-d35bbcaff3ca" containerName="istio-proxy"
Jul 09 20:40:26.179797 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:26.179778    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70a603a6-f438-4ed1-b185-f7cc2802b3d2" containerName="maas-api"
Jul 09 20:40:26.179797 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:26.179787    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="70a603a6-f438-4ed1-b185-f7cc2802b3d2" containerName="maas-api"
Jul 09 20:40:26.179797 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:26.179798    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b47ec9db-f060-4486-8599-d35bbcaff3ca" containerName="istio-proxy"
Jul 09 20:40:26.180052 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:26.179803    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b47ec9db-f060-4486-8599-d35bbcaff3ca" containerName="istio-proxy"
Jul 09 20:40:26.180052 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:26.179871    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="70a603a6-f438-4ed1-b185-f7cc2802b3d2" containerName="maas-api"
Jul 09 20:40:26.180052 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:26.179879    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="b47ec9db-f060-4486-8599-d35bbcaff3ca" containerName="istio-proxy"
Jul 09 20:40:26.180052 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:26.179884    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="b47ec9db-f060-4486-8599-d35bbcaff3ca" containerName="istio-proxy"
Jul 09 20:40:26.180052 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:26.179892    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b5a42ab-60d6-42f3-96b6-eaf3b22e8b08" containerName="maas-api"
Jul 09 20:40:26.183298 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:26.183279    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-a56db7-openshift-default-76dfdd4bf5-5rqvp"
Jul 09 20:40:26.186388 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:26.186367    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-a56db7-openshift-default-dockercfg-s48sl\""
Jul 09 20:40:26.201672 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:26.201652    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-a56db7-openshift-default-76dfdd4bf5-5rqvp"]
Jul 09 20:40:26.244928 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:26.244898    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b47ec9db-f060-4486-8599-d35bbcaff3ca" path="/var/lib/kubelet/pods/b47ec9db-f060-4486-8599-d35bbcaff3ca/volumes"
Jul 09 20:40:26.290041 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:26.290012    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d882374-1667-49f2-8c18-d7b8e159cb72-istio-token\") pod \"e2e-shared-a-a56db7-openshift-default-76dfdd4bf5-5rqvp\" (UID: \"7d882374-1667-49f2-8c18-d7b8e159cb72\") " pod="openshift-ingress/e2e-shared-a-a56db7-openshift-default-76dfdd4bf5-5rqvp"
Jul 09 20:40:26.290357 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:26.290044    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d882374-1667-49f2-8c18-d7b8e159cb72-credential-socket\") pod \"e2e-shared-a-a56db7-openshift-default-76dfdd4bf5-5rqvp\" (UID: \"7d882374-1667-49f2-8c18-d7b8e159cb72\") " pod="openshift-ingress/e2e-shared-a-a56db7-openshift-default-76dfdd4bf5-5rqvp"
Jul 09 20:40:26.290357 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:26.290065    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d882374-1667-49f2-8c18-d7b8e159cb72-istio-data\") pod \"e2e-shared-a-a56db7-openshift-default-76dfdd4bf5-5rqvp\" (UID: \"7d882374-1667-49f2-8c18-d7b8e159cb72\") " pod="openshift-ingress/e2e-shared-a-a56db7-openshift-default-76dfdd4bf5-5rqvp"
Jul 09 20:40:26.290357 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:26.290111    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d882374-1667-49f2-8c18-d7b8e159cb72-istio-podinfo\") pod \"e2e-shared-a-a56db7-openshift-default-76dfdd4bf5-5rqvp\" (UID: \"7d882374-1667-49f2-8c18-d7b8e159cb72\") " pod="openshift-ingress/e2e-shared-a-a56db7-openshift-default-76dfdd4bf5-5rqvp"
Jul 09 20:40:26.290357 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:26.290184    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-725ht\" (UniqueName: \"kubernetes.io/projected/7d882374-1667-49f2-8c18-d7b8e159cb72-kube-api-access-725ht\") pod \"e2e-shared-a-a56db7-openshift-default-76dfdd4bf5-5rqvp\" (UID: \"7d882374-1667-49f2-8c18-d7b8e159cb72\") " pod="openshift-ingress/e2e-shared-a-a56db7-openshift-default-76dfdd4bf5-5rqvp"
Jul 09 20:40:26.290357 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:26.290215    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d882374-1667-49f2-8c18-d7b8e159cb72-workload-socket\") pod \"e2e-shared-a-a56db7-openshift-default-76dfdd4bf5-5rqvp\" (UID: \"7d882374-1667-49f2-8c18-d7b8e159cb72\") " pod="openshift-ingress/e2e-shared-a-a56db7-openshift-default-76dfdd4bf5-5rqvp"
Jul 09 20:40:26.290357 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:26.290238    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d882374-1667-49f2-8c18-d7b8e159cb72-istio-envoy\") pod \"e2e-shared-a-a56db7-openshift-default-76dfdd4bf5-5rqvp\" (UID: \"7d882374-1667-49f2-8c18-d7b8e159cb72\") " pod="openshift-ingress/e2e-shared-a-a56db7-openshift-default-76dfdd4bf5-5rqvp"
Jul 09 20:40:26.290357 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:26.290257    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d882374-1667-49f2-8c18-d7b8e159cb72-istiod-ca-cert\") pod \"e2e-shared-a-a56db7-openshift-default-76dfdd4bf5-5rqvp\" (UID: \"7d882374-1667-49f2-8c18-d7b8e159cb72\") " pod="openshift-ingress/e2e-shared-a-a56db7-openshift-default-76dfdd4bf5-5rqvp"
Jul 09 20:40:26.290357 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:26.290277    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d882374-1667-49f2-8c18-d7b8e159cb72-workload-certs\") pod \"e2e-shared-a-a56db7-openshift-default-76dfdd4bf5-5rqvp\" (UID: \"7d882374-1667-49f2-8c18-d7b8e159cb72\") " pod="openshift-ingress/e2e-shared-a-a56db7-openshift-default-76dfdd4bf5-5rqvp"
Jul 09 20:40:26.391132 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:26.391099    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d882374-1667-49f2-8c18-d7b8e159cb72-istio-podinfo\") pod \"e2e-shared-a-a56db7-openshift-default-76dfdd4bf5-5rqvp\" (UID: \"7d882374-1667-49f2-8c18-d7b8e159cb72\") " pod="openshift-ingress/e2e-shared-a-a56db7-openshift-default-76dfdd4bf5-5rqvp"
Jul 09 20:40:26.391269 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:26.391201    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-725ht\" (UniqueName: \"kubernetes.io/projected/7d882374-1667-49f2-8c18-d7b8e159cb72-kube-api-access-725ht\") pod \"e2e-shared-a-a56db7-openshift-default-76dfdd4bf5-5rqvp\" (UID: \"7d882374-1667-49f2-8c18-d7b8e159cb72\") " pod="openshift-ingress/e2e-shared-a-a56db7-openshift-default-76dfdd4bf5-5rqvp"
Jul 09 20:40:26.391269 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:26.391260    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d882374-1667-49f2-8c18-d7b8e159cb72-workload-socket\") pod \"e2e-shared-a-a56db7-openshift-default-76dfdd4bf5-5rqvp\" (UID: \"7d882374-1667-49f2-8c18-d7b8e159cb72\") " pod="openshift-ingress/e2e-shared-a-a56db7-openshift-default-76dfdd4bf5-5rqvp"
Jul 09 20:40:26.391393 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:26.391284    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d882374-1667-49f2-8c18-d7b8e159cb72-istio-envoy\") pod \"e2e-shared-a-a56db7-openshift-default-76dfdd4bf5-5rqvp\" (UID: \"7d882374-1667-49f2-8c18-d7b8e159cb72\") " pod="openshift-ingress/e2e-shared-a-a56db7-openshift-default-76dfdd4bf5-5rqvp"
Jul 09 20:40:26.391393 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:26.391354    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d882374-1667-49f2-8c18-d7b8e159cb72-istiod-ca-cert\") pod \"e2e-shared-a-a56db7-openshift-default-76dfdd4bf5-5rqvp\" (UID: \"7d882374-1667-49f2-8c18-d7b8e159cb72\") " pod="openshift-ingress/e2e-shared-a-a56db7-openshift-default-76dfdd4bf5-5rqvp"
Jul 09 20:40:26.391492 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:26.391389    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d882374-1667-49f2-8c18-d7b8e159cb72-workload-certs\") pod \"e2e-shared-a-a56db7-openshift-default-76dfdd4bf5-5rqvp\" (UID: \"7d882374-1667-49f2-8c18-d7b8e159cb72\") " pod="openshift-ingress/e2e-shared-a-a56db7-openshift-default-76dfdd4bf5-5rqvp"
Jul 09 20:40:26.391542 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:26.391528    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d882374-1667-49f2-8c18-d7b8e159cb72-istio-token\") pod \"e2e-shared-a-a56db7-openshift-default-76dfdd4bf5-5rqvp\" (UID: \"7d882374-1667-49f2-8c18-d7b8e159cb72\") " pod="openshift-ingress/e2e-shared-a-a56db7-openshift-default-76dfdd4bf5-5rqvp"
Jul 09 20:40:26.391596 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:26.391559    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d882374-1667-49f2-8c18-d7b8e159cb72-credential-socket\") pod \"e2e-shared-a-a56db7-openshift-default-76dfdd4bf5-5rqvp\" (UID: \"7d882374-1667-49f2-8c18-d7b8e159cb72\") " pod="openshift-ingress/e2e-shared-a-a56db7-openshift-default-76dfdd4bf5-5rqvp"
Jul 09 20:40:26.391651 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:26.391618    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d882374-1667-49f2-8c18-d7b8e159cb72-istio-data\") pod \"e2e-shared-a-a56db7-openshift-default-76dfdd4bf5-5rqvp\" (UID: \"7d882374-1667-49f2-8c18-d7b8e159cb72\") " pod="openshift-ingress/e2e-shared-a-a56db7-openshift-default-76dfdd4bf5-5rqvp"
Jul 09 20:40:26.391766 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:26.391739    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d882374-1667-49f2-8c18-d7b8e159cb72-workload-socket\") pod \"e2e-shared-a-a56db7-openshift-default-76dfdd4bf5-5rqvp\" (UID: \"7d882374-1667-49f2-8c18-d7b8e159cb72\") " pod="openshift-ingress/e2e-shared-a-a56db7-openshift-default-76dfdd4bf5-5rqvp"
Jul 09 20:40:26.391848 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:26.391816    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d882374-1667-49f2-8c18-d7b8e159cb72-workload-certs\") pod \"e2e-shared-a-a56db7-openshift-default-76dfdd4bf5-5rqvp\" (UID: \"7d882374-1667-49f2-8c18-d7b8e159cb72\") " pod="openshift-ingress/e2e-shared-a-a56db7-openshift-default-76dfdd4bf5-5rqvp"
Jul 09 20:40:26.392060 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:26.392034    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d882374-1667-49f2-8c18-d7b8e159cb72-credential-socket\") pod \"e2e-shared-a-a56db7-openshift-default-76dfdd4bf5-5rqvp\" (UID: \"7d882374-1667-49f2-8c18-d7b8e159cb72\") " pod="openshift-ingress/e2e-shared-a-a56db7-openshift-default-76dfdd4bf5-5rqvp"
Jul 09 20:40:26.392155 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:26.392107    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d882374-1667-49f2-8c18-d7b8e159cb72-istio-data\") pod \"e2e-shared-a-a56db7-openshift-default-76dfdd4bf5-5rqvp\" (UID: \"7d882374-1667-49f2-8c18-d7b8e159cb72\") " pod="openshift-ingress/e2e-shared-a-a56db7-openshift-default-76dfdd4bf5-5rqvp"
Jul 09 20:40:26.392283 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:26.392265    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d882374-1667-49f2-8c18-d7b8e159cb72-istiod-ca-cert\") pod \"e2e-shared-a-a56db7-openshift-default-76dfdd4bf5-5rqvp\" (UID: \"7d882374-1667-49f2-8c18-d7b8e159cb72\") " pod="openshift-ingress/e2e-shared-a-a56db7-openshift-default-76dfdd4bf5-5rqvp"
Jul 09 20:40:26.393398 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:26.393379    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d882374-1667-49f2-8c18-d7b8e159cb72-istio-envoy\") pod \"e2e-shared-a-a56db7-openshift-default-76dfdd4bf5-5rqvp\" (UID: \"7d882374-1667-49f2-8c18-d7b8e159cb72\") " pod="openshift-ingress/e2e-shared-a-a56db7-openshift-default-76dfdd4bf5-5rqvp"
Jul 09 20:40:26.393732 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:26.393714    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d882374-1667-49f2-8c18-d7b8e159cb72-istio-podinfo\") pod \"e2e-shared-a-a56db7-openshift-default-76dfdd4bf5-5rqvp\" (UID: \"7d882374-1667-49f2-8c18-d7b8e159cb72\") " pod="openshift-ingress/e2e-shared-a-a56db7-openshift-default-76dfdd4bf5-5rqvp"
Jul 09 20:40:26.401853 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:26.401835    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d882374-1667-49f2-8c18-d7b8e159cb72-istio-token\") pod \"e2e-shared-a-a56db7-openshift-default-76dfdd4bf5-5rqvp\" (UID: \"7d882374-1667-49f2-8c18-d7b8e159cb72\") " pod="openshift-ingress/e2e-shared-a-a56db7-openshift-default-76dfdd4bf5-5rqvp"
Jul 09 20:40:26.402188 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:26.402164    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-725ht\" (UniqueName: \"kubernetes.io/projected/7d882374-1667-49f2-8c18-d7b8e159cb72-kube-api-access-725ht\") pod \"e2e-shared-a-a56db7-openshift-default-76dfdd4bf5-5rqvp\" (UID: \"7d882374-1667-49f2-8c18-d7b8e159cb72\") " pod="openshift-ingress/e2e-shared-a-a56db7-openshift-default-76dfdd4bf5-5rqvp"
Jul 09 20:40:26.493416 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:26.493402    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-a56db7-openshift-default-76dfdd4bf5-5rqvp"
Jul 09 20:40:26.625296 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:26.625271    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-a56db7-openshift-default-76dfdd4bf5-5rqvp"]
Jul 09 20:40:26.627786 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:40:26.627761    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d882374_1667_49f2_8c18_d7b8e159cb72.slice/crio-94484caa62497459178d447cd7a0a4f2d32f6735a573f42dd1c45dc614649456 WatchSource:0}: Error finding container 94484caa62497459178d447cd7a0a4f2d32f6735a573f42dd1c45dc614649456: Status 404 returned error can't find the container with id 94484caa62497459178d447cd7a0a4f2d32f6735a573f42dd1c45dc614649456
Jul 09 20:40:26.630259 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:26.630222    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 09 20:40:26.630395 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:26.630317    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 09 20:40:26.630395 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:26.630366    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 09 20:40:27.234474 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:27.234436    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-a56db7-openshift-default-76dfdd4bf5-5rqvp" event={"ID":"7d882374-1667-49f2-8c18-d7b8e159cb72","Type":"ContainerStarted","Data":"f45cf252d313c46abd2d2bb754fd48e07d73386f2452ffcf05c1877d8fbe0b0c"}
Jul 09 20:40:27.234636 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:27.234492    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-a56db7-openshift-default-76dfdd4bf5-5rqvp" event={"ID":"7d882374-1667-49f2-8c18-d7b8e159cb72","Type":"ContainerStarted","Data":"94484caa62497459178d447cd7a0a4f2d32f6735a573f42dd1c45dc614649456"}
Jul 09 20:40:27.257732 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:27.257678    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-a56db7-openshift-default-76dfdd4bf5-5rqvp" podStartSLOduration=1.2576595560000001 podStartE2EDuration="1.257659556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 20:40:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 20:40:27.25619071 +0000 UTC m=+2217.673860517" watchObservedRunningTime="2026-07-09 20:40:27.257659556 +0000 UTC m=+2217.675329370"
Jul 09 20:40:27.494355 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:27.494272    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-a56db7-openshift-default-76dfdd4bf5-5rqvp"
Jul 09 20:40:27.498771 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:27.498747    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-a56db7-openshift-default-76dfdd4bf5-5rqvp"
Jul 09 20:40:28.241096 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:28.241070    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-a56db7-openshift-default-76dfdd4bf5-5rqvp"
Jul 09 20:40:28.241218 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:28.241133    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-a56db7-openshift-default-76dfdd4bf5-5rqvp"
Jul 09 20:40:29.247633 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:29.247605    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-a56db7-586cd6c985-m9xm2"]
Jul 09 20:40:29.250516 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:29.250497    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a56db7-586cd6c985-m9xm2"
Jul 09 20:40:29.253582 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:29.253558    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-a56db7-serving-cert\""
Jul 09 20:40:29.263290 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:29.263270    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-a56db7-586cd6c985-m9xm2"]
Jul 09 20:40:29.413799 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:29.413761    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d684956-954c-47dd-bf3d-55a8fd570dac-openshift-service-ca\") pod \"maas-api-e2e-shared-a-a56db7-586cd6c985-m9xm2\" (UID: \"9d684956-954c-47dd-bf3d-55a8fd570dac\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a56db7-586cd6c985-m9xm2"
Jul 09 20:40:29.413976 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:29.413951    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d684956-954c-47dd-bf3d-55a8fd570dac-maas-api-tls\") pod \"maas-api-e2e-shared-a-a56db7-586cd6c985-m9xm2\" (UID: \"9d684956-954c-47dd-bf3d-55a8fd570dac\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a56db7-586cd6c985-m9xm2"
Jul 09 20:40:29.414073 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:29.414005    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lqxr\" (UniqueName: \"kubernetes.io/projected/9d684956-954c-47dd-bf3d-55a8fd570dac-kube-api-access-9lqxr\") pod \"maas-api-e2e-shared-a-a56db7-586cd6c985-m9xm2\" (UID: \"9d684956-954c-47dd-bf3d-55a8fd570dac\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a56db7-586cd6c985-m9xm2"
Jul 09 20:40:29.514762 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:29.514691    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lqxr\" (UniqueName: \"kubernetes.io/projected/9d684956-954c-47dd-bf3d-55a8fd570dac-kube-api-access-9lqxr\") pod \"maas-api-e2e-shared-a-a56db7-586cd6c985-m9xm2\" (UID: \"9d684956-954c-47dd-bf3d-55a8fd570dac\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a56db7-586cd6c985-m9xm2"
Jul 09 20:40:29.514762 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:29.514744    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d684956-954c-47dd-bf3d-55a8fd570dac-openshift-service-ca\") pod \"maas-api-e2e-shared-a-a56db7-586cd6c985-m9xm2\" (UID: \"9d684956-954c-47dd-bf3d-55a8fd570dac\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a56db7-586cd6c985-m9xm2"
Jul 09 20:40:29.514932 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:29.514845    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d684956-954c-47dd-bf3d-55a8fd570dac-maas-api-tls\") pod \"maas-api-e2e-shared-a-a56db7-586cd6c985-m9xm2\" (UID: \"9d684956-954c-47dd-bf3d-55a8fd570dac\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a56db7-586cd6c985-m9xm2"
Jul 09 20:40:29.514994 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:40:29.514946    2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-a56db7-serving-cert: secret "maas-api-e2e-shared-a-a56db7-serving-cert" not found
Jul 09 20:40:29.515047 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:40:29.515028    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d684956-954c-47dd-bf3d-55a8fd570dac-maas-api-tls podName:9d684956-954c-47dd-bf3d-55a8fd570dac nodeName:}" failed. No retries permitted until 2026-07-09 20:40:30.01500805 +0000 UTC m=+2220.432677855 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9d684956-954c-47dd-bf3d-55a8fd570dac-maas-api-tls") pod "maas-api-e2e-shared-a-a56db7-586cd6c985-m9xm2" (UID: "9d684956-954c-47dd-bf3d-55a8fd570dac") : secret "maas-api-e2e-shared-a-a56db7-serving-cert" not found
Jul 09 20:40:29.515365 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:29.515348    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d684956-954c-47dd-bf3d-55a8fd570dac-openshift-service-ca\") pod \"maas-api-e2e-shared-a-a56db7-586cd6c985-m9xm2\" (UID: \"9d684956-954c-47dd-bf3d-55a8fd570dac\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a56db7-586cd6c985-m9xm2"
Jul 09 20:40:29.523631 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:29.523612    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lqxr\" (UniqueName: \"kubernetes.io/projected/9d684956-954c-47dd-bf3d-55a8fd570dac-kube-api-access-9lqxr\") pod \"maas-api-e2e-shared-a-a56db7-586cd6c985-m9xm2\" (UID: \"9d684956-954c-47dd-bf3d-55a8fd570dac\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a56db7-586cd6c985-m9xm2"
Jul 09 20:40:30.019169 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:30.019134    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d684956-954c-47dd-bf3d-55a8fd570dac-maas-api-tls\") pod \"maas-api-e2e-shared-a-a56db7-586cd6c985-m9xm2\" (UID: \"9d684956-954c-47dd-bf3d-55a8fd570dac\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a56db7-586cd6c985-m9xm2"
Jul 09 20:40:30.021466 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:30.021435    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d684956-954c-47dd-bf3d-55a8fd570dac-maas-api-tls\") pod \"maas-api-e2e-shared-a-a56db7-586cd6c985-m9xm2\" (UID: \"9d684956-954c-47dd-bf3d-55a8fd570dac\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a56db7-586cd6c985-m9xm2"
Jul 09 20:40:30.162856 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:30.162828    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a56db7-586cd6c985-m9xm2"
Jul 09 20:40:30.295516 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:30.295488    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-a56db7-586cd6c985-m9xm2"]
Jul 09 20:40:30.296380 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:40:30.296356    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d684956_954c_47dd_bf3d_55a8fd570dac.slice/crio-1d89236c2849ab244a27f017ab8894ac3ddfc69bc7b6cd2e5b0060a59f5c3419 WatchSource:0}: Error finding container 1d89236c2849ab244a27f017ab8894ac3ddfc69bc7b6cd2e5b0060a59f5c3419: Status 404 returned error can't find the container with id 1d89236c2849ab244a27f017ab8894ac3ddfc69bc7b6cd2e5b0060a59f5c3419
Jul 09 20:40:30.380654 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:30.380637    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 09 20:40:30.511095 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:30.511071    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-9df7ce-openshift-default-b7fbc5db5-9j65l"
Jul 09 20:40:30.624524 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:30.624473    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/366ce8c4-6f65-4d55-849e-6fd02a8526a0-istiod-ca-cert\") pod \"366ce8c4-6f65-4d55-849e-6fd02a8526a0\" (UID: \"366ce8c4-6f65-4d55-849e-6fd02a8526a0\") "
Jul 09 20:40:30.624524 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:30.624521    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-54d77\" (UniqueName: \"kubernetes.io/projected/366ce8c4-6f65-4d55-849e-6fd02a8526a0-kube-api-access-54d77\") pod \"366ce8c4-6f65-4d55-849e-6fd02a8526a0\" (UID: \"366ce8c4-6f65-4d55-849e-6fd02a8526a0\") "
Jul 09 20:40:30.624728 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:30.624568    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/366ce8c4-6f65-4d55-849e-6fd02a8526a0-credential-socket\") pod \"366ce8c4-6f65-4d55-849e-6fd02a8526a0\" (UID: \"366ce8c4-6f65-4d55-849e-6fd02a8526a0\") "
Jul 09 20:40:30.624728 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:30.624590    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/366ce8c4-6f65-4d55-849e-6fd02a8526a0-workload-socket\") pod \"366ce8c4-6f65-4d55-849e-6fd02a8526a0\" (UID: \"366ce8c4-6f65-4d55-849e-6fd02a8526a0\") "
Jul 09 20:40:30.624728 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:30.624607    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/366ce8c4-6f65-4d55-849e-6fd02a8526a0-istio-token\") pod \"366ce8c4-6f65-4d55-849e-6fd02a8526a0\" (UID: \"366ce8c4-6f65-4d55-849e-6fd02a8526a0\") "
Jul 09 20:40:30.624728 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:30.624633    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/366ce8c4-6f65-4d55-849e-6fd02a8526a0-istio-envoy\") pod \"366ce8c4-6f65-4d55-849e-6fd02a8526a0\" (UID: \"366ce8c4-6f65-4d55-849e-6fd02a8526a0\") "
Jul 09 20:40:30.624929 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:30.624787    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/366ce8c4-6f65-4d55-849e-6fd02a8526a0-workload-certs\") pod \"366ce8c4-6f65-4d55-849e-6fd02a8526a0\" (UID: \"366ce8c4-6f65-4d55-849e-6fd02a8526a0\") "
Jul 09 20:40:30.624929 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:30.624830    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/366ce8c4-6f65-4d55-849e-6fd02a8526a0-istio-data\") pod \"366ce8c4-6f65-4d55-849e-6fd02a8526a0\" (UID: \"366ce8c4-6f65-4d55-849e-6fd02a8526a0\") "
Jul 09 20:40:30.624929 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:30.624873    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/366ce8c4-6f65-4d55-849e-6fd02a8526a0-istio-podinfo\") pod \"366ce8c4-6f65-4d55-849e-6fd02a8526a0\" (UID: \"366ce8c4-6f65-4d55-849e-6fd02a8526a0\") "
Jul 09 20:40:30.624929 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:30.624894    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/366ce8c4-6f65-4d55-849e-6fd02a8526a0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "366ce8c4-6f65-4d55-849e-6fd02a8526a0" (UID: "366ce8c4-6f65-4d55-849e-6fd02a8526a0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 20:40:30.624929 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:30.624905    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/366ce8c4-6f65-4d55-849e-6fd02a8526a0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "366ce8c4-6f65-4d55-849e-6fd02a8526a0" (UID: "366ce8c4-6f65-4d55-849e-6fd02a8526a0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 20:40:30.625125 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:30.624924    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/366ce8c4-6f65-4d55-849e-6fd02a8526a0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "366ce8c4-6f65-4d55-849e-6fd02a8526a0" (UID: "366ce8c4-6f65-4d55-849e-6fd02a8526a0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 20:40:30.625125 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:30.625027    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/366ce8c4-6f65-4d55-849e-6fd02a8526a0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "366ce8c4-6f65-4d55-849e-6fd02a8526a0" (UID: "366ce8c4-6f65-4d55-849e-6fd02a8526a0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 20:40:30.625241 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:30.625217    2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/366ce8c4-6f65-4d55-849e-6fd02a8526a0-workload-socket\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:40:30.625241 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:30.625240    2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/366ce8c4-6f65-4d55-849e-6fd02a8526a0-workload-certs\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:40:30.625417 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:30.625253    2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/366ce8c4-6f65-4d55-849e-6fd02a8526a0-istiod-ca-cert\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:40:30.625417 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:30.625264    2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/366ce8c4-6f65-4d55-849e-6fd02a8526a0-credential-socket\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:40:30.625669 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:30.625647    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/366ce8c4-6f65-4d55-849e-6fd02a8526a0-istio-data" (OuterVolumeSpecName: "istio-data") pod "366ce8c4-6f65-4d55-849e-6fd02a8526a0" (UID: "366ce8c4-6f65-4d55-849e-6fd02a8526a0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 20:40:30.626802 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:30.626780    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/366ce8c4-6f65-4d55-849e-6fd02a8526a0-kube-api-access-54d77" (OuterVolumeSpecName: "kube-api-access-54d77") pod "366ce8c4-6f65-4d55-849e-6fd02a8526a0" (UID: "366ce8c4-6f65-4d55-849e-6fd02a8526a0"). InnerVolumeSpecName "kube-api-access-54d77". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 20:40:30.626869 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:30.626806    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/366ce8c4-6f65-4d55-849e-6fd02a8526a0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "366ce8c4-6f65-4d55-849e-6fd02a8526a0" (UID: "366ce8c4-6f65-4d55-849e-6fd02a8526a0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 20:40:30.627186 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:30.627170    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/366ce8c4-6f65-4d55-849e-6fd02a8526a0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "366ce8c4-6f65-4d55-849e-6fd02a8526a0" (UID: "366ce8c4-6f65-4d55-849e-6fd02a8526a0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 09 20:40:30.627233 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:30.627203    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/366ce8c4-6f65-4d55-849e-6fd02a8526a0-istio-token" (OuterVolumeSpecName: "istio-token") pod "366ce8c4-6f65-4d55-849e-6fd02a8526a0" (UID: "366ce8c4-6f65-4d55-849e-6fd02a8526a0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 20:40:30.726482 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:30.726462    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54d77\" (UniqueName: \"kubernetes.io/projected/366ce8c4-6f65-4d55-849e-6fd02a8526a0-kube-api-access-54d77\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:40:30.726482 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:30.726480    2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/366ce8c4-6f65-4d55-849e-6fd02a8526a0-istio-token\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:40:30.726594 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:30.726490    2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/366ce8c4-6f65-4d55-849e-6fd02a8526a0-istio-envoy\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:40:30.726594 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:30.726497    2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/366ce8c4-6f65-4d55-849e-6fd02a8526a0-istio-data\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:40:30.726594 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:30.726504    2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/366ce8c4-6f65-4d55-849e-6fd02a8526a0-istio-podinfo\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:40:31.260122 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:31.260049    2562 generic.go:356] "Generic (PLEG): container finished" podID="366ce8c4-6f65-4d55-849e-6fd02a8526a0" containerID="6c5acf6cc5a133207a35c7a0d85445e8fab95bf49b1ff7d27e94861465fca8e5" exitCode=0
Jul 09 20:40:31.260122 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:31.260097    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-9df7ce-openshift-default-b7fbc5db5-9j65l"
Jul 09 20:40:31.260326 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:31.260136    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9df7ce-openshift-default-b7fbc5db5-9j65l" event={"ID":"366ce8c4-6f65-4d55-849e-6fd02a8526a0","Type":"ContainerDied","Data":"6c5acf6cc5a133207a35c7a0d85445e8fab95bf49b1ff7d27e94861465fca8e5"}
Jul 09 20:40:31.260326 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:31.260189    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9df7ce-openshift-default-b7fbc5db5-9j65l" event={"ID":"366ce8c4-6f65-4d55-849e-6fd02a8526a0","Type":"ContainerDied","Data":"326a4dd93e54ae5ab5b6c4bbcc06c4e2cabfae93ccea096e287dcf58a979beb6"}
Jul 09 20:40:31.260326 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:31.260221    2562 scope.go:117] "RemoveContainer" containerID="6c5acf6cc5a133207a35c7a0d85445e8fab95bf49b1ff7d27e94861465fca8e5"
Jul 09 20:40:31.261788 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:31.261760    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a56db7-586cd6c985-m9xm2" event={"ID":"9d684956-954c-47dd-bf3d-55a8fd570dac","Type":"ContainerStarted","Data":"df90f154dc6c58d10c3e2d86733ef736ec3e160a38ab0548a39b920fac1eeddf"}
Jul 09 20:40:31.261923 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:31.261812    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a56db7-586cd6c985-m9xm2" event={"ID":"9d684956-954c-47dd-bf3d-55a8fd570dac","Type":"ContainerStarted","Data":"1d89236c2849ab244a27f017ab8894ac3ddfc69bc7b6cd2e5b0060a59f5c3419"}
Jul 09 20:40:31.261923 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:31.261846    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a56db7-586cd6c985-m9xm2"
Jul 09 20:40:31.269643 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:31.269628    2562 scope.go:117] "RemoveContainer" containerID="3c5939cbb7801d52270847df51b7d95b3b5e8ff567e77ee82420e8744be03dd3"
Jul 09 20:40:31.276703 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:31.276686    2562 scope.go:117] "RemoveContainer" containerID="6c5acf6cc5a133207a35c7a0d85445e8fab95bf49b1ff7d27e94861465fca8e5"
Jul 09 20:40:31.276921 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:40:31.276905    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c5acf6cc5a133207a35c7a0d85445e8fab95bf49b1ff7d27e94861465fca8e5\": container with ID starting with 6c5acf6cc5a133207a35c7a0d85445e8fab95bf49b1ff7d27e94861465fca8e5 not found: ID does not exist" containerID="6c5acf6cc5a133207a35c7a0d85445e8fab95bf49b1ff7d27e94861465fca8e5"
Jul 09 20:40:31.276965 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:31.276928    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c5acf6cc5a133207a35c7a0d85445e8fab95bf49b1ff7d27e94861465fca8e5"} err="failed to get container status \"6c5acf6cc5a133207a35c7a0d85445e8fab95bf49b1ff7d27e94861465fca8e5\": rpc error: code = NotFound desc = could not find container \"6c5acf6cc5a133207a35c7a0d85445e8fab95bf49b1ff7d27e94861465fca8e5\": container with ID starting with 6c5acf6cc5a133207a35c7a0d85445e8fab95bf49b1ff7d27e94861465fca8e5 not found: ID does not exist"
Jul 09 20:40:31.276965 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:31.276942    2562 scope.go:117] "RemoveContainer" containerID="3c5939cbb7801d52270847df51b7d95b3b5e8ff567e77ee82420e8744be03dd3"
Jul 09 20:40:31.277178 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:40:31.277161    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c5939cbb7801d52270847df51b7d95b3b5e8ff567e77ee82420e8744be03dd3\": container with ID starting with 3c5939cbb7801d52270847df51b7d95b3b5e8ff567e77ee82420e8744be03dd3 not found: ID does not exist" containerID="3c5939cbb7801d52270847df51b7d95b3b5e8ff567e77ee82420e8744be03dd3"
Jul 09 20:40:31.277215 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:31.277185    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c5939cbb7801d52270847df51b7d95b3b5e8ff567e77ee82420e8744be03dd3"} err="failed to get container status \"3c5939cbb7801d52270847df51b7d95b3b5e8ff567e77ee82420e8744be03dd3\": rpc error: code = NotFound desc = could not find container \"3c5939cbb7801d52270847df51b7d95b3b5e8ff567e77ee82420e8744be03dd3\": container with ID starting with 3c5939cbb7801d52270847df51b7d95b3b5e8ff567e77ee82420e8744be03dd3 not found: ID does not exist"
Jul 09 20:40:31.281892 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:31.281854    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a56db7-586cd6c985-m9xm2" podStartSLOduration=1.706813333 podStartE2EDuration="2.281840163s" totalImagesPullingTime="575.02683ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 20:40:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 20:40:31.281287752 +0000 UTC m=+2221.698957581" watchObservedRunningTime="2026-07-09 20:40:31.281840163 +0000 UTC m=+2221.699509971"
Jul 09 20:40:31.295161 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:31.295143    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-9df7ce-openshift-default-b7fbc5db5-9j65l"]
Jul 09 20:40:31.299363 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:31.299345    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-9df7ce-openshift-default-b7fbc5db5-9j65l"]
Jul 09 20:40:32.241444 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:32.241413    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="366ce8c4-6f65-4d55-849e-6fd02a8526a0" path="/var/lib/kubelet/pods/366ce8c4-6f65-4d55-849e-6fd02a8526a0/volumes"
Jul 09 20:40:34.287685 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:34.287643    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-8c2e89-openshift-default-7c79cfb855-gvzzq"]
Jul 09 20:40:34.288172 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:34.288154    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="366ce8c4-6f65-4d55-849e-6fd02a8526a0" containerName="istio-proxy"
Jul 09 20:40:34.288258 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:34.288175    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="366ce8c4-6f65-4d55-849e-6fd02a8526a0" containerName="istio-proxy"
Jul 09 20:40:34.288258 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:34.288187    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="366ce8c4-6f65-4d55-849e-6fd02a8526a0" containerName="istio-proxy"
Jul 09 20:40:34.288258 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:34.288194    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="366ce8c4-6f65-4d55-849e-6fd02a8526a0" containerName="istio-proxy"
Jul 09 20:40:34.288419 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:34.288281    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="366ce8c4-6f65-4d55-849e-6fd02a8526a0" containerName="istio-proxy"
Jul 09 20:40:34.288419 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:34.288292    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="366ce8c4-6f65-4d55-849e-6fd02a8526a0" containerName="istio-proxy"
Jul 09 20:40:34.291247 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:34.291227    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-8c2e89-openshift-default-7c79cfb855-gvzzq"
Jul 09 20:40:34.294157 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:34.294134    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-8c2e89-openshift-default-dockercfg-cxv8h\""
Jul 09 20:40:34.300854 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:34.300835    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-8c2e89-openshift-default-7c79cfb855-gvzzq"]
Jul 09 20:40:34.457210 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:34.457176    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f06646f-6667-4d8b-88c4-caeccd277249-istio-data\") pod \"e2e-shared-b-8c2e89-openshift-default-7c79cfb855-gvzzq\" (UID: \"0f06646f-6667-4d8b-88c4-caeccd277249\") " pod="openshift-ingress/e2e-shared-b-8c2e89-openshift-default-7c79cfb855-gvzzq"
Jul 09 20:40:34.457396 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:34.457243    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f06646f-6667-4d8b-88c4-caeccd277249-workload-socket\") pod \"e2e-shared-b-8c2e89-openshift-default-7c79cfb855-gvzzq\" (UID: \"0f06646f-6667-4d8b-88c4-caeccd277249\") " pod="openshift-ingress/e2e-shared-b-8c2e89-openshift-default-7c79cfb855-gvzzq"
Jul 09 20:40:34.457396 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:34.457275    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kp5hm\" (UniqueName: \"kubernetes.io/projected/0f06646f-6667-4d8b-88c4-caeccd277249-kube-api-access-kp5hm\") pod \"e2e-shared-b-8c2e89-openshift-default-7c79cfb855-gvzzq\" (UID: \"0f06646f-6667-4d8b-88c4-caeccd277249\") " pod="openshift-ingress/e2e-shared-b-8c2e89-openshift-default-7c79cfb855-gvzzq"
Jul 09 20:40:34.457396 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:34.457319    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f06646f-6667-4d8b-88c4-caeccd277249-istio-token\") pod \"e2e-shared-b-8c2e89-openshift-default-7c79cfb855-gvzzq\" (UID: \"0f06646f-6667-4d8b-88c4-caeccd277249\") " pod="openshift-ingress/e2e-shared-b-8c2e89-openshift-default-7c79cfb855-gvzzq"
Jul 09 20:40:34.457396 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:34.457373    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f06646f-6667-4d8b-88c4-caeccd277249-istiod-ca-cert\") pod \"e2e-shared-b-8c2e89-openshift-default-7c79cfb855-gvzzq\" (UID: \"0f06646f-6667-4d8b-88c4-caeccd277249\") " pod="openshift-ingress/e2e-shared-b-8c2e89-openshift-default-7c79cfb855-gvzzq"
Jul 09 20:40:34.457615 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:34.457413    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f06646f-6667-4d8b-88c4-caeccd277249-istio-envoy\") pod \"e2e-shared-b-8c2e89-openshift-default-7c79cfb855-gvzzq\" (UID: \"0f06646f-6667-4d8b-88c4-caeccd277249\") " pod="openshift-ingress/e2e-shared-b-8c2e89-openshift-default-7c79cfb855-gvzzq"
Jul 09 20:40:34.457615 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:34.457494    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f06646f-6667-4d8b-88c4-caeccd277249-credential-socket\") pod \"e2e-shared-b-8c2e89-openshift-default-7c79cfb855-gvzzq\" (UID: \"0f06646f-6667-4d8b-88c4-caeccd277249\") " pod="openshift-ingress/e2e-shared-b-8c2e89-openshift-default-7c79cfb855-gvzzq"
Jul 09 20:40:34.457615 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:34.457556    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f06646f-6667-4d8b-88c4-caeccd277249-workload-certs\") pod \"e2e-shared-b-8c2e89-openshift-default-7c79cfb855-gvzzq\" (UID: \"0f06646f-6667-4d8b-88c4-caeccd277249\") " pod="openshift-ingress/e2e-shared-b-8c2e89-openshift-default-7c79cfb855-gvzzq"
Jul 09 20:40:34.457615 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:34.457589    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f06646f-6667-4d8b-88c4-caeccd277249-istio-podinfo\") pod \"e2e-shared-b-8c2e89-openshift-default-7c79cfb855-gvzzq\" (UID: \"0f06646f-6667-4d8b-88c4-caeccd277249\") " pod="openshift-ingress/e2e-shared-b-8c2e89-openshift-default-7c79cfb855-gvzzq"
Jul 09 20:40:34.558802 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:34.558716    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f06646f-6667-4d8b-88c4-caeccd277249-credential-socket\") pod \"e2e-shared-b-8c2e89-openshift-default-7c79cfb855-gvzzq\" (UID: \"0f06646f-6667-4d8b-88c4-caeccd277249\") " pod="openshift-ingress/e2e-shared-b-8c2e89-openshift-default-7c79cfb855-gvzzq"
Jul 09 20:40:34.558802 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:34.558759    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f06646f-6667-4d8b-88c4-caeccd277249-workload-certs\") pod \"e2e-shared-b-8c2e89-openshift-default-7c79cfb855-gvzzq\" (UID: \"0f06646f-6667-4d8b-88c4-caeccd277249\") " pod="openshift-ingress/e2e-shared-b-8c2e89-openshift-default-7c79cfb855-gvzzq"
Jul 09 20:40:34.558802 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:34.558782    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f06646f-6667-4d8b-88c4-caeccd277249-istio-podinfo\") pod \"e2e-shared-b-8c2e89-openshift-default-7c79cfb855-gvzzq\" (UID: \"0f06646f-6667-4d8b-88c4-caeccd277249\") " pod="openshift-ingress/e2e-shared-b-8c2e89-openshift-default-7c79cfb855-gvzzq"
Jul 09 20:40:34.559048 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:34.558825    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f06646f-6667-4d8b-88c4-caeccd277249-istio-data\") pod \"e2e-shared-b-8c2e89-openshift-default-7c79cfb855-gvzzq\" (UID: \"0f06646f-6667-4d8b-88c4-caeccd277249\") " pod="openshift-ingress/e2e-shared-b-8c2e89-openshift-default-7c79cfb855-gvzzq"
Jul 09 20:40:34.559048 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:34.558882    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f06646f-6667-4d8b-88c4-caeccd277249-workload-socket\") pod \"e2e-shared-b-8c2e89-openshift-default-7c79cfb855-gvzzq\" (UID: \"0f06646f-6667-4d8b-88c4-caeccd277249\") " pod="openshift-ingress/e2e-shared-b-8c2e89-openshift-default-7c79cfb855-gvzzq"
Jul 09 20:40:34.559048 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:34.558912    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kp5hm\" (UniqueName: \"kubernetes.io/projected/0f06646f-6667-4d8b-88c4-caeccd277249-kube-api-access-kp5hm\") pod \"e2e-shared-b-8c2e89-openshift-default-7c79cfb855-gvzzq\" (UID: \"0f06646f-6667-4d8b-88c4-caeccd277249\") " pod="openshift-ingress/e2e-shared-b-8c2e89-openshift-default-7c79cfb855-gvzzq"
Jul 09 20:40:34.559048 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:34.558942    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f06646f-6667-4d8b-88c4-caeccd277249-istio-token\") pod \"e2e-shared-b-8c2e89-openshift-default-7c79cfb855-gvzzq\" (UID: \"0f06646f-6667-4d8b-88c4-caeccd277249\") " pod="openshift-ingress/e2e-shared-b-8c2e89-openshift-default-7c79cfb855-gvzzq"
Jul 09 20:40:34.559048 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:34.558972    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f06646f-6667-4d8b-88c4-caeccd277249-istiod-ca-cert\") pod \"e2e-shared-b-8c2e89-openshift-default-7c79cfb855-gvzzq\" (UID: \"0f06646f-6667-4d8b-88c4-caeccd277249\") " pod="openshift-ingress/e2e-shared-b-8c2e89-openshift-default-7c79cfb855-gvzzq"
Jul 09 20:40:34.559048 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:34.558997    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f06646f-6667-4d8b-88c4-caeccd277249-istio-envoy\") pod \"e2e-shared-b-8c2e89-openshift-default-7c79cfb855-gvzzq\" (UID: \"0f06646f-6667-4d8b-88c4-caeccd277249\") " pod="openshift-ingress/e2e-shared-b-8c2e89-openshift-default-7c79cfb855-gvzzq"
Jul 09 20:40:34.559402 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:34.559148    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f06646f-6667-4d8b-88c4-caeccd277249-workload-certs\") pod \"e2e-shared-b-8c2e89-openshift-default-7c79cfb855-gvzzq\" (UID: \"0f06646f-6667-4d8b-88c4-caeccd277249\") " pod="openshift-ingress/e2e-shared-b-8c2e89-openshift-default-7c79cfb855-gvzzq"
Jul 09 20:40:34.559402 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:34.559165    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f06646f-6667-4d8b-88c4-caeccd277249-credential-socket\") pod \"e2e-shared-b-8c2e89-openshift-default-7c79cfb855-gvzzq\" (UID: \"0f06646f-6667-4d8b-88c4-caeccd277249\") " pod="openshift-ingress/e2e-shared-b-8c2e89-openshift-default-7c79cfb855-gvzzq"
Jul 09 20:40:34.559402 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:34.559257    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f06646f-6667-4d8b-88c4-caeccd277249-workload-socket\") pod \"e2e-shared-b-8c2e89-openshift-default-7c79cfb855-gvzzq\" (UID: \"0f06646f-6667-4d8b-88c4-caeccd277249\") " pod="openshift-ingress/e2e-shared-b-8c2e89-openshift-default-7c79cfb855-gvzzq"
Jul 09 20:40:34.559402 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:34.559353    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f06646f-6667-4d8b-88c4-caeccd277249-istio-data\") pod \"e2e-shared-b-8c2e89-openshift-default-7c79cfb855-gvzzq\" (UID: \"0f06646f-6667-4d8b-88c4-caeccd277249\") " pod="openshift-ingress/e2e-shared-b-8c2e89-openshift-default-7c79cfb855-gvzzq"
Jul 09 20:40:34.559758 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:34.559739    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f06646f-6667-4d8b-88c4-caeccd277249-istiod-ca-cert\") pod \"e2e-shared-b-8c2e89-openshift-default-7c79cfb855-gvzzq\" (UID: \"0f06646f-6667-4d8b-88c4-caeccd277249\") " pod="openshift-ingress/e2e-shared-b-8c2e89-openshift-default-7c79cfb855-gvzzq"
Jul 09 20:40:34.561070 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:34.561039    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f06646f-6667-4d8b-88c4-caeccd277249-istio-envoy\") pod \"e2e-shared-b-8c2e89-openshift-default-7c79cfb855-gvzzq\" (UID: \"0f06646f-6667-4d8b-88c4-caeccd277249\") " pod="openshift-ingress/e2e-shared-b-8c2e89-openshift-default-7c79cfb855-gvzzq"
Jul 09 20:40:34.561190 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:34.561171    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f06646f-6667-4d8b-88c4-caeccd277249-istio-podinfo\") pod \"e2e-shared-b-8c2e89-openshift-default-7c79cfb855-gvzzq\" (UID: \"0f06646f-6667-4d8b-88c4-caeccd277249\") " pod="openshift-ingress/e2e-shared-b-8c2e89-openshift-default-7c79cfb855-gvzzq"
Jul 09 20:40:34.566858 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:34.566833    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f06646f-6667-4d8b-88c4-caeccd277249-istio-token\") pod \"e2e-shared-b-8c2e89-openshift-default-7c79cfb855-gvzzq\" (UID: \"0f06646f-6667-4d8b-88c4-caeccd277249\") " pod="openshift-ingress/e2e-shared-b-8c2e89-openshift-default-7c79cfb855-gvzzq"
Jul 09 20:40:34.567023 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:34.567007    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kp5hm\" (UniqueName: \"kubernetes.io/projected/0f06646f-6667-4d8b-88c4-caeccd277249-kube-api-access-kp5hm\") pod \"e2e-shared-b-8c2e89-openshift-default-7c79cfb855-gvzzq\" (UID: \"0f06646f-6667-4d8b-88c4-caeccd277249\") " pod="openshift-ingress/e2e-shared-b-8c2e89-openshift-default-7c79cfb855-gvzzq"
Jul 09 20:40:34.602839 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:34.602818    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-8c2e89-openshift-default-7c79cfb855-gvzzq"
Jul 09 20:40:34.732088 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:34.732062    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-8c2e89-openshift-default-7c79cfb855-gvzzq"]
Jul 09 20:40:34.734176 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:40:34.734150    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f06646f_6667_4d8b_88c4_caeccd277249.slice/crio-9349ca53674b6078f4d58b8c1447086b0ce528d61174ca8caff6b0a075673f6d WatchSource:0}: Error finding container 9349ca53674b6078f4d58b8c1447086b0ce528d61174ca8caff6b0a075673f6d: Status 404 returned error can't find the container with id 9349ca53674b6078f4d58b8c1447086b0ce528d61174ca8caff6b0a075673f6d
Jul 09 20:40:34.736715 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:34.736683    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 09 20:40:34.736802 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:34.736751    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 09 20:40:34.736802 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:34.736784    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 09 20:40:35.283698 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:35.283668    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-8c2e89-openshift-default-7c79cfb855-gvzzq" event={"ID":"0f06646f-6667-4d8b-88c4-caeccd277249","Type":"ContainerStarted","Data":"6c03420022f567a8b7a583857ad6ec7737c73f117a745a506907c89e17050777"}
Jul 09 20:40:35.283858 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:35.283710    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-8c2e89-openshift-default-7c79cfb855-gvzzq" event={"ID":"0f06646f-6667-4d8b-88c4-caeccd277249","Type":"ContainerStarted","Data":"9349ca53674b6078f4d58b8c1447086b0ce528d61174ca8caff6b0a075673f6d"}
Jul 09 20:40:35.308135 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:35.308091    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-8c2e89-openshift-default-7c79cfb855-gvzzq" podStartSLOduration=1.308076714 podStartE2EDuration="1.308076714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 20:40:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 20:40:35.305001502 +0000 UTC m=+2225.722671310" watchObservedRunningTime="2026-07-09 20:40:35.308076714 +0000 UTC m=+2225.725746520"
Jul 09 20:40:35.603858 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:35.603790    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-8c2e89-openshift-default-7c79cfb855-gvzzq"
Jul 09 20:40:35.608233 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:35.608212    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-8c2e89-openshift-default-7c79cfb855-gvzzq"
Jul 09 20:40:36.288073 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:36.288047    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-8c2e89-openshift-default-7c79cfb855-gvzzq"
Jul 09 20:40:36.288900 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:36.288881    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-8c2e89-openshift-default-7c79cfb855-gvzzq"
Jul 09 20:40:37.273111 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:37.273081    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a56db7-586cd6c985-m9xm2"
Jul 09 20:40:37.278939 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:37.278908    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-8c2e89-8544fc76bd-5gwdp"]
Jul 09 20:40:37.283242 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:37.283220    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-8c2e89-8544fc76bd-5gwdp"
Jul 09 20:40:37.285876 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:37.285857    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-8c2e89-serving-cert\""
Jul 09 20:40:37.294070 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:37.293999    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-8c2e89-8544fc76bd-5gwdp"]
Jul 09 20:40:37.380755 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:37.380714    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67a91531-0e54-497d-a167-7b5398aff15a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-8c2e89-8544fc76bd-5gwdp\" (UID: \"67a91531-0e54-497d-a167-7b5398aff15a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-8c2e89-8544fc76bd-5gwdp"
Jul 09 20:40:37.380896 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:37.380862    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j29c2\" (UniqueName: \"kubernetes.io/projected/67a91531-0e54-497d-a167-7b5398aff15a-kube-api-access-j29c2\") pod \"maas-api-e2e-shared-b-8c2e89-8544fc76bd-5gwdp\" (UID: \"67a91531-0e54-497d-a167-7b5398aff15a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-8c2e89-8544fc76bd-5gwdp"
Jul 09 20:40:37.380958 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:37.380928    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67a91531-0e54-497d-a167-7b5398aff15a-maas-api-tls\") pod \"maas-api-e2e-shared-b-8c2e89-8544fc76bd-5gwdp\" (UID: \"67a91531-0e54-497d-a167-7b5398aff15a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-8c2e89-8544fc76bd-5gwdp"
Jul 09 20:40:37.482201 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:37.482172    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j29c2\" (UniqueName: \"kubernetes.io/projected/67a91531-0e54-497d-a167-7b5398aff15a-kube-api-access-j29c2\") pod \"maas-api-e2e-shared-b-8c2e89-8544fc76bd-5gwdp\" (UID: \"67a91531-0e54-497d-a167-7b5398aff15a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-8c2e89-8544fc76bd-5gwdp"
Jul 09 20:40:37.482412 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:37.482225    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67a91531-0e54-497d-a167-7b5398aff15a-maas-api-tls\") pod \"maas-api-e2e-shared-b-8c2e89-8544fc76bd-5gwdp\" (UID: \"67a91531-0e54-497d-a167-7b5398aff15a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-8c2e89-8544fc76bd-5gwdp"
Jul 09 20:40:37.482412 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:37.482290    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67a91531-0e54-497d-a167-7b5398aff15a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-8c2e89-8544fc76bd-5gwdp\" (UID: \"67a91531-0e54-497d-a167-7b5398aff15a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-8c2e89-8544fc76bd-5gwdp"
Jul 09 20:40:37.482412 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:40:37.482391    2562 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-8c2e89-serving-cert: secret "maas-api-e2e-shared-b-8c2e89-serving-cert" not found
Jul 09 20:40:37.482547 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:40:37.482457    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67a91531-0e54-497d-a167-7b5398aff15a-maas-api-tls podName:67a91531-0e54-497d-a167-7b5398aff15a nodeName:}" failed. No retries permitted until 2026-07-09 20:40:37.982441809 +0000 UTC m=+2228.400111596 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/67a91531-0e54-497d-a167-7b5398aff15a-maas-api-tls") pod "maas-api-e2e-shared-b-8c2e89-8544fc76bd-5gwdp" (UID: "67a91531-0e54-497d-a167-7b5398aff15a") : secret "maas-api-e2e-shared-b-8c2e89-serving-cert" not found
Jul 09 20:40:37.482937 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:37.482920    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67a91531-0e54-497d-a167-7b5398aff15a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-8c2e89-8544fc76bd-5gwdp\" (UID: \"67a91531-0e54-497d-a167-7b5398aff15a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-8c2e89-8544fc76bd-5gwdp"
Jul 09 20:40:37.493190 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:37.493164    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j29c2\" (UniqueName: \"kubernetes.io/projected/67a91531-0e54-497d-a167-7b5398aff15a-kube-api-access-j29c2\") pod \"maas-api-e2e-shared-b-8c2e89-8544fc76bd-5gwdp\" (UID: \"67a91531-0e54-497d-a167-7b5398aff15a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-8c2e89-8544fc76bd-5gwdp"
Jul 09 20:40:37.986997 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:37.986966    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67a91531-0e54-497d-a167-7b5398aff15a-maas-api-tls\") pod \"maas-api-e2e-shared-b-8c2e89-8544fc76bd-5gwdp\" (UID: \"67a91531-0e54-497d-a167-7b5398aff15a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-8c2e89-8544fc76bd-5gwdp"
Jul 09 20:40:37.989559 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:37.989502    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67a91531-0e54-497d-a167-7b5398aff15a-maas-api-tls\") pod \"maas-api-e2e-shared-b-8c2e89-8544fc76bd-5gwdp\" (UID: \"67a91531-0e54-497d-a167-7b5398aff15a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-8c2e89-8544fc76bd-5gwdp"
Jul 09 20:40:38.208690 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:38.208639    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-8c2e89-8544fc76bd-5gwdp"
Jul 09 20:40:38.387341 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:38.387148    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-8c2e89-8544fc76bd-5gwdp"]
Jul 09 20:40:38.390191 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:40:38.390015    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67a91531_0e54_497d_a167_7b5398aff15a.slice/crio-7d8fa18b88e21a3867b341f2fc0ff65c3769e6461ec1209ae7e2957dbd37360b WatchSource:0}: Error finding container 7d8fa18b88e21a3867b341f2fc0ff65c3769e6461ec1209ae7e2957dbd37360b: Status 404 returned error can't find the container with id 7d8fa18b88e21a3867b341f2fc0ff65c3769e6461ec1209ae7e2957dbd37360b
Jul 09 20:40:39.304776 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:39.304743    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-8c2e89-8544fc76bd-5gwdp" event={"ID":"67a91531-0e54-497d-a167-7b5398aff15a","Type":"ContainerStarted","Data":"0f9ae66f9943d2e95d1f544c3e34bf2e49f568c4e34f2cb982ec497a8a70d355"}
Jul 09 20:40:39.304949 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:39.304800    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-8c2e89-8544fc76bd-5gwdp" event={"ID":"67a91531-0e54-497d-a167-7b5398aff15a","Type":"ContainerStarted","Data":"7d8fa18b88e21a3867b341f2fc0ff65c3769e6461ec1209ae7e2957dbd37360b"}
Jul 09 20:40:39.304949 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:39.304912    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-8c2e89-8544fc76bd-5gwdp"
Jul 09 20:40:39.322096 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:39.322043    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-8c2e89-8544fc76bd-5gwdp" podStartSLOduration=1.795420419 podStartE2EDuration="2.322025238s" totalImagesPullingTime="526.604819ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 20:40:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 20:40:39.320216772 +0000 UTC m=+2229.737886580" watchObservedRunningTime="2026-07-09 20:40:39.322025238 +0000 UTC m=+2229.739695048"
Jul 09 20:40:44.281470 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:44.281433    2562 patch_prober.go:28] interesting pod/e2e-shared-a-a56db7-openshift-default-76dfdd4bf5-5rqvp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 09 20:40:44.281876 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:44.281503    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-a56db7-openshift-default-76dfdd4bf5-5rqvp" podUID="7d882374-1667-49f2-8c18-d7b8e159cb72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 09 20:40:45.329267 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:45.329172    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-8c2e89-8544fc76bd-5gwdp"
Jul 09 20:40:45.339799 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:45.339768    2562 generic.go:356] "Generic (PLEG): container finished" podID="7d882374-1667-49f2-8c18-d7b8e159cb72" containerID="f45cf252d313c46abd2d2bb754fd48e07d73386f2452ffcf05c1877d8fbe0b0c" exitCode=0
Jul 09 20:40:45.339976 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:45.339807    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-a56db7-openshift-default-76dfdd4bf5-5rqvp" event={"ID":"7d882374-1667-49f2-8c18-d7b8e159cb72","Type":"ContainerDied","Data":"f45cf252d313c46abd2d2bb754fd48e07d73386f2452ffcf05c1877d8fbe0b0c"}
Jul 09 20:40:45.340247 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:45.340228    2562 scope.go:117] "RemoveContainer" containerID="f45cf252d313c46abd2d2bb754fd48e07d73386f2452ffcf05c1877d8fbe0b0c"
Jul 09 20:40:45.341440 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:45.341411    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 09 20:40:45.341527 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:45.341490    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 09 20:40:45.341590 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:45.341560    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 09 20:40:46.350780 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:46.349832    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-a56db7-openshift-default-76dfdd4bf5-5rqvp" event={"ID":"7d882374-1667-49f2-8c18-d7b8e159cb72","Type":"ContainerStarted","Data":"08426554936b8963e67738e959cdc8ff0d6629258196afe5d297fab494bf0f95"}
Jul 09 20:40:46.494170 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:46.494138    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-a56db7-openshift-default-76dfdd4bf5-5rqvp"
Jul 09 20:40:46.494494 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:46.494471    2562 patch_prober.go:28] interesting pod/e2e-shared-a-a56db7-openshift-default-76dfdd4bf5-5rqvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Jul 09 20:40:46.494600 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:46.494524    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-a56db7-openshift-default-76dfdd4bf5-5rqvp" podUID="7d882374-1667-49f2-8c18-d7b8e159cb72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Jul 09 20:40:47.494193 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:47.494163    2562 patch_prober.go:28] interesting pod/e2e-shared-a-a56db7-openshift-default-76dfdd4bf5-5rqvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Jul 09 20:40:47.494596 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:47.494211    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-a56db7-openshift-default-76dfdd4bf5-5rqvp" podUID="7d882374-1667-49f2-8c18-d7b8e159cb72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Jul 09 20:40:48.493830 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:48.493798    2562 patch_prober.go:28] interesting pod/e2e-shared-a-a56db7-openshift-default-76dfdd4bf5-5rqvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Jul 09 20:40:48.494060 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:48.494039    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-a56db7-openshift-default-76dfdd4bf5-5rqvp" podUID="7d882374-1667-49f2-8c18-d7b8e159cb72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Jul 09 20:40:49.493912 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:49.493873    2562 patch_prober.go:28] interesting pod/e2e-shared-a-a56db7-openshift-default-76dfdd4bf5-5rqvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Jul 09 20:40:49.494325 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:49.493941    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-a56db7-openshift-default-76dfdd4bf5-5rqvp" podUID="7d882374-1667-49f2-8c18-d7b8e159cb72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Jul 09 20:40:50.496739 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:50.496705    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-a56db7-openshift-default-76dfdd4bf5-5rqvp"
Jul 09 20:40:50.497108 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:50.496995    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-a56db7-openshift-default-76dfdd4bf5-5rqvp"
Jul 09 20:40:50.497702 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:50.497683    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-a56db7-openshift-default-76dfdd4bf5-5rqvp"
Jul 09 20:40:51.669600 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:51.669265    2562 patch_prober.go:28] interesting pod/e2e-shared-b-8c2e89-openshift-default-7c79cfb855-gvzzq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": read tcp 10.134.0.2:50154->10.134.0.67:15021: read: connection reset by peer" start-of-body=
Jul 09 20:40:51.669600 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:51.669366    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-8c2e89-openshift-default-7c79cfb855-gvzzq" podUID="0f06646f-6667-4d8b-88c4-caeccd277249" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": read tcp 10.134.0.2:50154->10.134.0.67:15021: read: connection reset by peer"
Jul 09 20:40:52.389649 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:52.389622    2562 generic.go:356] "Generic (PLEG): container finished" podID="0f06646f-6667-4d8b-88c4-caeccd277249" containerID="6c03420022f567a8b7a583857ad6ec7737c73f117a745a506907c89e17050777" exitCode=0
Jul 09 20:40:52.389798 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:52.389703    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-8c2e89-openshift-default-7c79cfb855-gvzzq" event={"ID":"0f06646f-6667-4d8b-88c4-caeccd277249","Type":"ContainerDied","Data":"6c03420022f567a8b7a583857ad6ec7737c73f117a745a506907c89e17050777"}
Jul 09 20:40:52.390156 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:52.390141    2562 scope.go:117] "RemoveContainer" containerID="6c03420022f567a8b7a583857ad6ec7737c73f117a745a506907c89e17050777"
Jul 09 20:40:52.391043 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:52.391015    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 09 20:40:52.391149 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:52.391087    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 09 20:40:52.391149 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:52.391130    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 09 20:40:53.395959 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:53.395925    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-8c2e89-openshift-default-7c79cfb855-gvzzq" event={"ID":"0f06646f-6667-4d8b-88c4-caeccd277249","Type":"ContainerStarted","Data":"09ce4b255366412f649040f03b30582af180f0fe711c5c0c4bb6b912e560c3dc"}
Jul 09 20:40:53.603155 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:53.603124    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-8c2e89-openshift-default-7c79cfb855-gvzzq"
Jul 09 20:40:53.603348 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:53.603329    2562 patch_prober.go:28] interesting pod/e2e-shared-b-8c2e89-openshift-default-7c79cfb855-gvzzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Jul 09 20:40:53.603405 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:53.603383    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-8c2e89-openshift-default-7c79cfb855-gvzzq" podUID="0f06646f-6667-4d8b-88c4-caeccd277249" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Jul 09 20:40:54.603524 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:54.603493    2562 patch_prober.go:28] interesting pod/e2e-shared-b-8c2e89-openshift-default-7c79cfb855-gvzzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Jul 09 20:40:54.603981 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:54.603560    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-8c2e89-openshift-default-7c79cfb855-gvzzq" podUID="0f06646f-6667-4d8b-88c4-caeccd277249" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Jul 09 20:40:55.603522 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:55.603484    2562 patch_prober.go:28] interesting pod/e2e-shared-b-8c2e89-openshift-default-7c79cfb855-gvzzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Jul 09 20:40:55.603716 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:55.603528    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-8c2e89-openshift-default-7c79cfb855-gvzzq" podUID="0f06646f-6667-4d8b-88c4-caeccd277249" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Jul 09 20:40:56.603210 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:56.603178    2562 patch_prober.go:28] interesting pod/e2e-shared-b-8c2e89-openshift-default-7c79cfb855-gvzzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Jul 09 20:40:56.603385 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:56.603257    2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-8c2e89-openshift-default-7c79cfb855-gvzzq" podUID="0f06646f-6667-4d8b-88c4-caeccd277249" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Jul 09 20:40:57.606438 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:57.606409    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-8c2e89-openshift-default-7c79cfb855-gvzzq"
Jul 09 20:40:57.606908 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:57.606737    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-8c2e89-openshift-default-7c79cfb855-gvzzq"
Jul 09 20:40:57.607263 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:40:57.607244    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-8c2e89-openshift-default-7c79cfb855-gvzzq"
Jul 09 20:43:30.245631 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:43:30.245601    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-cqs6d_578878d0-e6b3-4725-a698-796cfc169fa6/console-operator/1.log"
Jul 09 20:43:30.251027 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:43:30.251007    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hds4g_4b4468fd-1708-487b-8d69-bb7d817e25de/ovn-acl-logging/0.log"
Jul 09 20:43:30.251709 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:43:30.251691    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-cqs6d_578878d0-e6b3-4725-a698-796cfc169fa6/console-operator/1.log"
Jul 09 20:43:30.256704 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:43:30.256687    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hds4g_4b4468fd-1708-487b-8d69-bb7d817e25de/ovn-acl-logging/0.log"
Jul 09 20:46:46.587396 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:46:46.587363    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-a56db7-586cd6c985-m9xm2"]
Jul 09 20:46:46.591010 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:46:46.587602    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a56db7-586cd6c985-m9xm2" podUID="9d684956-954c-47dd-bf3d-55a8fd570dac" containerName="maas-api" containerID="cri-o://df90f154dc6c58d10c3e2d86733ef736ec3e160a38ab0548a39b920fac1eeddf" gracePeriod=30
Jul 09 20:46:46.962064 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:46:46.962030    2562 generic.go:356] "Generic (PLEG): container finished" podID="9d684956-954c-47dd-bf3d-55a8fd570dac" containerID="df90f154dc6c58d10c3e2d86733ef736ec3e160a38ab0548a39b920fac1eeddf" exitCode=0
Jul 09 20:46:46.962204 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:46:46.962079    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a56db7-586cd6c985-m9xm2" event={"ID":"9d684956-954c-47dd-bf3d-55a8fd570dac","Type":"ContainerDied","Data":"df90f154dc6c58d10c3e2d86733ef736ec3e160a38ab0548a39b920fac1eeddf"}
Jul 09 20:46:47.028328 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:46:47.028293    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a56db7-586cd6c985-m9xm2"
Jul 09 20:46:47.173794 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:46:47.173735    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d684956-954c-47dd-bf3d-55a8fd570dac-maas-api-tls\") pod \"9d684956-954c-47dd-bf3d-55a8fd570dac\" (UID: \"9d684956-954c-47dd-bf3d-55a8fd570dac\") "
Jul 09 20:46:47.173920 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:46:47.173828    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9lqxr\" (UniqueName: \"kubernetes.io/projected/9d684956-954c-47dd-bf3d-55a8fd570dac-kube-api-access-9lqxr\") pod \"9d684956-954c-47dd-bf3d-55a8fd570dac\" (UID: \"9d684956-954c-47dd-bf3d-55a8fd570dac\") "
Jul 09 20:46:47.173920 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:46:47.173868    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d684956-954c-47dd-bf3d-55a8fd570dac-openshift-service-ca\") pod \"9d684956-954c-47dd-bf3d-55a8fd570dac\" (UID: \"9d684956-954c-47dd-bf3d-55a8fd570dac\") "
Jul 09 20:46:47.174254 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:46:47.174224    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d684956-954c-47dd-bf3d-55a8fd570dac-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9d684956-954c-47dd-bf3d-55a8fd570dac" (UID: "9d684956-954c-47dd-bf3d-55a8fd570dac"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 20:46:47.175592 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:46:47.175571    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d684956-954c-47dd-bf3d-55a8fd570dac-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9d684956-954c-47dd-bf3d-55a8fd570dac" (UID: "9d684956-954c-47dd-bf3d-55a8fd570dac"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 20:46:47.175688 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:46:47.175675    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d684956-954c-47dd-bf3d-55a8fd570dac-kube-api-access-9lqxr" (OuterVolumeSpecName: "kube-api-access-9lqxr") pod "9d684956-954c-47dd-bf3d-55a8fd570dac" (UID: "9d684956-954c-47dd-bf3d-55a8fd570dac"). InnerVolumeSpecName "kube-api-access-9lqxr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 20:46:47.275427 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:46:47.275409    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9lqxr\" (UniqueName: \"kubernetes.io/projected/9d684956-954c-47dd-bf3d-55a8fd570dac-kube-api-access-9lqxr\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:46:47.275427 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:46:47.275427    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d684956-954c-47dd-bf3d-55a8fd570dac-openshift-service-ca\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:46:47.275544 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:46:47.275436    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d684956-954c-47dd-bf3d-55a8fd570dac-maas-api-tls\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:46:47.967990 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:46:47.967961    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a56db7-586cd6c985-m9xm2" event={"ID":"9d684956-954c-47dd-bf3d-55a8fd570dac","Type":"ContainerDied","Data":"1d89236c2849ab244a27f017ab8894ac3ddfc69bc7b6cd2e5b0060a59f5c3419"}
Jul 09 20:46:47.968377 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:46:47.968003    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a56db7-586cd6c985-m9xm2"
Jul 09 20:46:47.968377 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:46:47.968035    2562 scope.go:117] "RemoveContainer" containerID="df90f154dc6c58d10c3e2d86733ef736ec3e160a38ab0548a39b920fac1eeddf"
Jul 09 20:46:47.989361 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:46:47.989337    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-a56db7-586cd6c985-m9xm2"]
Jul 09 20:46:47.993091 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:46:47.993071    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-a56db7-586cd6c985-m9xm2"]
Jul 09 20:46:48.241484 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:46:48.241427    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d684956-954c-47dd-bf3d-55a8fd570dac" path="/var/lib/kubelet/pods/9d684956-954c-47dd-bf3d-55a8fd570dac/volumes"
Jul 09 20:46:52.214767 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:46:52.214678    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-a56db7-openshift-default-76dfdd4bf5-5rqvp"]
Jul 09 20:46:52.215184 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:46:52.214914    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-a56db7-openshift-default-76dfdd4bf5-5rqvp" podUID="7d882374-1667-49f2-8c18-d7b8e159cb72" containerName="istio-proxy" containerID="cri-o://08426554936b8963e67738e959cdc8ff0d6629258196afe5d297fab494bf0f95" gracePeriod=30
Jul 09 20:46:53.218797 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:46:53.218765    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-8c2e89-8544fc76bd-5gwdp"]
Jul 09 20:46:53.219263 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:46:53.218975    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-8c2e89-8544fc76bd-5gwdp" podUID="67a91531-0e54-497d-a167-7b5398aff15a" containerName="maas-api" containerID="cri-o://0f9ae66f9943d2e95d1f544c3e34bf2e49f568c4e34f2cb982ec497a8a70d355" gracePeriod=30
Jul 09 20:46:53.669862 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:46:53.669839    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-8c2e89-8544fc76bd-5gwdp"
Jul 09 20:46:53.827037 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:46:53.826976    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67a91531-0e54-497d-a167-7b5398aff15a-maas-api-tls\") pod \"67a91531-0e54-497d-a167-7b5398aff15a\" (UID: \"67a91531-0e54-497d-a167-7b5398aff15a\") "
Jul 09 20:46:53.827037 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:46:53.827028    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67a91531-0e54-497d-a167-7b5398aff15a-openshift-service-ca\") pod \"67a91531-0e54-497d-a167-7b5398aff15a\" (UID: \"67a91531-0e54-497d-a167-7b5398aff15a\") "
Jul 09 20:46:53.827177 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:46:53.827131    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j29c2\" (UniqueName: \"kubernetes.io/projected/67a91531-0e54-497d-a167-7b5398aff15a-kube-api-access-j29c2\") pod \"67a91531-0e54-497d-a167-7b5398aff15a\" (UID: \"67a91531-0e54-497d-a167-7b5398aff15a\") "
Jul 09 20:46:53.827340 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:46:53.827321    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67a91531-0e54-497d-a167-7b5398aff15a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "67a91531-0e54-497d-a167-7b5398aff15a" (UID: "67a91531-0e54-497d-a167-7b5398aff15a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 20:46:53.828900 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:46:53.828871    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67a91531-0e54-497d-a167-7b5398aff15a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "67a91531-0e54-497d-a167-7b5398aff15a" (UID: "67a91531-0e54-497d-a167-7b5398aff15a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 20:46:53.828995 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:46:53.828947    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67a91531-0e54-497d-a167-7b5398aff15a-kube-api-access-j29c2" (OuterVolumeSpecName: "kube-api-access-j29c2") pod "67a91531-0e54-497d-a167-7b5398aff15a" (UID: "67a91531-0e54-497d-a167-7b5398aff15a"). InnerVolumeSpecName "kube-api-access-j29c2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 20:46:53.927783 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:46:53.927764    2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67a91531-0e54-497d-a167-7b5398aff15a-maas-api-tls\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:46:53.927783 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:46:53.927783    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67a91531-0e54-497d-a167-7b5398aff15a-openshift-service-ca\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:46:53.927895 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:46:53.927792    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j29c2\" (UniqueName: \"kubernetes.io/projected/67a91531-0e54-497d-a167-7b5398aff15a-kube-api-access-j29c2\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:46:53.998816 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:46:53.998789    2562 generic.go:356] "Generic (PLEG): container finished" podID="67a91531-0e54-497d-a167-7b5398aff15a" containerID="0f9ae66f9943d2e95d1f544c3e34bf2e49f568c4e34f2cb982ec497a8a70d355" exitCode=0
Jul 09 20:46:53.998898 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:46:53.998836    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-8c2e89-8544fc76bd-5gwdp"
Jul 09 20:46:53.998937 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:46:53.998836    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-8c2e89-8544fc76bd-5gwdp" event={"ID":"67a91531-0e54-497d-a167-7b5398aff15a","Type":"ContainerDied","Data":"0f9ae66f9943d2e95d1f544c3e34bf2e49f568c4e34f2cb982ec497a8a70d355"}
Jul 09 20:46:53.999001 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:46:53.998984    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-8c2e89-8544fc76bd-5gwdp" event={"ID":"67a91531-0e54-497d-a167-7b5398aff15a","Type":"ContainerDied","Data":"7d8fa18b88e21a3867b341f2fc0ff65c3769e6461ec1209ae7e2957dbd37360b"}
Jul 09 20:46:53.999034 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:46:53.999017    2562 scope.go:117] "RemoveContainer" containerID="0f9ae66f9943d2e95d1f544c3e34bf2e49f568c4e34f2cb982ec497a8a70d355"
Jul 09 20:46:54.010126 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:46:54.010108    2562 scope.go:117] "RemoveContainer" containerID="0f9ae66f9943d2e95d1f544c3e34bf2e49f568c4e34f2cb982ec497a8a70d355"
Jul 09 20:46:54.010402 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:46:54.010378    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f9ae66f9943d2e95d1f544c3e34bf2e49f568c4e34f2cb982ec497a8a70d355\": container with ID starting with 0f9ae66f9943d2e95d1f544c3e34bf2e49f568c4e34f2cb982ec497a8a70d355 not found: ID does not exist" containerID="0f9ae66f9943d2e95d1f544c3e34bf2e49f568c4e34f2cb982ec497a8a70d355"
Jul 09 20:46:54.010473 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:46:54.010405    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f9ae66f9943d2e95d1f544c3e34bf2e49f568c4e34f2cb982ec497a8a70d355"} err="failed to get container status \"0f9ae66f9943d2e95d1f544c3e34bf2e49f568c4e34f2cb982ec497a8a70d355\": rpc error: code = NotFound desc = could not find container \"0f9ae66f9943d2e95d1f544c3e34bf2e49f568c4e34f2cb982ec497a8a70d355\": container with ID starting with 0f9ae66f9943d2e95d1f544c3e34bf2e49f568c4e34f2cb982ec497a8a70d355 not found: ID does not exist"
Jul 09 20:46:54.022778 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:46:54.022748    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-8c2e89-8544fc76bd-5gwdp"]
Jul 09 20:46:54.027206 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:46:54.027183    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-8c2e89-8544fc76bd-5gwdp"]
Jul 09 20:46:54.242220 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:46:54.242199    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67a91531-0e54-497d-a167-7b5398aff15a" path="/var/lib/kubelet/pods/67a91531-0e54-497d-a167-7b5398aff15a/volumes"
Jul 09 20:46:57.453702 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:46:57.453681    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-a56db7-openshift-default-76dfdd4bf5-5rqvp"
Jul 09 20:46:57.555038 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:46:57.554987    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d882374-1667-49f2-8c18-d7b8e159cb72-workload-certs\") pod \"7d882374-1667-49f2-8c18-d7b8e159cb72\" (UID: \"7d882374-1667-49f2-8c18-d7b8e159cb72\") "
Jul 09 20:46:57.555038 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:46:57.555015    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d882374-1667-49f2-8c18-d7b8e159cb72-istio-token\") pod \"7d882374-1667-49f2-8c18-d7b8e159cb72\" (UID: \"7d882374-1667-49f2-8c18-d7b8e159cb72\") "
Jul 09 20:46:57.555170 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:46:57.555054    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d882374-1667-49f2-8c18-d7b8e159cb72-workload-socket\") pod \"7d882374-1667-49f2-8c18-d7b8e159cb72\" (UID: \"7d882374-1667-49f2-8c18-d7b8e159cb72\") "
Jul 09 20:46:57.555170 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:46:57.555076    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d882374-1667-49f2-8c18-d7b8e159cb72-istio-envoy\") pod \"7d882374-1667-49f2-8c18-d7b8e159cb72\" (UID: \"7d882374-1667-49f2-8c18-d7b8e159cb72\") "
Jul 09 20:46:57.555170 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:46:57.555122    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d882374-1667-49f2-8c18-d7b8e159cb72-istio-data\") pod \"7d882374-1667-49f2-8c18-d7b8e159cb72\" (UID: \"7d882374-1667-49f2-8c18-d7b8e159cb72\") "
Jul 09 20:46:57.555170 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:46:57.555141    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-725ht\" (UniqueName: \"kubernetes.io/projected/7d882374-1667-49f2-8c18-d7b8e159cb72-kube-api-access-725ht\") pod \"7d882374-1667-49f2-8c18-d7b8e159cb72\" (UID: \"7d882374-1667-49f2-8c18-d7b8e159cb72\") "
Jul 09 20:46:57.555393 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:46:57.555181    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d882374-1667-49f2-8c18-d7b8e159cb72-credential-socket\") pod \"7d882374-1667-49f2-8c18-d7b8e159cb72\" (UID: \"7d882374-1667-49f2-8c18-d7b8e159cb72\") "
Jul 09 20:46:57.555393 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:46:57.555207    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d882374-1667-49f2-8c18-d7b8e159cb72-istio-podinfo\") pod \"7d882374-1667-49f2-8c18-d7b8e159cb72\" (UID: \"7d882374-1667-49f2-8c18-d7b8e159cb72\") "
Jul 09 20:46:57.555393 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:46:57.555236    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d882374-1667-49f2-8c18-d7b8e159cb72-istiod-ca-cert\") pod \"7d882374-1667-49f2-8c18-d7b8e159cb72\" (UID: \"7d882374-1667-49f2-8c18-d7b8e159cb72\") "
Jul 09 20:46:57.555544 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:46:57.555290    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d882374-1667-49f2-8c18-d7b8e159cb72-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7d882374-1667-49f2-8c18-d7b8e159cb72" (UID: "7d882374-1667-49f2-8c18-d7b8e159cb72"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 20:46:57.555544 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:46:57.555415    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d882374-1667-49f2-8c18-d7b8e159cb72-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7d882374-1667-49f2-8c18-d7b8e159cb72" (UID: "7d882374-1667-49f2-8c18-d7b8e159cb72"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 20:46:57.555544 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:46:57.555483    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d882374-1667-49f2-8c18-d7b8e159cb72-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7d882374-1667-49f2-8c18-d7b8e159cb72" (UID: "7d882374-1667-49f2-8c18-d7b8e159cb72"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 20:46:57.555694 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:46:57.555569    2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d882374-1667-49f2-8c18-d7b8e159cb72-workload-certs\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:46:57.555694 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:46:57.555589    2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d882374-1667-49f2-8c18-d7b8e159cb72-workload-socket\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:46:57.555694 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:46:57.555602    2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d882374-1667-49f2-8c18-d7b8e159cb72-credential-socket\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:46:57.555694 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:46:57.555666    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d882374-1667-49f2-8c18-d7b8e159cb72-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7d882374-1667-49f2-8c18-d7b8e159cb72" (UID: "7d882374-1667-49f2-8c18-d7b8e159cb72"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 20:46:57.556147 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:46:57.556121    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d882374-1667-49f2-8c18-d7b8e159cb72-istio-data" (OuterVolumeSpecName: "istio-data") pod "7d882374-1667-49f2-8c18-d7b8e159cb72" (UID: "7d882374-1667-49f2-8c18-d7b8e159cb72"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 20:46:57.557253 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:46:57.557227    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7d882374-1667-49f2-8c18-d7b8e159cb72-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7d882374-1667-49f2-8c18-d7b8e159cb72" (UID: "7d882374-1667-49f2-8c18-d7b8e159cb72"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 09 20:46:57.557403 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:46:57.557253    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d882374-1667-49f2-8c18-d7b8e159cb72-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7d882374-1667-49f2-8c18-d7b8e159cb72" (UID: "7d882374-1667-49f2-8c18-d7b8e159cb72"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 20:46:57.557403 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:46:57.557322    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d882374-1667-49f2-8c18-d7b8e159cb72-kube-api-access-725ht" (OuterVolumeSpecName: "kube-api-access-725ht") pod "7d882374-1667-49f2-8c18-d7b8e159cb72" (UID: "7d882374-1667-49f2-8c18-d7b8e159cb72"). InnerVolumeSpecName "kube-api-access-725ht". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 20:46:57.557403 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:46:57.557382    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d882374-1667-49f2-8c18-d7b8e159cb72-istio-token" (OuterVolumeSpecName: "istio-token") pod "7d882374-1667-49f2-8c18-d7b8e159cb72" (UID: "7d882374-1667-49f2-8c18-d7b8e159cb72"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 20:46:57.656849 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:46:57.656827    2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d882374-1667-49f2-8c18-d7b8e159cb72-istio-podinfo\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:46:57.656849 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:46:57.656847    2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d882374-1667-49f2-8c18-d7b8e159cb72-istiod-ca-cert\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:46:57.656965 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:46:57.656855    2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d882374-1667-49f2-8c18-d7b8e159cb72-istio-token\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:46:57.656965 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:46:57.656862    2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d882374-1667-49f2-8c18-d7b8e159cb72-istio-envoy\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:46:57.656965 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:46:57.656869    2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d882374-1667-49f2-8c18-d7b8e159cb72-istio-data\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:46:57.656965 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:46:57.656876    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-725ht\" (UniqueName: \"kubernetes.io/projected/7d882374-1667-49f2-8c18-d7b8e159cb72-kube-api-access-725ht\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:46:58.023766 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:46:58.023741    2562 generic.go:356] "Generic (PLEG): container finished" podID="7d882374-1667-49f2-8c18-d7b8e159cb72" containerID="08426554936b8963e67738e959cdc8ff0d6629258196afe5d297fab494bf0f95" exitCode=0
Jul 09 20:46:58.023867 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:46:58.023798    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-a56db7-openshift-default-76dfdd4bf5-5rqvp"
Jul 09 20:46:58.023867 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:46:58.023833    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-a56db7-openshift-default-76dfdd4bf5-5rqvp" event={"ID":"7d882374-1667-49f2-8c18-d7b8e159cb72","Type":"ContainerDied","Data":"08426554936b8963e67738e959cdc8ff0d6629258196afe5d297fab494bf0f95"}
Jul 09 20:46:58.023939 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:46:58.023917    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-a56db7-openshift-default-76dfdd4bf5-5rqvp" event={"ID":"7d882374-1667-49f2-8c18-d7b8e159cb72","Type":"ContainerDied","Data":"94484caa62497459178d447cd7a0a4f2d32f6735a573f42dd1c45dc614649456"}
Jul 09 20:46:58.023977 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:46:58.023944    2562 scope.go:117] "RemoveContainer" containerID="08426554936b8963e67738e959cdc8ff0d6629258196afe5d297fab494bf0f95"
Jul 09 20:46:58.032543 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:46:58.032528    2562 scope.go:117] "RemoveContainer" containerID="f45cf252d313c46abd2d2bb754fd48e07d73386f2452ffcf05c1877d8fbe0b0c"
Jul 09 20:46:58.039984 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:46:58.039967    2562 scope.go:117] "RemoveContainer" containerID="08426554936b8963e67738e959cdc8ff0d6629258196afe5d297fab494bf0f95"
Jul 09 20:46:58.040187 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:46:58.040167    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08426554936b8963e67738e959cdc8ff0d6629258196afe5d297fab494bf0f95\": container with ID starting with 08426554936b8963e67738e959cdc8ff0d6629258196afe5d297fab494bf0f95 not found: ID does not exist" containerID="08426554936b8963e67738e959cdc8ff0d6629258196afe5d297fab494bf0f95"
Jul 09 20:46:58.040253 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:46:58.040198    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08426554936b8963e67738e959cdc8ff0d6629258196afe5d297fab494bf0f95"} err="failed to get container status \"08426554936b8963e67738e959cdc8ff0d6629258196afe5d297fab494bf0f95\": rpc error: code = NotFound desc = could not find container \"08426554936b8963e67738e959cdc8ff0d6629258196afe5d297fab494bf0f95\": container with ID starting with 08426554936b8963e67738e959cdc8ff0d6629258196afe5d297fab494bf0f95 not found: ID does not exist"
Jul 09 20:46:58.040253 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:46:58.040220    2562 scope.go:117] "RemoveContainer" containerID="f45cf252d313c46abd2d2bb754fd48e07d73386f2452ffcf05c1877d8fbe0b0c"
Jul 09 20:46:58.040456 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:46:58.040442    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f45cf252d313c46abd2d2bb754fd48e07d73386f2452ffcf05c1877d8fbe0b0c\": container with ID starting with f45cf252d313c46abd2d2bb754fd48e07d73386f2452ffcf05c1877d8fbe0b0c not found: ID does not exist" containerID="f45cf252d313c46abd2d2bb754fd48e07d73386f2452ffcf05c1877d8fbe0b0c"
Jul 09 20:46:58.040497 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:46:58.040460    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f45cf252d313c46abd2d2bb754fd48e07d73386f2452ffcf05c1877d8fbe0b0c"} err="failed to get container status \"f45cf252d313c46abd2d2bb754fd48e07d73386f2452ffcf05c1877d8fbe0b0c\": rpc error: code = NotFound desc = could not find container \"f45cf252d313c46abd2d2bb754fd48e07d73386f2452ffcf05c1877d8fbe0b0c\": container with ID starting with f45cf252d313c46abd2d2bb754fd48e07d73386f2452ffcf05c1877d8fbe0b0c not found: ID does not exist"
Jul 09 20:46:58.044431 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:46:58.044413    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-a56db7-openshift-default-76dfdd4bf5-5rqvp"]
Jul 09 20:46:58.049689 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:46:58.049668    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-a56db7-openshift-default-76dfdd4bf5-5rqvp"]
Jul 09 20:46:58.241441 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:46:58.241421    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d882374-1667-49f2-8c18-d7b8e159cb72" path="/var/lib/kubelet/pods/7d882374-1667-49f2-8c18-d7b8e159cb72/volumes"
Jul 09 20:46:59.374822 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:46:59.374796    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-8c2e89-openshift-default-7c79cfb855-gvzzq"]
Jul 09 20:46:59.375191 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:46:59.374991    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-8c2e89-openshift-default-7c79cfb855-gvzzq" podUID="0f06646f-6667-4d8b-88c4-caeccd277249" containerName="istio-proxy" containerID="cri-o://09ce4b255366412f649040f03b30582af180f0fe711c5c0c4bb6b912e560c3dc" gracePeriod=30
Jul 09 20:47:04.626205 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:04.626184    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-8c2e89-openshift-default-7c79cfb855-gvzzq"
Jul 09 20:47:04.806585 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:04.806562    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f06646f-6667-4d8b-88c4-caeccd277249-workload-socket\") pod \"0f06646f-6667-4d8b-88c4-caeccd277249\" (UID: \"0f06646f-6667-4d8b-88c4-caeccd277249\") "
Jul 09 20:47:04.806694 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:04.806593    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f06646f-6667-4d8b-88c4-caeccd277249-istio-token\") pod \"0f06646f-6667-4d8b-88c4-caeccd277249\" (UID: \"0f06646f-6667-4d8b-88c4-caeccd277249\") "
Jul 09 20:47:04.806694 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:04.806614    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f06646f-6667-4d8b-88c4-caeccd277249-istio-data\") pod \"0f06646f-6667-4d8b-88c4-caeccd277249\" (UID: \"0f06646f-6667-4d8b-88c4-caeccd277249\") "
Jul 09 20:47:04.806694 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:04.806660    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f06646f-6667-4d8b-88c4-caeccd277249-workload-certs\") pod \"0f06646f-6667-4d8b-88c4-caeccd277249\" (UID: \"0f06646f-6667-4d8b-88c4-caeccd277249\") "
Jul 09 20:47:04.806694 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:04.806676    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f06646f-6667-4d8b-88c4-caeccd277249-istio-envoy\") pod \"0f06646f-6667-4d8b-88c4-caeccd277249\" (UID: \"0f06646f-6667-4d8b-88c4-caeccd277249\") "
Jul 09 20:47:04.806694 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:04.806690    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f06646f-6667-4d8b-88c4-caeccd277249-istio-podinfo\") pod \"0f06646f-6667-4d8b-88c4-caeccd277249\" (UID: \"0f06646f-6667-4d8b-88c4-caeccd277249\") "
Jul 09 20:47:04.806932 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:04.806714    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kp5hm\" (UniqueName: \"kubernetes.io/projected/0f06646f-6667-4d8b-88c4-caeccd277249-kube-api-access-kp5hm\") pod \"0f06646f-6667-4d8b-88c4-caeccd277249\" (UID: \"0f06646f-6667-4d8b-88c4-caeccd277249\") "
Jul 09 20:47:04.806932 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:04.806745    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f06646f-6667-4d8b-88c4-caeccd277249-credential-socket\") pod \"0f06646f-6667-4d8b-88c4-caeccd277249\" (UID: \"0f06646f-6667-4d8b-88c4-caeccd277249\") "
Jul 09 20:47:04.806932 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:04.806792    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f06646f-6667-4d8b-88c4-caeccd277249-istiod-ca-cert\") pod \"0f06646f-6667-4d8b-88c4-caeccd277249\" (UID: \"0f06646f-6667-4d8b-88c4-caeccd277249\") "
Jul 09 20:47:04.807077 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:04.806891    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f06646f-6667-4d8b-88c4-caeccd277249-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0f06646f-6667-4d8b-88c4-caeccd277249" (UID: "0f06646f-6667-4d8b-88c4-caeccd277249"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 20:47:04.807077 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:04.807023    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f06646f-6667-4d8b-88c4-caeccd277249-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0f06646f-6667-4d8b-88c4-caeccd277249" (UID: "0f06646f-6667-4d8b-88c4-caeccd277249"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 20:47:04.807077 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:04.807062    2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f06646f-6667-4d8b-88c4-caeccd277249-workload-socket\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:47:04.807264 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:04.807241    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f06646f-6667-4d8b-88c4-caeccd277249-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0f06646f-6667-4d8b-88c4-caeccd277249" (UID: "0f06646f-6667-4d8b-88c4-caeccd277249"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 20:47:04.807343 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:04.807280    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f06646f-6667-4d8b-88c4-caeccd277249-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0f06646f-6667-4d8b-88c4-caeccd277249" (UID: "0f06646f-6667-4d8b-88c4-caeccd277249"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 20:47:04.807468 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:04.807445    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f06646f-6667-4d8b-88c4-caeccd277249-istio-data" (OuterVolumeSpecName: "istio-data") pod "0f06646f-6667-4d8b-88c4-caeccd277249" (UID: "0f06646f-6667-4d8b-88c4-caeccd277249"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 20:47:04.808777 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:04.808759    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f06646f-6667-4d8b-88c4-caeccd277249-kube-api-access-kp5hm" (OuterVolumeSpecName: "kube-api-access-kp5hm") pod "0f06646f-6667-4d8b-88c4-caeccd277249" (UID: "0f06646f-6667-4d8b-88c4-caeccd277249"). InnerVolumeSpecName "kube-api-access-kp5hm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 20:47:04.808858 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:04.808782    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0f06646f-6667-4d8b-88c4-caeccd277249-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0f06646f-6667-4d8b-88c4-caeccd277249" (UID: "0f06646f-6667-4d8b-88c4-caeccd277249"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 09 20:47:04.808996 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:04.808964    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f06646f-6667-4d8b-88c4-caeccd277249-istio-token" (OuterVolumeSpecName: "istio-token") pod "0f06646f-6667-4d8b-88c4-caeccd277249" (UID: "0f06646f-6667-4d8b-88c4-caeccd277249"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 20:47:04.809062 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:04.809028    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f06646f-6667-4d8b-88c4-caeccd277249-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0f06646f-6667-4d8b-88c4-caeccd277249" (UID: "0f06646f-6667-4d8b-88c4-caeccd277249"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 20:47:04.907646 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:04.907627    2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f06646f-6667-4d8b-88c4-caeccd277249-credential-socket\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:47:04.907646 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:04.907646    2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f06646f-6667-4d8b-88c4-caeccd277249-istiod-ca-cert\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:47:04.907750 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:04.907655    2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f06646f-6667-4d8b-88c4-caeccd277249-istio-token\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:47:04.907750 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:04.907663    2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f06646f-6667-4d8b-88c4-caeccd277249-istio-data\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:47:04.907750 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:04.907670    2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f06646f-6667-4d8b-88c4-caeccd277249-workload-certs\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:47:04.907750 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:04.907678    2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f06646f-6667-4d8b-88c4-caeccd277249-istio-envoy\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:47:04.907750 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:04.907685    2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f06646f-6667-4d8b-88c4-caeccd277249-istio-podinfo\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:47:04.907750 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:04.907693    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kp5hm\" (UniqueName: \"kubernetes.io/projected/0f06646f-6667-4d8b-88c4-caeccd277249-kube-api-access-kp5hm\") on node \"ip-10-0-140-63.ec2.internal\" DevicePath \"\""
Jul 09 20:47:05.056228 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:05.056205    2562 generic.go:356] "Generic (PLEG): container finished" podID="0f06646f-6667-4d8b-88c4-caeccd277249" containerID="09ce4b255366412f649040f03b30582af180f0fe711c5c0c4bb6b912e560c3dc" exitCode=0
Jul 09 20:47:05.056326 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:05.056249    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-8c2e89-openshift-default-7c79cfb855-gvzzq"
Jul 09 20:47:05.056326 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:05.056276    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-8c2e89-openshift-default-7c79cfb855-gvzzq" event={"ID":"0f06646f-6667-4d8b-88c4-caeccd277249","Type":"ContainerDied","Data":"09ce4b255366412f649040f03b30582af180f0fe711c5c0c4bb6b912e560c3dc"}
Jul 09 20:47:05.056437 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:05.056368    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-8c2e89-openshift-default-7c79cfb855-gvzzq" event={"ID":"0f06646f-6667-4d8b-88c4-caeccd277249","Type":"ContainerDied","Data":"9349ca53674b6078f4d58b8c1447086b0ce528d61174ca8caff6b0a075673f6d"}
Jul 09 20:47:05.056437 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:05.056397    2562 scope.go:117] "RemoveContainer" containerID="09ce4b255366412f649040f03b30582af180f0fe711c5c0c4bb6b912e560c3dc"
Jul 09 20:47:05.064488 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:05.064474    2562 scope.go:117] "RemoveContainer" containerID="6c03420022f567a8b7a583857ad6ec7737c73f117a745a506907c89e17050777"
Jul 09 20:47:05.072376 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:05.072352    2562 scope.go:117] "RemoveContainer" containerID="09ce4b255366412f649040f03b30582af180f0fe711c5c0c4bb6b912e560c3dc"
Jul 09 20:47:05.072661 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:47:05.072636    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09ce4b255366412f649040f03b30582af180f0fe711c5c0c4bb6b912e560c3dc\": container with ID starting with 09ce4b255366412f649040f03b30582af180f0fe711c5c0c4bb6b912e560c3dc not found: ID does not exist" containerID="09ce4b255366412f649040f03b30582af180f0fe711c5c0c4bb6b912e560c3dc"
Jul 09 20:47:05.072741 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:05.072670    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09ce4b255366412f649040f03b30582af180f0fe711c5c0c4bb6b912e560c3dc"} err="failed to get container status \"09ce4b255366412f649040f03b30582af180f0fe711c5c0c4bb6b912e560c3dc\": rpc error: code = NotFound desc = could not find container \"09ce4b255366412f649040f03b30582af180f0fe711c5c0c4bb6b912e560c3dc\": container with ID starting with 09ce4b255366412f649040f03b30582af180f0fe711c5c0c4bb6b912e560c3dc not found: ID does not exist"
Jul 09 20:47:05.072741 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:05.072685    2562 scope.go:117] "RemoveContainer" containerID="6c03420022f567a8b7a583857ad6ec7737c73f117a745a506907c89e17050777"
Jul 09 20:47:05.072949 ip-10-0-140-63 kubenswrapper[2562]: E0709 20:47:05.072931    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c03420022f567a8b7a583857ad6ec7737c73f117a745a506907c89e17050777\": container with ID starting with 6c03420022f567a8b7a583857ad6ec7737c73f117a745a506907c89e17050777 not found: ID does not exist" containerID="6c03420022f567a8b7a583857ad6ec7737c73f117a745a506907c89e17050777"
Jul 09 20:47:05.073011 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:05.072959    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c03420022f567a8b7a583857ad6ec7737c73f117a745a506907c89e17050777"} err="failed to get container status \"6c03420022f567a8b7a583857ad6ec7737c73f117a745a506907c89e17050777\": rpc error: code = NotFound desc = could not find container \"6c03420022f567a8b7a583857ad6ec7737c73f117a745a506907c89e17050777\": container with ID starting with 6c03420022f567a8b7a583857ad6ec7737c73f117a745a506907c89e17050777 not found: ID does not exist"
Jul 09 20:47:05.080381 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:05.080361    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-8c2e89-openshift-default-7c79cfb855-gvzzq"]
Jul 09 20:47:05.090743 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:05.090715    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-8c2e89-openshift-default-7c79cfb855-gvzzq"]
Jul 09 20:47:05.217258 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:05.217237    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5ff76ccd7-gntk4_be8647ac-8106-4927-b1e6-b1c604e87ccd/manager/0.log"
Jul 09 20:47:05.455283 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:05.455266    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-swz8d_3344c0a8-3dcd-4c0b-879e-2491d0aea694/manager/0.log"
Jul 09 20:47:05.709294 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:05.709249    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-64f986597c-htv4s_48e85b41-042f-4b1f-a2bc-b7a246286070/copy-manifests/0.log"
Jul 09 20:47:05.715385 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:05.715370    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-64f986597c-htv4s_48e85b41-042f-4b1f-a2bc-b7a246286070/manager/0.log"
Jul 09 20:47:06.189431 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:06.189372    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-6bc8548564-gvhbr_dbf505e3-ecfb-4dc4-92f7-78f52be1c12f/maas-api/0.log"
Jul 09 20:47:06.241345 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:06.241295    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f06646f-6667-4d8b-88c4-caeccd277249" path="/var/lib/kubelet/pods/0f06646f-6667-4d8b-88c4-caeccd277249/volumes"
Jul 09 20:47:06.305073 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:06.305055    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-k2rgs_737886eb-3b43-45da-b81d-ea174d47a0ae/postgres/0.log"
Jul 09 20:47:08.110040 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:08.110021    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-dlkf9_203f88dd-f8b7-4843-a4b2-246a3583cb83/registry-server/0.log"
Jul 09 20:47:08.482697 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:08.482673    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-g4vck_f5ed03c8-ae4f-42cd-89e1-4573415a3463/manager/0.log"
Jul 09 20:47:08.978474 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:08.978454    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-cqlmf_b1381926-955e-4bd3-a477-bcfafa860d42/discovery/0.log"
Jul 09 20:47:09.210664 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:09.210647    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-8d5dd4c74-cgxgl_4bd539c1-f6e7-4b25-a378-29b483b20851/kube-auth-proxy/0.log"
Jul 09 20:47:09.328732 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:09.328681    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-phr4z_5fa2660d-7348-4743-9447-c477bb97a722/istio-proxy/1.log"
Jul 09 20:47:09.448064 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:09.448042    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-jfpjl_3da2fa45-fd15-4a70-ba90-9ed26ea33d79/payload-pre-processing/0.log"
Jul 09 20:47:09.563235 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:09.563216    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-8xb6m_2548ca62-2892-4717-869c-c78ed5445438/payload-processing/0.log"
Jul 09 20:47:09.692810 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:09.692763    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-b66d5d5c8-9kl6t_e5773e57-8d05-422d-84b1-7c98763683f7/router/0.log"
Jul 09 20:47:10.039649 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:10.039631    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-5wbtg_1343760b-26ed-44de-96f5-87bb0e9b67f9/main/0.log"
Jul 09 20:47:10.160267 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:10.160249    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-x455w_3b0b703d-1f9a-4487-b9cd-8ea2dddb4ee5/main/0.log"
Jul 09 20:47:17.154640 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:17.154610    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-dccxz_13454dc6-77a2-4dc9-adc2-81791d1cd463/global-pull-secret-syncer/0.log"
Jul 09 20:47:17.316372 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:17.316341    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-vwttv_00761abf-a23d-4f99-ba0f-bcb623060f98/konnectivity-agent/0.log"
Jul 09 20:47:17.366045 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:17.366019    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-63.ec2.internal_61c3c5dbc96e6f9e2e03d9211834c2ca/haproxy/0.log"
Jul 09 20:47:22.639630 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:22.639601    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-dlkf9_203f88dd-f8b7-4843-a4b2-246a3583cb83/registry-server/0.log"
Jul 09 20:47:22.880153 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:22.880124    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-g4vck_f5ed03c8-ae4f-42cd-89e1-4573415a3463/manager/0.log"
Jul 09 20:47:25.064863 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:25.064795    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-grp5l_09dfd499-f51c-4d1b-ad6e-9fad60d7798e/node-exporter/0.log"
Jul 09 20:47:25.091541 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:25.091519    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-grp5l_09dfd499-f51c-4d1b-ad6e-9fad60d7798e/kube-rbac-proxy/0.log"
Jul 09 20:47:25.119487 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:25.119471    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-grp5l_09dfd499-f51c-4d1b-ad6e-9fad60d7798e/init-textfile/0.log"
Jul 09 20:47:25.241388 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:25.241372    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-ww6bh_da490631-59e9-45ed-9083-c8949f2ca1af/kube-rbac-proxy-main/0.log"
Jul 09 20:47:25.267675 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:25.267660    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-ww6bh_da490631-59e9-45ed-9083-c8949f2ca1af/kube-rbac-proxy-self/0.log"
Jul 09 20:47:25.290669 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:25.290654    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-ww6bh_da490631-59e9-45ed-9083-c8949f2ca1af/openshift-state-metrics/0.log"
Jul 09 20:47:25.562331 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:25.562285    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-77d98bc786-mh8xw_cad231e2-9767-458e-af67-8ca2505a24c6/prometheus-operator-admission-webhook/0.log"
Jul 09 20:47:26.999816 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:26.999790    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6cf856758c-t9d7w_3b4c1edc-a174-4e8e-9f15-eccad2f85d52/networking-console-plugin/0.log"
Jul 09 20:47:27.457719 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:27.457693    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-76kn5/perf-node-gather-daemonset-hkw6w"]
Jul 09 20:47:27.458013 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:27.458002    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f06646f-6667-4d8b-88c4-caeccd277249" containerName="istio-proxy"
Jul 09 20:47:27.458055 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:27.458015    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f06646f-6667-4d8b-88c4-caeccd277249" containerName="istio-proxy"
Jul 09 20:47:27.458055 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:27.458025    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d684956-954c-47dd-bf3d-55a8fd570dac" containerName="maas-api"
Jul 09 20:47:27.458055 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:27.458030    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d684956-954c-47dd-bf3d-55a8fd570dac" containerName="maas-api"
Jul 09 20:47:27.458055 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:27.458048    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d882374-1667-49f2-8c18-d7b8e159cb72" containerName="istio-proxy"
Jul 09 20:47:27.458173 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:27.458056    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d882374-1667-49f2-8c18-d7b8e159cb72" containerName="istio-proxy"
Jul 09 20:47:27.458173 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:27.458067    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67a91531-0e54-497d-a167-7b5398aff15a" containerName="maas-api"
Jul 09 20:47:27.458173 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:27.458072    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="67a91531-0e54-497d-a167-7b5398aff15a" containerName="maas-api"
Jul 09 20:47:27.458173 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:27.458080    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d882374-1667-49f2-8c18-d7b8e159cb72" containerName="istio-proxy"
Jul 09 20:47:27.458173 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:27.458085    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d882374-1667-49f2-8c18-d7b8e159cb72" containerName="istio-proxy"
Jul 09 20:47:27.458173 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:27.458135    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="67a91531-0e54-497d-a167-7b5398aff15a" containerName="maas-api"
Jul 09 20:47:27.458173 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:27.458143    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f06646f-6667-4d8b-88c4-caeccd277249" containerName="istio-proxy"
Jul 09 20:47:27.458173 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:27.458148    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d882374-1667-49f2-8c18-d7b8e159cb72" containerName="istio-proxy"
Jul 09 20:47:27.458173 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:27.458156    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f06646f-6667-4d8b-88c4-caeccd277249" containerName="istio-proxy"
Jul 09 20:47:27.458173 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:27.458161    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d684956-954c-47dd-bf3d-55a8fd570dac" containerName="maas-api"
Jul 09 20:47:27.458173 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:27.458165    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d882374-1667-49f2-8c18-d7b8e159cb72" containerName="istio-proxy"
Jul 09 20:47:27.461150 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:27.461136    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-76kn5/perf-node-gather-daemonset-hkw6w"
Jul 09 20:47:27.463998 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:27.463979    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-76kn5\"/\"openshift-service-ca.crt\""
Jul 09 20:47:27.464127 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:27.464002    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-76kn5\"/\"kube-root-ca.crt\""
Jul 09 20:47:27.465531 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:27.465507    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-76kn5\"/\"default-dockercfg-cblvf\""
Jul 09 20:47:27.470352 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:27.470333    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-76kn5/perf-node-gather-daemonset-hkw6w"]
Jul 09 20:47:27.558706 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:27.558685    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4af8582f-7cfa-48a0-8443-c874e256864b-sys\") pod \"perf-node-gather-daemonset-hkw6w\" (UID: \"4af8582f-7cfa-48a0-8443-c874e256864b\") " pod="openshift-must-gather-76kn5/perf-node-gather-daemonset-hkw6w"
Jul 09 20:47:27.558823 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:27.558714    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vt6kv\" (UniqueName: \"kubernetes.io/projected/4af8582f-7cfa-48a0-8443-c874e256864b-kube-api-access-vt6kv\") pod \"perf-node-gather-daemonset-hkw6w\" (UID: \"4af8582f-7cfa-48a0-8443-c874e256864b\") " pod="openshift-must-gather-76kn5/perf-node-gather-daemonset-hkw6w"
Jul 09 20:47:27.558823 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:27.558748    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4af8582f-7cfa-48a0-8443-c874e256864b-proc\") pod \"perf-node-gather-daemonset-hkw6w\" (UID: \"4af8582f-7cfa-48a0-8443-c874e256864b\") " pod="openshift-must-gather-76kn5/perf-node-gather-daemonset-hkw6w"
Jul 09 20:47:27.558940 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:27.558833    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4af8582f-7cfa-48a0-8443-c874e256864b-lib-modules\") pod \"perf-node-gather-daemonset-hkw6w\" (UID: \"4af8582f-7cfa-48a0-8443-c874e256864b\") " pod="openshift-must-gather-76kn5/perf-node-gather-daemonset-hkw6w"
Jul 09 20:47:27.558940 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:27.558876    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4af8582f-7cfa-48a0-8443-c874e256864b-podres\") pod \"perf-node-gather-daemonset-hkw6w\" (UID: \"4af8582f-7cfa-48a0-8443-c874e256864b\") " pod="openshift-must-gather-76kn5/perf-node-gather-daemonset-hkw6w"
Jul 09 20:47:27.616780 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:27.616764    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-cqs6d_578878d0-e6b3-4725-a698-796cfc169fa6/console-operator/1.log"
Jul 09 20:47:27.620522 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:27.620505    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-cqs6d_578878d0-e6b3-4725-a698-796cfc169fa6/console-operator/2.log"
Jul 09 20:47:27.659976 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:27.659952    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4af8582f-7cfa-48a0-8443-c874e256864b-sys\") pod \"perf-node-gather-daemonset-hkw6w\" (UID: \"4af8582f-7cfa-48a0-8443-c874e256864b\") " pod="openshift-must-gather-76kn5/perf-node-gather-daemonset-hkw6w"
Jul 09 20:47:27.660073 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:27.659981    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vt6kv\" (UniqueName: \"kubernetes.io/projected/4af8582f-7cfa-48a0-8443-c874e256864b-kube-api-access-vt6kv\") pod \"perf-node-gather-daemonset-hkw6w\" (UID: \"4af8582f-7cfa-48a0-8443-c874e256864b\") " pod="openshift-must-gather-76kn5/perf-node-gather-daemonset-hkw6w"
Jul 09 20:47:27.660073 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:27.660007    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4af8582f-7cfa-48a0-8443-c874e256864b-proc\") pod \"perf-node-gather-daemonset-hkw6w\" (UID: \"4af8582f-7cfa-48a0-8443-c874e256864b\") " pod="openshift-must-gather-76kn5/perf-node-gather-daemonset-hkw6w"
Jul 09 20:47:27.660073 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:27.660042    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4af8582f-7cfa-48a0-8443-c874e256864b-lib-modules\") pod \"perf-node-gather-daemonset-hkw6w\" (UID: \"4af8582f-7cfa-48a0-8443-c874e256864b\") " pod="openshift-must-gather-76kn5/perf-node-gather-daemonset-hkw6w"
Jul 09 20:47:27.660073 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:27.660054    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4af8582f-7cfa-48a0-8443-c874e256864b-sys\") pod \"perf-node-gather-daemonset-hkw6w\" (UID: \"4af8582f-7cfa-48a0-8443-c874e256864b\") " pod="openshift-must-gather-76kn5/perf-node-gather-daemonset-hkw6w"
Jul 09 20:47:27.660273 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:27.660073    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4af8582f-7cfa-48a0-8443-c874e256864b-podres\") pod \"perf-node-gather-daemonset-hkw6w\" (UID: \"4af8582f-7cfa-48a0-8443-c874e256864b\") " pod="openshift-must-gather-76kn5/perf-node-gather-daemonset-hkw6w"
Jul 09 20:47:27.660273 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:27.660094    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4af8582f-7cfa-48a0-8443-c874e256864b-proc\") pod \"perf-node-gather-daemonset-hkw6w\" (UID: \"4af8582f-7cfa-48a0-8443-c874e256864b\") " pod="openshift-must-gather-76kn5/perf-node-gather-daemonset-hkw6w"
Jul 09 20:47:27.660273 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:27.660181    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4af8582f-7cfa-48a0-8443-c874e256864b-lib-modules\") pod \"perf-node-gather-daemonset-hkw6w\" (UID: \"4af8582f-7cfa-48a0-8443-c874e256864b\") " pod="openshift-must-gather-76kn5/perf-node-gather-daemonset-hkw6w"
Jul 09 20:47:27.660273 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:27.660185    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4af8582f-7cfa-48a0-8443-c874e256864b-podres\") pod \"perf-node-gather-daemonset-hkw6w\" (UID: \"4af8582f-7cfa-48a0-8443-c874e256864b\") " pod="openshift-must-gather-76kn5/perf-node-gather-daemonset-hkw6w"
Jul 09 20:47:27.666946 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:27.666929    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vt6kv\" (UniqueName: \"kubernetes.io/projected/4af8582f-7cfa-48a0-8443-c874e256864b-kube-api-access-vt6kv\") pod \"perf-node-gather-daemonset-hkw6w\" (UID: \"4af8582f-7cfa-48a0-8443-c874e256864b\") " pod="openshift-must-gather-76kn5/perf-node-gather-daemonset-hkw6w"
Jul 09 20:47:27.770859 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:27.770802    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-76kn5/perf-node-gather-daemonset-hkw6w"
Jul 09 20:47:27.894993 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:27.894967    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-76kn5/perf-node-gather-daemonset-hkw6w"]
Jul 09 20:47:27.896681 ip-10-0-140-63 kubenswrapper[2562]: W0709 20:47:27.896647    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod4af8582f_7cfa_48a0_8443_c874e256864b.slice/crio-790306c866bccc6e5ee50d02e0b99480945ccb746ab613ff543ee76158df1ffb WatchSource:0}: Error finding container 790306c866bccc6e5ee50d02e0b99480945ccb746ab613ff543ee76158df1ffb: Status 404 returned error can't find the container with id 790306c866bccc6e5ee50d02e0b99480945ccb746ab613ff543ee76158df1ffb
Jul 09 20:47:28.161392 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:28.161331    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-76kn5/perf-node-gather-daemonset-hkw6w" event={"ID":"4af8582f-7cfa-48a0-8443-c874e256864b","Type":"ContainerStarted","Data":"154b199610f66b95a037cbd4c55689e623be3b4daa26ce4d2577198b499bc45d"}
Jul 09 20:47:28.161392 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:28.161379    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-76kn5/perf-node-gather-daemonset-hkw6w" event={"ID":"4af8582f-7cfa-48a0-8443-c874e256864b","Type":"ContainerStarted","Data":"790306c866bccc6e5ee50d02e0b99480945ccb746ab613ff543ee76158df1ffb"}
Jul 09 20:47:28.161738 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:28.161399    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-76kn5/perf-node-gather-daemonset-hkw6w"
Jul 09 20:47:28.177172 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:28.177136    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-76kn5/perf-node-gather-daemonset-hkw6w" podStartSLOduration=1.177121877 podStartE2EDuration="1.177121877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 20:47:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 20:47:28.176343143 +0000 UTC m=+2638.594012950" watchObservedRunningTime="2026-07-09 20:47:28.177121877 +0000 UTC m=+2638.594791683"
Jul 09 20:47:30.027866 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:30.027839    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-vhphk_1f3c5da7-a9c8-4238-840b-df34177f1096/dns/0.log"
Jul 09 20:47:30.049773 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:30.049753    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-vhphk_1f3c5da7-a9c8-4238-840b-df34177f1096/kube-rbac-proxy/0.log"
Jul 09 20:47:30.101421 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:30.101398    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-8r78k_dddbfeb6-339f-4aa6-b74f-26dfe9438a2c/dns-node-resolver/0.log"
Jul 09 20:47:30.714892 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:30.714845    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-n2tvb_785ecd20-a92f-4924-b2f3-663697381982/node-ca/0.log"
Jul 09 20:47:31.780523 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:31.780499    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-cqlmf_b1381926-955e-4bd3-a477-bcfafa860d42/discovery/0.log"
Jul 09 20:47:31.831160 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:31.831136    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-8d5dd4c74-cgxgl_4bd539c1-f6e7-4b25-a378-29b483b20851/kube-auth-proxy/0.log"
Jul 09 20:47:31.859129 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:31.859103    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-phr4z_5fa2660d-7348-4743-9447-c477bb97a722/istio-proxy/0.log"
Jul 09 20:47:31.884767 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:31.884748    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-phr4z_5fa2660d-7348-4743-9447-c477bb97a722/istio-proxy/1.log"
Jul 09 20:47:31.921443 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:31.921426    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-jfpjl_3da2fa45-fd15-4a70-ba90-9ed26ea33d79/payload-pre-processing/0.log"
Jul 09 20:47:31.964838 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:31.964809    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-8xb6m_2548ca62-2892-4717-869c-c78ed5445438/payload-processing/0.log"
Jul 09 20:47:31.989534 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:31.989508    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-b66d5d5c8-9kl6t_e5773e57-8d05-422d-84b1-7c98763683f7/router/0.log"
Jul 09 20:47:32.759048 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:32.759027    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-pwjnv_fd8d29db-3983-4f37-9476-6dca7b1b33b7/serve-healthcheck-canary/0.log"
Jul 09 20:47:33.293819 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:33.293796    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7qx9m_09094efa-769d-4669-944e-a76a98aa185e/kube-rbac-proxy/0.log"
Jul 09 20:47:33.321765 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:33.321740    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7qx9m_09094efa-769d-4669-944e-a76a98aa185e/exporter/0.log"
Jul 09 20:47:33.361053 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:33.361032    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7qx9m_09094efa-769d-4669-944e-a76a98aa185e/extractor/0.log"
Jul 09 20:47:34.174752 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:34.174722    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-76kn5/perf-node-gather-daemonset-hkw6w"
Jul 09 20:47:35.917265 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:35.917239    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5ff76ccd7-gntk4_be8647ac-8106-4927-b1e6-b1c604e87ccd/manager/0.log"
Jul 09 20:47:36.007620 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:36.007599    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-swz8d_3344c0a8-3dcd-4c0b-879e-2491d0aea694/manager/0.log"
Jul 09 20:47:36.160801 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:36.160782    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-64f986597c-htv4s_48e85b41-042f-4b1f-a2bc-b7a246286070/manager/0.log"
Jul 09 20:47:36.184494 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:36.184437    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-64f986597c-htv4s_48e85b41-042f-4b1f-a2bc-b7a246286070/copy-manifests/0.log"
Jul 09 20:47:37.732198 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:37.732173    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-vj4qk_eb47f6d7-b5d5-4ed4-94f0-8635b1b89199/openshift-lws-operator/0.log"
Jul 09 20:47:38.557521 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:38.557450    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-cqlmf_b1381926-955e-4bd3-a477-bcfafa860d42/discovery/0.log"
Jul 09 20:47:38.589066 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:38.589048    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-8d5dd4c74-cgxgl_4bd539c1-f6e7-4b25-a378-29b483b20851/kube-auth-proxy/0.log"
Jul 09 20:47:38.613204 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:38.613185    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-phr4z_5fa2660d-7348-4743-9447-c477bb97a722/istio-proxy/0.log"
Jul 09 20:47:38.639173 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:38.639158    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-phr4z_5fa2660d-7348-4743-9447-c477bb97a722/istio-proxy/1.log"
Jul 09 20:47:38.669044 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:38.669025    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-jfpjl_3da2fa45-fd15-4a70-ba90-9ed26ea33d79/payload-pre-processing/0.log"
Jul 09 20:47:38.706716 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:38.706697    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-8xb6m_2548ca62-2892-4717-869c-c78ed5445438/payload-processing/0.log"
Jul 09 20:47:38.722286 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:38.722272    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-b66d5d5c8-9kl6t_e5773e57-8d05-422d-84b1-7c98763683f7/router/0.log"
Jul 09 20:47:40.048136 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:40.048111    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-cqlmf_b1381926-955e-4bd3-a477-bcfafa860d42/discovery/0.log"
Jul 09 20:47:40.087973 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:40.087955    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-8d5dd4c74-cgxgl_4bd539c1-f6e7-4b25-a378-29b483b20851/kube-auth-proxy/0.log"
Jul 09 20:47:40.115367 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:40.115345    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-phr4z_5fa2660d-7348-4743-9447-c477bb97a722/istio-proxy/0.log"
Jul 09 20:47:40.142631 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:40.142614    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-phr4z_5fa2660d-7348-4743-9447-c477bb97a722/istio-proxy/1.log"
Jul 09 20:47:40.170483 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:40.170450    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-jfpjl_3da2fa45-fd15-4a70-ba90-9ed26ea33d79/payload-pre-processing/0.log"
Jul 09 20:47:40.206381 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:40.206334    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-8xb6m_2548ca62-2892-4717-869c-c78ed5445438/payload-processing/0.log"
Jul 09 20:47:40.222718 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:40.222699    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-b66d5d5c8-9kl6t_e5773e57-8d05-422d-84b1-7c98763683f7/router/0.log"
Jul 09 20:47:43.003203 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:43.003176    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-8gtgc_89b4a7d6-9699-455b-af5f-473a56d4f1fc/migrator/0.log"
Jul 09 20:47:43.024590 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:43.024570    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-8gtgc_89b4a7d6-9699-455b-af5f-473a56d4f1fc/graceful-termination/0.log"
Jul 09 20:47:44.522916 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:44.522892    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dz4cq_d592535b-5457-48d0-ae48-caed8efc04fc/kube-multus-additional-cni-plugins/0.log"
Jul 09 20:47:44.548016 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:44.547984    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dz4cq_d592535b-5457-48d0-ae48-caed8efc04fc/egress-router-binary-copy/0.log"
Jul 09 20:47:44.569326 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:44.569293    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dz4cq_d592535b-5457-48d0-ae48-caed8efc04fc/cni-plugins/0.log"
Jul 09 20:47:44.592242 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:44.592221    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dz4cq_d592535b-5457-48d0-ae48-caed8efc04fc/bond-cni-plugin/0.log"
Jul 09 20:47:44.612881 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:44.612861    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dz4cq_d592535b-5457-48d0-ae48-caed8efc04fc/routeoverride-cni/0.log"
Jul 09 20:47:44.635875 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:44.635858    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dz4cq_d592535b-5457-48d0-ae48-caed8efc04fc/whereabouts-cni-bincopy/0.log"
Jul 09 20:47:44.660853 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:44.660833    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dz4cq_d592535b-5457-48d0-ae48-caed8efc04fc/whereabouts-cni/0.log"
Jul 09 20:47:44.860368 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:44.860321    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-rvxnk_5204f9ff-c575-4889-91ff-3cf3ffcd032b/kube-multus/0.log"
Jul 09 20:47:45.009758 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:45.009740    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-ksd8v_14a1d22f-43ff-4eb8-a89d-920551b2ef1f/network-metrics-daemon/0.log"
Jul 09 20:47:45.031866 ip-10-0-140-63 kubenswrapper[2562]: I0709 20:47:45.031848    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-ksd8v_14a1d22f-43ff-4eb8-a89d-920551b2ef1f/kube-rbac-proxy/0.log"