Sep 11 15:54:51.066352 ip-10-0-136-96 systemd[1]: Starting Kubernetes Kubelet... Sep 11 15:54:51.486189 ip-10-0-136-96 kubenswrapper[2566]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 11 15:54:51.486189 ip-10-0-136-96 kubenswrapper[2566]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 11 15:54:51.486189 ip-10-0-136-96 kubenswrapper[2566]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 11 15:54:51.486189 ip-10-0-136-96 kubenswrapper[2566]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 11 15:54:51.486189 ip-10-0-136-96 kubenswrapper[2566]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 11 15:54:51.487523 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.487435 2566 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 11 15:54:51.492940 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.492925 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 15:54:51.492940 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.492941 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 15:54:51.493003 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.492945 2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 15:54:51.493003 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.492949 2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 15:54:51.493003 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.492952 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 15:54:51.493003 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.492955 2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 15:54:51.493003 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.492958 2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 15:54:51.493003 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.492961 2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 15:54:51.493003 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.492963 2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 15:54:51.493003 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.492966 2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 15:54:51.493003 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.492969 2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 15:54:51.493003 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.492972 2566 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 15:54:51.493003 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.492975 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 15:54:51.493003 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.492978 2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 15:54:51.493003 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.492981 2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 15:54:51.493003 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.492983 2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 15:54:51.493003 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.492987 2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 15:54:51.493003 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.492989 2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 15:54:51.493003 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.492992 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 15:54:51.493003 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.492995 2566 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 15:54:51.493003 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.492999 2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 15:54:51.493003 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.493001 2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 15:54:51.493577 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.493004 2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 15:54:51.493577 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.493007 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 15:54:51.493577 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.493010 2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 15:54:51.493577 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.493013 2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 15:54:51.493577 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.493016 2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 15:54:51.493577 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.493019 2566 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 15:54:51.493577 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.493021 2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 15:54:51.493577 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.493024 2566 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 15:54:51.493577 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.493027 2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 15:54:51.493577 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.493029 2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 15:54:51.493577 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.493032 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 15:54:51.493577 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.493034 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 15:54:51.493577 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.493038 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 15:54:51.493577 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.493051 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 15:54:51.493577 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.493054 2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 15:54:51.493577 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.493057 2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 15:54:51.493577 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.493060 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 15:54:51.493577 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.493063 2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 15:54:51.493577 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.493066 2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 15:54:51.493577 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.493068 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 15:54:51.494083 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.493074 2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 15:54:51.494083 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.493079 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 15:54:51.494083 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.493082 2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 15:54:51.494083 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.493085 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 15:54:51.494083 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.493087 2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 15:54:51.494083 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.493090 2566 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 15:54:51.494083 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.493093 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 15:54:51.494083 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.493096 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 15:54:51.494083 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.493098 2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 15:54:51.494083 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.493101 2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 15:54:51.494083 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.493104 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 15:54:51.494083 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.493106 2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 15:54:51.494083 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.493110 2566 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 15:54:51.494083 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.493112 2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 15:54:51.494083 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.493115 2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 15:54:51.494083 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.493118 2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 15:54:51.494083 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.493121 2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 15:54:51.494083 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.493123 2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 15:54:51.494083 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.493126 2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 15:54:51.494083 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.493128 2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 15:54:51.494583 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.493131 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 15:54:51.494583 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.493134 2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 15:54:51.494583 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.493136 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 15:54:51.494583 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.493139 2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 15:54:51.494583 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.493143 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 15:54:51.494583 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.493148 2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 15:54:51.494583 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.493151 2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 15:54:51.494583 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.493154 2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 15:54:51.494583 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.493158 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 15:54:51.494583 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.493161 2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 15:54:51.494583 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.493164 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 15:54:51.494583 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.493167 2566 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 15:54:51.494583 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.493170 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 15:54:51.494583 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.493174 2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 15:54:51.494583 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.493177 2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 15:54:51.494583 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.493179 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 15:54:51.494583 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.493182 2566 feature_gate.go:328] unrecognized feature gate: Example Sep 11 15:54:51.494583 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.493185 2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 15:54:51.494583 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.493187 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 15:54:51.495037 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.493190 2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 15:54:51.495037 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.493193 2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 15:54:51.495037 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.493195 2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 15:54:51.495037 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.493198 2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 15:54:51.495037 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.493201 2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 15:54:51.495037 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.493206 2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 15:54:51.495037 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.493208 2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 15:54:51.495037 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.493211 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 15:54:51.495037 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.493215 2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 15:54:51.495037 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.493219 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 15:54:51.495037 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.493222 2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 15:54:51.495037 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.493225 2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 15:54:51.495037 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.493228 2566 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 15:54:51.495037 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.493230 2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 15:54:51.495037 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.493234 2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 15:54:51.495037 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.493236 2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 15:54:51.495037 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.493239 2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 15:54:51.495037 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.493242 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 15:54:51.495037 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.493958 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 15:54:51.495513 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.493964 2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 15:54:51.495513 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.493967 2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 15:54:51.495513 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.493970 2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 15:54:51.495513 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.493973 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 15:54:51.495513 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.493978 2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 15:54:51.495513 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.493982 2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 15:54:51.495513 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.493985 2566 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 15:54:51.495513 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.493988 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 15:54:51.495513 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.493991 2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 15:54:51.495513 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.493994 2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 15:54:51.495513 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.493997 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 15:54:51.495513 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.493999 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 15:54:51.495513 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494002 2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 15:54:51.495513 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494004 2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 15:54:51.495513 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494007 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 15:54:51.495513 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494010 2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 15:54:51.495513 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494012 2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 15:54:51.495513 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494015 2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 15:54:51.495513 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494018 2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 15:54:51.496026 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494020 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 15:54:51.496026 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494023 2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 15:54:51.496026 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494026 2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 15:54:51.496026 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494029 2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 15:54:51.496026 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494032 2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 15:54:51.496026 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494034 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 15:54:51.496026 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494037 2566 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 15:54:51.496026 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494054 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 15:54:51.496026 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494057 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 15:54:51.496026 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494059 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 15:54:51.496026 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494062 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 15:54:51.496026 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494064 2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 15:54:51.496026 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494067 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 15:54:51.496026 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494070 2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 15:54:51.496026 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494072 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 15:54:51.496026 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494075 2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 15:54:51.496026 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494078 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 15:54:51.496026 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494080 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 15:54:51.496026 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494087 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 15:54:51.496026 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494090 2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 15:54:51.496531 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494093 2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 15:54:51.496531 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494096 2566 feature_gate.go:328] unrecognized feature gate: Example Sep 11 15:54:51.496531 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494098 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 15:54:51.496531 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494101 2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 15:54:51.496531 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494104 2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 15:54:51.496531 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494106 2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 15:54:51.496531 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494109 2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 15:54:51.496531 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494112 2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 15:54:51.496531 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494114 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 15:54:51.496531 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494117 2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 15:54:51.496531 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494119 2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 15:54:51.496531 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494122 2566 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 15:54:51.496531 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494125 2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 15:54:51.496531 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494130 2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 15:54:51.496531 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494133 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 15:54:51.496531 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494137 2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 15:54:51.496531 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494140 2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 15:54:51.496531 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494144 2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 15:54:51.496531 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494147 2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 15:54:51.496999 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494149 2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 15:54:51.496999 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494152 2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 15:54:51.496999 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494154 2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 15:54:51.496999 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494157 2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 15:54:51.496999 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494159 2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 15:54:51.496999 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494162 2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 15:54:51.496999 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494172 2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 15:54:51.496999 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494185 2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 15:54:51.496999 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494189 2566 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 15:54:51.496999 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494193 2566 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 15:54:51.496999 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494198 2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 15:54:51.496999 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494202 2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 15:54:51.496999 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494212 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 15:54:51.496999 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494215 2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 15:54:51.496999 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494218 2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 15:54:51.496999 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494220 2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 15:54:51.496999 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494223 2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 15:54:51.496999 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494225 2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 15:54:51.496999 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494228 2566 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 15:54:51.496999 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494230 2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 15:54:51.497494 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494233 2566 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 15:54:51.497494 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494236 2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 15:54:51.497494 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494238 2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 15:54:51.497494 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494242 2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 15:54:51.497494 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494245 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 15:54:51.497494 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494248 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 15:54:51.497494 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494250 2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 15:54:51.497494 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494253 2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 15:54:51.497494 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494255 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 15:54:51.497494 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494258 2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 15:54:51.497494 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494261 2566 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 15:54:51.497494 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494263 2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 15:54:51.497494 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494266 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 15:54:51.497494 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494268 2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 15:54:51.497494 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494271 2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 15:54:51.497494 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494273 2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 15:54:51.497494 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494276 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 15:54:51.497494 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494278 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 15:54:51.497494 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494281 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 15:54:51.497494 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494283 2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 15:54:51.497985 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494668 2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 15:54:51.497985 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494674 2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 15:54:51.497985 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494676 2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 15:54:51.497985 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494679 2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 15:54:51.497985 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494683 2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 15:54:51.497985 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494686 2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 15:54:51.497985 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494689 2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 15:54:51.497985 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494692 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 15:54:51.497985 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494694 2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 15:54:51.497985 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494697 2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 15:54:51.497985 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494699 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 15:54:51.497985 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494702 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 15:54:51.497985 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494705 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 15:54:51.497985 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494708 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 15:54:51.497985 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494710 2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 15:54:51.497985 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494717 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 15:54:51.497985 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494720 2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 15:54:51.497985 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494723 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 15:54:51.497985 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494726 2566 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 15:54:51.497985 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494728 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 15:54:51.498482 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494731 2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 15:54:51.498482 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494733 2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 15:54:51.498482 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494736 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 15:54:51.498482 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494739 2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 15:54:51.498482 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494741 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 15:54:51.498482 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494744 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 15:54:51.498482 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494746 2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 15:54:51.498482 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494749 2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 15:54:51.498482 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494752 2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 15:54:51.498482 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494754 2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 15:54:51.498482 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494757 2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 15:54:51.498482 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494759 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 15:54:51.498482 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494762 2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 15:54:51.498482 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494765 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 15:54:51.498482 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494767 2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 15:54:51.498482 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494770 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 15:54:51.498482 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494773 2566 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 15:54:51.498482 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494776 2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 15:54:51.498482 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494778 2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 15:54:51.498482 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494781 2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 15:54:51.498959 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494784 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 15:54:51.498959 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494786 2566 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 15:54:51.498959 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494789 2566 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 15:54:51.498959 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494792 2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 15:54:51.498959 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494794 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 15:54:51.498959 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494797 2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 15:54:51.498959 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494799 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 15:54:51.498959 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494803 2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 15:54:51.498959 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494806 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 15:54:51.498959 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494811 2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 15:54:51.498959 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494814 2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 15:54:51.498959 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494818 2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 15:54:51.498959 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494820 2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 15:54:51.498959 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494824 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 15:54:51.498959 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494826 2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 15:54:51.498959 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494829 2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 15:54:51.498959 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494831 2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 15:54:51.498959 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494834 2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 15:54:51.498959 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494837 2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 15:54:51.499420 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494839 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 15:54:51.499420 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494842 2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 15:54:51.499420 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494844 2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 15:54:51.499420 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494847 2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 15:54:51.499420 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494850 2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 15:54:51.499420 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494853 2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 15:54:51.499420 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494856 2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 15:54:51.499420 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494858 2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 15:54:51.499420 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494861 2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 15:54:51.499420 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494863 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 15:54:51.499420 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494866 2566 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 15:54:51.499420 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494869 2566 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 15:54:51.499420 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494871 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 15:54:51.499420 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494874 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 15:54:51.499420 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494877 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 15:54:51.499420 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494879 2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 15:54:51.499420 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494882 2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 15:54:51.499420 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494884 2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 15:54:51.499420 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494887 2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 15:54:51.499420 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494889 2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 15:54:51.499911 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494893 2566 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 15:54:51.499911 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494896 2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 15:54:51.499911 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494900 2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 15:54:51.499911 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494903 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 15:54:51.499911 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494906 2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 15:54:51.499911 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494910 2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 15:54:51.499911 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494913 2566 feature_gate.go:328] unrecognized feature gate: Example Sep 11 15:54:51.499911 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494917 2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 15:54:51.499911 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494920 2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 15:54:51.499911 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494923 2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 15:54:51.499911 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494926 2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 15:54:51.499911 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494929 2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 15:54:51.499911 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494931 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 15:54:51.499911 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494934 2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 15:54:51.499911 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494937 2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 15:54:51.499911 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494939 2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 15:54:51.499911 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494942 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 15:54:51.499911 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494944 2566 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 15:54:51.499911 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494947 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 15:54:51.500381 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.494949 2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 15:54:51.500381 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495031 2566 flags.go:64] FLAG: --address="0.0.0.0" Sep 11 15:54:51.500381 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495083 2566 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 11 15:54:51.500381 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495090 2566 flags.go:64] FLAG: --anonymous-auth="true" Sep 11 15:54:51.500381 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495095 2566 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 11 15:54:51.500381 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495100 2566 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 11 15:54:51.500381 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495103 2566 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 11 15:54:51.500381 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495107 2566 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 11 15:54:51.500381 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495112 2566 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 11 15:54:51.500381 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495115 2566 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 11 15:54:51.500381 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495118 2566 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 11 15:54:51.500381 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495121 2566 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 11 15:54:51.500381 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495125 2566 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 11 15:54:51.500381 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495130 2566 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 11 15:54:51.500381 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495134 2566 flags.go:64] FLAG: --cgroup-root="" Sep 11 15:54:51.500381 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495137 2566 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 11 15:54:51.500381 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495140 2566 flags.go:64] FLAG: --client-ca-file="" Sep 11 15:54:51.500381 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495143 2566 flags.go:64] FLAG: --cloud-provider="external" Sep 11 15:54:51.500381 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495146 2566 flags.go:64] FLAG: --cluster-dns="[]" Sep 11 15:54:51.500381 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495150 2566 flags.go:64] FLAG: --cluster-domain="" Sep 11 15:54:51.500381 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495154 2566 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 11 15:54:51.500381 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495157 2566 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 11 15:54:51.500381 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495160 2566 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 11 15:54:51.500381 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495164 2566 flags.go:64] FLAG: --container-log-max-files="5" Sep 11 15:54:51.500945 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495168 2566 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 11 15:54:51.500945 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495171 2566 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 11 15:54:51.500945 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495174 2566 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 11 15:54:51.500945 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495177 2566 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 11 15:54:51.500945 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495180 2566 flags.go:64] FLAG: --contention-profiling="false" Sep 11 15:54:51.500945 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495183 2566 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 11 15:54:51.500945 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495186 2566 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 11 15:54:51.500945 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495189 2566 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 11 15:54:51.500945 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495192 2566 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 11 15:54:51.500945 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495196 2566 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 11 15:54:51.500945 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495199 2566 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 11 15:54:51.500945 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495202 2566 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 11 15:54:51.500945 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495205 2566 flags.go:64] FLAG: --enable-load-reader="false" Sep 11 15:54:51.500945 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495208 2566 flags.go:64] FLAG: --enable-server="true" Sep 11 15:54:51.500945 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495212 2566 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 11 15:54:51.500945 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495216 2566 flags.go:64] FLAG: --event-burst="100" Sep 11 15:54:51.500945 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495219 2566 flags.go:64] FLAG: --event-qps="50" Sep 11 15:54:51.500945 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495222 2566 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 11 15:54:51.500945 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495225 2566 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 11 15:54:51.500945 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495228 2566 flags.go:64] FLAG: --eviction-hard="" Sep 11 15:54:51.500945 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495232 2566 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 11 15:54:51.500945 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495237 2566 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 11 15:54:51.500945 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495241 2566 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 11 15:54:51.500945 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495244 2566 flags.go:64] FLAG: --eviction-soft="" Sep 11 15:54:51.500945 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495246 2566 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 11 15:54:51.501582 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495250 2566 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 11 15:54:51.501582 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495252 2566 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 11 15:54:51.501582 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495255 2566 flags.go:64] FLAG: --experimental-mounter-path="" Sep 11 15:54:51.501582 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495258 2566 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 11 15:54:51.501582 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495261 2566 flags.go:64] FLAG: --fail-swap-on="true" Sep 11 15:54:51.501582 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495264 2566 flags.go:64] FLAG: --feature-gates="" Sep 11 15:54:51.501582 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495268 2566 flags.go:64] FLAG: --file-check-frequency="20s" Sep 11 15:54:51.501582 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495271 2566 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 11 15:54:51.501582 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495274 2566 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 11 15:54:51.501582 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495277 2566 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 11 15:54:51.501582 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495280 2566 flags.go:64] FLAG: --healthz-port="10248" Sep 11 15:54:51.501582 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495283 2566 flags.go:64] FLAG: --help="false" Sep 11 15:54:51.501582 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495286 2566 flags.go:64] FLAG: --hostname-override="ip-10-0-136-96.ec2.internal" Sep 11 15:54:51.501582 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495289 2566 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 11 15:54:51.501582 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495292 2566 flags.go:64] FLAG: --http-check-frequency="20s" Sep 11 15:54:51.501582 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495295 2566 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 11 15:54:51.501582 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495298 2566 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 11 15:54:51.501582 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495301 2566 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 11 15:54:51.501582 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495305 2566 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 11 15:54:51.501582 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495307 2566 flags.go:64] FLAG: --image-service-endpoint="" Sep 11 15:54:51.501582 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495310 2566 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 11 15:54:51.501582 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495313 2566 flags.go:64] FLAG: --kube-api-burst="100" Sep 11 15:54:51.501582 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495316 2566 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 11 15:54:51.502169 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495319 2566 flags.go:64] FLAG: --kube-api-qps="50" Sep 11 15:54:51.502169 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495322 2566 flags.go:64] FLAG: --kube-reserved="" Sep 11 15:54:51.502169 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495325 2566 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 11 15:54:51.502169 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495328 2566 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 11 15:54:51.502169 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495331 2566 flags.go:64] FLAG: --kubelet-cgroups="" Sep 11 15:54:51.502169 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495336 2566 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 11 15:54:51.502169 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495339 2566 flags.go:64] FLAG: --lock-file="" Sep 11 15:54:51.502169 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495342 2566 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 11 15:54:51.502169 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495345 2566 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 11 15:54:51.502169 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495348 2566 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 11 15:54:51.502169 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495353 2566 flags.go:64] FLAG: --log-json-split-stream="false" Sep 11 15:54:51.502169 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495356 2566 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 11 15:54:51.502169 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495359 2566 flags.go:64] FLAG: --log-text-split-stream="false" Sep 11 15:54:51.502169 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495362 2566 flags.go:64] FLAG: --logging-format="text" Sep 11 15:54:51.502169 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495365 2566 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 11 15:54:51.502169 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495368 2566 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 11 15:54:51.502169 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495371 2566 flags.go:64] FLAG: --manifest-url="" Sep 11 15:54:51.502169 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495374 2566 flags.go:64] FLAG: --manifest-url-header="" Sep 11 15:54:51.502169 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495379 2566 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 11 15:54:51.502169 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495382 2566 flags.go:64] FLAG: --max-open-files="1000000" Sep 11 15:54:51.502169 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495387 2566 flags.go:64] FLAG: --max-pods="110" Sep 11 15:54:51.502169 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495389 2566 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 11 15:54:51.502169 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495393 2566 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 11 15:54:51.502169 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495396 2566 flags.go:64] FLAG: --memory-manager-policy="None" Sep 11 15:54:51.502169 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495399 2566 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 11 15:54:51.502816 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495402 2566 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 11 15:54:51.502816 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495405 2566 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 11 15:54:51.502816 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495407 2566 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 11 15:54:51.502816 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495414 2566 flags.go:64] FLAG: --node-status-max-images="50" Sep 11 15:54:51.502816 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495417 2566 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 11 15:54:51.502816 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495421 2566 flags.go:64] FLAG: --oom-score-adj="-999" Sep 11 15:54:51.502816 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495424 2566 flags.go:64] FLAG: --pod-cidr="" Sep 11 15:54:51.502816 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495428 2566 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10" Sep 11 15:54:51.502816 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495433 2566 flags.go:64] FLAG: --pod-manifest-path="" Sep 11 15:54:51.502816 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495437 2566 flags.go:64] FLAG: --pod-max-pids="-1" Sep 11 15:54:51.502816 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495440 2566 flags.go:64] FLAG: --pods-per-core="0" Sep 11 15:54:51.502816 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495443 2566 flags.go:64] FLAG: --port="10250" Sep 11 15:54:51.502816 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495447 2566 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 11 15:54:51.502816 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495451 2566 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0dc233a2b0c778bef" Sep 11 15:54:51.502816 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495455 2566 flags.go:64] FLAG: --qos-reserved="" Sep 11 15:54:51.502816 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495458 2566 flags.go:64] FLAG: --read-only-port="10255" Sep 11 15:54:51.502816 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495461 2566 flags.go:64] FLAG: --register-node="true" Sep 11 15:54:51.502816 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495464 2566 flags.go:64] FLAG: --register-with-taints="" Sep 11 15:54:51.502816 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495468 2566 flags.go:64] FLAG: --registry-burst="10" Sep 11 15:54:51.502816 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495471 2566 flags.go:64] FLAG: --registry-qps="5" Sep 11 15:54:51.502816 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495473 2566 flags.go:64] FLAG: --reserved-cpus="" Sep 11 15:54:51.502816 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495476 2566 flags.go:64] FLAG: --reserved-memory="" Sep 11 15:54:51.502816 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495480 2566 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 11 15:54:51.502816 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495483 2566 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 11 15:54:51.502816 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495486 2566 flags.go:64] FLAG: --rotate-certificates="false" Sep 11 15:54:51.503421 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495489 2566 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 11 15:54:51.503421 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495492 2566 flags.go:64] FLAG: --runonce="false" Sep 11 15:54:51.503421 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495495 2566 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 11 15:54:51.503421 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495498 2566 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 11 15:54:51.503421 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495501 2566 flags.go:64] FLAG: --seccomp-default="false" Sep 11 15:54:51.503421 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495504 2566 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 11 15:54:51.503421 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495507 2566 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 11 15:54:51.503421 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495510 2566 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 11 15:54:51.503421 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495513 2566 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 11 15:54:51.503421 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495517 2566 flags.go:64] FLAG: --storage-driver-password="root" Sep 11 15:54:51.503421 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495520 2566 flags.go:64] FLAG: --storage-driver-secure="false" Sep 11 15:54:51.503421 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495522 2566 flags.go:64] FLAG: --storage-driver-table="stats" Sep 11 15:54:51.503421 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495525 2566 flags.go:64] FLAG: --storage-driver-user="root" Sep 11 15:54:51.503421 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495528 2566 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 11 15:54:51.503421 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495532 2566 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 11 15:54:51.503421 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495535 2566 flags.go:64] FLAG: --system-cgroups="" Sep 11 15:54:51.503421 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495538 2566 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 11 15:54:51.503421 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495543 2566 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 11 15:54:51.503421 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495546 2566 flags.go:64] FLAG: --tls-cert-file="" Sep 11 15:54:51.503421 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495550 2566 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 11 15:54:51.503421 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495556 2566 flags.go:64] FLAG: --tls-min-version="" Sep 11 15:54:51.503421 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495559 2566 flags.go:64] FLAG: --tls-private-key-file="" Sep 11 15:54:51.503421 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495562 2566 flags.go:64] FLAG: --topology-manager-policy="none" Sep 11 15:54:51.503421 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495565 2566 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 11 15:54:51.503421 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495568 2566 flags.go:64] FLAG: --topology-manager-scope="container" Sep 11 15:54:51.504060 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495571 2566 flags.go:64] FLAG: --v="2" Sep 11 15:54:51.504060 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495575 2566 flags.go:64] FLAG: --version="false" Sep 11 15:54:51.504060 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495579 2566 flags.go:64] FLAG: --vmodule="" Sep 11 15:54:51.504060 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495588 2566 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 11 15:54:51.504060 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.495592 2566 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 11 15:54:51.504060 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.495713 2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 15:54:51.504060 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.495717 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 15:54:51.504060 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.495720 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 15:54:51.504060 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.495723 2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 15:54:51.504060 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.495726 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 15:54:51.504060 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.495729 2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 15:54:51.504060 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.495732 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 15:54:51.504060 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.495735 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 15:54:51.504060 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.495738 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 15:54:51.504060 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.495741 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 15:54:51.504060 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.495744 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 15:54:51.504060 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.495747 2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 15:54:51.504060 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.495750 2566 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 15:54:51.504060 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.495753 2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 15:54:51.504060 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.495756 2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 15:54:51.504558 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.495758 2566 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 15:54:51.504558 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.495761 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 15:54:51.504558 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.495764 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 15:54:51.504558 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.495773 2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 15:54:51.504558 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.495775 2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 15:54:51.504558 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.495778 2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 15:54:51.504558 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.495782 2566 feature_gate.go:328] unrecognized feature gate: Example Sep 11 15:54:51.504558 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.495786 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 15:54:51.504558 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.495789 2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 15:54:51.504558 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.495792 2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 15:54:51.504558 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.495794 2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 15:54:51.504558 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.495797 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 15:54:51.504558 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.495800 2566 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 15:54:51.504558 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.495802 2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 15:54:51.504558 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.495805 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 15:54:51.504558 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.495807 2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 15:54:51.504558 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.495810 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 15:54:51.504558 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.495812 2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 15:54:51.504558 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.495816 2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 15:54:51.504558 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.495820 2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 15:54:51.505208 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.495823 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 15:54:51.505208 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.495826 2566 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 15:54:51.505208 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.495828 2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 15:54:51.505208 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.495831 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 15:54:51.505208 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.495833 2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 15:54:51.505208 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.495836 2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 15:54:51.505208 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.495839 2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 15:54:51.505208 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.495841 2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 15:54:51.505208 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.495844 2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 15:54:51.505208 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.495846 2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 15:54:51.505208 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.495856 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 15:54:51.505208 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.495858 2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 15:54:51.505208 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.495861 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 15:54:51.505208 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.495864 2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 15:54:51.505208 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.495866 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 15:54:51.505208 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.495869 2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 15:54:51.505208 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.495878 2566 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 15:54:51.505208 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.495881 2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 15:54:51.505208 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.495885 2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 15:54:51.505673 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.495889 2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 15:54:51.505673 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.495893 2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 15:54:51.505673 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.495896 2566 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 15:54:51.505673 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.495899 2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 15:54:51.505673 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.495902 2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 15:54:51.505673 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.495905 2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 15:54:51.505673 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.495907 2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 15:54:51.505673 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.495910 2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 15:54:51.505673 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.495913 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 15:54:51.505673 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.495915 2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 15:54:51.505673 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.495918 2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 15:54:51.505673 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.495921 2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 15:54:51.505673 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.495924 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 15:54:51.505673 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.495926 2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 15:54:51.505673 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.495928 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 15:54:51.505673 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.495931 2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 15:54:51.505673 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.495934 2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 15:54:51.505673 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.495936 2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 15:54:51.505673 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.495939 2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 15:54:51.505673 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.495941 2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 15:54:51.506185 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.495944 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 15:54:51.506185 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.495947 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 15:54:51.506185 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.495949 2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 15:54:51.506185 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.495952 2566 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 15:54:51.506185 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.495954 2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 15:54:51.506185 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.495957 2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 15:54:51.506185 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.495959 2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 15:54:51.506185 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.495962 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 15:54:51.506185 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.495983 2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 15:54:51.506185 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.495988 2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 15:54:51.506185 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.495993 2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 15:54:51.506185 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.495997 2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 15:54:51.506185 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.496002 2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 15:54:51.506185 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.496004 2566 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 15:54:51.506185 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.496007 2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 15:54:51.506185 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.496010 2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 15:54:51.506185 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.496014 2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 15:54:51.506185 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.496017 2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 15:54:51.506185 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.496020 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 15:54:51.506185 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.496023 2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 15:54:51.506680 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.496026 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 15:54:51.506680 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.496030 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 15:54:51.506680 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.496033 2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 15:54:51.506680 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.496036 2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 15:54:51.506680 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.496039 2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 15:54:51.506680 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.496617 2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 11 15:54:51.506680 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.503698 2566 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 11 15:54:51.506680 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.503831 2566 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 11 15:54:51.506680 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.504590 2566 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 11 15:54:51.506680 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.504598 2566 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 11 15:54:51.506680 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.504656 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 15:54:51.506680 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.504662 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 15:54:51.506680 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.504666 2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 15:54:51.506680 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.504669 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 15:54:51.507030 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.504673 2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 15:54:51.507030 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.504675 2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 15:54:51.507030 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.504678 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 15:54:51.507030 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.504681 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 15:54:51.507030 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.504684 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 15:54:51.507030 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.504686 2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 15:54:51.507030 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.504689 2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 15:54:51.507030 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.504692 2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 15:54:51.507030 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.504695 2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 15:54:51.507030 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.504698 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 15:54:51.507030 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.504700 2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 15:54:51.507030 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.504703 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 15:54:51.507030 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.504706 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 15:54:51.507030 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.504709 2566 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 15:54:51.507030 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.504711 2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 15:54:51.507030 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.504714 2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 15:54:51.507030 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.504717 2566 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 15:54:51.507030 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.504719 2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 15:54:51.507030 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.504722 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 15:54:51.507030 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.504725 2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 15:54:51.507526 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.504727 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 15:54:51.507526 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.504730 2566 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 15:54:51.507526 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.504732 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 15:54:51.507526 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.504736 2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 15:54:51.507526 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.504740 2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 15:54:51.507526 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.504744 2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 15:54:51.507526 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.504747 2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 15:54:51.507526 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.504751 2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 15:54:51.507526 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.504755 2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 15:54:51.507526 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.504759 2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 15:54:51.507526 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.504762 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 15:54:51.507526 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.504765 2566 feature_gate.go:328] unrecognized feature gate: Example Sep 11 15:54:51.507526 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.504768 2566 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 15:54:51.507526 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.504771 2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 15:54:51.507526 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.504774 2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 15:54:51.507526 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.504776 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 15:54:51.507526 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.504779 2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 15:54:51.507526 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.504782 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 15:54:51.507526 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.504785 2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 15:54:51.507996 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.504788 2566 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 15:54:51.507996 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.504791 2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 15:54:51.507996 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.504794 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 15:54:51.507996 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.504796 2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 15:54:51.507996 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.504799 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 15:54:51.507996 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.504802 2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 15:54:51.507996 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.504804 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 15:54:51.507996 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.504807 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 15:54:51.507996 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.504809 2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 15:54:51.507996 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.504812 2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 15:54:51.507996 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.504814 2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 15:54:51.507996 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.504817 2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 15:54:51.507996 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.504819 2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 15:54:51.507996 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.504822 2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 15:54:51.507996 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.504825 2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 15:54:51.507996 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.504828 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 15:54:51.507996 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.504830 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 15:54:51.507996 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.504833 2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 15:54:51.507996 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.504837 2566 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 15:54:51.507996 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.504840 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 15:54:51.508594 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.504842 2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 15:54:51.508594 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.504845 2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 15:54:51.508594 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.504849 2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 15:54:51.508594 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.504852 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 15:54:51.508594 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.504854 2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 15:54:51.508594 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.504857 2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 15:54:51.508594 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.504860 2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 15:54:51.508594 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.504863 2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 15:54:51.508594 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.504866 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 15:54:51.508594 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.504869 2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 15:54:51.508594 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.504872 2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 15:54:51.508594 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.504875 2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 15:54:51.508594 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.504877 2566 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 15:54:51.508594 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.504880 2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 15:54:51.508594 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.504882 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 15:54:51.508594 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.504885 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 15:54:51.508594 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.504887 2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 15:54:51.508594 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.504890 2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 15:54:51.508594 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.504893 2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 15:54:51.509085 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.504895 2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 15:54:51.509085 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.504897 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 15:54:51.509085 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.504900 2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 15:54:51.509085 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.504903 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 15:54:51.509085 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.504905 2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 15:54:51.509085 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.504908 2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 15:54:51.509085 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.504910 2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 15:54:51.509085 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.504912 2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 15:54:51.509085 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.504915 2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 15:54:51.509085 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.504918 2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 15:54:51.509085 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.504921 2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 15:54:51.509085 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.504924 2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 15:54:51.509085 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.504927 2566 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 15:54:51.509085 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.504930 2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 15:54:51.509085 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.504933 2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 15:54:51.509085 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.504936 2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 15:54:51.509085 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.504938 2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 15:54:51.509600 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.504944 2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 11 15:54:51.509600 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.505071 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 15:54:51.509600 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.505078 2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 15:54:51.509600 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.505082 2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 15:54:51.509600 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.505085 2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 15:54:51.509600 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.505088 2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 15:54:51.509600 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.505091 2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 15:54:51.509600 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.505094 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 15:54:51.509600 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.505096 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 15:54:51.509600 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.505099 2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 15:54:51.509600 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.505102 2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 15:54:51.509600 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.505104 2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 15:54:51.509600 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.505107 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 15:54:51.509600 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.505110 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 15:54:51.509961 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.505113 2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 15:54:51.509961 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.505115 2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 15:54:51.509961 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.505118 2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 15:54:51.509961 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.505120 2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 15:54:51.509961 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.505123 2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 15:54:51.509961 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.505126 2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 15:54:51.509961 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.505128 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 15:54:51.509961 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.505132 2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 15:54:51.509961 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.505135 2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 15:54:51.509961 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.505137 2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 15:54:51.509961 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.505141 2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 15:54:51.509961 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.505144 2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 15:54:51.509961 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.505147 2566 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 15:54:51.509961 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.505150 2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 15:54:51.509961 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.505153 2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 15:54:51.509961 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.505156 2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 15:54:51.509961 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.505159 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 15:54:51.509961 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.505161 2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 15:54:51.509961 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.505164 2566 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 15:54:51.509961 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.505166 2566 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 15:54:51.510553 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.505169 2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 15:54:51.510553 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.505171 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 15:54:51.510553 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.505174 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 15:54:51.510553 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.505176 2566 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 15:54:51.510553 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.505179 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 15:54:51.510553 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.505181 2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 15:54:51.510553 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.505184 2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 15:54:51.510553 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.505186 2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 15:54:51.510553 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.505189 2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 15:54:51.510553 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.505191 2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 15:54:51.510553 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.505194 2566 feature_gate.go:328] unrecognized feature gate: Example Sep 11 15:54:51.510553 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.505197 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 15:54:51.510553 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.505199 2566 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 15:54:51.510553 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.505202 2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 15:54:51.510553 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.505204 2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 15:54:51.510553 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.505207 2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 15:54:51.510553 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.505209 2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 15:54:51.510553 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.505212 2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 15:54:51.510553 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.505215 2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 15:54:51.510553 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.505217 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 15:54:51.511099 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.505220 2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 15:54:51.511099 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.505223 2566 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 15:54:51.511099 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.505225 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 15:54:51.511099 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.505228 2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 15:54:51.511099 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.505231 2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 15:54:51.511099 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.505233 2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 15:54:51.511099 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.505236 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 15:54:51.511099 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.505238 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 15:54:51.511099 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.505241 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 15:54:51.511099 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.505244 2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 15:54:51.511099 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.505247 2566 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 15:54:51.511099 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.505249 2566 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 15:54:51.511099 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.505251 2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 15:54:51.511099 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.505254 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 15:54:51.511099 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.505257 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 15:54:51.511099 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.505259 2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 15:54:51.511099 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.505262 2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 15:54:51.511099 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.505264 2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 15:54:51.511099 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.505266 2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 15:54:51.511099 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.505269 2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 15:54:51.511896 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.505272 2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 15:54:51.511896 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.505274 2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 15:54:51.511896 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.505277 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 15:54:51.511896 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.505279 2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 15:54:51.511896 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.505282 2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 15:54:51.511896 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.505284 2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 15:54:51.511896 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.505287 2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 15:54:51.511896 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.505290 2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 15:54:51.511896 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.505292 2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 15:54:51.511896 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.505294 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 15:54:51.511896 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.505298 2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 15:54:51.511896 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.505302 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 15:54:51.511896 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.505304 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 15:54:51.511896 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.505307 2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 15:54:51.511896 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.505310 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 15:54:51.511896 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.505313 2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 15:54:51.511896 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.505317 2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 15:54:51.511896 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.505319 2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 15:54:51.511896 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.505322 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 15:54:51.512676 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.505324 2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 15:54:51.512676 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.505326 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 15:54:51.512676 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.505330 2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 15:54:51.512676 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.505332 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 15:54:51.512676 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.505335 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 15:54:51.512676 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.505337 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 15:54:51.512676 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:51.505339 2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 15:54:51.512676 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.505344 2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 11 15:54:51.512676 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.505460 2566 server.go:962] "Client rotation is on, will bootstrap in background" Sep 11 15:54:51.512676 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.508258 2566 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 11 15:54:51.512676 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.509057 2566 server.go:1019] "Starting client certificate rotation" Sep 11 15:54:51.512676 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.509171 2566 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 11 15:54:51.512676 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.509222 2566 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 11 15:54:51.513340 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.511531 2566 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 11 15:54:51.516241 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.516212 2566 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 11 15:54:51.526243 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.526220 2566 log.go:25] "Validated CRI v1 runtime API" Sep 11 15:54:51.528976 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.528961 2566 log.go:25] "Validated CRI v1 image API" Sep 11 15:54:51.529922 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.529905 2566 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 11 15:54:51.533333 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.533313 2566 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 ae9d8903-f632-44ce-b513-3f57ee363530:/dev/nvme0n1p4 c5d2e532-14a1-4711-b21f-43fd39f51871:/dev/nvme0n1p3] Sep 11 15:54:51.533391 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.533333 2566 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 11 15:54:51.538383 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.538366 2566 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 11 15:54:51.538521 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.538381 2566 manager.go:216] Machine: {Timestamp:2026-09-11 15:54:51.53708035 +0000 UTC m=+0.348840969 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101569 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2741b034650e1690edf9787a6968a4 SystemUUID:ec2741b0-3465-0e16-90ed-f9787a6968a4 BootID:ca091847-9ecc-4719-84ba-6f8b0d238d30 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:49:37:b4:3e:c9 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:49:37:b4:3e:c9 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:b2:98:83:e9:98:bf Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 11 15:54:51.538521 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.538519 2566 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 11 15:54:51.538652 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.538641 2566 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 11 15:54:51.540463 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.540436 2566 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 11 15:54:51.540602 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.540467 2566 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-96.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 11 15:54:51.541371 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.541361 2566 topology_manager.go:138] "Creating topology manager with none policy" Sep 11 15:54:51.541411 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.541374 2566 container_manager_linux.go:309] "Creating device plugin manager" Sep 11 15:54:51.541411 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.541385 2566 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 11 15:54:51.541965 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.541955 2566 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 11 15:54:51.542675 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.542665 2566 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 11 15:54:51.546318 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.546306 2566 state_mem.go:36] "Initialized new in-memory state store" Sep 11 15:54:51.546434 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.546425 2566 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 11 15:54:51.546469 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.546455 2566 kubelet.go:486] "Attempting to sync node with API server" Sep 11 15:54:51.546497 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.546468 2566 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 11 15:54:51.546497 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.546484 2566 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 11 15:54:51.546497 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.546496 2566 kubelet.go:398] "Adding apiserver pod source" Sep 11 15:54:51.546572 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.546505 2566 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 11 15:54:51.546990 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.546975 2566 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 11 15:54:51.547036 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.546995 2566 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 11 15:54:51.549397 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.549378 2566 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 11 15:54:51.551530 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.551516 2566 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 11 15:54:51.551578 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.551549 2566 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 11 15:54:51.553542 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.553529 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 11 15:54:51.553585 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.553549 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 11 15:54:51.553585 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.553555 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 11 15:54:51.553585 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.553561 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 11 15:54:51.553585 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.553570 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 11 15:54:51.553585 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.553586 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 11 15:54:51.553726 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.553605 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 11 15:54:51.553726 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.553610 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 11 15:54:51.553726 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.553617 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 11 15:54:51.553726 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.553622 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 11 15:54:51.553726 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.553635 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 11 15:54:51.553726 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.553640 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 11 15:54:51.554903 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.554890 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 11 15:54:51.554903 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.554901 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 11 15:54:51.559096 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.559078 2566 server.go:1268] "Started kubelet" Sep 11 15:54:51.559174 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.559111 2566 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 11 15:54:51.559248 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.559200 2566 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 11 15:54:51.561830 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.561805 2566 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 11 15:54:51.562184 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.562131 2566 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-136-96.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 11 15:54:51.562518 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:54:51.562482 2566 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-136-96.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 11 15:54:51.562635 ip-10-0-136-96 systemd[1]: Started Kubernetes Kubelet. Sep 11 15:54:51.562765 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:54:51.562726 2566 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 11 15:54:51.564363 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.564339 2566 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 11 15:54:51.566525 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.566432 2566 server.go:310] "Adding debug handlers to kubelet server" Sep 11 15:54:51.568193 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:54:51.567301 2566 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-96.ec2.internal.18d44f40c41266da default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-96.ec2.internal,UID:ip-10-0-136-96.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-136-96.ec2.internal,},FirstTimestamp:2026-09-11 15:54:51.558971098 +0000 UTC m=+0.370731713,LastTimestamp:2026-09-11 15:54:51.558971098 +0000 UTC m=+0.370731713,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-96.ec2.internal,}" Sep 11 15:54:51.569942 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.569929 2566 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 11 15:54:51.570006 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.569990 2566 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 11 15:54:51.570293 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:54:51.570261 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-96.ec2.internal\" not found" Sep 11 15:54:51.570803 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.570727 2566 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 11 15:54:51.570803 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.570745 2566 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 11 15:54:51.571003 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.570878 2566 reconstruct.go:97] "Volume reconstruction finished" Sep 11 15:54:51.571003 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.570886 2566 reconciler.go:29] "Reconciler: start to sync state" Sep 11 15:54:51.572432 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.572157 2566 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 11 15:54:51.572827 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.572799 2566 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 11 15:54:51.572827 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.572818 2566 factory.go:55] Registering systemd factory Sep 11 15:54:51.572827 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.572828 2566 factory.go:223] Registration of the systemd container factory successfully Sep 11 15:54:51.573099 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.573082 2566 factory.go:153] Registering CRI-O factory Sep 11 15:54:51.573099 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.573101 2566 factory.go:223] Registration of the crio container factory successfully Sep 11 15:54:51.573244 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.573121 2566 factory.go:103] Registering Raw factory Sep 11 15:54:51.573244 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.573133 2566 manager.go:1196] Started watching for new ooms in manager Sep 11 15:54:51.573635 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.573623 2566 manager.go:349] Starting recovery of all containers Sep 11 15:54:51.573689 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:54:51.573655 2566 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 11 15:54:51.580224 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:54:51.580199 2566 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-136-96.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 11 15:54:51.580329 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:54:51.580306 2566 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 11 15:54:51.585380 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.585359 2566 manager.go:354] Recovery completed Sep 11 15:54:51.589884 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.589865 2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 15:54:51.591967 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.591133 2566 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-m7pj9" Sep 11 15:54:51.592148 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.592126 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-96.ec2.internal" event="NodeHasSufficientMemory" Sep 11 15:54:51.592265 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.592163 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-96.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 15:54:51.592265 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.592178 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-96.ec2.internal" event="NodeHasSufficientPID" Sep 11 15:54:51.592910 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.592896 2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 15:54:51.592993 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.592896 2566 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 11 15:54:51.592993 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.592944 2566 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 11 15:54:51.592993 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.592961 2566 state_mem.go:36] "Initialized new in-memory state store" Sep 11 15:54:51.593728 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.593714 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-96.ec2.internal" event="NodeHasSufficientMemory" Sep 11 15:54:51.593792 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.593739 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-96.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 15:54:51.593792 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.593752 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-96.ec2.internal" event="NodeHasSufficientPID" Sep 11 15:54:51.594766 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:54:51.594698 2566 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-96.ec2.internal.18d44f40c60c9c1f default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-96.ec2.internal,UID:ip-10-0-136-96.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-136-96.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-136-96.ec2.internal,},FirstTimestamp:2026-09-11 15:54:51.592145951 +0000 UTC m=+0.403906604,LastTimestamp:2026-09-11 15:54:51.592145951 +0000 UTC m=+0.403906604,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-96.ec2.internal,}" Sep 11 15:54:51.595130 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.595118 2566 policy_none.go:49] "None policy: Start" Sep 11 15:54:51.595172 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.595135 2566 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 11 15:54:51.595172 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.595146 2566 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 11 15:54:51.597159 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.597144 2566 policy_none.go:47] "Start" Sep 11 15:54:51.600580 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.600557 2566 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-m7pj9" Sep 11 15:54:51.602025 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:54:51.601942 2566 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-96.ec2.internal.18d44f40c60cf84e default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-96.ec2.internal,UID:ip-10-0-136-96.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-136-96.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-136-96.ec2.internal,},FirstTimestamp:2026-09-11 15:54:51.59216955 +0000 UTC m=+0.403930166,LastTimestamp:2026-09-11 15:54:51.59216955 +0000 UTC m=+0.403930166,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-96.ec2.internal,}" Sep 11 15:54:51.604367 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:54:51.604276 2566 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-96.ec2.internal.18d44f40c60d2d50 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-96.ec2.internal,UID:ip-10-0-136-96.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-136-96.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-136-96.ec2.internal,},FirstTimestamp:2026-09-11 15:54:51.59218312 +0000 UTC m=+0.403943736,LastTimestamp:2026-09-11 15:54:51.59218312 +0000 UTC m=+0.403943736,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-96.ec2.internal,}" Sep 11 15:54:51.647271 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.633807 2566 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 11 15:54:51.647271 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.635073 2566 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 11 15:54:51.647271 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.635134 2566 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 11 15:54:51.647271 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.635160 2566 kubelet.go:2448] "Starting kubelet main sync loop" Sep 11 15:54:51.647271 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:54:51.635206 2566 kubelet.go:2472] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Sep 11 15:54:51.647271 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.637752 2566 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 15:54:51.647271 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.639688 2566 manager.go:337] "Starting Device Plugin manager" Sep 11 15:54:51.647271 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:54:51.639727 2566 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 11 15:54:51.647271 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.639740 2566 server.go:87] "Starting device plugin registration server" Sep 11 15:54:51.647271 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.640065 2566 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 11 15:54:51.647271 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.640077 2566 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 11 15:54:51.647271 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.640176 2566 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 11 15:54:51.647271 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.640253 2566 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 11 15:54:51.647271 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.640264 2566 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 11 15:54:51.647271 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:54:51.640778 2566 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 11 15:54:51.647271 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:54:51.640821 2566 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-96.ec2.internal\" not found" Sep 11 15:54:51.736089 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.736030 2566 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-96.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-96.ec2.internal"] Sep 11 15:54:51.736287 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.736144 2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 15:54:51.737868 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.737843 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-96.ec2.internal" event="NodeHasSufficientMemory" Sep 11 15:54:51.737972 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.737885 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-96.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 15:54:51.737972 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.737896 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-96.ec2.internal" event="NodeHasSufficientPID" Sep 11 15:54:51.740376 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.740362 2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 15:54:51.740470 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.740451 2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 15:54:51.740536 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.740520 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-96.ec2.internal" Sep 11 15:54:51.740572 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.740563 2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 15:54:51.741270 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.741254 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-96.ec2.internal" event="NodeHasSufficientMemory" Sep 11 15:54:51.741336 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.741254 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-96.ec2.internal" event="NodeHasSufficientMemory" Sep 11 15:54:51.741336 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.741307 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-96.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 15:54:51.741394 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.741337 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-96.ec2.internal" event="NodeHasSufficientPID" Sep 11 15:54:51.741394 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.741256 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-96.ec2.internal" event="NodeHasSufficientMemory" Sep 11 15:54:51.741394 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.741357 2566 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-136-96.ec2.internal" Sep 11 15:54:51.741394 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.741362 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-96.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 15:54:51.741394 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.741379 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-96.ec2.internal" event="NodeHasSufficientPID" Sep 11 15:54:51.741550 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.741285 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-96.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 15:54:51.741550 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.741451 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-96.ec2.internal" event="NodeHasSufficientPID" Sep 11 15:54:51.743684 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.743669 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-96.ec2.internal" Sep 11 15:54:51.743746 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.743696 2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 15:54:51.744413 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.744399 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-96.ec2.internal" event="NodeHasSufficientMemory" Sep 11 15:54:51.744483 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.744428 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-96.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 15:54:51.744483 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.744438 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-96.ec2.internal" event="NodeHasSufficientPID" Sep 11 15:54:51.753513 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.753498 2566 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-136-96.ec2.internal" Sep 11 15:54:51.753604 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:54:51.753522 2566 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-96.ec2.internal\": node \"ip-10-0-136-96.ec2.internal\" not found" Sep 11 15:54:51.765601 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:54:51.765582 2566 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-96.ec2.internal\" not found" node="ip-10-0-136-96.ec2.internal" Sep 11 15:54:51.768920 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:54:51.768905 2566 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-96.ec2.internal\" not found" node="ip-10-0-136-96.ec2.internal" Sep 11 15:54:51.791218 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:54:51.791199 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-96.ec2.internal\" not found" Sep 11 15:54:51.872932 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.872906 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e46b070927ba07e09c7fdfca6f7f4f91-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-96.ec2.internal\" (UID: \"e46b070927ba07e09c7fdfca6f7f4f91\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-96.ec2.internal" Sep 11 15:54:51.873020 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.872945 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e46b070927ba07e09c7fdfca6f7f4f91-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-96.ec2.internal\" (UID: \"e46b070927ba07e09c7fdfca6f7f4f91\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-96.ec2.internal" Sep 11 15:54:51.873020 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.872995 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bd507eff9b3716da03d385309d794351-config\") pod \"kube-apiserver-proxy-ip-10-0-136-96.ec2.internal\" (UID: \"bd507eff9b3716da03d385309d794351\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-96.ec2.internal" Sep 11 15:54:51.892032 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:54:51.892009 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-96.ec2.internal\" not found" Sep 11 15:54:51.973394 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.973357 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e46b070927ba07e09c7fdfca6f7f4f91-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-96.ec2.internal\" (UID: \"e46b070927ba07e09c7fdfca6f7f4f91\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-96.ec2.internal" Sep 11 15:54:51.973394 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.973398 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e46b070927ba07e09c7fdfca6f7f4f91-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-96.ec2.internal\" (UID: \"e46b070927ba07e09c7fdfca6f7f4f91\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-96.ec2.internal" Sep 11 15:54:51.973504 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.973451 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bd507eff9b3716da03d385309d794351-config\") pod \"kube-apiserver-proxy-ip-10-0-136-96.ec2.internal\" (UID: \"bd507eff9b3716da03d385309d794351\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-96.ec2.internal" Sep 11 15:54:51.973504 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.973495 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bd507eff9b3716da03d385309d794351-config\") pod \"kube-apiserver-proxy-ip-10-0-136-96.ec2.internal\" (UID: \"bd507eff9b3716da03d385309d794351\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-96.ec2.internal" Sep 11 15:54:51.973570 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.973504 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e46b070927ba07e09c7fdfca6f7f4f91-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-96.ec2.internal\" (UID: \"e46b070927ba07e09c7fdfca6f7f4f91\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-96.ec2.internal" Sep 11 15:54:51.973570 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:51.973548 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e46b070927ba07e09c7fdfca6f7f4f91-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-96.ec2.internal\" (UID: \"e46b070927ba07e09c7fdfca6f7f4f91\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-96.ec2.internal" Sep 11 15:54:51.992535 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:54:51.992466 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-96.ec2.internal\" not found" Sep 11 15:54:52.067946 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:52.067901 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-96.ec2.internal" Sep 11 15:54:52.071276 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:52.071254 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-96.ec2.internal" Sep 11 15:54:52.092916 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:54:52.092891 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-96.ec2.internal\" not found" Sep 11 15:54:52.193511 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:54:52.193468 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-96.ec2.internal\" not found" Sep 11 15:54:52.294163 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:54:52.294083 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-96.ec2.internal\" not found" Sep 11 15:54:53.991811 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:54:53.991763 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-96.ec2.internal\" not found" Sep 11 15:54:54.092423 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:54:54.092393 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-96.ec2.internal\" not found" Sep 11 15:54:54.105526 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:54.105500 2566 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 11 15:54:54.105672 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:54.105654 2566 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 11 15:54:54.192910 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:54:54.192832 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-96.ec2.internal\" not found" Sep 11 15:54:54.204037 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:54.203992 2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-10 15:49:53 +0000 UTC" deadline="2028-03-08 20:49:45.895946488 +0000 UTC" Sep 11 15:54:54.204037 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:54.204030 2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13060h54m51.691919666s" Sep 11 15:54:54.278618 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:54.278443 2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 15:54:54.279662 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:54.279642 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-96.ec2.internal" event="NodeHasSufficientMemory" Sep 11 15:54:54.279783 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:54.279676 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-96.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 15:54:54.279783 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:54.279687 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-96.ec2.internal" event="NodeHasSufficientPID" Sep 11 15:54:54.293123 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:54:54.293094 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-96.ec2.internal\" not found" Sep 11 15:54:54.346727 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:54.346694 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode46b070927ba07e09c7fdfca6f7f4f91.slice/crio-e509d109296443bf7ffdf77431854bf774d00967a5c087f4b5489f098da45170 WatchSource:0}: Error finding container e509d109296443bf7ffdf77431854bf774d00967a5c087f4b5489f098da45170: Status 404 returned error can't find the container with id e509d109296443bf7ffdf77431854bf774d00967a5c087f4b5489f098da45170 Sep 11 15:54:54.349190 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:54.349150 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd507eff9b3716da03d385309d794351.slice/crio-4a6142c1b763d85cdf255e33d297b5513815136f75cf57bcb06a498abd77c983 WatchSource:0}: Error finding container 4a6142c1b763d85cdf255e33d297b5513815136f75cf57bcb06a498abd77c983: Status 404 returned error can't find the container with id 4a6142c1b763d85cdf255e33d297b5513815136f75cf57bcb06a498abd77c983 Sep 11 15:54:54.352574 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:54.352559 2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 15:54:54.393781 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:54:54.393753 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-96.ec2.internal\" not found" Sep 11 15:54:54.474102 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:54.474024 2566 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 15:54:54.494825 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:54:54.494785 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-96.ec2.internal\" not found" Sep 11 15:54:54.550468 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:54.550445 2566 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 15:54:54.595722 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:54:54.595694 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-96.ec2.internal\" not found" Sep 11 15:54:54.696610 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:54:54.696580 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-96.ec2.internal\" not found" Sep 11 15:54:54.761838 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:54.761764 2566 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 15:54:54.767910 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:54.767892 2566 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-96.ec2.internal" Sep 11 15:54:54.786095 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:54.786075 2566 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 11 15:54:54.787143 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:54.787131 2566 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-96.ec2.internal" Sep 11 15:54:54.796301 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:54.796287 2566 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 11 15:54:55.144640 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.144562 2566 apiserver.go:52] "Watching apiserver" Sep 11 15:54:55.150331 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.150301 2566 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 11 15:54:55.150754 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.150728 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-nzcjl","openshift-network-operator/iptables-alerter-5hlvj","openshift-ovn-kubernetes/ovnkube-node-tgq5c","kube-system/konnectivity-agent-spxb6","kube-system/kube-apiserver-proxy-ip-10-0-136-96.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xkvw8","openshift-cluster-node-tuning-operator/tuned-sn7js","openshift-dns/node-resolver-sxqj2","openshift-multus/multus-4vhlr","openshift-multus/multus-additional-cni-plugins-b7ftm","openshift-multus/network-metrics-daemon-plkhx","kube-system/global-pull-secret-syncer-95zvh","openshift-image-registry/node-ca-bctd5","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-96.ec2.internal"] Sep 11 15:54:55.153607 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.153219 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-sn7js" Sep 11 15:54:55.155663 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.155366 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-plkhx" Sep 11 15:54:55.155663 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:54:55.155438 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-plkhx" podUID="331632d0-fa59-4462-805e-bce0e18b95bc" Sep 11 15:54:55.155817 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.155672 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-h56bx\"" Sep 11 15:54:55.156006 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.155932 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 11 15:54:55.156006 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.155935 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 11 15:54:55.158169 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.157626 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-tgq5c" Sep 11 15:54:55.159852 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.159830 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 11 15:54:55.159937 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.159894 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 11 15:54:55.160045 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.160023 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 11 15:54:55.160117 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.160094 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 11 15:54:55.160171 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.160033 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 11 15:54:55.160171 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.159841 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 11 15:54:55.160171 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.160153 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-h99f2\"" Sep 11 15:54:55.160364 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.160346 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-spxb6" Sep 11 15:54:55.162156 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.162137 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-sxqj2" Sep 11 15:54:55.162599 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.162580 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 11 15:54:55.162694 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.162648 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 11 15:54:55.162845 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.162829 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-m7rnp\"" Sep 11 15:54:55.164930 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.163881 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 11 15:54:55.164930 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.164163 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-pssvp\"" Sep 11 15:54:55.164930 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.164382 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 11 15:54:55.164930 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.164660 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xkvw8" Sep 11 15:54:55.166901 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.166877 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 11 15:54:55.167459 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.167197 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-rl7kn\"" Sep 11 15:54:55.167459 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.167303 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 11 15:54:55.167459 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.167382 2566 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 11 15:54:55.167459 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.167399 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 11 15:54:55.169624 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.169605 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4vhlr" Sep 11 15:54:55.173322 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.171815 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 11 15:54:55.173322 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.171946 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 11 15:54:55.173322 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.172093 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 11 15:54:55.173322 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.172494 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 11 15:54:55.173322 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.172495 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-fpwg7\"" Sep 11 15:54:55.173895 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.173872 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bctd5" Sep 11 15:54:55.176056 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.176034 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 11 15:54:55.176704 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.176359 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 11 15:54:55.176704 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.176397 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 11 15:54:55.176704 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.176432 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-xppsj\"" Sep 11 15:54:55.177174 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.177155 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5hlvj" Sep 11 15:54:55.177616 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.177489 2566 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 11 15:54:55.179026 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.178958 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 11 15:54:55.179182 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.179031 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eab992d2-f32f-41bc-a028-c3fa544c4fec-lib-modules\") pod \"tuned-sn7js\" (UID: \"eab992d2-f32f-41bc-a028-c3fa544c4fec\") " pod="openshift-cluster-node-tuning-operator/tuned-sn7js" Sep 11 15:54:55.179253 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.179210 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/981d7d1e-f81f-48fc-8962-7831bc5c2b27-ovn-node-metrics-cert\") pod \"ovnkube-node-tgq5c\" (UID: \"981d7d1e-f81f-48fc-8962-7831bc5c2b27\") " pod="openshift-ovn-kubernetes/ovnkube-node-tgq5c" Sep 11 15:54:55.179253 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.179233 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-m5lk7\"" Sep 11 15:54:55.179253 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.179236 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcpmk\" (UniqueName: \"kubernetes.io/projected/ed90592a-177c-4128-b153-b51d18b8f619-kube-api-access-jcpmk\") pod \"node-resolver-sxqj2\" (UID: \"ed90592a-177c-4128-b153-b51d18b8f619\") " pod="openshift-dns/node-resolver-sxqj2" Sep 11 15:54:55.179405 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.179261 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xknqc\" (UniqueName: \"kubernetes.io/projected/249a0e7e-1da6-422a-8492-1820f6437318-kube-api-access-xknqc\") pod \"node-ca-bctd5\" (UID: \"249a0e7e-1da6-422a-8492-1820f6437318\") " pod="openshift-image-registry/node-ca-bctd5" Sep 11 15:54:55.179405 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.179295 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/eab992d2-f32f-41bc-a028-c3fa544c4fec-etc-modprobe-d\") pod \"tuned-sn7js\" (UID: \"eab992d2-f32f-41bc-a028-c3fa544c4fec\") " pod="openshift-cluster-node-tuning-operator/tuned-sn7js" Sep 11 15:54:55.179405 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.179319 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/eab992d2-f32f-41bc-a028-c3fa544c4fec-etc-sysconfig\") pod \"tuned-sn7js\" (UID: \"eab992d2-f32f-41bc-a028-c3fa544c4fec\") " pod="openshift-cluster-node-tuning-operator/tuned-sn7js" Sep 11 15:54:55.179405 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.179342 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/981d7d1e-f81f-48fc-8962-7831bc5c2b27-etc-openvswitch\") pod \"ovnkube-node-tgq5c\" (UID: \"981d7d1e-f81f-48fc-8962-7831bc5c2b27\") " pod="openshift-ovn-kubernetes/ovnkube-node-tgq5c" Sep 11 15:54:55.179405 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.179363 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/981d7d1e-f81f-48fc-8962-7831bc5c2b27-run-ovn\") pod \"ovnkube-node-tgq5c\" (UID: \"981d7d1e-f81f-48fc-8962-7831bc5c2b27\") " pod="openshift-ovn-kubernetes/ovnkube-node-tgq5c" Sep 11 15:54:55.179405 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.179397 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/981d7d1e-f81f-48fc-8962-7831bc5c2b27-ovnkube-config\") pod \"ovnkube-node-tgq5c\" (UID: \"981d7d1e-f81f-48fc-8962-7831bc5c2b27\") " pod="openshift-ovn-kubernetes/ovnkube-node-tgq5c" Sep 11 15:54:55.179713 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.179431 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e05877f1-2ad9-43d5-8523-4749d6b08782-registration-dir\") pod \"aws-ebs-csi-driver-node-xkvw8\" (UID: \"e05877f1-2ad9-43d5-8523-4749d6b08782\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xkvw8" Sep 11 15:54:55.179713 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.179441 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 11 15:54:55.179713 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.179454 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eab992d2-f32f-41bc-a028-c3fa544c4fec-sys\") pod \"tuned-sn7js\" (UID: \"eab992d2-f32f-41bc-a028-c3fa544c4fec\") " pod="openshift-cluster-node-tuning-operator/tuned-sn7js" Sep 11 15:54:55.179713 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.179475 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/981d7d1e-f81f-48fc-8962-7831bc5c2b27-node-log\") pod \"ovnkube-node-tgq5c\" (UID: \"981d7d1e-f81f-48fc-8962-7831bc5c2b27\") " pod="openshift-ovn-kubernetes/ovnkube-node-tgq5c" Sep 11 15:54:55.179713 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.179517 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bba741b5-3455-493b-a0df-5e7177b832d8-multus-socket-dir-parent\") pod \"multus-4vhlr\" (UID: \"bba741b5-3455-493b-a0df-5e7177b832d8\") " pod="openshift-multus/multus-4vhlr" Sep 11 15:54:55.179713 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.179546 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bba741b5-3455-493b-a0df-5e7177b832d8-host-var-lib-cni-multus\") pod \"multus-4vhlr\" (UID: \"bba741b5-3455-493b-a0df-5e7177b832d8\") " pod="openshift-multus/multus-4vhlr" Sep 11 15:54:55.179713 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.179567 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/249a0e7e-1da6-422a-8492-1820f6437318-host\") pod \"node-ca-bctd5\" (UID: \"249a0e7e-1da6-422a-8492-1820f6437318\") " pod="openshift-image-registry/node-ca-bctd5" Sep 11 15:54:55.179713 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.179588 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/981d7d1e-f81f-48fc-8962-7831bc5c2b27-host-run-netns\") pod \"ovnkube-node-tgq5c\" (UID: \"981d7d1e-f81f-48fc-8962-7831bc5c2b27\") " pod="openshift-ovn-kubernetes/ovnkube-node-tgq5c" Sep 11 15:54:55.179713 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.179611 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f86rv\" (UniqueName: \"kubernetes.io/projected/981d7d1e-f81f-48fc-8962-7831bc5c2b27-kube-api-access-f86rv\") pod \"ovnkube-node-tgq5c\" (UID: \"981d7d1e-f81f-48fc-8962-7831bc5c2b27\") " pod="openshift-ovn-kubernetes/ovnkube-node-tgq5c" Sep 11 15:54:55.179713 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.179621 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 11 15:54:55.179713 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.179634 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed90592a-177c-4128-b153-b51d18b8f619-tmp-dir\") pod \"node-resolver-sxqj2\" (UID: \"ed90592a-177c-4128-b153-b51d18b8f619\") " pod="openshift-dns/node-resolver-sxqj2" Sep 11 15:54:55.179713 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.179687 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bba741b5-3455-493b-a0df-5e7177b832d8-system-cni-dir\") pod \"multus-4vhlr\" (UID: \"bba741b5-3455-493b-a0df-5e7177b832d8\") " pod="openshift-multus/multus-4vhlr" Sep 11 15:54:55.179713 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.179710 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eab992d2-f32f-41bc-a028-c3fa544c4fec-tmp\") pod \"tuned-sn7js\" (UID: \"eab992d2-f32f-41bc-a028-c3fa544c4fec\") " pod="openshift-cluster-node-tuning-operator/tuned-sn7js" Sep 11 15:54:55.180258 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.179734 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/94ab0396-7020-4ad3-b1b7-63cd05490e84-konnectivity-ca\") pod \"konnectivity-agent-spxb6\" (UID: \"94ab0396-7020-4ad3-b1b7-63cd05490e84\") " pod="kube-system/konnectivity-agent-spxb6" Sep 11 15:54:55.180258 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.179758 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/981d7d1e-f81f-48fc-8962-7831bc5c2b27-var-lib-openvswitch\") pod \"ovnkube-node-tgq5c\" (UID: \"981d7d1e-f81f-48fc-8962-7831bc5c2b27\") " pod="openshift-ovn-kubernetes/ovnkube-node-tgq5c" Sep 11 15:54:55.180258 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.179790 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ed90592a-177c-4128-b153-b51d18b8f619-hosts-file\") pod \"node-resolver-sxqj2\" (UID: \"ed90592a-177c-4128-b153-b51d18b8f619\") " pod="openshift-dns/node-resolver-sxqj2" Sep 11 15:54:55.180258 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.179868 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bba741b5-3455-493b-a0df-5e7177b832d8-host-var-lib-cni-bin\") pod \"multus-4vhlr\" (UID: \"bba741b5-3455-493b-a0df-5e7177b832d8\") " pod="openshift-multus/multus-4vhlr" Sep 11 15:54:55.180258 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.179903 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bba741b5-3455-493b-a0df-5e7177b832d8-hostroot\") pod \"multus-4vhlr\" (UID: \"bba741b5-3455-493b-a0df-5e7177b832d8\") " pod="openshift-multus/multus-4vhlr" Sep 11 15:54:55.180258 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.179930 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/eab992d2-f32f-41bc-a028-c3fa544c4fec-etc-sysctl-d\") pod \"tuned-sn7js\" (UID: \"eab992d2-f32f-41bc-a028-c3fa544c4fec\") " pod="openshift-cluster-node-tuning-operator/tuned-sn7js" Sep 11 15:54:55.180258 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.179958 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/eab992d2-f32f-41bc-a028-c3fa544c4fec-etc-sysctl-conf\") pod \"tuned-sn7js\" (UID: \"eab992d2-f32f-41bc-a028-c3fa544c4fec\") " pod="openshift-cluster-node-tuning-operator/tuned-sn7js" Sep 11 15:54:55.180258 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.180023 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eab992d2-f32f-41bc-a028-c3fa544c4fec-var-lib-kubelet\") pod \"tuned-sn7js\" (UID: \"eab992d2-f32f-41bc-a028-c3fa544c4fec\") " pod="openshift-cluster-node-tuning-operator/tuned-sn7js" Sep 11 15:54:55.180258 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.180048 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/eab992d2-f32f-41bc-a028-c3fa544c4fec-etc-tuned\") pod \"tuned-sn7js\" (UID: \"eab992d2-f32f-41bc-a028-c3fa544c4fec\") " pod="openshift-cluster-node-tuning-operator/tuned-sn7js" Sep 11 15:54:55.180258 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.180062 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-b7ftm" Sep 11 15:54:55.180258 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.180077 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/981d7d1e-f81f-48fc-8962-7831bc5c2b27-systemd-units\") pod \"ovnkube-node-tgq5c\" (UID: \"981d7d1e-f81f-48fc-8962-7831bc5c2b27\") " pod="openshift-ovn-kubernetes/ovnkube-node-tgq5c" Sep 11 15:54:55.180258 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.180101 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/981d7d1e-f81f-48fc-8962-7831bc5c2b27-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tgq5c\" (UID: \"981d7d1e-f81f-48fc-8962-7831bc5c2b27\") " pod="openshift-ovn-kubernetes/ovnkube-node-tgq5c" Sep 11 15:54:55.180951 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.180455 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/94ab0396-7020-4ad3-b1b7-63cd05490e84-agent-certs\") pod \"konnectivity-agent-spxb6\" (UID: \"94ab0396-7020-4ad3-b1b7-63cd05490e84\") " pod="kube-system/konnectivity-agent-spxb6" Sep 11 15:54:55.180951 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.180511 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e05877f1-2ad9-43d5-8523-4749d6b08782-socket-dir\") pod \"aws-ebs-csi-driver-node-xkvw8\" (UID: \"e05877f1-2ad9-43d5-8523-4749d6b08782\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xkvw8" Sep 11 15:54:55.180951 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.180565 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e05877f1-2ad9-43d5-8523-4749d6b08782-sys-fs\") pod \"aws-ebs-csi-driver-node-xkvw8\" (UID: \"e05877f1-2ad9-43d5-8523-4749d6b08782\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xkvw8" Sep 11 15:54:55.180951 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.180589 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bba741b5-3455-493b-a0df-5e7177b832d8-multus-cni-dir\") pod \"multus-4vhlr\" (UID: \"bba741b5-3455-493b-a0df-5e7177b832d8\") " pod="openshift-multus/multus-4vhlr" Sep 11 15:54:55.180951 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.180625 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bba741b5-3455-493b-a0df-5e7177b832d8-multus-daemon-config\") pod \"multus-4vhlr\" (UID: \"bba741b5-3455-493b-a0df-5e7177b832d8\") " pod="openshift-multus/multus-4vhlr" Sep 11 15:54:55.180951 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.180647 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/eab992d2-f32f-41bc-a028-c3fa544c4fec-etc-systemd\") pod \"tuned-sn7js\" (UID: \"eab992d2-f32f-41bc-a028-c3fa544c4fec\") " pod="openshift-cluster-node-tuning-operator/tuned-sn7js" Sep 11 15:54:55.180951 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.180679 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5d8bj\" (UniqueName: \"kubernetes.io/projected/eab992d2-f32f-41bc-a028-c3fa544c4fec-kube-api-access-5d8bj\") pod \"tuned-sn7js\" (UID: \"eab992d2-f32f-41bc-a028-c3fa544c4fec\") " pod="openshift-cluster-node-tuning-operator/tuned-sn7js" Sep 11 15:54:55.180951 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.180712 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/981d7d1e-f81f-48fc-8962-7831bc5c2b27-host-cni-netd\") pod \"ovnkube-node-tgq5c\" (UID: \"981d7d1e-f81f-48fc-8962-7831bc5c2b27\") " pod="openshift-ovn-kubernetes/ovnkube-node-tgq5c" Sep 11 15:54:55.180951 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.180744 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/331632d0-fa59-4462-805e-bce0e18b95bc-metrics-certs\") pod \"network-metrics-daemon-plkhx\" (UID: \"331632d0-fa59-4462-805e-bce0e18b95bc\") " pod="openshift-multus/network-metrics-daemon-plkhx" Sep 11 15:54:55.180951 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.180766 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bba741b5-3455-493b-a0df-5e7177b832d8-etc-kubernetes\") pod \"multus-4vhlr\" (UID: \"bba741b5-3455-493b-a0df-5e7177b832d8\") " pod="openshift-multus/multus-4vhlr" Sep 11 15:54:55.180951 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.180794 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/981d7d1e-f81f-48fc-8962-7831bc5c2b27-host-kubelet\") pod \"ovnkube-node-tgq5c\" (UID: \"981d7d1e-f81f-48fc-8962-7831bc5c2b27\") " pod="openshift-ovn-kubernetes/ovnkube-node-tgq5c" Sep 11 15:54:55.180951 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.180816 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/981d7d1e-f81f-48fc-8962-7831bc5c2b27-host-slash\") pod \"ovnkube-node-tgq5c\" (UID: \"981d7d1e-f81f-48fc-8962-7831bc5c2b27\") " pod="openshift-ovn-kubernetes/ovnkube-node-tgq5c" Sep 11 15:54:55.180951 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.180873 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/981d7d1e-f81f-48fc-8962-7831bc5c2b27-run-systemd\") pod \"ovnkube-node-tgq5c\" (UID: \"981d7d1e-f81f-48fc-8962-7831bc5c2b27\") " pod="openshift-ovn-kubernetes/ovnkube-node-tgq5c" Sep 11 15:54:55.180951 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.180933 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/981d7d1e-f81f-48fc-8962-7831bc5c2b27-env-overrides\") pod \"ovnkube-node-tgq5c\" (UID: \"981d7d1e-f81f-48fc-8962-7831bc5c2b27\") " pod="openshift-ovn-kubernetes/ovnkube-node-tgq5c" Sep 11 15:54:55.181699 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.180965 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/981d7d1e-f81f-48fc-8962-7831bc5c2b27-ovnkube-script-lib\") pod \"ovnkube-node-tgq5c\" (UID: \"981d7d1e-f81f-48fc-8962-7831bc5c2b27\") " pod="openshift-ovn-kubernetes/ovnkube-node-tgq5c" Sep 11 15:54:55.181699 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.181009 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bba741b5-3455-493b-a0df-5e7177b832d8-cni-binary-copy\") pod \"multus-4vhlr\" (UID: \"bba741b5-3455-493b-a0df-5e7177b832d8\") " pod="openshift-multus/multus-4vhlr" Sep 11 15:54:55.181699 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.181093 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/eab992d2-f32f-41bc-a028-c3fa544c4fec-run\") pod \"tuned-sn7js\" (UID: \"eab992d2-f32f-41bc-a028-c3fa544c4fec\") " pod="openshift-cluster-node-tuning-operator/tuned-sn7js" Sep 11 15:54:55.181699 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.181130 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e05877f1-2ad9-43d5-8523-4749d6b08782-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xkvw8\" (UID: \"e05877f1-2ad9-43d5-8523-4749d6b08782\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xkvw8" Sep 11 15:54:55.181699 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.181177 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6hzg\" (UniqueName: \"kubernetes.io/projected/e05877f1-2ad9-43d5-8523-4749d6b08782-kube-api-access-w6hzg\") pod \"aws-ebs-csi-driver-node-xkvw8\" (UID: \"e05877f1-2ad9-43d5-8523-4749d6b08782\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xkvw8" Sep 11 15:54:55.181699 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.181231 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bba741b5-3455-493b-a0df-5e7177b832d8-multus-conf-dir\") pod \"multus-4vhlr\" (UID: \"bba741b5-3455-493b-a0df-5e7177b832d8\") " pod="openshift-multus/multus-4vhlr" Sep 11 15:54:55.181699 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.181271 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grtpf\" (UniqueName: \"kubernetes.io/projected/bba741b5-3455-493b-a0df-5e7177b832d8-kube-api-access-grtpf\") pod \"multus-4vhlr\" (UID: \"bba741b5-3455-493b-a0df-5e7177b832d8\") " pod="openshift-multus/multus-4vhlr" Sep 11 15:54:55.181699 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.181347 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/249a0e7e-1da6-422a-8492-1820f6437318-serviceca\") pod \"node-ca-bctd5\" (UID: \"249a0e7e-1da6-422a-8492-1820f6437318\") " pod="openshift-image-registry/node-ca-bctd5" Sep 11 15:54:55.181699 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.181374 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eab992d2-f32f-41bc-a028-c3fa544c4fec-host\") pod \"tuned-sn7js\" (UID: \"eab992d2-f32f-41bc-a028-c3fa544c4fec\") " pod="openshift-cluster-node-tuning-operator/tuned-sn7js" Sep 11 15:54:55.181699 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.181399 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/981d7d1e-f81f-48fc-8962-7831bc5c2b27-log-socket\") pod \"ovnkube-node-tgq5c\" (UID: \"981d7d1e-f81f-48fc-8962-7831bc5c2b27\") " pod="openshift-ovn-kubernetes/ovnkube-node-tgq5c" Sep 11 15:54:55.181699 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.181424 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/981d7d1e-f81f-48fc-8962-7831bc5c2b27-host-run-ovn-kubernetes\") pod \"ovnkube-node-tgq5c\" (UID: \"981d7d1e-f81f-48fc-8962-7831bc5c2b27\") " pod="openshift-ovn-kubernetes/ovnkube-node-tgq5c" Sep 11 15:54:55.181699 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.181452 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/981d7d1e-f81f-48fc-8962-7831bc5c2b27-host-cni-bin\") pod \"ovnkube-node-tgq5c\" (UID: \"981d7d1e-f81f-48fc-8962-7831bc5c2b27\") " pod="openshift-ovn-kubernetes/ovnkube-node-tgq5c" Sep 11 15:54:55.181699 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.181477 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bba741b5-3455-493b-a0df-5e7177b832d8-host-run-netns\") pod \"multus-4vhlr\" (UID: \"bba741b5-3455-493b-a0df-5e7177b832d8\") " pod="openshift-multus/multus-4vhlr" Sep 11 15:54:55.181699 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.181517 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/981d7d1e-f81f-48fc-8962-7831bc5c2b27-run-openvswitch\") pod \"ovnkube-node-tgq5c\" (UID: \"981d7d1e-f81f-48fc-8962-7831bc5c2b27\") " pod="openshift-ovn-kubernetes/ovnkube-node-tgq5c" Sep 11 15:54:55.181699 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.181541 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e05877f1-2ad9-43d5-8523-4749d6b08782-etc-selinux\") pod \"aws-ebs-csi-driver-node-xkvw8\" (UID: \"e05877f1-2ad9-43d5-8523-4749d6b08782\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xkvw8" Sep 11 15:54:55.181699 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.181566 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bba741b5-3455-493b-a0df-5e7177b832d8-host-run-k8s-cni-cncf-io\") pod \"multus-4vhlr\" (UID: \"bba741b5-3455-493b-a0df-5e7177b832d8\") " pod="openshift-multus/multus-4vhlr" Sep 11 15:54:55.182555 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.181607 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bba741b5-3455-493b-a0df-5e7177b832d8-host-var-lib-kubelet\") pod \"multus-4vhlr\" (UID: \"bba741b5-3455-493b-a0df-5e7177b832d8\") " pod="openshift-multus/multus-4vhlr" Sep 11 15:54:55.182555 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.181633 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bba741b5-3455-493b-a0df-5e7177b832d8-host-run-multus-certs\") pod \"multus-4vhlr\" (UID: \"bba741b5-3455-493b-a0df-5e7177b832d8\") " pod="openshift-multus/multus-4vhlr" Sep 11 15:54:55.182555 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.181688 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eab992d2-f32f-41bc-a028-c3fa544c4fec-etc-kubernetes\") pod \"tuned-sn7js\" (UID: \"eab992d2-f32f-41bc-a028-c3fa544c4fec\") " pod="openshift-cluster-node-tuning-operator/tuned-sn7js" Sep 11 15:54:55.182555 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.181748 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjvb6\" (UniqueName: \"kubernetes.io/projected/331632d0-fa59-4462-805e-bce0e18b95bc-kube-api-access-zjvb6\") pod \"network-metrics-daemon-plkhx\" (UID: \"331632d0-fa59-4462-805e-bce0e18b95bc\") " pod="openshift-multus/network-metrics-daemon-plkhx" Sep 11 15:54:55.182555 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.181781 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e05877f1-2ad9-43d5-8523-4749d6b08782-device-dir\") pod \"aws-ebs-csi-driver-node-xkvw8\" (UID: \"e05877f1-2ad9-43d5-8523-4749d6b08782\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xkvw8" Sep 11 15:54:55.182555 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.181803 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bba741b5-3455-493b-a0df-5e7177b832d8-cnibin\") pod \"multus-4vhlr\" (UID: \"bba741b5-3455-493b-a0df-5e7177b832d8\") " pod="openshift-multus/multus-4vhlr" Sep 11 15:54:55.182555 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.181865 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 11 15:54:55.182555 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.181935 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 11 15:54:55.182555 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.182081 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-b549p\"" Sep 11 15:54:55.182555 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.182189 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nzcjl" Sep 11 15:54:55.182555 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:54:55.182247 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nzcjl" podUID="f6c05fce-e71f-4a9c-b219-a77b1df84ac8" Sep 11 15:54:55.184472 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.184374 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-95zvh" Sep 11 15:54:55.184472 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:54:55.184434 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-95zvh" podUID="1c358fa6-a923-4589-953f-bd49ddd4235c" Sep 11 15:54:55.204585 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.204480 2566 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-nc2wv" Sep 11 15:54:55.214093 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.214071 2566 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-nc2wv" Sep 11 15:54:55.238308 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.238226 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-96.ec2.internal" event={"ID":"bd507eff9b3716da03d385309d794351","Type":"ContainerStarted","Data":"4a6142c1b763d85cdf255e33d297b5513815136f75cf57bcb06a498abd77c983"} Sep 11 15:54:55.240326 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.240237 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-96.ec2.internal" event={"ID":"e46b070927ba07e09c7fdfca6f7f4f91","Type":"ContainerStarted","Data":"e509d109296443bf7ffdf77431854bf774d00967a5c087f4b5489f098da45170"} Sep 11 15:54:55.247181 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.247162 2566 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 15:54:55.273424 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.273346 2566 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 11 15:54:55.282747 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.282718 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bba741b5-3455-493b-a0df-5e7177b832d8-host-var-lib-kubelet\") pod \"multus-4vhlr\" (UID: \"bba741b5-3455-493b-a0df-5e7177b832d8\") " pod="openshift-multus/multus-4vhlr" Sep 11 15:54:55.282871 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.282760 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bba741b5-3455-493b-a0df-5e7177b832d8-host-run-multus-certs\") pod \"multus-4vhlr\" (UID: \"bba741b5-3455-493b-a0df-5e7177b832d8\") " pod="openshift-multus/multus-4vhlr" Sep 11 15:54:55.282871 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.282785 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eab992d2-f32f-41bc-a028-c3fa544c4fec-etc-kubernetes\") pod \"tuned-sn7js\" (UID: \"eab992d2-f32f-41bc-a028-c3fa544c4fec\") " pod="openshift-cluster-node-tuning-operator/tuned-sn7js" Sep 11 15:54:55.282871 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.282813 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fe6dccde-412f-483d-a914-5d46e9922b69-cnibin\") pod \"multus-additional-cni-plugins-b7ftm\" (UID: \"fe6dccde-412f-483d-a914-5d46e9922b69\") " pod="openshift-multus/multus-additional-cni-plugins-b7ftm" Sep 11 15:54:55.282871 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.282842 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fe6dccde-412f-483d-a914-5d46e9922b69-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-b7ftm\" (UID: \"fe6dccde-412f-483d-a914-5d46e9922b69\") " pod="openshift-multus/multus-additional-cni-plugins-b7ftm" Sep 11 15:54:55.282871 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.282868 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjvb6\" (UniqueName: \"kubernetes.io/projected/331632d0-fa59-4462-805e-bce0e18b95bc-kube-api-access-zjvb6\") pod \"network-metrics-daemon-plkhx\" (UID: \"331632d0-fa59-4462-805e-bce0e18b95bc\") " pod="openshift-multus/network-metrics-daemon-plkhx" Sep 11 15:54:55.283146 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.282893 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e05877f1-2ad9-43d5-8523-4749d6b08782-device-dir\") pod \"aws-ebs-csi-driver-node-xkvw8\" (UID: \"e05877f1-2ad9-43d5-8523-4749d6b08782\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xkvw8" Sep 11 15:54:55.283146 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.282915 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bba741b5-3455-493b-a0df-5e7177b832d8-cnibin\") pod \"multus-4vhlr\" (UID: \"bba741b5-3455-493b-a0df-5e7177b832d8\") " pod="openshift-multus/multus-4vhlr" Sep 11 15:54:55.283146 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.282937 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eab992d2-f32f-41bc-a028-c3fa544c4fec-lib-modules\") pod \"tuned-sn7js\" (UID: \"eab992d2-f32f-41bc-a028-c3fa544c4fec\") " pod="openshift-cluster-node-tuning-operator/tuned-sn7js" Sep 11 15:54:55.283146 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.282959 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/981d7d1e-f81f-48fc-8962-7831bc5c2b27-ovn-node-metrics-cert\") pod \"ovnkube-node-tgq5c\" (UID: \"981d7d1e-f81f-48fc-8962-7831bc5c2b27\") " pod="openshift-ovn-kubernetes/ovnkube-node-tgq5c" Sep 11 15:54:55.283146 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.283003 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcpmk\" (UniqueName: \"kubernetes.io/projected/ed90592a-177c-4128-b153-b51d18b8f619-kube-api-access-jcpmk\") pod \"node-resolver-sxqj2\" (UID: \"ed90592a-177c-4128-b153-b51d18b8f619\") " pod="openshift-dns/node-resolver-sxqj2" Sep 11 15:54:55.283146 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.283029 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xknqc\" (UniqueName: \"kubernetes.io/projected/249a0e7e-1da6-422a-8492-1820f6437318-kube-api-access-xknqc\") pod \"node-ca-bctd5\" (UID: \"249a0e7e-1da6-422a-8492-1820f6437318\") " pod="openshift-image-registry/node-ca-bctd5" Sep 11 15:54:55.283146 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.283049 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/eab992d2-f32f-41bc-a028-c3fa544c4fec-etc-modprobe-d\") pod \"tuned-sn7js\" (UID: \"eab992d2-f32f-41bc-a028-c3fa544c4fec\") " pod="openshift-cluster-node-tuning-operator/tuned-sn7js" Sep 11 15:54:55.283146 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.283062 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/eab992d2-f32f-41bc-a028-c3fa544c4fec-etc-sysconfig\") pod \"tuned-sn7js\" (UID: \"eab992d2-f32f-41bc-a028-c3fa544c4fec\") " pod="openshift-cluster-node-tuning-operator/tuned-sn7js" Sep 11 15:54:55.283146 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.283079 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/981d7d1e-f81f-48fc-8962-7831bc5c2b27-etc-openvswitch\") pod \"ovnkube-node-tgq5c\" (UID: \"981d7d1e-f81f-48fc-8962-7831bc5c2b27\") " pod="openshift-ovn-kubernetes/ovnkube-node-tgq5c" Sep 11 15:54:55.283146 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.283099 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/981d7d1e-f81f-48fc-8962-7831bc5c2b27-run-ovn\") pod \"ovnkube-node-tgq5c\" (UID: \"981d7d1e-f81f-48fc-8962-7831bc5c2b27\") " pod="openshift-ovn-kubernetes/ovnkube-node-tgq5c" Sep 11 15:54:55.283146 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.283122 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/981d7d1e-f81f-48fc-8962-7831bc5c2b27-ovnkube-config\") pod \"ovnkube-node-tgq5c\" (UID: \"981d7d1e-f81f-48fc-8962-7831bc5c2b27\") " pod="openshift-ovn-kubernetes/ovnkube-node-tgq5c" Sep 11 15:54:55.283146 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.283146 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e05877f1-2ad9-43d5-8523-4749d6b08782-registration-dir\") pod \"aws-ebs-csi-driver-node-xkvw8\" (UID: \"e05877f1-2ad9-43d5-8523-4749d6b08782\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xkvw8" Sep 11 15:54:55.283767 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.283168 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eab992d2-f32f-41bc-a028-c3fa544c4fec-sys\") pod \"tuned-sn7js\" (UID: \"eab992d2-f32f-41bc-a028-c3fa544c4fec\") " pod="openshift-cluster-node-tuning-operator/tuned-sn7js" Sep 11 15:54:55.283767 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.283189 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/981d7d1e-f81f-48fc-8962-7831bc5c2b27-node-log\") pod \"ovnkube-node-tgq5c\" (UID: \"981d7d1e-f81f-48fc-8962-7831bc5c2b27\") " pod="openshift-ovn-kubernetes/ovnkube-node-tgq5c" Sep 11 15:54:55.283767 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.283214 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gwrb\" (UniqueName: \"kubernetes.io/projected/b02518a9-32bc-4627-97c8-4535c8333875-kube-api-access-6gwrb\") pod \"iptables-alerter-5hlvj\" (UID: \"b02518a9-32bc-4627-97c8-4535c8333875\") " pod="openshift-network-operator/iptables-alerter-5hlvj" Sep 11 15:54:55.283767 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.283238 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9g7p\" (UniqueName: \"kubernetes.io/projected/f6c05fce-e71f-4a9c-b219-a77b1df84ac8-kube-api-access-h9g7p\") pod \"network-check-target-nzcjl\" (UID: \"f6c05fce-e71f-4a9c-b219-a77b1df84ac8\") " pod="openshift-network-diagnostics/network-check-target-nzcjl" Sep 11 15:54:55.283767 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.283261 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bba741b5-3455-493b-a0df-5e7177b832d8-multus-socket-dir-parent\") pod \"multus-4vhlr\" (UID: \"bba741b5-3455-493b-a0df-5e7177b832d8\") " pod="openshift-multus/multus-4vhlr" Sep 11 15:54:55.283767 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.283284 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bba741b5-3455-493b-a0df-5e7177b832d8-host-var-lib-cni-multus\") pod \"multus-4vhlr\" (UID: \"bba741b5-3455-493b-a0df-5e7177b832d8\") " pod="openshift-multus/multus-4vhlr" Sep 11 15:54:55.283767 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.283306 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/249a0e7e-1da6-422a-8492-1820f6437318-host\") pod \"node-ca-bctd5\" (UID: \"249a0e7e-1da6-422a-8492-1820f6437318\") " pod="openshift-image-registry/node-ca-bctd5" Sep 11 15:54:55.283767 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.283327 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/981d7d1e-f81f-48fc-8962-7831bc5c2b27-host-run-netns\") pod \"ovnkube-node-tgq5c\" (UID: \"981d7d1e-f81f-48fc-8962-7831bc5c2b27\") " pod="openshift-ovn-kubernetes/ovnkube-node-tgq5c" Sep 11 15:54:55.283767 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.283350 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f86rv\" (UniqueName: \"kubernetes.io/projected/981d7d1e-f81f-48fc-8962-7831bc5c2b27-kube-api-access-f86rv\") pod \"ovnkube-node-tgq5c\" (UID: \"981d7d1e-f81f-48fc-8962-7831bc5c2b27\") " pod="openshift-ovn-kubernetes/ovnkube-node-tgq5c" Sep 11 15:54:55.283767 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.283372 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fe6dccde-412f-483d-a914-5d46e9922b69-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-b7ftm\" (UID: \"fe6dccde-412f-483d-a914-5d46e9922b69\") " pod="openshift-multus/multus-additional-cni-plugins-b7ftm" Sep 11 15:54:55.283767 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.283394 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed90592a-177c-4128-b153-b51d18b8f619-tmp-dir\") pod \"node-resolver-sxqj2\" (UID: \"ed90592a-177c-4128-b153-b51d18b8f619\") " pod="openshift-dns/node-resolver-sxqj2" Sep 11 15:54:55.283767 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.283417 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bba741b5-3455-493b-a0df-5e7177b832d8-system-cni-dir\") pod \"multus-4vhlr\" (UID: \"bba741b5-3455-493b-a0df-5e7177b832d8\") " pod="openshift-multus/multus-4vhlr" Sep 11 15:54:55.283767 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.283446 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eab992d2-f32f-41bc-a028-c3fa544c4fec-tmp\") pod \"tuned-sn7js\" (UID: \"eab992d2-f32f-41bc-a028-c3fa544c4fec\") " pod="openshift-cluster-node-tuning-operator/tuned-sn7js" Sep 11 15:54:55.283767 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.283469 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96sg9\" (UniqueName: \"kubernetes.io/projected/fe6dccde-412f-483d-a914-5d46e9922b69-kube-api-access-96sg9\") pod \"multus-additional-cni-plugins-b7ftm\" (UID: \"fe6dccde-412f-483d-a914-5d46e9922b69\") " pod="openshift-multus/multus-additional-cni-plugins-b7ftm" Sep 11 15:54:55.283767 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.283496 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1c358fa6-a923-4589-953f-bd49ddd4235c-kubelet-config\") pod \"global-pull-secret-syncer-95zvh\" (UID: \"1c358fa6-a923-4589-953f-bd49ddd4235c\") " pod="kube-system/global-pull-secret-syncer-95zvh" Sep 11 15:54:55.283767 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.283535 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/94ab0396-7020-4ad3-b1b7-63cd05490e84-konnectivity-ca\") pod \"konnectivity-agent-spxb6\" (UID: \"94ab0396-7020-4ad3-b1b7-63cd05490e84\") " pod="kube-system/konnectivity-agent-spxb6" Sep 11 15:54:55.283767 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.283562 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fe6dccde-412f-483d-a914-5d46e9922b69-tuning-conf-dir\") pod \"multus-additional-cni-plugins-b7ftm\" (UID: \"fe6dccde-412f-483d-a914-5d46e9922b69\") " pod="openshift-multus/multus-additional-cni-plugins-b7ftm" Sep 11 15:54:55.284608 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.283585 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/981d7d1e-f81f-48fc-8962-7831bc5c2b27-var-lib-openvswitch\") pod \"ovnkube-node-tgq5c\" (UID: \"981d7d1e-f81f-48fc-8962-7831bc5c2b27\") " pod="openshift-ovn-kubernetes/ovnkube-node-tgq5c" Sep 11 15:54:55.284608 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.283610 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ed90592a-177c-4128-b153-b51d18b8f619-hosts-file\") pod \"node-resolver-sxqj2\" (UID: \"ed90592a-177c-4128-b153-b51d18b8f619\") " pod="openshift-dns/node-resolver-sxqj2" Sep 11 15:54:55.284608 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.283640 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bba741b5-3455-493b-a0df-5e7177b832d8-host-var-lib-cni-bin\") pod \"multus-4vhlr\" (UID: \"bba741b5-3455-493b-a0df-5e7177b832d8\") " pod="openshift-multus/multus-4vhlr" Sep 11 15:54:55.284608 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.283664 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bba741b5-3455-493b-a0df-5e7177b832d8-hostroot\") pod \"multus-4vhlr\" (UID: \"bba741b5-3455-493b-a0df-5e7177b832d8\") " pod="openshift-multus/multus-4vhlr" Sep 11 15:54:55.284608 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.283687 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/eab992d2-f32f-41bc-a028-c3fa544c4fec-etc-sysctl-d\") pod \"tuned-sn7js\" (UID: \"eab992d2-f32f-41bc-a028-c3fa544c4fec\") " pod="openshift-cluster-node-tuning-operator/tuned-sn7js" Sep 11 15:54:55.284608 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.283709 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b02518a9-32bc-4627-97c8-4535c8333875-host-slash\") pod \"iptables-alerter-5hlvj\" (UID: \"b02518a9-32bc-4627-97c8-4535c8333875\") " pod="openshift-network-operator/iptables-alerter-5hlvj" Sep 11 15:54:55.284608 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.283733 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/eab992d2-f32f-41bc-a028-c3fa544c4fec-etc-sysctl-conf\") pod \"tuned-sn7js\" (UID: \"eab992d2-f32f-41bc-a028-c3fa544c4fec\") " pod="openshift-cluster-node-tuning-operator/tuned-sn7js" Sep 11 15:54:55.284608 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.283756 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eab992d2-f32f-41bc-a028-c3fa544c4fec-var-lib-kubelet\") pod \"tuned-sn7js\" (UID: \"eab992d2-f32f-41bc-a028-c3fa544c4fec\") " pod="openshift-cluster-node-tuning-operator/tuned-sn7js" Sep 11 15:54:55.284608 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.283779 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/eab992d2-f32f-41bc-a028-c3fa544c4fec-etc-tuned\") pod \"tuned-sn7js\" (UID: \"eab992d2-f32f-41bc-a028-c3fa544c4fec\") " pod="openshift-cluster-node-tuning-operator/tuned-sn7js" Sep 11 15:54:55.284608 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.283802 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/981d7d1e-f81f-48fc-8962-7831bc5c2b27-systemd-units\") pod \"ovnkube-node-tgq5c\" (UID: \"981d7d1e-f81f-48fc-8962-7831bc5c2b27\") " pod="openshift-ovn-kubernetes/ovnkube-node-tgq5c" Sep 11 15:54:55.284608 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.283828 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/981d7d1e-f81f-48fc-8962-7831bc5c2b27-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tgq5c\" (UID: \"981d7d1e-f81f-48fc-8962-7831bc5c2b27\") " pod="openshift-ovn-kubernetes/ovnkube-node-tgq5c" Sep 11 15:54:55.284608 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.283853 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/94ab0396-7020-4ad3-b1b7-63cd05490e84-agent-certs\") pod \"konnectivity-agent-spxb6\" (UID: \"94ab0396-7020-4ad3-b1b7-63cd05490e84\") " pod="kube-system/konnectivity-agent-spxb6" Sep 11 15:54:55.284608 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.283878 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e05877f1-2ad9-43d5-8523-4749d6b08782-socket-dir\") pod \"aws-ebs-csi-driver-node-xkvw8\" (UID: \"e05877f1-2ad9-43d5-8523-4749d6b08782\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xkvw8" Sep 11 15:54:55.284608 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.283901 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e05877f1-2ad9-43d5-8523-4749d6b08782-sys-fs\") pod \"aws-ebs-csi-driver-node-xkvw8\" (UID: \"e05877f1-2ad9-43d5-8523-4749d6b08782\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xkvw8" Sep 11 15:54:55.284608 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.283929 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bba741b5-3455-493b-a0df-5e7177b832d8-multus-cni-dir\") pod \"multus-4vhlr\" (UID: \"bba741b5-3455-493b-a0df-5e7177b832d8\") " pod="openshift-multus/multus-4vhlr" Sep 11 15:54:55.284608 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.283954 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bba741b5-3455-493b-a0df-5e7177b832d8-multus-daemon-config\") pod \"multus-4vhlr\" (UID: \"bba741b5-3455-493b-a0df-5e7177b832d8\") " pod="openshift-multus/multus-4vhlr" Sep 11 15:54:55.284608 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.283995 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/eab992d2-f32f-41bc-a028-c3fa544c4fec-etc-systemd\") pod \"tuned-sn7js\" (UID: \"eab992d2-f32f-41bc-a028-c3fa544c4fec\") " pod="openshift-cluster-node-tuning-operator/tuned-sn7js" Sep 11 15:54:55.285400 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.284019 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5d8bj\" (UniqueName: \"kubernetes.io/projected/eab992d2-f32f-41bc-a028-c3fa544c4fec-kube-api-access-5d8bj\") pod \"tuned-sn7js\" (UID: \"eab992d2-f32f-41bc-a028-c3fa544c4fec\") " pod="openshift-cluster-node-tuning-operator/tuned-sn7js" Sep 11 15:54:55.285400 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.284050 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/981d7d1e-f81f-48fc-8962-7831bc5c2b27-host-cni-netd\") pod \"ovnkube-node-tgq5c\" (UID: \"981d7d1e-f81f-48fc-8962-7831bc5c2b27\") " pod="openshift-ovn-kubernetes/ovnkube-node-tgq5c" Sep 11 15:54:55.285400 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.284076 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/331632d0-fa59-4462-805e-bce0e18b95bc-metrics-certs\") pod \"network-metrics-daemon-plkhx\" (UID: \"331632d0-fa59-4462-805e-bce0e18b95bc\") " pod="openshift-multus/network-metrics-daemon-plkhx" Sep 11 15:54:55.285400 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.284101 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bba741b5-3455-493b-a0df-5e7177b832d8-etc-kubernetes\") pod \"multus-4vhlr\" (UID: \"bba741b5-3455-493b-a0df-5e7177b832d8\") " pod="openshift-multus/multus-4vhlr" Sep 11 15:54:55.285400 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.284126 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/981d7d1e-f81f-48fc-8962-7831bc5c2b27-host-kubelet\") pod \"ovnkube-node-tgq5c\" (UID: \"981d7d1e-f81f-48fc-8962-7831bc5c2b27\") " pod="openshift-ovn-kubernetes/ovnkube-node-tgq5c" Sep 11 15:54:55.285400 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.284150 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/981d7d1e-f81f-48fc-8962-7831bc5c2b27-host-slash\") pod \"ovnkube-node-tgq5c\" (UID: \"981d7d1e-f81f-48fc-8962-7831bc5c2b27\") " pod="openshift-ovn-kubernetes/ovnkube-node-tgq5c" Sep 11 15:54:55.285400 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.284172 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/981d7d1e-f81f-48fc-8962-7831bc5c2b27-run-systemd\") pod \"ovnkube-node-tgq5c\" (UID: \"981d7d1e-f81f-48fc-8962-7831bc5c2b27\") " pod="openshift-ovn-kubernetes/ovnkube-node-tgq5c" Sep 11 15:54:55.285400 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.284196 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/981d7d1e-f81f-48fc-8962-7831bc5c2b27-env-overrides\") pod \"ovnkube-node-tgq5c\" (UID: \"981d7d1e-f81f-48fc-8962-7831bc5c2b27\") " pod="openshift-ovn-kubernetes/ovnkube-node-tgq5c" Sep 11 15:54:55.285400 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.284226 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/981d7d1e-f81f-48fc-8962-7831bc5c2b27-ovnkube-script-lib\") pod \"ovnkube-node-tgq5c\" (UID: \"981d7d1e-f81f-48fc-8962-7831bc5c2b27\") " pod="openshift-ovn-kubernetes/ovnkube-node-tgq5c" Sep 11 15:54:55.285400 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.284251 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1c358fa6-a923-4589-953f-bd49ddd4235c-dbus\") pod \"global-pull-secret-syncer-95zvh\" (UID: \"1c358fa6-a923-4589-953f-bd49ddd4235c\") " pod="kube-system/global-pull-secret-syncer-95zvh" Sep 11 15:54:55.285400 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.284298 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bba741b5-3455-493b-a0df-5e7177b832d8-cni-binary-copy\") pod \"multus-4vhlr\" (UID: \"bba741b5-3455-493b-a0df-5e7177b832d8\") " pod="openshift-multus/multus-4vhlr" Sep 11 15:54:55.285400 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.284321 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/eab992d2-f32f-41bc-a028-c3fa544c4fec-run\") pod \"tuned-sn7js\" (UID: \"eab992d2-f32f-41bc-a028-c3fa544c4fec\") " pod="openshift-cluster-node-tuning-operator/tuned-sn7js" Sep 11 15:54:55.285400 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.284345 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1c358fa6-a923-4589-953f-bd49ddd4235c-original-pull-secret\") pod \"global-pull-secret-syncer-95zvh\" (UID: \"1c358fa6-a923-4589-953f-bd49ddd4235c\") " pod="kube-system/global-pull-secret-syncer-95zvh" Sep 11 15:54:55.285400 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.284372 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e05877f1-2ad9-43d5-8523-4749d6b08782-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xkvw8\" (UID: \"e05877f1-2ad9-43d5-8523-4749d6b08782\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xkvw8" Sep 11 15:54:55.285400 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.284398 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6hzg\" (UniqueName: \"kubernetes.io/projected/e05877f1-2ad9-43d5-8523-4749d6b08782-kube-api-access-w6hzg\") pod \"aws-ebs-csi-driver-node-xkvw8\" (UID: \"e05877f1-2ad9-43d5-8523-4749d6b08782\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xkvw8" Sep 11 15:54:55.285400 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.284429 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fe6dccde-412f-483d-a914-5d46e9922b69-system-cni-dir\") pod \"multus-additional-cni-plugins-b7ftm\" (UID: \"fe6dccde-412f-483d-a914-5d46e9922b69\") " pod="openshift-multus/multus-additional-cni-plugins-b7ftm" Sep 11 15:54:55.285400 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.284481 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bba741b5-3455-493b-a0df-5e7177b832d8-multus-conf-dir\") pod \"multus-4vhlr\" (UID: \"bba741b5-3455-493b-a0df-5e7177b832d8\") " pod="openshift-multus/multus-4vhlr" Sep 11 15:54:55.286252 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.284509 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grtpf\" (UniqueName: \"kubernetes.io/projected/bba741b5-3455-493b-a0df-5e7177b832d8-kube-api-access-grtpf\") pod \"multus-4vhlr\" (UID: \"bba741b5-3455-493b-a0df-5e7177b832d8\") " pod="openshift-multus/multus-4vhlr" Sep 11 15:54:55.286252 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.284534 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/249a0e7e-1da6-422a-8492-1820f6437318-serviceca\") pod \"node-ca-bctd5\" (UID: \"249a0e7e-1da6-422a-8492-1820f6437318\") " pod="openshift-image-registry/node-ca-bctd5" Sep 11 15:54:55.286252 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.284599 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/981d7d1e-f81f-48fc-8962-7831bc5c2b27-etc-openvswitch\") pod \"ovnkube-node-tgq5c\" (UID: \"981d7d1e-f81f-48fc-8962-7831bc5c2b27\") " pod="openshift-ovn-kubernetes/ovnkube-node-tgq5c" Sep 11 15:54:55.286252 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.284654 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eab992d2-f32f-41bc-a028-c3fa544c4fec-host\") pod \"tuned-sn7js\" (UID: \"eab992d2-f32f-41bc-a028-c3fa544c4fec\") " pod="openshift-cluster-node-tuning-operator/tuned-sn7js" Sep 11 15:54:55.286252 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.284684 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/981d7d1e-f81f-48fc-8962-7831bc5c2b27-log-socket\") pod \"ovnkube-node-tgq5c\" (UID: \"981d7d1e-f81f-48fc-8962-7831bc5c2b27\") " pod="openshift-ovn-kubernetes/ovnkube-node-tgq5c" Sep 11 15:54:55.286252 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.284737 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/981d7d1e-f81f-48fc-8962-7831bc5c2b27-host-run-ovn-kubernetes\") pod \"ovnkube-node-tgq5c\" (UID: \"981d7d1e-f81f-48fc-8962-7831bc5c2b27\") " pod="openshift-ovn-kubernetes/ovnkube-node-tgq5c" Sep 11 15:54:55.286252 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.284764 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/981d7d1e-f81f-48fc-8962-7831bc5c2b27-host-cni-bin\") pod \"ovnkube-node-tgq5c\" (UID: \"981d7d1e-f81f-48fc-8962-7831bc5c2b27\") " pod="openshift-ovn-kubernetes/ovnkube-node-tgq5c" Sep 11 15:54:55.286252 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.284790 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bba741b5-3455-493b-a0df-5e7177b832d8-host-run-netns\") pod \"multus-4vhlr\" (UID: \"bba741b5-3455-493b-a0df-5e7177b832d8\") " pod="openshift-multus/multus-4vhlr" Sep 11 15:54:55.286252 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.284816 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/981d7d1e-f81f-48fc-8962-7831bc5c2b27-run-openvswitch\") pod \"ovnkube-node-tgq5c\" (UID: \"981d7d1e-f81f-48fc-8962-7831bc5c2b27\") " pod="openshift-ovn-kubernetes/ovnkube-node-tgq5c" Sep 11 15:54:55.286252 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.284860 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b02518a9-32bc-4627-97c8-4535c8333875-iptables-alerter-script\") pod \"iptables-alerter-5hlvj\" (UID: \"b02518a9-32bc-4627-97c8-4535c8333875\") " pod="openshift-network-operator/iptables-alerter-5hlvj" Sep 11 15:54:55.286252 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.284882 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fe6dccde-412f-483d-a914-5d46e9922b69-cni-binary-copy\") pod \"multus-additional-cni-plugins-b7ftm\" (UID: \"fe6dccde-412f-483d-a914-5d46e9922b69\") " pod="openshift-multus/multus-additional-cni-plugins-b7ftm" Sep 11 15:54:55.286252 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.284899 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e05877f1-2ad9-43d5-8523-4749d6b08782-etc-selinux\") pod \"aws-ebs-csi-driver-node-xkvw8\" (UID: \"e05877f1-2ad9-43d5-8523-4749d6b08782\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xkvw8" Sep 11 15:54:55.286252 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.284917 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bba741b5-3455-493b-a0df-5e7177b832d8-host-run-k8s-cni-cncf-io\") pod \"multus-4vhlr\" (UID: \"bba741b5-3455-493b-a0df-5e7177b832d8\") " pod="openshift-multus/multus-4vhlr" Sep 11 15:54:55.286252 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.284945 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/249a0e7e-1da6-422a-8492-1820f6437318-serviceca\") pod \"node-ca-bctd5\" (UID: \"249a0e7e-1da6-422a-8492-1820f6437318\") " pod="openshift-image-registry/node-ca-bctd5" Sep 11 15:54:55.286252 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.284994 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bba741b5-3455-493b-a0df-5e7177b832d8-host-run-k8s-cni-cncf-io\") pod \"multus-4vhlr\" (UID: \"bba741b5-3455-493b-a0df-5e7177b832d8\") " pod="openshift-multus/multus-4vhlr" Sep 11 15:54:55.286252 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.285020 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bba741b5-3455-493b-a0df-5e7177b832d8-host-var-lib-kubelet\") pod \"multus-4vhlr\" (UID: \"bba741b5-3455-493b-a0df-5e7177b832d8\") " pod="openshift-multus/multus-4vhlr" Sep 11 15:54:55.286252 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.285049 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bba741b5-3455-493b-a0df-5e7177b832d8-host-run-multus-certs\") pod \"multus-4vhlr\" (UID: \"bba741b5-3455-493b-a0df-5e7177b832d8\") " pod="openshift-multus/multus-4vhlr" Sep 11 15:54:55.286252 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.285051 2566 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 11 15:54:55.287113 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.285092 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eab992d2-f32f-41bc-a028-c3fa544c4fec-etc-kubernetes\") pod \"tuned-sn7js\" (UID: \"eab992d2-f32f-41bc-a028-c3fa544c4fec\") " pod="openshift-cluster-node-tuning-operator/tuned-sn7js" Sep 11 15:54:55.287113 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.285212 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/981d7d1e-f81f-48fc-8962-7831bc5c2b27-var-lib-openvswitch\") pod \"ovnkube-node-tgq5c\" (UID: \"981d7d1e-f81f-48fc-8962-7831bc5c2b27\") " pod="openshift-ovn-kubernetes/ovnkube-node-tgq5c" Sep 11 15:54:55.287113 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.285255 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bba741b5-3455-493b-a0df-5e7177b832d8-multus-socket-dir-parent\") pod \"multus-4vhlr\" (UID: \"bba741b5-3455-493b-a0df-5e7177b832d8\") " pod="openshift-multus/multus-4vhlr" Sep 11 15:54:55.287113 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.285294 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ed90592a-177c-4128-b153-b51d18b8f619-hosts-file\") pod \"node-resolver-sxqj2\" (UID: \"ed90592a-177c-4128-b153-b51d18b8f619\") " pod="openshift-dns/node-resolver-sxqj2" Sep 11 15:54:55.287113 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.285327 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e05877f1-2ad9-43d5-8523-4749d6b08782-device-dir\") pod \"aws-ebs-csi-driver-node-xkvw8\" (UID: \"e05877f1-2ad9-43d5-8523-4749d6b08782\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xkvw8" Sep 11 15:54:55.287113 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.285342 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bba741b5-3455-493b-a0df-5e7177b832d8-host-var-lib-cni-bin\") pod \"multus-4vhlr\" (UID: \"bba741b5-3455-493b-a0df-5e7177b832d8\") " pod="openshift-multus/multus-4vhlr" Sep 11 15:54:55.287113 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.285364 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bba741b5-3455-493b-a0df-5e7177b832d8-cnibin\") pod \"multus-4vhlr\" (UID: \"bba741b5-3455-493b-a0df-5e7177b832d8\") " pod="openshift-multus/multus-4vhlr" Sep 11 15:54:55.287113 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.285390 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/eab992d2-f32f-41bc-a028-c3fa544c4fec-etc-sysctl-conf\") pod \"tuned-sn7js\" (UID: \"eab992d2-f32f-41bc-a028-c3fa544c4fec\") " pod="openshift-cluster-node-tuning-operator/tuned-sn7js" Sep 11 15:54:55.287113 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.285477 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bba741b5-3455-493b-a0df-5e7177b832d8-multus-daemon-config\") pod \"multus-4vhlr\" (UID: \"bba741b5-3455-493b-a0df-5e7177b832d8\") " pod="openshift-multus/multus-4vhlr" Sep 11 15:54:55.287113 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.285487 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eab992d2-f32f-41bc-a028-c3fa544c4fec-lib-modules\") pod \"tuned-sn7js\" (UID: \"eab992d2-f32f-41bc-a028-c3fa544c4fec\") " pod="openshift-cluster-node-tuning-operator/tuned-sn7js" Sep 11 15:54:55.287113 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.285497 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/eab992d2-f32f-41bc-a028-c3fa544c4fec-etc-sysctl-d\") pod \"tuned-sn7js\" (UID: \"eab992d2-f32f-41bc-a028-c3fa544c4fec\") " pod="openshift-cluster-node-tuning-operator/tuned-sn7js" Sep 11 15:54:55.287113 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.285540 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/981d7d1e-f81f-48fc-8962-7831bc5c2b27-run-ovn\") pod \"ovnkube-node-tgq5c\" (UID: \"981d7d1e-f81f-48fc-8962-7831bc5c2b27\") " pod="openshift-ovn-kubernetes/ovnkube-node-tgq5c" Sep 11 15:54:55.287113 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.285563 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/eab992d2-f32f-41bc-a028-c3fa544c4fec-etc-systemd\") pod \"tuned-sn7js\" (UID: \"eab992d2-f32f-41bc-a028-c3fa544c4fec\") " pod="openshift-cluster-node-tuning-operator/tuned-sn7js" Sep 11 15:54:55.287113 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.285566 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/eab992d2-f32f-41bc-a028-c3fa544c4fec-etc-modprobe-d\") pod \"tuned-sn7js\" (UID: \"eab992d2-f32f-41bc-a028-c3fa544c4fec\") " pod="openshift-cluster-node-tuning-operator/tuned-sn7js" Sep 11 15:54:55.287113 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.285591 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/eab992d2-f32f-41bc-a028-c3fa544c4fec-etc-sysconfig\") pod \"tuned-sn7js\" (UID: \"eab992d2-f32f-41bc-a028-c3fa544c4fec\") " pod="openshift-cluster-node-tuning-operator/tuned-sn7js" Sep 11 15:54:55.287113 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.285640 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eab992d2-f32f-41bc-a028-c3fa544c4fec-host\") pod \"tuned-sn7js\" (UID: \"eab992d2-f32f-41bc-a028-c3fa544c4fec\") " pod="openshift-cluster-node-tuning-operator/tuned-sn7js" Sep 11 15:54:55.287113 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.285663 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e05877f1-2ad9-43d5-8523-4749d6b08782-etc-selinux\") pod \"aws-ebs-csi-driver-node-xkvw8\" (UID: \"e05877f1-2ad9-43d5-8523-4749d6b08782\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xkvw8" Sep 11 15:54:55.287113 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.285687 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/981d7d1e-f81f-48fc-8962-7831bc5c2b27-host-run-ovn-kubernetes\") pod \"ovnkube-node-tgq5c\" (UID: \"981d7d1e-f81f-48fc-8962-7831bc5c2b27\") " pod="openshift-ovn-kubernetes/ovnkube-node-tgq5c" Sep 11 15:54:55.287956 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.285752 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/981d7d1e-f81f-48fc-8962-7831bc5c2b27-host-cni-bin\") pod \"ovnkube-node-tgq5c\" (UID: \"981d7d1e-f81f-48fc-8962-7831bc5c2b27\") " pod="openshift-ovn-kubernetes/ovnkube-node-tgq5c" Sep 11 15:54:55.287956 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.285786 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bba741b5-3455-493b-a0df-5e7177b832d8-host-run-netns\") pod \"multus-4vhlr\" (UID: \"bba741b5-3455-493b-a0df-5e7177b832d8\") " pod="openshift-multus/multus-4vhlr" Sep 11 15:54:55.287956 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.285821 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/981d7d1e-f81f-48fc-8962-7831bc5c2b27-host-cni-netd\") pod \"ovnkube-node-tgq5c\" (UID: \"981d7d1e-f81f-48fc-8962-7831bc5c2b27\") " pod="openshift-ovn-kubernetes/ovnkube-node-tgq5c" Sep 11 15:54:55.287956 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.285825 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bba741b5-3455-493b-a0df-5e7177b832d8-host-var-lib-cni-multus\") pod \"multus-4vhlr\" (UID: \"bba741b5-3455-493b-a0df-5e7177b832d8\") " pod="openshift-multus/multus-4vhlr" Sep 11 15:54:55.287956 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.285861 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/981d7d1e-f81f-48fc-8962-7831bc5c2b27-host-run-netns\") pod \"ovnkube-node-tgq5c\" (UID: \"981d7d1e-f81f-48fc-8962-7831bc5c2b27\") " pod="openshift-ovn-kubernetes/ovnkube-node-tgq5c" Sep 11 15:54:55.287956 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:54:55.285897 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 15:54:55.287956 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.285911 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bba741b5-3455-493b-a0df-5e7177b832d8-etc-kubernetes\") pod \"multus-4vhlr\" (UID: \"bba741b5-3455-493b-a0df-5e7177b832d8\") " pod="openshift-multus/multus-4vhlr" Sep 11 15:54:55.287956 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.285942 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/981d7d1e-f81f-48fc-8962-7831bc5c2b27-host-kubelet\") pod \"ovnkube-node-tgq5c\" (UID: \"981d7d1e-f81f-48fc-8962-7831bc5c2b27\") " pod="openshift-ovn-kubernetes/ovnkube-node-tgq5c" Sep 11 15:54:55.287956 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.285942 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/981d7d1e-f81f-48fc-8962-7831bc5c2b27-ovnkube-config\") pod \"ovnkube-node-tgq5c\" (UID: \"981d7d1e-f81f-48fc-8962-7831bc5c2b27\") " pod="openshift-ovn-kubernetes/ovnkube-node-tgq5c" Sep 11 15:54:55.287956 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:54:55.285958 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/331632d0-fa59-4462-805e-bce0e18b95bc-metrics-certs podName:331632d0-fa59-4462-805e-bce0e18b95bc nodeName:}" failed. No retries permitted until 2026-09-11 15:54:55.785940944 +0000 UTC m=+3.000774843 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/331632d0-fa59-4462-805e-bce0e18b95bc-metrics-certs") pod "network-metrics-daemon-plkhx" (UID: "331632d0-fa59-4462-805e-bce0e18b95bc") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 15:54:55.287956 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.285965 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/981d7d1e-f81f-48fc-8962-7831bc5c2b27-host-slash\") pod \"ovnkube-node-tgq5c\" (UID: \"981d7d1e-f81f-48fc-8962-7831bc5c2b27\") " pod="openshift-ovn-kubernetes/ovnkube-node-tgq5c" Sep 11 15:54:55.287956 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.286016 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e05877f1-2ad9-43d5-8523-4749d6b08782-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xkvw8\" (UID: \"e05877f1-2ad9-43d5-8523-4749d6b08782\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xkvw8" Sep 11 15:54:55.287956 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.286041 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e05877f1-2ad9-43d5-8523-4749d6b08782-registration-dir\") pod \"aws-ebs-csi-driver-node-xkvw8\" (UID: \"e05877f1-2ad9-43d5-8523-4749d6b08782\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xkvw8" Sep 11 15:54:55.287956 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.286058 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/981d7d1e-f81f-48fc-8962-7831bc5c2b27-run-systemd\") pod \"ovnkube-node-tgq5c\" (UID: \"981d7d1e-f81f-48fc-8962-7831bc5c2b27\") " pod="openshift-ovn-kubernetes/ovnkube-node-tgq5c" Sep 11 15:54:55.287956 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.286096 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eab992d2-f32f-41bc-a028-c3fa544c4fec-var-lib-kubelet\") pod \"tuned-sn7js\" (UID: \"eab992d2-f32f-41bc-a028-c3fa544c4fec\") " pod="openshift-cluster-node-tuning-operator/tuned-sn7js" Sep 11 15:54:55.287956 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.286101 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eab992d2-f32f-41bc-a028-c3fa544c4fec-sys\") pod \"tuned-sn7js\" (UID: \"eab992d2-f32f-41bc-a028-c3fa544c4fec\") " pod="openshift-cluster-node-tuning-operator/tuned-sn7js" Sep 11 15:54:55.287956 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.286158 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/981d7d1e-f81f-48fc-8962-7831bc5c2b27-node-log\") pod \"ovnkube-node-tgq5c\" (UID: \"981d7d1e-f81f-48fc-8962-7831bc5c2b27\") " pod="openshift-ovn-kubernetes/ovnkube-node-tgq5c" Sep 11 15:54:55.287956 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.285395 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bba741b5-3455-493b-a0df-5e7177b832d8-hostroot\") pod \"multus-4vhlr\" (UID: \"bba741b5-3455-493b-a0df-5e7177b832d8\") " pod="openshift-multus/multus-4vhlr" Sep 11 15:54:55.288848 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.286242 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bba741b5-3455-493b-a0df-5e7177b832d8-multus-conf-dir\") pod \"multus-4vhlr\" (UID: \"bba741b5-3455-493b-a0df-5e7177b832d8\") " pod="openshift-multus/multus-4vhlr" Sep 11 15:54:55.288848 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.286411 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/981d7d1e-f81f-48fc-8962-7831bc5c2b27-env-overrides\") pod \"ovnkube-node-tgq5c\" (UID: \"981d7d1e-f81f-48fc-8962-7831bc5c2b27\") " pod="openshift-ovn-kubernetes/ovnkube-node-tgq5c" Sep 11 15:54:55.288848 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.286466 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/981d7d1e-f81f-48fc-8962-7831bc5c2b27-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tgq5c\" (UID: \"981d7d1e-f81f-48fc-8962-7831bc5c2b27\") " pod="openshift-ovn-kubernetes/ovnkube-node-tgq5c" Sep 11 15:54:55.288848 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.286534 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed90592a-177c-4128-b153-b51d18b8f619-tmp-dir\") pod \"node-resolver-sxqj2\" (UID: \"ed90592a-177c-4128-b153-b51d18b8f619\") " pod="openshift-dns/node-resolver-sxqj2" Sep 11 15:54:55.288848 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.286538 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bba741b5-3455-493b-a0df-5e7177b832d8-multus-cni-dir\") pod \"multus-4vhlr\" (UID: \"bba741b5-3455-493b-a0df-5e7177b832d8\") " pod="openshift-multus/multus-4vhlr" Sep 11 15:54:55.288848 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.286604 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/981d7d1e-f81f-48fc-8962-7831bc5c2b27-log-socket\") pod \"ovnkube-node-tgq5c\" (UID: \"981d7d1e-f81f-48fc-8962-7831bc5c2b27\") " pod="openshift-ovn-kubernetes/ovnkube-node-tgq5c" Sep 11 15:54:55.288848 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.286636 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/981d7d1e-f81f-48fc-8962-7831bc5c2b27-run-openvswitch\") pod \"ovnkube-node-tgq5c\" (UID: \"981d7d1e-f81f-48fc-8962-7831bc5c2b27\") " pod="openshift-ovn-kubernetes/ovnkube-node-tgq5c" Sep 11 15:54:55.288848 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.286666 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/249a0e7e-1da6-422a-8492-1820f6437318-host\") pod \"node-ca-bctd5\" (UID: \"249a0e7e-1da6-422a-8492-1820f6437318\") " pod="openshift-image-registry/node-ca-bctd5" Sep 11 15:54:55.288848 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.286708 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/981d7d1e-f81f-48fc-8962-7831bc5c2b27-ovnkube-script-lib\") pod \"ovnkube-node-tgq5c\" (UID: \"981d7d1e-f81f-48fc-8962-7831bc5c2b27\") " pod="openshift-ovn-kubernetes/ovnkube-node-tgq5c" Sep 11 15:54:55.288848 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.286782 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e05877f1-2ad9-43d5-8523-4749d6b08782-sys-fs\") pod \"aws-ebs-csi-driver-node-xkvw8\" (UID: \"e05877f1-2ad9-43d5-8523-4749d6b08782\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xkvw8" Sep 11 15:54:55.288848 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.286818 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/981d7d1e-f81f-48fc-8962-7831bc5c2b27-systemd-units\") pod \"ovnkube-node-tgq5c\" (UID: \"981d7d1e-f81f-48fc-8962-7831bc5c2b27\") " pod="openshift-ovn-kubernetes/ovnkube-node-tgq5c" Sep 11 15:54:55.288848 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.286818 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bba741b5-3455-493b-a0df-5e7177b832d8-cni-binary-copy\") pod \"multus-4vhlr\" (UID: \"bba741b5-3455-493b-a0df-5e7177b832d8\") " pod="openshift-multus/multus-4vhlr" Sep 11 15:54:55.288848 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.286828 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e05877f1-2ad9-43d5-8523-4749d6b08782-socket-dir\") pod \"aws-ebs-csi-driver-node-xkvw8\" (UID: \"e05877f1-2ad9-43d5-8523-4749d6b08782\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xkvw8" Sep 11 15:54:55.288848 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.286863 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/eab992d2-f32f-41bc-a028-c3fa544c4fec-run\") pod \"tuned-sn7js\" (UID: \"eab992d2-f32f-41bc-a028-c3fa544c4fec\") " pod="openshift-cluster-node-tuning-operator/tuned-sn7js" Sep 11 15:54:55.288848 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.286890 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bba741b5-3455-493b-a0df-5e7177b832d8-system-cni-dir\") pod \"multus-4vhlr\" (UID: \"bba741b5-3455-493b-a0df-5e7177b832d8\") " pod="openshift-multus/multus-4vhlr" Sep 11 15:54:55.288848 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.287365 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/94ab0396-7020-4ad3-b1b7-63cd05490e84-konnectivity-ca\") pod \"konnectivity-agent-spxb6\" (UID: \"94ab0396-7020-4ad3-b1b7-63cd05490e84\") " pod="kube-system/konnectivity-agent-spxb6" Sep 11 15:54:55.289599 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.289472 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/eab992d2-f32f-41bc-a028-c3fa544c4fec-etc-tuned\") pod \"tuned-sn7js\" (UID: \"eab992d2-f32f-41bc-a028-c3fa544c4fec\") " pod="openshift-cluster-node-tuning-operator/tuned-sn7js" Sep 11 15:54:55.289651 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.289621 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eab992d2-f32f-41bc-a028-c3fa544c4fec-tmp\") pod \"tuned-sn7js\" (UID: \"eab992d2-f32f-41bc-a028-c3fa544c4fec\") " pod="openshift-cluster-node-tuning-operator/tuned-sn7js" Sep 11 15:54:55.289651 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.289637 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/981d7d1e-f81f-48fc-8962-7831bc5c2b27-ovn-node-metrics-cert\") pod \"ovnkube-node-tgq5c\" (UID: \"981d7d1e-f81f-48fc-8962-7831bc5c2b27\") " pod="openshift-ovn-kubernetes/ovnkube-node-tgq5c" Sep 11 15:54:55.291694 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.291672 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/94ab0396-7020-4ad3-b1b7-63cd05490e84-agent-certs\") pod \"konnectivity-agent-spxb6\" (UID: \"94ab0396-7020-4ad3-b1b7-63cd05490e84\") " pod="kube-system/konnectivity-agent-spxb6" Sep 11 15:54:55.294252 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.294230 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjvb6\" (UniqueName: \"kubernetes.io/projected/331632d0-fa59-4462-805e-bce0e18b95bc-kube-api-access-zjvb6\") pod \"network-metrics-daemon-plkhx\" (UID: \"331632d0-fa59-4462-805e-bce0e18b95bc\") " pod="openshift-multus/network-metrics-daemon-plkhx" Sep 11 15:54:55.298086 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.297957 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6hzg\" (UniqueName: \"kubernetes.io/projected/e05877f1-2ad9-43d5-8523-4749d6b08782-kube-api-access-w6hzg\") pod \"aws-ebs-csi-driver-node-xkvw8\" (UID: \"e05877f1-2ad9-43d5-8523-4749d6b08782\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xkvw8" Sep 11 15:54:55.298334 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.298308 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5d8bj\" (UniqueName: \"kubernetes.io/projected/eab992d2-f32f-41bc-a028-c3fa544c4fec-kube-api-access-5d8bj\") pod \"tuned-sn7js\" (UID: \"eab992d2-f32f-41bc-a028-c3fa544c4fec\") " pod="openshift-cluster-node-tuning-operator/tuned-sn7js" Sep 11 15:54:55.299186 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.299152 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcpmk\" (UniqueName: \"kubernetes.io/projected/ed90592a-177c-4128-b153-b51d18b8f619-kube-api-access-jcpmk\") pod \"node-resolver-sxqj2\" (UID: \"ed90592a-177c-4128-b153-b51d18b8f619\") " pod="openshift-dns/node-resolver-sxqj2" Sep 11 15:54:55.299344 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.299323 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f86rv\" (UniqueName: \"kubernetes.io/projected/981d7d1e-f81f-48fc-8962-7831bc5c2b27-kube-api-access-f86rv\") pod \"ovnkube-node-tgq5c\" (UID: \"981d7d1e-f81f-48fc-8962-7831bc5c2b27\") " pod="openshift-ovn-kubernetes/ovnkube-node-tgq5c" Sep 11 15:54:55.300380 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.300340 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xknqc\" (UniqueName: \"kubernetes.io/projected/249a0e7e-1da6-422a-8492-1820f6437318-kube-api-access-xknqc\") pod \"node-ca-bctd5\" (UID: \"249a0e7e-1da6-422a-8492-1820f6437318\") " pod="openshift-image-registry/node-ca-bctd5" Sep 11 15:54:55.300851 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.300826 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-grtpf\" (UniqueName: \"kubernetes.io/projected/bba741b5-3455-493b-a0df-5e7177b832d8-kube-api-access-grtpf\") pod \"multus-4vhlr\" (UID: \"bba741b5-3455-493b-a0df-5e7177b832d8\") " pod="openshift-multus/multus-4vhlr" Sep 11 15:54:55.385463 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.385428 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gwrb\" (UniqueName: \"kubernetes.io/projected/b02518a9-32bc-4627-97c8-4535c8333875-kube-api-access-6gwrb\") pod \"iptables-alerter-5hlvj\" (UID: \"b02518a9-32bc-4627-97c8-4535c8333875\") " pod="openshift-network-operator/iptables-alerter-5hlvj" Sep 11 15:54:55.385463 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.385471 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9g7p\" (UniqueName: \"kubernetes.io/projected/f6c05fce-e71f-4a9c-b219-a77b1df84ac8-kube-api-access-h9g7p\") pod \"network-check-target-nzcjl\" (UID: \"f6c05fce-e71f-4a9c-b219-a77b1df84ac8\") " pod="openshift-network-diagnostics/network-check-target-nzcjl" Sep 11 15:54:55.385715 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.385503 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fe6dccde-412f-483d-a914-5d46e9922b69-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-b7ftm\" (UID: \"fe6dccde-412f-483d-a914-5d46e9922b69\") " pod="openshift-multus/multus-additional-cni-plugins-b7ftm" Sep 11 15:54:55.385715 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.385531 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96sg9\" (UniqueName: \"kubernetes.io/projected/fe6dccde-412f-483d-a914-5d46e9922b69-kube-api-access-96sg9\") pod \"multus-additional-cni-plugins-b7ftm\" (UID: \"fe6dccde-412f-483d-a914-5d46e9922b69\") " pod="openshift-multus/multus-additional-cni-plugins-b7ftm" Sep 11 15:54:55.385715 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.385554 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1c358fa6-a923-4589-953f-bd49ddd4235c-kubelet-config\") pod \"global-pull-secret-syncer-95zvh\" (UID: \"1c358fa6-a923-4589-953f-bd49ddd4235c\") " pod="kube-system/global-pull-secret-syncer-95zvh" Sep 11 15:54:55.385715 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.385579 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fe6dccde-412f-483d-a914-5d46e9922b69-tuning-conf-dir\") pod \"multus-additional-cni-plugins-b7ftm\" (UID: \"fe6dccde-412f-483d-a914-5d46e9922b69\") " pod="openshift-multus/multus-additional-cni-plugins-b7ftm" Sep 11 15:54:55.385715 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.385606 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b02518a9-32bc-4627-97c8-4535c8333875-host-slash\") pod \"iptables-alerter-5hlvj\" (UID: \"b02518a9-32bc-4627-97c8-4535c8333875\") " pod="openshift-network-operator/iptables-alerter-5hlvj" Sep 11 15:54:55.385715 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.385655 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1c358fa6-a923-4589-953f-bd49ddd4235c-dbus\") pod \"global-pull-secret-syncer-95zvh\" (UID: \"1c358fa6-a923-4589-953f-bd49ddd4235c\") " pod="kube-system/global-pull-secret-syncer-95zvh" Sep 11 15:54:55.385715 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.385684 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1c358fa6-a923-4589-953f-bd49ddd4235c-original-pull-secret\") pod \"global-pull-secret-syncer-95zvh\" (UID: \"1c358fa6-a923-4589-953f-bd49ddd4235c\") " pod="kube-system/global-pull-secret-syncer-95zvh" Sep 11 15:54:55.385715 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.385677 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1c358fa6-a923-4589-953f-bd49ddd4235c-kubelet-config\") pod \"global-pull-secret-syncer-95zvh\" (UID: \"1c358fa6-a923-4589-953f-bd49ddd4235c\") " pod="kube-system/global-pull-secret-syncer-95zvh" Sep 11 15:54:55.385715 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.385711 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fe6dccde-412f-483d-a914-5d46e9922b69-system-cni-dir\") pod \"multus-additional-cni-plugins-b7ftm\" (UID: \"fe6dccde-412f-483d-a914-5d46e9922b69\") " pod="openshift-multus/multus-additional-cni-plugins-b7ftm" Sep 11 15:54:55.386152 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.385767 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b02518a9-32bc-4627-97c8-4535c8333875-iptables-alerter-script\") pod \"iptables-alerter-5hlvj\" (UID: \"b02518a9-32bc-4627-97c8-4535c8333875\") " pod="openshift-network-operator/iptables-alerter-5hlvj" Sep 11 15:54:55.386152 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.385790 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b02518a9-32bc-4627-97c8-4535c8333875-host-slash\") pod \"iptables-alerter-5hlvj\" (UID: \"b02518a9-32bc-4627-97c8-4535c8333875\") " pod="openshift-network-operator/iptables-alerter-5hlvj" Sep 11 15:54:55.386152 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.385797 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fe6dccde-412f-483d-a914-5d46e9922b69-cni-binary-copy\") pod \"multus-additional-cni-plugins-b7ftm\" (UID: \"fe6dccde-412f-483d-a914-5d46e9922b69\") " pod="openshift-multus/multus-additional-cni-plugins-b7ftm" Sep 11 15:54:55.386152 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.385846 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fe6dccde-412f-483d-a914-5d46e9922b69-cnibin\") pod \"multus-additional-cni-plugins-b7ftm\" (UID: \"fe6dccde-412f-483d-a914-5d46e9922b69\") " pod="openshift-multus/multus-additional-cni-plugins-b7ftm" Sep 11 15:54:55.386152 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:54:55.385860 2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 15:54:55.386152 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.385868 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fe6dccde-412f-483d-a914-5d46e9922b69-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-b7ftm\" (UID: \"fe6dccde-412f-483d-a914-5d46e9922b69\") " pod="openshift-multus/multus-additional-cni-plugins-b7ftm" Sep 11 15:54:55.386152 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:54:55.385918 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c358fa6-a923-4589-953f-bd49ddd4235c-original-pull-secret podName:1c358fa6-a923-4589-953f-bd49ddd4235c nodeName:}" failed. No retries permitted until 2026-09-11 15:54:55.885899233 +0000 UTC m=+3.100733133 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1c358fa6-a923-4589-953f-bd49ddd4235c-original-pull-secret") pod "global-pull-secret-syncer-95zvh" (UID: "1c358fa6-a923-4589-953f-bd49ddd4235c") : object "kube-system"/"original-pull-secret" not registered Sep 11 15:54:55.386503 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.386239 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fe6dccde-412f-483d-a914-5d46e9922b69-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-b7ftm\" (UID: \"fe6dccde-412f-483d-a914-5d46e9922b69\") " pod="openshift-multus/multus-additional-cni-plugins-b7ftm" Sep 11 15:54:55.386503 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.386276 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fe6dccde-412f-483d-a914-5d46e9922b69-tuning-conf-dir\") pod \"multus-additional-cni-plugins-b7ftm\" (UID: \"fe6dccde-412f-483d-a914-5d46e9922b69\") " pod="openshift-multus/multus-additional-cni-plugins-b7ftm" Sep 11 15:54:55.386503 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.386334 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fe6dccde-412f-483d-a914-5d46e9922b69-cnibin\") pod \"multus-additional-cni-plugins-b7ftm\" (UID: \"fe6dccde-412f-483d-a914-5d46e9922b69\") " pod="openshift-multus/multus-additional-cni-plugins-b7ftm" Sep 11 15:54:55.386503 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.385770 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fe6dccde-412f-483d-a914-5d46e9922b69-system-cni-dir\") pod \"multus-additional-cni-plugins-b7ftm\" (UID: \"fe6dccde-412f-483d-a914-5d46e9922b69\") " pod="openshift-multus/multus-additional-cni-plugins-b7ftm" Sep 11 15:54:55.386503 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.386388 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fe6dccde-412f-483d-a914-5d46e9922b69-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-b7ftm\" (UID: \"fe6dccde-412f-483d-a914-5d46e9922b69\") " pod="openshift-multus/multus-additional-cni-plugins-b7ftm" Sep 11 15:54:55.386503 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.386401 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1c358fa6-a923-4589-953f-bd49ddd4235c-dbus\") pod \"global-pull-secret-syncer-95zvh\" (UID: \"1c358fa6-a923-4589-953f-bd49ddd4235c\") " pod="kube-system/global-pull-secret-syncer-95zvh" Sep 11 15:54:55.386800 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.386598 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b02518a9-32bc-4627-97c8-4535c8333875-iptables-alerter-script\") pod \"iptables-alerter-5hlvj\" (UID: \"b02518a9-32bc-4627-97c8-4535c8333875\") " pod="openshift-network-operator/iptables-alerter-5hlvj" Sep 11 15:54:55.386850 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.386816 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fe6dccde-412f-483d-a914-5d46e9922b69-cni-binary-copy\") pod \"multus-additional-cni-plugins-b7ftm\" (UID: \"fe6dccde-412f-483d-a914-5d46e9922b69\") " pod="openshift-multus/multus-additional-cni-plugins-b7ftm" Sep 11 15:54:55.392715 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:54:55.392691 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 15:54:55.392835 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:54:55.392721 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 15:54:55.392835 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:54:55.392736 2566 projected.go:196] Error preparing data for projected volume kube-api-access-h9g7p for pod openshift-network-diagnostics/network-check-target-nzcjl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 15:54:55.392835 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:54:55.392802 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f6c05fce-e71f-4a9c-b219-a77b1df84ac8-kube-api-access-h9g7p podName:f6c05fce-e71f-4a9c-b219-a77b1df84ac8 nodeName:}" failed. No retries permitted until 2026-09-11 15:54:55.892786532 +0000 UTC m=+3.107620433 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-h9g7p" (UniqueName: "kubernetes.io/projected/f6c05fce-e71f-4a9c-b219-a77b1df84ac8-kube-api-access-h9g7p") pod "network-check-target-nzcjl" (UID: "f6c05fce-e71f-4a9c-b219-a77b1df84ac8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 15:54:55.395510 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.395456 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gwrb\" (UniqueName: \"kubernetes.io/projected/b02518a9-32bc-4627-97c8-4535c8333875-kube-api-access-6gwrb\") pod \"iptables-alerter-5hlvj\" (UID: \"b02518a9-32bc-4627-97c8-4535c8333875\") " pod="openshift-network-operator/iptables-alerter-5hlvj" Sep 11 15:54:55.395890 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.395866 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-96sg9\" (UniqueName: \"kubernetes.io/projected/fe6dccde-412f-483d-a914-5d46e9922b69-kube-api-access-96sg9\") pod \"multus-additional-cni-plugins-b7ftm\" (UID: \"fe6dccde-412f-483d-a914-5d46e9922b69\") " pod="openshift-multus/multus-additional-cni-plugins-b7ftm" Sep 11 15:54:55.468299 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.468265 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-sn7js" Sep 11 15:54:55.477050 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.476916 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-tgq5c" Sep 11 15:54:55.479863 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:55.479834 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeab992d2_f32f_41bc_a028_c3fa544c4fec.slice/crio-5e5ae7c5463d97186e75bdf4087fe34a7d531dc27017b4651b37abf9c0e11c38 WatchSource:0}: Error finding container 5e5ae7c5463d97186e75bdf4087fe34a7d531dc27017b4651b37abf9c0e11c38: Status 404 returned error can't find the container with id 5e5ae7c5463d97186e75bdf4087fe34a7d531dc27017b4651b37abf9c0e11c38 Sep 11 15:54:55.487390 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.487365 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-spxb6" Sep 11 15:54:55.492649 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:55.492620 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod981d7d1e_f81f_48fc_8962_7831bc5c2b27.slice/crio-f4a6ab8c0cc33ea2e5e054c053243a446d8e0cf209ca454e4eaaa6ba969db524 WatchSource:0}: Error finding container f4a6ab8c0cc33ea2e5e054c053243a446d8e0cf209ca454e4eaaa6ba969db524: Status 404 returned error can't find the container with id f4a6ab8c0cc33ea2e5e054c053243a446d8e0cf209ca454e4eaaa6ba969db524 Sep 11 15:54:55.496812 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.496639 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-sxqj2" Sep 11 15:54:55.497616 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:55.497578 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94ab0396_7020_4ad3_b1b7_63cd05490e84.slice/crio-63047d4a66ae9825be72d3ab2b2588814e633b015737ba0611271f1517136481 WatchSource:0}: Error finding container 63047d4a66ae9825be72d3ab2b2588814e633b015737ba0611271f1517136481: Status 404 returned error can't find the container with id 63047d4a66ae9825be72d3ab2b2588814e633b015737ba0611271f1517136481 Sep 11 15:54:55.504885 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.504866 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xkvw8" Sep 11 15:54:55.507630 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:55.507601 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded90592a_177c_4128_b153_b51d18b8f619.slice/crio-70a37faa10872ddf3b67374aa2153141b0e58c4859da201ccea9c1e22bad0479 WatchSource:0}: Error finding container 70a37faa10872ddf3b67374aa2153141b0e58c4859da201ccea9c1e22bad0479: Status 404 returned error can't find the container with id 70a37faa10872ddf3b67374aa2153141b0e58c4859da201ccea9c1e22bad0479 Sep 11 15:54:55.512258 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.512236 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4vhlr" Sep 11 15:54:55.521627 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.521524 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bctd5" Sep 11 15:54:55.524838 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:55.524805 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbba741b5_3455_493b_a0df_5e7177b832d8.slice/crio-2a212440b6cb259f08f0ca2b32370cf6f31a2d3b54233957d735fafd50de00c8 WatchSource:0}: Error finding container 2a212440b6cb259f08f0ca2b32370cf6f31a2d3b54233957d735fafd50de00c8: Status 404 returned error can't find the container with id 2a212440b6cb259f08f0ca2b32370cf6f31a2d3b54233957d735fafd50de00c8 Sep 11 15:54:55.532281 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.532255 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5hlvj" Sep 11 15:54:55.533100 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:55.533070 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod249a0e7e_1da6_422a_8492_1820f6437318.slice/crio-c3340bb8c9718ea74bf08bf1b9834e65e2cff36d82b4dd2631cd9c7e96ea9852 WatchSource:0}: Error finding container c3340bb8c9718ea74bf08bf1b9834e65e2cff36d82b4dd2631cd9c7e96ea9852: Status 404 returned error can't find the container with id c3340bb8c9718ea74bf08bf1b9834e65e2cff36d82b4dd2631cd9c7e96ea9852 Sep 11 15:54:55.538355 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.538334 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-b7ftm" Sep 11 15:54:55.543367 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:55.543345 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb02518a9_32bc_4627_97c8_4535c8333875.slice/crio-893a4ea4e397adeaa25721a6378741bd9d78e225d81a7ca05341db9b14c4d574 WatchSource:0}: Error finding container 893a4ea4e397adeaa25721a6378741bd9d78e225d81a7ca05341db9b14c4d574: Status 404 returned error can't find the container with id 893a4ea4e397adeaa25721a6378741bd9d78e225d81a7ca05341db9b14c4d574 Sep 11 15:54:55.549859 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:54:55.549831 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe6dccde_412f_483d_a914_5d46e9922b69.slice/crio-6fb6e302aba3a0f3b97a49c8a881b34b08ece40231463cf7a16a370a6d0cbfdc WatchSource:0}: Error finding container 6fb6e302aba3a0f3b97a49c8a881b34b08ece40231463cf7a16a370a6d0cbfdc: Status 404 returned error can't find the container with id 6fb6e302aba3a0f3b97a49c8a881b34b08ece40231463cf7a16a370a6d0cbfdc Sep 11 15:54:55.788054 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.787955 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/331632d0-fa59-4462-805e-bce0e18b95bc-metrics-certs\") pod \"network-metrics-daemon-plkhx\" (UID: \"331632d0-fa59-4462-805e-bce0e18b95bc\") " pod="openshift-multus/network-metrics-daemon-plkhx" Sep 11 15:54:55.788203 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:54:55.788115 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 15:54:55.788203 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:54:55.788184 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/331632d0-fa59-4462-805e-bce0e18b95bc-metrics-certs podName:331632d0-fa59-4462-805e-bce0e18b95bc nodeName:}" failed. No retries permitted until 2026-09-11 15:54:56.788168389 +0000 UTC m=+4.003002277 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/331632d0-fa59-4462-805e-bce0e18b95bc-metrics-certs") pod "network-metrics-daemon-plkhx" (UID: "331632d0-fa59-4462-805e-bce0e18b95bc") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 15:54:55.888563 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.888516 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1c358fa6-a923-4589-953f-bd49ddd4235c-original-pull-secret\") pod \"global-pull-secret-syncer-95zvh\" (UID: \"1c358fa6-a923-4589-953f-bd49ddd4235c\") " pod="kube-system/global-pull-secret-syncer-95zvh" Sep 11 15:54:55.888724 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:54:55.888654 2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 15:54:55.888724 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:54:55.888723 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c358fa6-a923-4589-953f-bd49ddd4235c-original-pull-secret podName:1c358fa6-a923-4589-953f-bd49ddd4235c nodeName:}" failed. No retries permitted until 2026-09-11 15:54:56.888706712 +0000 UTC m=+4.103540616 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1c358fa6-a923-4589-953f-bd49ddd4235c-original-pull-secret") pod "global-pull-secret-syncer-95zvh" (UID: "1c358fa6-a923-4589-953f-bd49ddd4235c") : object "kube-system"/"original-pull-secret" not registered Sep 11 15:54:55.989566 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:55.989523 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9g7p\" (UniqueName: \"kubernetes.io/projected/f6c05fce-e71f-4a9c-b219-a77b1df84ac8-kube-api-access-h9g7p\") pod \"network-check-target-nzcjl\" (UID: \"f6c05fce-e71f-4a9c-b219-a77b1df84ac8\") " pod="openshift-network-diagnostics/network-check-target-nzcjl" Sep 11 15:54:55.989736 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:54:55.989702 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 15:54:55.989784 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:54:55.989735 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 15:54:55.989784 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:54:55.989751 2566 projected.go:196] Error preparing data for projected volume kube-api-access-h9g7p for pod openshift-network-diagnostics/network-check-target-nzcjl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 15:54:55.989862 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:54:55.989823 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f6c05fce-e71f-4a9c-b219-a77b1df84ac8-kube-api-access-h9g7p podName:f6c05fce-e71f-4a9c-b219-a77b1df84ac8 nodeName:}" failed. No retries permitted until 2026-09-11 15:54:56.98980563 +0000 UTC m=+4.204639527 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-h9g7p" (UniqueName: "kubernetes.io/projected/f6c05fce-e71f-4a9c-b219-a77b1df84ac8-kube-api-access-h9g7p") pod "network-check-target-nzcjl" (UID: "f6c05fce-e71f-4a9c-b219-a77b1df84ac8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 15:54:56.217418 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:56.217252 2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 15:49:55 +0000 UTC" deadline="2028-02-26 07:30:40.882933712 +0000 UTC" Sep 11 15:54:56.217418 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:56.217316 2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12783h35m44.66562158s" Sep 11 15:54:56.244854 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:56.244749 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b7ftm" event={"ID":"fe6dccde-412f-483d-a914-5d46e9922b69","Type":"ContainerStarted","Data":"6fb6e302aba3a0f3b97a49c8a881b34b08ece40231463cf7a16a370a6d0cbfdc"} Sep 11 15:54:56.246409 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:56.246362 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5hlvj" event={"ID":"b02518a9-32bc-4627-97c8-4535c8333875","Type":"ContainerStarted","Data":"893a4ea4e397adeaa25721a6378741bd9d78e225d81a7ca05341db9b14c4d574"} Sep 11 15:54:56.247853 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:56.247806 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4vhlr" event={"ID":"bba741b5-3455-493b-a0df-5e7177b832d8","Type":"ContainerStarted","Data":"2a212440b6cb259f08f0ca2b32370cf6f31a2d3b54233957d735fafd50de00c8"} Sep 11 15:54:56.249144 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:56.249122 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-sxqj2" event={"ID":"ed90592a-177c-4128-b153-b51d18b8f619","Type":"ContainerStarted","Data":"70a37faa10872ddf3b67374aa2153141b0e58c4859da201ccea9c1e22bad0479"} Sep 11 15:54:56.250601 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:56.250572 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-spxb6" event={"ID":"94ab0396-7020-4ad3-b1b7-63cd05490e84","Type":"ContainerStarted","Data":"63047d4a66ae9825be72d3ab2b2588814e633b015737ba0611271f1517136481"} Sep 11 15:54:56.252232 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:56.252153 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tgq5c" event={"ID":"981d7d1e-f81f-48fc-8962-7831bc5c2b27","Type":"ContainerStarted","Data":"f4a6ab8c0cc33ea2e5e054c053243a446d8e0cf209ca454e4eaaa6ba969db524"} Sep 11 15:54:56.253606 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:56.253581 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-sn7js" event={"ID":"eab992d2-f32f-41bc-a028-c3fa544c4fec","Type":"ContainerStarted","Data":"5e5ae7c5463d97186e75bdf4087fe34a7d531dc27017b4651b37abf9c0e11c38"} Sep 11 15:54:56.255702 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:56.255677 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bctd5" event={"ID":"249a0e7e-1da6-422a-8492-1820f6437318","Type":"ContainerStarted","Data":"c3340bb8c9718ea74bf08bf1b9834e65e2cff36d82b4dd2631cd9c7e96ea9852"} Sep 11 15:54:56.257245 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:56.257222 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xkvw8" event={"ID":"e05877f1-2ad9-43d5-8523-4749d6b08782","Type":"ContainerStarted","Data":"f05a4781f6dd821147a902406c8f0d0c3acd799d59ea5e946254002a57d1b98f"} Sep 11 15:54:56.797865 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:56.797559 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/331632d0-fa59-4462-805e-bce0e18b95bc-metrics-certs\") pod \"network-metrics-daemon-plkhx\" (UID: \"331632d0-fa59-4462-805e-bce0e18b95bc\") " pod="openshift-multus/network-metrics-daemon-plkhx" Sep 11 15:54:56.797865 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:54:56.797760 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 15:54:56.797865 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:54:56.797823 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/331632d0-fa59-4462-805e-bce0e18b95bc-metrics-certs podName:331632d0-fa59-4462-805e-bce0e18b95bc nodeName:}" failed. No retries permitted until 2026-09-11 15:54:58.797804092 +0000 UTC m=+6.012637980 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/331632d0-fa59-4462-805e-bce0e18b95bc-metrics-certs") pod "network-metrics-daemon-plkhx" (UID: "331632d0-fa59-4462-805e-bce0e18b95bc") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 15:54:56.898509 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:56.898471 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1c358fa6-a923-4589-953f-bd49ddd4235c-original-pull-secret\") pod \"global-pull-secret-syncer-95zvh\" (UID: \"1c358fa6-a923-4589-953f-bd49ddd4235c\") " pod="kube-system/global-pull-secret-syncer-95zvh" Sep 11 15:54:56.898698 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:54:56.898664 2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 15:54:56.898760 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:54:56.898737 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c358fa6-a923-4589-953f-bd49ddd4235c-original-pull-secret podName:1c358fa6-a923-4589-953f-bd49ddd4235c nodeName:}" failed. No retries permitted until 2026-09-11 15:54:58.898718114 +0000 UTC m=+6.113552021 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1c358fa6-a923-4589-953f-bd49ddd4235c-original-pull-secret") pod "global-pull-secret-syncer-95zvh" (UID: "1c358fa6-a923-4589-953f-bd49ddd4235c") : object "kube-system"/"original-pull-secret" not registered Sep 11 15:54:57.000537 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:57.000495 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9g7p\" (UniqueName: \"kubernetes.io/projected/f6c05fce-e71f-4a9c-b219-a77b1df84ac8-kube-api-access-h9g7p\") pod \"network-check-target-nzcjl\" (UID: \"f6c05fce-e71f-4a9c-b219-a77b1df84ac8\") " pod="openshift-network-diagnostics/network-check-target-nzcjl" Sep 11 15:54:57.000753 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:54:57.000720 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 15:54:57.000753 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:54:57.000743 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 15:54:57.000860 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:54:57.000757 2566 projected.go:196] Error preparing data for projected volume kube-api-access-h9g7p for pod openshift-network-diagnostics/network-check-target-nzcjl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 15:54:57.000860 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:54:57.000815 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f6c05fce-e71f-4a9c-b219-a77b1df84ac8-kube-api-access-h9g7p podName:f6c05fce-e71f-4a9c-b219-a77b1df84ac8 nodeName:}" failed. No retries permitted until 2026-09-11 15:54:59.000797111 +0000 UTC m=+6.215631026 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-h9g7p" (UniqueName: "kubernetes.io/projected/f6c05fce-e71f-4a9c-b219-a77b1df84ac8-kube-api-access-h9g7p") pod "network-check-target-nzcjl" (UID: "f6c05fce-e71f-4a9c-b219-a77b1df84ac8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 15:54:57.217653 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:57.217579 2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 15:49:55 +0000 UTC" deadline="2028-03-05 15:15:39.348020085 +0000 UTC" Sep 11 15:54:57.217653 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:57.217620 2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12983h20m42.130405789s" Sep 11 15:54:57.240631 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:57.240154 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nzcjl" Sep 11 15:54:57.240631 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:57.240192 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-95zvh" Sep 11 15:54:57.240631 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:54:57.240274 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nzcjl" podUID="f6c05fce-e71f-4a9c-b219-a77b1df84ac8" Sep 11 15:54:57.240631 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:57.240154 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-plkhx" Sep 11 15:54:57.240631 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:54:57.240378 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-plkhx" podUID="331632d0-fa59-4462-805e-bce0e18b95bc" Sep 11 15:54:57.240631 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:54:57.240448 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-95zvh" podUID="1c358fa6-a923-4589-953f-bd49ddd4235c" Sep 11 15:54:57.269375 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:57.269334 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-96.ec2.internal" event={"ID":"bd507eff9b3716da03d385309d794351","Type":"ContainerStarted","Data":"b503204691a092073ac584f6c1a46f9c02406753dbe33b651d518c0f7cc01246"} Sep 11 15:54:57.286876 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:57.285940 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-96.ec2.internal" podStartSLOduration=3.285920711 podStartE2EDuration="3.285920711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 15:54:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:54:57.285847279 +0000 UTC m=+4.500681183" watchObservedRunningTime="2026-09-11 15:54:57.285920711 +0000 UTC m=+4.500754617" Sep 11 15:54:58.280498 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:58.280123 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-96.ec2.internal" event={"ID":"e46b070927ba07e09c7fdfca6f7f4f91","Type":"ContainerStarted","Data":"e5c9f21edbcf0dcbf152154f5207c8af1f31df4addd7baefbeec5566788c3a03"} Sep 11 15:54:58.822960 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:58.822481 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/331632d0-fa59-4462-805e-bce0e18b95bc-metrics-certs\") pod \"network-metrics-daemon-plkhx\" (UID: \"331632d0-fa59-4462-805e-bce0e18b95bc\") " pod="openshift-multus/network-metrics-daemon-plkhx" Sep 11 15:54:58.822960 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:54:58.822598 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 15:54:58.822960 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:54:58.822642 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/331632d0-fa59-4462-805e-bce0e18b95bc-metrics-certs podName:331632d0-fa59-4462-805e-bce0e18b95bc nodeName:}" failed. No retries permitted until 2026-09-11 15:55:02.822628721 +0000 UTC m=+10.037462604 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/331632d0-fa59-4462-805e-bce0e18b95bc-metrics-certs") pod "network-metrics-daemon-plkhx" (UID: "331632d0-fa59-4462-805e-bce0e18b95bc") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 15:54:58.923173 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:58.923139 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1c358fa6-a923-4589-953f-bd49ddd4235c-original-pull-secret\") pod \"global-pull-secret-syncer-95zvh\" (UID: \"1c358fa6-a923-4589-953f-bd49ddd4235c\") " pod="kube-system/global-pull-secret-syncer-95zvh" Sep 11 15:54:58.923331 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:54:58.923301 2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 15:54:58.923389 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:54:58.923360 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c358fa6-a923-4589-953f-bd49ddd4235c-original-pull-secret podName:1c358fa6-a923-4589-953f-bd49ddd4235c nodeName:}" failed. No retries permitted until 2026-09-11 15:55:02.923342424 +0000 UTC m=+10.138176310 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1c358fa6-a923-4589-953f-bd49ddd4235c-original-pull-secret") pod "global-pull-secret-syncer-95zvh" (UID: "1c358fa6-a923-4589-953f-bd49ddd4235c") : object "kube-system"/"original-pull-secret" not registered Sep 11 15:54:59.024834 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:59.024086 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9g7p\" (UniqueName: \"kubernetes.io/projected/f6c05fce-e71f-4a9c-b219-a77b1df84ac8-kube-api-access-h9g7p\") pod \"network-check-target-nzcjl\" (UID: \"f6c05fce-e71f-4a9c-b219-a77b1df84ac8\") " pod="openshift-network-diagnostics/network-check-target-nzcjl" Sep 11 15:54:59.024834 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:54:59.024267 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 15:54:59.024834 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:54:59.024289 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 15:54:59.024834 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:54:59.024302 2566 projected.go:196] Error preparing data for projected volume kube-api-access-h9g7p for pod openshift-network-diagnostics/network-check-target-nzcjl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 15:54:59.024834 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:54:59.024358 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f6c05fce-e71f-4a9c-b219-a77b1df84ac8-kube-api-access-h9g7p podName:f6c05fce-e71f-4a9c-b219-a77b1df84ac8 nodeName:}" failed. No retries permitted until 2026-09-11 15:55:03.024340788 +0000 UTC m=+10.239174700 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-h9g7p" (UniqueName: "kubernetes.io/projected/f6c05fce-e71f-4a9c-b219-a77b1df84ac8-kube-api-access-h9g7p") pod "network-check-target-nzcjl" (UID: "f6c05fce-e71f-4a9c-b219-a77b1df84ac8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 15:54:59.233287 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:59.233203 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nzcjl" Sep 11 15:54:59.233287 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:59.233225 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-95zvh" Sep 11 15:54:59.233511 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:54:59.233327 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nzcjl" podUID="f6c05fce-e71f-4a9c-b219-a77b1df84ac8" Sep 11 15:54:59.233511 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:54:59.233414 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-95zvh" podUID="1c358fa6-a923-4589-953f-bd49ddd4235c" Sep 11 15:54:59.233511 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:54:59.233444 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-plkhx" Sep 11 15:54:59.233649 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:54:59.233541 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-plkhx" podUID="331632d0-fa59-4462-805e-bce0e18b95bc" Sep 11 15:55:00.287271 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:00.287228 2566 generic.go:356] "Generic (PLEG): container finished" podID="e46b070927ba07e09c7fdfca6f7f4f91" containerID="e5c9f21edbcf0dcbf152154f5207c8af1f31df4addd7baefbeec5566788c3a03" exitCode=0 Sep 11 15:55:00.287709 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:00.287281 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-96.ec2.internal" event={"ID":"e46b070927ba07e09c7fdfca6f7f4f91","Type":"ContainerDied","Data":"e5c9f21edbcf0dcbf152154f5207c8af1f31df4addd7baefbeec5566788c3a03"} Sep 11 15:55:01.233103 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:01.233072 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-plkhx" Sep 11 15:55:01.233103 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:01.233106 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-95zvh" Sep 11 15:55:01.233332 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:01.233074 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nzcjl" Sep 11 15:55:01.233332 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:01.233197 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-plkhx" podUID="331632d0-fa59-4462-805e-bce0e18b95bc" Sep 11 15:55:01.233332 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:01.233293 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nzcjl" podUID="f6c05fce-e71f-4a9c-b219-a77b1df84ac8" Sep 11 15:55:01.233457 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:01.233390 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-95zvh" podUID="1c358fa6-a923-4589-953f-bd49ddd4235c" Sep 11 15:55:02.857151 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:02.857105 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/331632d0-fa59-4462-805e-bce0e18b95bc-metrics-certs\") pod \"network-metrics-daemon-plkhx\" (UID: \"331632d0-fa59-4462-805e-bce0e18b95bc\") " pod="openshift-multus/network-metrics-daemon-plkhx" Sep 11 15:55:02.857552 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:02.857269 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 15:55:02.857552 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:02.857334 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/331632d0-fa59-4462-805e-bce0e18b95bc-metrics-certs podName:331632d0-fa59-4462-805e-bce0e18b95bc nodeName:}" failed. No retries permitted until 2026-09-11 15:55:10.857313451 +0000 UTC m=+18.072147354 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/331632d0-fa59-4462-805e-bce0e18b95bc-metrics-certs") pod "network-metrics-daemon-plkhx" (UID: "331632d0-fa59-4462-805e-bce0e18b95bc") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 15:55:02.958726 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:02.958125 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1c358fa6-a923-4589-953f-bd49ddd4235c-original-pull-secret\") pod \"global-pull-secret-syncer-95zvh\" (UID: \"1c358fa6-a923-4589-953f-bd49ddd4235c\") " pod="kube-system/global-pull-secret-syncer-95zvh" Sep 11 15:55:02.958726 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:02.958316 2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 15:55:02.958726 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:02.958379 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c358fa6-a923-4589-953f-bd49ddd4235c-original-pull-secret podName:1c358fa6-a923-4589-953f-bd49ddd4235c nodeName:}" failed. No retries permitted until 2026-09-11 15:55:10.958360038 +0000 UTC m=+18.173193924 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1c358fa6-a923-4589-953f-bd49ddd4235c-original-pull-secret") pod "global-pull-secret-syncer-95zvh" (UID: "1c358fa6-a923-4589-953f-bd49ddd4235c") : object "kube-system"/"original-pull-secret" not registered Sep 11 15:55:03.058619 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:03.058581 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9g7p\" (UniqueName: \"kubernetes.io/projected/f6c05fce-e71f-4a9c-b219-a77b1df84ac8-kube-api-access-h9g7p\") pod \"network-check-target-nzcjl\" (UID: \"f6c05fce-e71f-4a9c-b219-a77b1df84ac8\") " pod="openshift-network-diagnostics/network-check-target-nzcjl" Sep 11 15:55:03.058797 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:03.058735 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 15:55:03.058797 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:03.058757 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 15:55:03.058797 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:03.058767 2566 projected.go:196] Error preparing data for projected volume kube-api-access-h9g7p for pod openshift-network-diagnostics/network-check-target-nzcjl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 15:55:03.058918 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:03.058831 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f6c05fce-e71f-4a9c-b219-a77b1df84ac8-kube-api-access-h9g7p podName:f6c05fce-e71f-4a9c-b219-a77b1df84ac8 nodeName:}" failed. No retries permitted until 2026-09-11 15:55:11.058811304 +0000 UTC m=+18.273645210 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-h9g7p" (UniqueName: "kubernetes.io/projected/f6c05fce-e71f-4a9c-b219-a77b1df84ac8-kube-api-access-h9g7p") pod "network-check-target-nzcjl" (UID: "f6c05fce-e71f-4a9c-b219-a77b1df84ac8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 15:55:03.235043 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:03.234436 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nzcjl" Sep 11 15:55:03.235043 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:03.234611 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nzcjl" podUID="f6c05fce-e71f-4a9c-b219-a77b1df84ac8" Sep 11 15:55:03.235256 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:03.235191 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-95zvh" Sep 11 15:55:03.235256 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:03.235248 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-plkhx" Sep 11 15:55:03.236025 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:03.235477 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-plkhx" podUID="331632d0-fa59-4462-805e-bce0e18b95bc" Sep 11 15:55:03.236025 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:03.235651 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-95zvh" podUID="1c358fa6-a923-4589-953f-bd49ddd4235c" Sep 11 15:55:05.232986 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:05.232931 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-plkhx" Sep 11 15:55:05.233494 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:05.233092 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-plkhx" podUID="331632d0-fa59-4462-805e-bce0e18b95bc" Sep 11 15:55:05.233494 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:05.233117 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nzcjl" Sep 11 15:55:05.233494 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:05.233172 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-95zvh" Sep 11 15:55:05.233494 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:05.233265 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-95zvh" podUID="1c358fa6-a923-4589-953f-bd49ddd4235c" Sep 11 15:55:05.233494 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:05.233349 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nzcjl" podUID="f6c05fce-e71f-4a9c-b219-a77b1df84ac8" Sep 11 15:55:07.232800 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:07.232696 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nzcjl" Sep 11 15:55:07.232800 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:07.232724 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-plkhx" Sep 11 15:55:07.232800 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:07.232753 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-95zvh" Sep 11 15:55:07.233325 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:07.232889 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nzcjl" podUID="f6c05fce-e71f-4a9c-b219-a77b1df84ac8" Sep 11 15:55:07.233325 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:07.233027 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-plkhx" podUID="331632d0-fa59-4462-805e-bce0e18b95bc" Sep 11 15:55:07.233325 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:07.233120 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-95zvh" podUID="1c358fa6-a923-4589-953f-bd49ddd4235c" Sep 11 15:55:09.233185 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:09.233148 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nzcjl" Sep 11 15:55:09.233604 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:09.233151 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-plkhx" Sep 11 15:55:09.233604 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:09.233273 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nzcjl" podUID="f6c05fce-e71f-4a9c-b219-a77b1df84ac8" Sep 11 15:55:09.233604 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:09.233385 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-plkhx" podUID="331632d0-fa59-4462-805e-bce0e18b95bc" Sep 11 15:55:09.233604 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:09.233151 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-95zvh" Sep 11 15:55:09.233604 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:09.233482 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-95zvh" podUID="1c358fa6-a923-4589-953f-bd49ddd4235c" Sep 11 15:55:10.919415 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:10.919375 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/331632d0-fa59-4462-805e-bce0e18b95bc-metrics-certs\") pod \"network-metrics-daemon-plkhx\" (UID: \"331632d0-fa59-4462-805e-bce0e18b95bc\") " pod="openshift-multus/network-metrics-daemon-plkhx" Sep 11 15:55:10.919927 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:10.919552 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 15:55:10.919927 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:10.919648 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/331632d0-fa59-4462-805e-bce0e18b95bc-metrics-certs podName:331632d0-fa59-4462-805e-bce0e18b95bc nodeName:}" failed. No retries permitted until 2026-09-11 15:55:26.919624562 +0000 UTC m=+34.134458461 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/331632d0-fa59-4462-805e-bce0e18b95bc-metrics-certs") pod "network-metrics-daemon-plkhx" (UID: "331632d0-fa59-4462-805e-bce0e18b95bc") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 15:55:11.020187 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:11.020143 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1c358fa6-a923-4589-953f-bd49ddd4235c-original-pull-secret\") pod \"global-pull-secret-syncer-95zvh\" (UID: \"1c358fa6-a923-4589-953f-bd49ddd4235c\") " pod="kube-system/global-pull-secret-syncer-95zvh" Sep 11 15:55:11.020370 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:11.020308 2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 15:55:11.020430 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:11.020393 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c358fa6-a923-4589-953f-bd49ddd4235c-original-pull-secret podName:1c358fa6-a923-4589-953f-bd49ddd4235c nodeName:}" failed. No retries permitted until 2026-09-11 15:55:27.020373802 +0000 UTC m=+34.235207686 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1c358fa6-a923-4589-953f-bd49ddd4235c-original-pull-secret") pod "global-pull-secret-syncer-95zvh" (UID: "1c358fa6-a923-4589-953f-bd49ddd4235c") : object "kube-system"/"original-pull-secret" not registered Sep 11 15:55:11.120638 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:11.120600 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9g7p\" (UniqueName: \"kubernetes.io/projected/f6c05fce-e71f-4a9c-b219-a77b1df84ac8-kube-api-access-h9g7p\") pod \"network-check-target-nzcjl\" (UID: \"f6c05fce-e71f-4a9c-b219-a77b1df84ac8\") " pod="openshift-network-diagnostics/network-check-target-nzcjl" Sep 11 15:55:11.120821 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:11.120785 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 15:55:11.120821 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:11.120812 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 15:55:11.120896 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:11.120823 2566 projected.go:196] Error preparing data for projected volume kube-api-access-h9g7p for pod openshift-network-diagnostics/network-check-target-nzcjl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 15:55:11.120896 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:11.120880 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f6c05fce-e71f-4a9c-b219-a77b1df84ac8-kube-api-access-h9g7p podName:f6c05fce-e71f-4a9c-b219-a77b1df84ac8 nodeName:}" failed. No retries permitted until 2026-09-11 15:55:27.120866757 +0000 UTC m=+34.335700639 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-h9g7p" (UniqueName: "kubernetes.io/projected/f6c05fce-e71f-4a9c-b219-a77b1df84ac8-kube-api-access-h9g7p") pod "network-check-target-nzcjl" (UID: "f6c05fce-e71f-4a9c-b219-a77b1df84ac8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 15:55:11.233112 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:11.233030 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-95zvh" Sep 11 15:55:11.233273 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:11.233037 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-plkhx" Sep 11 15:55:11.233273 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:11.233159 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-95zvh" podUID="1c358fa6-a923-4589-953f-bd49ddd4235c" Sep 11 15:55:11.233375 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:11.233267 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-plkhx" podUID="331632d0-fa59-4462-805e-bce0e18b95bc" Sep 11 15:55:11.233375 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:11.233037 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nzcjl" Sep 11 15:55:11.233375 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:11.233361 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nzcjl" podUID="f6c05fce-e71f-4a9c-b219-a77b1df84ac8" Sep 11 15:55:13.234064 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:13.234025 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-95zvh" Sep 11 15:55:13.234519 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:13.234138 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-plkhx" Sep 11 15:55:13.234519 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:13.234146 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-95zvh" podUID="1c358fa6-a923-4589-953f-bd49ddd4235c" Sep 11 15:55:13.234519 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:13.234261 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-plkhx" podUID="331632d0-fa59-4462-805e-bce0e18b95bc" Sep 11 15:55:13.234519 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:13.234292 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nzcjl" Sep 11 15:55:13.234519 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:13.234379 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nzcjl" podUID="f6c05fce-e71f-4a9c-b219-a77b1df84ac8" Sep 11 15:55:14.317836 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:14.317474 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-96.ec2.internal" event={"ID":"e46b070927ba07e09c7fdfca6f7f4f91","Type":"ContainerStarted","Data":"de1aae80df511c239476a00b524f7319f24c8e51802fef47f79d9321d1a4c707"} Sep 11 15:55:14.319307 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:14.319276 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bctd5" event={"ID":"249a0e7e-1da6-422a-8492-1820f6437318","Type":"ContainerStarted","Data":"c0fb1c2a427caf976e432ab18862c9ca434e4f796f209b0b76df875d7f8e182e"} Sep 11 15:55:14.320812 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:14.320783 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xkvw8" event={"ID":"e05877f1-2ad9-43d5-8523-4749d6b08782","Type":"ContainerStarted","Data":"f3e45b0a2abfb7faa303e149bedb4f75a1b915a89e0230439a5563677fc14766"} Sep 11 15:55:14.322554 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:14.322523 2566 generic.go:356] "Generic (PLEG): container finished" podID="fe6dccde-412f-483d-a914-5d46e9922b69" containerID="6991acd8c86fb6b3ae21f9670eb59684761931c8e7ec3ffb8ba9ee31379e2615" exitCode=0 Sep 11 15:55:14.322661 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:14.322559 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b7ftm" event={"ID":"fe6dccde-412f-483d-a914-5d46e9922b69","Type":"ContainerDied","Data":"6991acd8c86fb6b3ae21f9670eb59684761931c8e7ec3ffb8ba9ee31379e2615"} Sep 11 15:55:14.324406 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:14.324369 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4vhlr" event={"ID":"bba741b5-3455-493b-a0df-5e7177b832d8","Type":"ContainerStarted","Data":"d34596d3c51bd5aa858e83b0fda05403a7e522c94c84d26c04c3a6473b700642"} Sep 11 15:55:14.326470 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:14.326442 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-sxqj2" event={"ID":"ed90592a-177c-4128-b153-b51d18b8f619","Type":"ContainerStarted","Data":"bdadf84429761e0a8a07e8101f33729228deac52ee5da605f0f6aed51c47662a"} Sep 11 15:55:14.328046 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:14.328001 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-spxb6" event={"ID":"94ab0396-7020-4ad3-b1b7-63cd05490e84","Type":"ContainerStarted","Data":"01d3cefa886f21488437e8dc89367674cd36ee5f09950436a9fffeeb4d07dbf9"} Sep 11 15:55:14.331432 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:14.331413 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tgq5c_981d7d1e-f81f-48fc-8962-7831bc5c2b27/ovn-acl-logging/0.log" Sep 11 15:55:14.331809 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:14.331783 2566 generic.go:356] "Generic (PLEG): container finished" podID="981d7d1e-f81f-48fc-8962-7831bc5c2b27" containerID="880d46347e055d61250e59c1e9b2060667f1e459f90f588fd9904c14c5c9d79d" exitCode=1 Sep 11 15:55:14.331905 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:14.331851 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tgq5c" event={"ID":"981d7d1e-f81f-48fc-8962-7831bc5c2b27","Type":"ContainerStarted","Data":"878fac889189b4654d6afecf774fda1d2f673b147e84a61414042ffd825ad42f"} Sep 11 15:55:14.331946 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:14.331925 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tgq5c" event={"ID":"981d7d1e-f81f-48fc-8962-7831bc5c2b27","Type":"ContainerStarted","Data":"85039b0f1258c7e70b515aa5a65d6c33adea9beade899e43c3054576bd5ac008"} Sep 11 15:55:14.332016 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:14.332000 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tgq5c" event={"ID":"981d7d1e-f81f-48fc-8962-7831bc5c2b27","Type":"ContainerStarted","Data":"b92c8bf5cb10870991858d181c7f64beedc4c605d4b01cd2e29e1f48a0a3b630"} Sep 11 15:55:14.332060 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:14.332042 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tgq5c" event={"ID":"981d7d1e-f81f-48fc-8962-7831bc5c2b27","Type":"ContainerStarted","Data":"463e8f251050ec7752f13d0088867876e32d92e41c24f2c9d5667fba93e59858"} Sep 11 15:55:14.332089 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:14.332075 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tgq5c" event={"ID":"981d7d1e-f81f-48fc-8962-7831bc5c2b27","Type":"ContainerDied","Data":"880d46347e055d61250e59c1e9b2060667f1e459f90f588fd9904c14c5c9d79d"} Sep 11 15:55:14.332130 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:14.332123 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tgq5c" event={"ID":"981d7d1e-f81f-48fc-8962-7831bc5c2b27","Type":"ContainerStarted","Data":"94996ac9c419a60f29c6fb049f2ed89d4b4484be5bafcba98c4407749ca79292"} Sep 11 15:55:14.335336 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:14.335280 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-96.ec2.internal" podStartSLOduration=20.335262056 podStartE2EDuration="20.335262056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 15:54:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:55:14.334540651 +0000 UTC m=+21.549374557" watchObservedRunningTime="2026-09-11 15:55:14.335262056 +0000 UTC m=+21.550095961" Sep 11 15:55:14.335812 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:14.335788 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-sn7js" event={"ID":"eab992d2-f32f-41bc-a028-c3fa544c4fec","Type":"ContainerStarted","Data":"d659774cc8167b8aba2b63087b30e36b4e069ddfd71dc3bb3fd1a7b351c9148c"} Sep 11 15:55:14.349917 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:14.349852 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-sxqj2" podStartSLOduration=3.436114185 podStartE2EDuration="21.349836071s" totalImagesPullingTime="17.913721886s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 15:54:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:55:14.349692796 +0000 UTC m=+21.564526700" watchObservedRunningTime="2026-09-11 15:55:14.349836071 +0000 UTC m=+21.564669976" Sep 11 15:55:14.367733 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:14.367674 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-4vhlr" podStartSLOduration=2.915677772 podStartE2EDuration="21.367656681s" totalImagesPullingTime="18.451978909s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 15:54:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:55:14.367317979 +0000 UTC m=+21.582151883" watchObservedRunningTime="2026-09-11 15:55:14.367656681 +0000 UTC m=+21.582490585" Sep 11 15:55:14.383065 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:14.383016 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-bctd5" podStartSLOduration=3.491546891 podStartE2EDuration="21.382994055s" totalImagesPullingTime="17.891447164s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 15:54:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:55:14.382656048 +0000 UTC m=+21.597489953" watchObservedRunningTime="2026-09-11 15:55:14.382994055 +0000 UTC m=+21.597827953" Sep 11 15:55:14.400054 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:14.400003 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-spxb6" podStartSLOduration=3.4738671500000002 podStartE2EDuration="21.399965459s" totalImagesPullingTime="17.926098309s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:54:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:55:14.399672103 +0000 UTC m=+21.614506007" watchObservedRunningTime="2026-09-11 15:55:14.399965459 +0000 UTC m=+21.614799365" Sep 11 15:55:14.442098 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:14.442046 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-sn7js" podStartSLOduration=3.4960295009999998 podStartE2EDuration="21.442029853s" totalImagesPullingTime="17.946000352s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 15:54:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:55:14.441763038 +0000 UTC m=+21.656596940" watchObservedRunningTime="2026-09-11 15:55:14.442029853 +0000 UTC m=+21.656863757" Sep 11 15:55:14.790951 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:14.790929 2566 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 11 15:55:15.233279 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:15.233245 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-plkhx" Sep 11 15:55:15.233541 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:15.233254 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nzcjl" Sep 11 15:55:15.233541 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:15.233389 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-plkhx" podUID="331632d0-fa59-4462-805e-bce0e18b95bc" Sep 11 15:55:15.233541 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:15.233254 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-95zvh" Sep 11 15:55:15.233541 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:15.233450 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nzcjl" podUID="f6c05fce-e71f-4a9c-b219-a77b1df84ac8" Sep 11 15:55:15.233692 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:15.233544 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-95zvh" podUID="1c358fa6-a923-4589-953f-bd49ddd4235c" Sep 11 15:55:15.249522 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:15.249412 2566 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-11T15:55:14.790947862Z","UUID":"ac3b8b68-3263-47c0-8045-d767e3684d75","Handler":null,"Name":"","Endpoint":""} Sep 11 15:55:15.251289 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:15.251268 2566 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 11 15:55:15.251432 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:15.251305 2566 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 11 15:55:15.339531 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:15.339499 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5hlvj" event={"ID":"b02518a9-32bc-4627-97c8-4535c8333875","Type":"ContainerStarted","Data":"f6edf6beb77e215d37f24600a984543792aba2e641a92f5ffcbf001a1c588444"} Sep 11 15:55:15.341498 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:15.341468 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xkvw8" event={"ID":"e05877f1-2ad9-43d5-8523-4749d6b08782","Type":"ContainerStarted","Data":"9253492807bc99bcc53e786a26c486c42354bd7c61e7f9d3b2049e30312c355c"} Sep 11 15:55:15.362531 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:15.362481 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-5hlvj" podStartSLOduration=4.482277284 podStartE2EDuration="22.362465882s" totalImagesPullingTime="17.880188598s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 15:54:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:55:15.36222582 +0000 UTC m=+22.577059725" watchObservedRunningTime="2026-09-11 15:55:15.362465882 +0000 UTC m=+22.577299786" Sep 11 15:55:15.631073 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:15.630934 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-spxb6" Sep 11 15:55:15.631583 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:15.631565 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-spxb6" Sep 11 15:55:16.346753 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:16.346711 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xkvw8" event={"ID":"e05877f1-2ad9-43d5-8523-4749d6b08782","Type":"ContainerStarted","Data":"bae358679d62006d1a3d6c1b75ca6fb2134ff9993040088aa5f08daeb4686096"} Sep 11 15:55:16.350703 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:16.350683 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tgq5c_981d7d1e-f81f-48fc-8962-7831bc5c2b27/ovn-acl-logging/0.log" Sep 11 15:55:16.351151 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:16.351126 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tgq5c" event={"ID":"981d7d1e-f81f-48fc-8962-7831bc5c2b27","Type":"ContainerStarted","Data":"089579ca6de1508631993da14c6ae3ee86b2dcd8fe62acfbcc7973a3193dd4d0"} Sep 11 15:55:16.351383 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:16.351359 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-spxb6" Sep 11 15:55:16.351878 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:16.351860 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-spxb6" Sep 11 15:55:16.367367 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:16.367316 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xkvw8" podStartSLOduration=3.2351737050000002 podStartE2EDuration="23.367300162s" totalImagesPullingTime="20.132126457s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 15:54:53 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:55:16.366938428 +0000 UTC m=+23.581772333" watchObservedRunningTime="2026-09-11 15:55:16.367300162 +0000 UTC m=+23.582134067" Sep 11 15:55:17.233065 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:17.233004 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nzcjl" Sep 11 15:55:17.233065 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:17.233025 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-plkhx" Sep 11 15:55:17.233466 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:17.233121 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nzcjl" podUID="f6c05fce-e71f-4a9c-b219-a77b1df84ac8" Sep 11 15:55:17.233466 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:17.233004 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-95zvh" Sep 11 15:55:17.233466 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:17.233282 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-plkhx" podUID="331632d0-fa59-4462-805e-bce0e18b95bc" Sep 11 15:55:17.233466 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:17.233388 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-95zvh" podUID="1c358fa6-a923-4589-953f-bd49ddd4235c" Sep 11 15:55:19.233337 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:19.233120 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nzcjl" Sep 11 15:55:19.233826 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:19.233130 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-95zvh" Sep 11 15:55:19.233826 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:19.233442 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nzcjl" podUID="f6c05fce-e71f-4a9c-b219-a77b1df84ac8" Sep 11 15:55:19.233826 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:19.233194 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-plkhx" Sep 11 15:55:19.233826 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:19.233517 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-95zvh" podUID="1c358fa6-a923-4589-953f-bd49ddd4235c" Sep 11 15:55:19.233826 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:19.233641 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-plkhx" podUID="331632d0-fa59-4462-805e-bce0e18b95bc" Sep 11 15:55:20.362507 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:20.362472 2566 generic.go:356] "Generic (PLEG): container finished" podID="fe6dccde-412f-483d-a914-5d46e9922b69" containerID="981bab7eba2570cd879bbb1b9278db2dd9f3d5bf13dc3c038c849f9fb564f1f1" exitCode=0 Sep 11 15:55:20.363463 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:20.362532 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b7ftm" event={"ID":"fe6dccde-412f-483d-a914-5d46e9922b69","Type":"ContainerDied","Data":"981bab7eba2570cd879bbb1b9278db2dd9f3d5bf13dc3c038c849f9fb564f1f1"} Sep 11 15:55:20.366211 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:20.366191 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tgq5c_981d7d1e-f81f-48fc-8962-7831bc5c2b27/ovn-acl-logging/0.log" Sep 11 15:55:20.366601 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:20.366572 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tgq5c" event={"ID":"981d7d1e-f81f-48fc-8962-7831bc5c2b27","Type":"ContainerStarted","Data":"6c069e162842249391b1f3792b1eb6d2066ee2312f9c6b319460cc2b12d3e934"} Sep 11 15:55:20.366840 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:20.366824 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tgq5c" Sep 11 15:55:20.370487 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:20.367350 2566 scope.go:117] "RemoveContainer" containerID="880d46347e055d61250e59c1e9b2060667f1e459f90f588fd9904c14c5c9d79d" Sep 11 15:55:20.383414 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:20.383392 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tgq5c" Sep 11 15:55:21.233529 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:21.233306 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nzcjl" Sep 11 15:55:21.233733 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:21.233315 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-95zvh" Sep 11 15:55:21.233733 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:21.233634 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nzcjl" podUID="f6c05fce-e71f-4a9c-b219-a77b1df84ac8" Sep 11 15:55:21.233733 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:21.233367 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-plkhx" Sep 11 15:55:21.233733 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:21.233714 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-95zvh" podUID="1c358fa6-a923-4589-953f-bd49ddd4235c" Sep 11 15:55:21.233961 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:21.233812 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-plkhx" podUID="331632d0-fa59-4462-805e-bce0e18b95bc" Sep 11 15:55:21.372801 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:21.372777 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tgq5c_981d7d1e-f81f-48fc-8962-7831bc5c2b27/ovn-acl-logging/0.log" Sep 11 15:55:21.373272 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:21.373250 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tgq5c" event={"ID":"981d7d1e-f81f-48fc-8962-7831bc5c2b27","Type":"ContainerStarted","Data":"d02934c91c74130d82563aaaf8d229eb8e08b9a02d6b8413293cf1964cd5786b"} Sep 11 15:55:21.373463 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:21.373436 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tgq5c" Sep 11 15:55:21.373532 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:21.373515 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tgq5c" Sep 11 15:55:21.375587 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:21.375565 2566 generic.go:356] "Generic (PLEG): container finished" podID="fe6dccde-412f-483d-a914-5d46e9922b69" containerID="4724c6a1c596477b7cb37575cfe0f16d25c22ffe7e4e588eab6c67bd024eba7e" exitCode=0 Sep 11 15:55:21.375687 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:21.375600 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b7ftm" event={"ID":"fe6dccde-412f-483d-a914-5d46e9922b69","Type":"ContainerDied","Data":"4724c6a1c596477b7cb37575cfe0f16d25c22ffe7e4e588eab6c67bd024eba7e"} Sep 11 15:55:21.388679 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:21.388652 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tgq5c" Sep 11 15:55:21.401345 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:21.401297 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-tgq5c" podStartSLOduration=10.420237442 podStartE2EDuration="28.40127699s" totalImagesPullingTime="17.981039548s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 15:54:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:55:21.400881585 +0000 UTC m=+28.615715511" watchObservedRunningTime="2026-09-11 15:55:21.40127699 +0000 UTC m=+28.616110895" Sep 11 15:55:21.409349 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:21.409321 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-95zvh"] Sep 11 15:55:21.409493 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:21.409421 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-95zvh" Sep 11 15:55:21.409559 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:21.409514 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-95zvh" podUID="1c358fa6-a923-4589-953f-bd49ddd4235c" Sep 11 15:55:21.412912 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:21.412888 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-nzcjl"] Sep 11 15:55:21.413053 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:21.413009 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nzcjl" Sep 11 15:55:21.413197 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:21.413123 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nzcjl" podUID="f6c05fce-e71f-4a9c-b219-a77b1df84ac8" Sep 11 15:55:21.413685 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:21.413662 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-plkhx"] Sep 11 15:55:21.413767 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:21.413754 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-plkhx" Sep 11 15:55:21.413885 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:21.413865 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-plkhx" podUID="331632d0-fa59-4462-805e-bce0e18b95bc" Sep 11 15:55:22.380547 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:22.380461 2566 generic.go:356] "Generic (PLEG): container finished" podID="fe6dccde-412f-483d-a914-5d46e9922b69" containerID="d8daa6007725c563c0ce72b7affc9108e6890a9ff160b91d78283e34c26a9af1" exitCode=0 Sep 11 15:55:22.380991 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:22.380546 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b7ftm" event={"ID":"fe6dccde-412f-483d-a914-5d46e9922b69","Type":"ContainerDied","Data":"d8daa6007725c563c0ce72b7affc9108e6890a9ff160b91d78283e34c26a9af1"} Sep 11 15:55:23.234031 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:23.233999 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nzcjl" Sep 11 15:55:23.234236 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:23.234077 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-plkhx" Sep 11 15:55:23.234236 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:23.234115 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-95zvh" Sep 11 15:55:23.234236 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:23.234125 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nzcjl" podUID="f6c05fce-e71f-4a9c-b219-a77b1df84ac8" Sep 11 15:55:23.234236 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:23.234189 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-95zvh" podUID="1c358fa6-a923-4589-953f-bd49ddd4235c" Sep 11 15:55:23.234433 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:23.234278 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-plkhx" podUID="331632d0-fa59-4462-805e-bce0e18b95bc" Sep 11 15:55:25.232713 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:25.232620 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-plkhx" Sep 11 15:55:25.232713 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:25.232663 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-95zvh" Sep 11 15:55:25.233209 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:25.232620 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nzcjl" Sep 11 15:55:25.233209 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:25.232779 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-plkhx" podUID="331632d0-fa59-4462-805e-bce0e18b95bc" Sep 11 15:55:25.233209 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:25.232914 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nzcjl" podUID="f6c05fce-e71f-4a9c-b219-a77b1df84ac8" Sep 11 15:55:25.233209 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:25.233020 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-95zvh" podUID="1c358fa6-a923-4589-953f-bd49ddd4235c" Sep 11 15:55:26.603225 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.603144 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-96.ec2.internal" event="NodeReady" Sep 11 15:55:26.603653 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.603291 2566 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 11 15:55:26.643484 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.643454 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-56c5d9df5d-7vqnb"] Sep 11 15:55:26.682277 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.682243 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-844dfdb99b-766v5"] Sep 11 15:55:26.682437 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.682363 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-56c5d9df5d-7vqnb" Sep 11 15:55:26.696170 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.696138 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-n6wjl\"" Sep 11 15:55:26.699323 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.699304 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 11 15:55:26.703101 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.703082 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-kqtwh"] Sep 11 15:55:26.703227 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.703210 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-844dfdb99b-766v5" Sep 11 15:55:26.705582 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.705564 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 11 15:55:26.707376 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.707355 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 11 15:55:26.707488 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.707356 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Sep 11 15:55:26.710782 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.710764 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Sep 11 15:55:26.711425 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.711405 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Sep 11 15:55:26.711528 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.711423 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 11 15:55:26.713814 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.713799 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 11 15:55:26.715345 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.715329 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Sep 11 15:55:26.715449 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.715365 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-zk78d\"" Sep 11 15:55:26.718934 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.718915 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-qjtlt"] Sep 11 15:55:26.719079 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.719058 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-kqtwh" Sep 11 15:55:26.726772 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.726279 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Sep 11 15:55:26.726772 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.726357 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-gpzqq\"" Sep 11 15:55:26.726772 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.726570 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 11 15:55:26.729350 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.729329 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 11 15:55:26.738589 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.737789 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-djghf"] Sep 11 15:55:26.738589 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.737965 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-qjtlt" Sep 11 15:55:26.745732 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.745506 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Sep 11 15:55:26.746547 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.746482 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Sep 11 15:55:26.753031 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.753006 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-4hdz4"] Sep 11 15:55:26.753175 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.753156 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-djghf" Sep 11 15:55:26.762183 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.762163 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Sep 11 15:55:26.763524 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.763499 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Sep 11 15:55:26.763833 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.763811 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Sep 11 15:55:26.763833 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.763827 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-rvvft\"" Sep 11 15:55:26.764099 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.764081 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Sep 11 15:55:26.771412 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.771388 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5844b856db-mjh4m"] Sep 11 15:55:26.771533 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.771515 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-4hdz4" Sep 11 15:55:26.771987 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.771958 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Sep 11 15:55:26.775430 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.775406 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Sep 11 15:55:26.776334 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.776316 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Sep 11 15:55:26.776691 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.776645 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-m665j\"" Sep 11 15:55:26.777104 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.777086 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Sep 11 15:55:26.777406 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.777388 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Sep 11 15:55:26.777948 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.777892 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-vj99t\"" Sep 11 15:55:26.778579 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.778557 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 11 15:55:26.780021 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.780002 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Sep 11 15:55:26.804322 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.804294 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-t82dg"] Sep 11 15:55:26.804480 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.804438 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-mjh4m" Sep 11 15:55:26.806952 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.806933 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Sep 11 15:55:26.807727 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.807600 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-j55nk\"" Sep 11 15:55:26.807727 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.807667 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 11 15:55:26.807727 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.807687 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 11 15:55:26.808032 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.807963 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Sep 11 15:55:26.815327 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.815309 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Sep 11 15:55:26.820353 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.820332 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-qjtlt"] Sep 11 15:55:26.820446 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.820367 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-djghf"] Sep 11 15:55:26.820446 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.820423 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-56c5d9df5d-7vqnb"] Sep 11 15:55:26.820446 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.820441 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-xjvl5"] Sep 11 15:55:26.820588 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.820571 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-t82dg" Sep 11 15:55:26.822634 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.822614 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 11 15:55:26.822733 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.822716 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-wlz5b\"" Sep 11 15:55:26.823119 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.823105 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 11 15:55:26.832470 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.832452 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-8q8dx"] Sep 11 15:55:26.832597 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.832582 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-xjvl5" Sep 11 15:55:26.835323 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.835304 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Sep 11 15:55:26.835412 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.835341 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Sep 11 15:55:26.835412 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.835399 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-52hb4\"" Sep 11 15:55:26.835691 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.835671 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Sep 11 15:55:26.835691 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.835687 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Sep 11 15:55:26.836991 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.836950 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5429cde4-cb80-4579-9687-e71617b15b9f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-kqtwh\" (UID: \"5429cde4-cb80-4579-9687-e71617b15b9f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-kqtwh" Sep 11 15:55:26.837090 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.837003 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ae9022dc-da99-4acf-af85-3cf9858737b4-image-registry-private-configuration\") pod \"image-registry-56c5d9df5d-7vqnb\" (UID: \"ae9022dc-da99-4acf-af85-3cf9858737b4\") " pod="openshift-image-registry/image-registry-56c5d9df5d-7vqnb" Sep 11 15:55:26.837090 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.837031 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxrcg\" (UniqueName: \"kubernetes.io/projected/ae9022dc-da99-4acf-af85-3cf9858737b4-kube-api-access-jxrcg\") pod \"image-registry-56c5d9df5d-7vqnb\" (UID: \"ae9022dc-da99-4acf-af85-3cf9858737b4\") " pod="openshift-image-registry/image-registry-56c5d9df5d-7vqnb" Sep 11 15:55:26.837090 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.837060 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-699kq\" (UniqueName: \"kubernetes.io/projected/07d36306-7ec8-4c74-9b97-7545503a2783-kube-api-access-699kq\") pod \"kube-storage-version-migrator-operator-6c8f84f897-djghf\" (UID: \"07d36306-7ec8-4c74-9b97-7545503a2783\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-djghf" Sep 11 15:55:26.837090 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.837086 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ae9022dc-da99-4acf-af85-3cf9858737b4-trusted-ca\") pod \"image-registry-56c5d9df5d-7vqnb\" (UID: \"ae9022dc-da99-4acf-af85-3cf9858737b4\") " pod="openshift-image-registry/image-registry-56c5d9df5d-7vqnb" Sep 11 15:55:26.837296 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.837165 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/26aadde0-9d1e-4368-86b0-4a2eb04656c3-config\") pod \"service-ca-operator-cf7c776b7-qjtlt\" (UID: \"26aadde0-9d1e-4368-86b0-4a2eb04656c3\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-qjtlt" Sep 11 15:55:26.837296 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.837189 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/313238e9-79d7-45f7-8d96-beb55feb415a-service-ca-bundle\") pod \"router-default-844dfdb99b-766v5\" (UID: \"313238e9-79d7-45f7-8d96-beb55feb415a\") " pod="openshift-ingress/router-default-844dfdb99b-766v5" Sep 11 15:55:26.837296 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.837218 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ae9022dc-da99-4acf-af85-3cf9858737b4-installation-pull-secrets\") pod \"image-registry-56c5d9df5d-7vqnb\" (UID: \"ae9022dc-da99-4acf-af85-3cf9858737b4\") " pod="openshift-image-registry/image-registry-56c5d9df5d-7vqnb" Sep 11 15:55:26.837441 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.837292 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ae9022dc-da99-4acf-af85-3cf9858737b4-ca-trust-extracted\") pod \"image-registry-56c5d9df5d-7vqnb\" (UID: \"ae9022dc-da99-4acf-af85-3cf9858737b4\") " pod="openshift-image-registry/image-registry-56c5d9df5d-7vqnb" Sep 11 15:55:26.837441 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.837330 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/5429cde4-cb80-4579-9687-e71617b15b9f-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-kqtwh\" (UID: \"5429cde4-cb80-4579-9687-e71617b15b9f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-kqtwh" Sep 11 15:55:26.837441 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.837359 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfmk4\" (UniqueName: \"kubernetes.io/projected/5429cde4-cb80-4579-9687-e71617b15b9f-kube-api-access-sfmk4\") pod \"cluster-monitoring-operator-6cd954b897-kqtwh\" (UID: \"5429cde4-cb80-4579-9687-e71617b15b9f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-kqtwh" Sep 11 15:55:26.837441 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.837393 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ae9022dc-da99-4acf-af85-3cf9858737b4-bound-sa-token\") pod \"image-registry-56c5d9df5d-7vqnb\" (UID: \"ae9022dc-da99-4acf-af85-3cf9858737b4\") " pod="openshift-image-registry/image-registry-56c5d9df5d-7vqnb" Sep 11 15:55:26.837441 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.837419 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/313238e9-79d7-45f7-8d96-beb55feb415a-stats-auth\") pod \"router-default-844dfdb99b-766v5\" (UID: \"313238e9-79d7-45f7-8d96-beb55feb415a\") " pod="openshift-ingress/router-default-844dfdb99b-766v5" Sep 11 15:55:26.837672 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.837447 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tr2kr\" (UniqueName: \"kubernetes.io/projected/313238e9-79d7-45f7-8d96-beb55feb415a-kube-api-access-tr2kr\") pod \"router-default-844dfdb99b-766v5\" (UID: \"313238e9-79d7-45f7-8d96-beb55feb415a\") " pod="openshift-ingress/router-default-844dfdb99b-766v5" Sep 11 15:55:26.837672 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.837510 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdrfw\" (UniqueName: \"kubernetes.io/projected/26aadde0-9d1e-4368-86b0-4a2eb04656c3-kube-api-access-zdrfw\") pod \"service-ca-operator-cf7c776b7-qjtlt\" (UID: \"26aadde0-9d1e-4368-86b0-4a2eb04656c3\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-qjtlt" Sep 11 15:55:26.837672 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.837538 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ae9022dc-da99-4acf-af85-3cf9858737b4-registry-certificates\") pod \"image-registry-56c5d9df5d-7vqnb\" (UID: \"ae9022dc-da99-4acf-af85-3cf9858737b4\") " pod="openshift-image-registry/image-registry-56c5d9df5d-7vqnb" Sep 11 15:55:26.837672 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.837566 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/313238e9-79d7-45f7-8d96-beb55feb415a-default-certificate\") pod \"router-default-844dfdb99b-766v5\" (UID: \"313238e9-79d7-45f7-8d96-beb55feb415a\") " pod="openshift-ingress/router-default-844dfdb99b-766v5" Sep 11 15:55:26.837672 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.837603 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/07d36306-7ec8-4c74-9b97-7545503a2783-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-djghf\" (UID: \"07d36306-7ec8-4c74-9b97-7545503a2783\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-djghf" Sep 11 15:55:26.837672 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.837632 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/26aadde0-9d1e-4368-86b0-4a2eb04656c3-serving-cert\") pod \"service-ca-operator-cf7c776b7-qjtlt\" (UID: \"26aadde0-9d1e-4368-86b0-4a2eb04656c3\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-qjtlt" Sep 11 15:55:26.837672 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.837661 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ae9022dc-da99-4acf-af85-3cf9858737b4-registry-tls\") pod \"image-registry-56c5d9df5d-7vqnb\" (UID: \"ae9022dc-da99-4acf-af85-3cf9858737b4\") " pod="openshift-image-registry/image-registry-56c5d9df5d-7vqnb" Sep 11 15:55:26.837946 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.837698 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/313238e9-79d7-45f7-8d96-beb55feb415a-metrics-certs\") pod \"router-default-844dfdb99b-766v5\" (UID: \"313238e9-79d7-45f7-8d96-beb55feb415a\") " pod="openshift-ingress/router-default-844dfdb99b-766v5" Sep 11 15:55:26.837946 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.837736 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/07d36306-7ec8-4c74-9b97-7545503a2783-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-djghf\" (UID: \"07d36306-7ec8-4c74-9b97-7545503a2783\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-djghf" Sep 11 15:55:26.844769 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.844743 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Sep 11 15:55:26.848491 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.848472 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-ssz6b"] Sep 11 15:55:26.848621 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.848602 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-8q8dx" Sep 11 15:55:26.850900 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.850884 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Sep 11 15:55:26.851531 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.851510 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Sep 11 15:55:26.851628 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.851560 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-lhmcj\"" Sep 11 15:55:26.862124 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.862105 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-mjh4m"] Sep 11 15:55:26.862218 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.862138 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-kqtwh"] Sep 11 15:55:26.862218 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.862161 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-4hdz4"] Sep 11 15:55:26.862325 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.862236 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-ssz6b" Sep 11 15:55:26.862325 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.862269 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-gx9fm"] Sep 11 15:55:26.864853 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.864836 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-jt5ll\"" Sep 11 15:55:26.865214 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.865196 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Sep 11 15:55:26.865300 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.865218 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Sep 11 15:55:26.878477 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.878455 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-844dfdb99b-766v5"] Sep 11 15:55:26.878591 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.878497 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-4kl5b"] Sep 11 15:55:26.878639 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.878583 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gx9fm" Sep 11 15:55:26.880931 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.880914 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 11 15:55:26.882045 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.882022 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 11 15:55:26.882143 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.882030 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-jvv4c\"" Sep 11 15:55:26.889338 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.889316 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-t82dg"] Sep 11 15:55:26.889441 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.889357 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-ssz6b"] Sep 11 15:55:26.889441 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.889374 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-8q8dx"] Sep 11 15:55:26.889441 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.889388 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-xjvl5"] Sep 11 15:55:26.889441 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.889400 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-gx9fm"] Sep 11 15:55:26.889441 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.889417 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-4kl5b"] Sep 11 15:55:26.889664 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.889443 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-4kl5b" Sep 11 15:55:26.892478 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.892462 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 11 15:55:26.892690 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.892537 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-dxw2k\"" Sep 11 15:55:26.892793 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.892567 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 11 15:55:26.892793 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.892749 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 11 15:55:26.939079 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.939043 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/313238e9-79d7-45f7-8d96-beb55feb415a-metrics-certs\") pod \"router-default-844dfdb99b-766v5\" (UID: \"313238e9-79d7-45f7-8d96-beb55feb415a\") " pod="openshift-ingress/router-default-844dfdb99b-766v5" Sep 11 15:55:26.939238 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.939085 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/07d36306-7ec8-4c74-9b97-7545503a2783-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-djghf\" (UID: \"07d36306-7ec8-4c74-9b97-7545503a2783\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-djghf" Sep 11 15:55:26.939238 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.939118 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5429cde4-cb80-4579-9687-e71617b15b9f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-kqtwh\" (UID: \"5429cde4-cb80-4579-9687-e71617b15b9f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-kqtwh" Sep 11 15:55:26.939238 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.939191 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqhgw\" (UniqueName: \"kubernetes.io/projected/7542ad77-c817-4370-91de-3b08db264753-kube-api-access-xqhgw\") pod \"console-operator-696884fb9c-xjvl5\" (UID: \"7542ad77-c817-4370-91de-3b08db264753\") " pod="openshift-console-operator/console-operator-696884fb9c-xjvl5" Sep 11 15:55:26.939393 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.939237 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-699kq\" (UniqueName: \"kubernetes.io/projected/07d36306-7ec8-4c74-9b97-7545503a2783-kube-api-access-699kq\") pod \"kube-storage-version-migrator-operator-6c8f84f897-djghf\" (UID: \"07d36306-7ec8-4c74-9b97-7545503a2783\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-djghf" Sep 11 15:55:26.939393 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.939272 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ae9022dc-da99-4acf-af85-3cf9858737b4-trusted-ca\") pod \"image-registry-56c5d9df5d-7vqnb\" (UID: \"ae9022dc-da99-4acf-af85-3cf9858737b4\") " pod="openshift-image-registry/image-registry-56c5d9df5d-7vqnb" Sep 11 15:55:26.939393 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:26.939245 2566 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 11 15:55:26.939545 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:26.939398 2566 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 11 15:55:26.939545 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.939443 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/313238e9-79d7-45f7-8d96-beb55feb415a-service-ca-bundle\") pod \"router-default-844dfdb99b-766v5\" (UID: \"313238e9-79d7-45f7-8d96-beb55feb415a\") " pod="openshift-ingress/router-default-844dfdb99b-766v5" Sep 11 15:55:26.939545 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:26.939469 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5429cde4-cb80-4579-9687-e71617b15b9f-cluster-monitoring-operator-tls podName:5429cde4-cb80-4579-9687-e71617b15b9f nodeName:}" failed. No retries permitted until 2026-09-11 15:55:27.439449339 +0000 UTC m=+34.654283239 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/5429cde4-cb80-4579-9687-e71617b15b9f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-kqtwh" (UID: "5429cde4-cb80-4579-9687-e71617b15b9f") : secret "cluster-monitoring-operator-tls" not found Sep 11 15:55:26.939545 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.939507 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7542ad77-c817-4370-91de-3b08db264753-config\") pod \"console-operator-696884fb9c-xjvl5\" (UID: \"7542ad77-c817-4370-91de-3b08db264753\") " pod="openshift-console-operator/console-operator-696884fb9c-xjvl5" Sep 11 15:55:26.939545 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:26.939529 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/313238e9-79d7-45f7-8d96-beb55feb415a-metrics-certs podName:313238e9-79d7-45f7-8d96-beb55feb415a nodeName:}" failed. No retries permitted until 2026-09-11 15:55:27.439510183 +0000 UTC m=+34.654344080 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/313238e9-79d7-45f7-8d96-beb55feb415a-metrics-certs") pod "router-default-844dfdb99b-766v5" (UID: "313238e9-79d7-45f7-8d96-beb55feb415a") : secret "router-metrics-certs-default" not found Sep 11 15:55:26.939797 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:26.939553 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/313238e9-79d7-45f7-8d96-beb55feb415a-service-ca-bundle podName:313238e9-79d7-45f7-8d96-beb55feb415a nodeName:}" failed. No retries permitted until 2026-09-11 15:55:27.43954391 +0000 UTC m=+34.654377806 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/313238e9-79d7-45f7-8d96-beb55feb415a-service-ca-bundle") pod "router-default-844dfdb99b-766v5" (UID: "313238e9-79d7-45f7-8d96-beb55feb415a") : configmap references non-existent config key: service-ca.crt Sep 11 15:55:26.939797 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.939587 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ae9022dc-da99-4acf-af85-3cf9858737b4-ca-trust-extracted\") pod \"image-registry-56c5d9df5d-7vqnb\" (UID: \"ae9022dc-da99-4acf-af85-3cf9858737b4\") " pod="openshift-image-registry/image-registry-56c5d9df5d-7vqnb" Sep 11 15:55:26.939797 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.939617 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/5429cde4-cb80-4579-9687-e71617b15b9f-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-kqtwh\" (UID: \"5429cde4-cb80-4579-9687-e71617b15b9f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-kqtwh" Sep 11 15:55:26.939797 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.939644 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfmk4\" (UniqueName: \"kubernetes.io/projected/5429cde4-cb80-4579-9687-e71617b15b9f-kube-api-access-sfmk4\") pod \"cluster-monitoring-operator-6cd954b897-kqtwh\" (UID: \"5429cde4-cb80-4579-9687-e71617b15b9f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-kqtwh" Sep 11 15:55:26.939797 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.939669 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/313238e9-79d7-45f7-8d96-beb55feb415a-stats-auth\") pod \"router-default-844dfdb99b-766v5\" (UID: \"313238e9-79d7-45f7-8d96-beb55feb415a\") " pod="openshift-ingress/router-default-844dfdb99b-766v5" Sep 11 15:55:26.940058 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.939695 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tr2kr\" (UniqueName: \"kubernetes.io/projected/313238e9-79d7-45f7-8d96-beb55feb415a-kube-api-access-tr2kr\") pod \"router-default-844dfdb99b-766v5\" (UID: \"313238e9-79d7-45f7-8d96-beb55feb415a\") " pod="openshift-ingress/router-default-844dfdb99b-766v5" Sep 11 15:55:26.940058 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.939989 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f6c5948-c774-49b6-aaf5-026c7b5db718-trusted-ca-bundle\") pod \"insights-operator-5844b856db-mjh4m\" (UID: \"9f6c5948-c774-49b6-aaf5-026c7b5db718\") " pod="openshift-insights/insights-operator-5844b856db-mjh4m" Sep 11 15:55:26.940058 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.939994 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ae9022dc-da99-4acf-af85-3cf9858737b4-ca-trust-extracted\") pod \"image-registry-56c5d9df5d-7vqnb\" (UID: \"ae9022dc-da99-4acf-af85-3cf9858737b4\") " pod="openshift-image-registry/image-registry-56c5d9df5d-7vqnb" Sep 11 15:55:26.940058 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.940022 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/331632d0-fa59-4462-805e-bce0e18b95bc-metrics-certs\") pod \"network-metrics-daemon-plkhx\" (UID: \"331632d0-fa59-4462-805e-bce0e18b95bc\") " pod="openshift-multus/network-metrics-daemon-plkhx" Sep 11 15:55:26.940058 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.940052 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/9f6c5948-c774-49b6-aaf5-026c7b5db718-snapshots\") pod \"insights-operator-5844b856db-mjh4m\" (UID: \"9f6c5948-c774-49b6-aaf5-026c7b5db718\") " pod="openshift-insights/insights-operator-5844b856db-mjh4m" Sep 11 15:55:26.940309 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.940082 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ae9022dc-da99-4acf-af85-3cf9858737b4-registry-tls\") pod \"image-registry-56c5d9df5d-7vqnb\" (UID: \"ae9022dc-da99-4acf-af85-3cf9858737b4\") " pod="openshift-image-registry/image-registry-56c5d9df5d-7vqnb" Sep 11 15:55:26.940309 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.940107 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7542ad77-c817-4370-91de-3b08db264753-trusted-ca\") pod \"console-operator-696884fb9c-xjvl5\" (UID: \"7542ad77-c817-4370-91de-3b08db264753\") " pod="openshift-console-operator/console-operator-696884fb9c-xjvl5" Sep 11 15:55:26.940309 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:26.940143 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 15:55:26.940309 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.940143 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9g2s\" (UniqueName: \"kubernetes.io/projected/c2a34499-68a0-42b4-b77a-edc8b3b964c4-kube-api-access-b9g2s\") pod \"cluster-samples-operator-86499c89c4-4hdz4\" (UID: \"c2a34499-68a0-42b4-b77a-edc8b3b964c4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-4hdz4" Sep 11 15:55:26.940309 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:26.940193 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/331632d0-fa59-4462-805e-bce0e18b95bc-metrics-certs podName:331632d0-fa59-4462-805e-bce0e18b95bc nodeName:}" failed. No retries permitted until 2026-09-11 15:55:58.940180061 +0000 UTC m=+66.155013974 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/331632d0-fa59-4462-805e-bce0e18b95bc-metrics-certs") pod "network-metrics-daemon-plkhx" (UID: "331632d0-fa59-4462-805e-bce0e18b95bc") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 15:55:26.940309 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:26.940204 2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 11 15:55:26.940309 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.940212 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7542ad77-c817-4370-91de-3b08db264753-serving-cert\") pod \"console-operator-696884fb9c-xjvl5\" (UID: \"7542ad77-c817-4370-91de-3b08db264753\") " pod="openshift-console-operator/console-operator-696884fb9c-xjvl5" Sep 11 15:55:26.940309 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.940242 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f6c5948-c774-49b6-aaf5-026c7b5db718-service-ca-bundle\") pod \"insights-operator-5844b856db-mjh4m\" (UID: \"9f6c5948-c774-49b6-aaf5-026c7b5db718\") " pod="openshift-insights/insights-operator-5844b856db-mjh4m" Sep 11 15:55:26.940309 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:26.940220 2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-56c5d9df5d-7vqnb: secret "image-registry-tls" not found Sep 11 15:55:26.940734 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.940312 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ae9022dc-da99-4acf-af85-3cf9858737b4-image-registry-private-configuration\") pod \"image-registry-56c5d9df5d-7vqnb\" (UID: \"ae9022dc-da99-4acf-af85-3cf9858737b4\") " pod="openshift-image-registry/image-registry-56c5d9df5d-7vqnb" Sep 11 15:55:26.940734 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:26.940331 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ae9022dc-da99-4acf-af85-3cf9858737b4-registry-tls podName:ae9022dc-da99-4acf-af85-3cf9858737b4 nodeName:}" failed. No retries permitted until 2026-09-11 15:55:27.440317083 +0000 UTC m=+34.655150980 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ae9022dc-da99-4acf-af85-3cf9858737b4-registry-tls") pod "image-registry-56c5d9df5d-7vqnb" (UID: "ae9022dc-da99-4acf-af85-3cf9858737b4") : secret "image-registry-tls" not found Sep 11 15:55:26.940734 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.940361 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxrcg\" (UniqueName: \"kubernetes.io/projected/ae9022dc-da99-4acf-af85-3cf9858737b4-kube-api-access-jxrcg\") pod \"image-registry-56c5d9df5d-7vqnb\" (UID: \"ae9022dc-da99-4acf-af85-3cf9858737b4\") " pod="openshift-image-registry/image-registry-56c5d9df5d-7vqnb" Sep 11 15:55:26.940734 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.940372 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/5429cde4-cb80-4579-9687-e71617b15b9f-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-kqtwh\" (UID: \"5429cde4-cb80-4579-9687-e71617b15b9f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-kqtwh" Sep 11 15:55:26.940734 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.940392 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9f6c5948-c774-49b6-aaf5-026c7b5db718-serving-cert\") pod \"insights-operator-5844b856db-mjh4m\" (UID: \"9f6c5948-c774-49b6-aaf5-026c7b5db718\") " pod="openshift-insights/insights-operator-5844b856db-mjh4m" Sep 11 15:55:26.940734 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.940451 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ae9022dc-da99-4acf-af85-3cf9858737b4-trusted-ca\") pod \"image-registry-56c5d9df5d-7vqnb\" (UID: \"ae9022dc-da99-4acf-af85-3cf9858737b4\") " pod="openshift-image-registry/image-registry-56c5d9df5d-7vqnb" Sep 11 15:55:26.940734 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.940509 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g59z4\" (UniqueName: \"kubernetes.io/projected/9f6c5948-c774-49b6-aaf5-026c7b5db718-kube-api-access-g59z4\") pod \"insights-operator-5844b856db-mjh4m\" (UID: \"9f6c5948-c774-49b6-aaf5-026c7b5db718\") " pod="openshift-insights/insights-operator-5844b856db-mjh4m" Sep 11 15:55:26.940734 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.940550 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c2a34499-68a0-42b4-b77a-edc8b3b964c4-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-4hdz4\" (UID: \"c2a34499-68a0-42b4-b77a-edc8b3b964c4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-4hdz4" Sep 11 15:55:26.940734 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.940579 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/09f64aa8-4936-49d3-96f7-9895410e5dda-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-8q8dx\" (UID: \"09f64aa8-4936-49d3-96f7-9895410e5dda\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-8q8dx" Sep 11 15:55:26.940734 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.940620 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/26aadde0-9d1e-4368-86b0-4a2eb04656c3-config\") pod \"service-ca-operator-cf7c776b7-qjtlt\" (UID: \"26aadde0-9d1e-4368-86b0-4a2eb04656c3\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-qjtlt" Sep 11 15:55:26.940734 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.940651 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bd62\" (UniqueName: \"kubernetes.io/projected/00183511-0324-43ff-9a5f-26dfacb9401c-kube-api-access-6bd62\") pod \"network-check-source-6f8b768455-t82dg\" (UID: \"00183511-0324-43ff-9a5f-26dfacb9401c\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-t82dg" Sep 11 15:55:26.940734 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.940681 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ae9022dc-da99-4acf-af85-3cf9858737b4-installation-pull-secrets\") pod \"image-registry-56c5d9df5d-7vqnb\" (UID: \"ae9022dc-da99-4acf-af85-3cf9858737b4\") " pod="openshift-image-registry/image-registry-56c5d9df5d-7vqnb" Sep 11 15:55:26.940734 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.940706 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ae9022dc-da99-4acf-af85-3cf9858737b4-bound-sa-token\") pod \"image-registry-56c5d9df5d-7vqnb\" (UID: \"ae9022dc-da99-4acf-af85-3cf9858737b4\") " pod="openshift-image-registry/image-registry-56c5d9df5d-7vqnb" Sep 11 15:55:26.940734 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.940734 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/26aadde0-9d1e-4368-86b0-4a2eb04656c3-serving-cert\") pod \"service-ca-operator-cf7c776b7-qjtlt\" (UID: \"26aadde0-9d1e-4368-86b0-4a2eb04656c3\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-qjtlt" Sep 11 15:55:26.941331 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.940777 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdrfw\" (UniqueName: \"kubernetes.io/projected/26aadde0-9d1e-4368-86b0-4a2eb04656c3-kube-api-access-zdrfw\") pod \"service-ca-operator-cf7c776b7-qjtlt\" (UID: \"26aadde0-9d1e-4368-86b0-4a2eb04656c3\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-qjtlt" Sep 11 15:55:26.941331 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.940800 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9f6c5948-c774-49b6-aaf5-026c7b5db718-tmp\") pod \"insights-operator-5844b856db-mjh4m\" (UID: \"9f6c5948-c774-49b6-aaf5-026c7b5db718\") " pod="openshift-insights/insights-operator-5844b856db-mjh4m" Sep 11 15:55:26.941331 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.940825 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/09f64aa8-4936-49d3-96f7-9895410e5dda-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-8q8dx\" (UID: \"09f64aa8-4936-49d3-96f7-9895410e5dda\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-8q8dx" Sep 11 15:55:26.941331 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.940852 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ae9022dc-da99-4acf-af85-3cf9858737b4-registry-certificates\") pod \"image-registry-56c5d9df5d-7vqnb\" (UID: \"ae9022dc-da99-4acf-af85-3cf9858737b4\") " pod="openshift-image-registry/image-registry-56c5d9df5d-7vqnb" Sep 11 15:55:26.941331 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.940876 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/313238e9-79d7-45f7-8d96-beb55feb415a-default-certificate\") pod \"router-default-844dfdb99b-766v5\" (UID: \"313238e9-79d7-45f7-8d96-beb55feb415a\") " pod="openshift-ingress/router-default-844dfdb99b-766v5" Sep 11 15:55:26.941331 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.940905 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/07d36306-7ec8-4c74-9b97-7545503a2783-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-djghf\" (UID: \"07d36306-7ec8-4c74-9b97-7545503a2783\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-djghf" Sep 11 15:55:26.941331 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.941176 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/26aadde0-9d1e-4368-86b0-4a2eb04656c3-config\") pod \"service-ca-operator-cf7c776b7-qjtlt\" (UID: \"26aadde0-9d1e-4368-86b0-4a2eb04656c3\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-qjtlt" Sep 11 15:55:26.941643 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.941518 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/07d36306-7ec8-4c74-9b97-7545503a2783-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-djghf\" (UID: \"07d36306-7ec8-4c74-9b97-7545503a2783\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-djghf" Sep 11 15:55:26.942072 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.942041 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ae9022dc-da99-4acf-af85-3cf9858737b4-registry-certificates\") pod \"image-registry-56c5d9df5d-7vqnb\" (UID: \"ae9022dc-da99-4acf-af85-3cf9858737b4\") " pod="openshift-image-registry/image-registry-56c5d9df5d-7vqnb" Sep 11 15:55:26.944608 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.944565 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/26aadde0-9d1e-4368-86b0-4a2eb04656c3-serving-cert\") pod \"service-ca-operator-cf7c776b7-qjtlt\" (UID: \"26aadde0-9d1e-4368-86b0-4a2eb04656c3\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-qjtlt" Sep 11 15:55:26.945034 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.945012 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/07d36306-7ec8-4c74-9b97-7545503a2783-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-djghf\" (UID: \"07d36306-7ec8-4c74-9b97-7545503a2783\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-djghf" Sep 11 15:55:26.945123 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.945031 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ae9022dc-da99-4acf-af85-3cf9858737b4-image-registry-private-configuration\") pod \"image-registry-56c5d9df5d-7vqnb\" (UID: \"ae9022dc-da99-4acf-af85-3cf9858737b4\") " pod="openshift-image-registry/image-registry-56c5d9df5d-7vqnb" Sep 11 15:55:26.945345 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.945321 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ae9022dc-da99-4acf-af85-3cf9858737b4-installation-pull-secrets\") pod \"image-registry-56c5d9df5d-7vqnb\" (UID: \"ae9022dc-da99-4acf-af85-3cf9858737b4\") " pod="openshift-image-registry/image-registry-56c5d9df5d-7vqnb" Sep 11 15:55:26.951727 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.951699 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ae9022dc-da99-4acf-af85-3cf9858737b4-bound-sa-token\") pod \"image-registry-56c5d9df5d-7vqnb\" (UID: \"ae9022dc-da99-4acf-af85-3cf9858737b4\") " pod="openshift-image-registry/image-registry-56c5d9df5d-7vqnb" Sep 11 15:55:26.955816 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.955787 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/313238e9-79d7-45f7-8d96-beb55feb415a-stats-auth\") pod \"router-default-844dfdb99b-766v5\" (UID: \"313238e9-79d7-45f7-8d96-beb55feb415a\") " pod="openshift-ingress/router-default-844dfdb99b-766v5" Sep 11 15:55:26.956213 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.956190 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/313238e9-79d7-45f7-8d96-beb55feb415a-default-certificate\") pod \"router-default-844dfdb99b-766v5\" (UID: \"313238e9-79d7-45f7-8d96-beb55feb415a\") " pod="openshift-ingress/router-default-844dfdb99b-766v5" Sep 11 15:55:26.957902 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.957858 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxrcg\" (UniqueName: \"kubernetes.io/projected/ae9022dc-da99-4acf-af85-3cf9858737b4-kube-api-access-jxrcg\") pod \"image-registry-56c5d9df5d-7vqnb\" (UID: \"ae9022dc-da99-4acf-af85-3cf9858737b4\") " pod="openshift-image-registry/image-registry-56c5d9df5d-7vqnb" Sep 11 15:55:26.958450 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.958420 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-699kq\" (UniqueName: \"kubernetes.io/projected/07d36306-7ec8-4c74-9b97-7545503a2783-kube-api-access-699kq\") pod \"kube-storage-version-migrator-operator-6c8f84f897-djghf\" (UID: \"07d36306-7ec8-4c74-9b97-7545503a2783\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-djghf" Sep 11 15:55:26.958893 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.958849 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfmk4\" (UniqueName: \"kubernetes.io/projected/5429cde4-cb80-4579-9687-e71617b15b9f-kube-api-access-sfmk4\") pod \"cluster-monitoring-operator-6cd954b897-kqtwh\" (UID: \"5429cde4-cb80-4579-9687-e71617b15b9f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-kqtwh" Sep 11 15:55:26.958993 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.958910 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tr2kr\" (UniqueName: \"kubernetes.io/projected/313238e9-79d7-45f7-8d96-beb55feb415a-kube-api-access-tr2kr\") pod \"router-default-844dfdb99b-766v5\" (UID: \"313238e9-79d7-45f7-8d96-beb55feb415a\") " pod="openshift-ingress/router-default-844dfdb99b-766v5" Sep 11 15:55:26.966520 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:26.966497 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdrfw\" (UniqueName: \"kubernetes.io/projected/26aadde0-9d1e-4368-86b0-4a2eb04656c3-kube-api-access-zdrfw\") pod \"service-ca-operator-cf7c776b7-qjtlt\" (UID: \"26aadde0-9d1e-4368-86b0-4a2eb04656c3\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-qjtlt" Sep 11 15:55:27.017095 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:27.017050 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spxb6_94ab0396-7020-4ad3-b1b7-63cd05490e84/konnectivity-agent/0.log" limit=1024 Sep 11 15:55:27.041722 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:27.041690 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/9f6c5948-c774-49b6-aaf5-026c7b5db718-snapshots\") pod \"insights-operator-5844b856db-mjh4m\" (UID: \"9f6c5948-c774-49b6-aaf5-026c7b5db718\") " pod="openshift-insights/insights-operator-5844b856db-mjh4m" Sep 11 15:55:27.041902 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:27.041732 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7542ad77-c817-4370-91de-3b08db264753-trusted-ca\") pod \"console-operator-696884fb9c-xjvl5\" (UID: \"7542ad77-c817-4370-91de-3b08db264753\") " pod="openshift-console-operator/console-operator-696884fb9c-xjvl5" Sep 11 15:55:27.041902 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:27.041752 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9g2s\" (UniqueName: \"kubernetes.io/projected/c2a34499-68a0-42b4-b77a-edc8b3b964c4-kube-api-access-b9g2s\") pod \"cluster-samples-operator-86499c89c4-4hdz4\" (UID: \"c2a34499-68a0-42b4-b77a-edc8b3b964c4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-4hdz4" Sep 11 15:55:27.041902 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:27.041768 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7542ad77-c817-4370-91de-3b08db264753-serving-cert\") pod \"console-operator-696884fb9c-xjvl5\" (UID: \"7542ad77-c817-4370-91de-3b08db264753\") " pod="openshift-console-operator/console-operator-696884fb9c-xjvl5" Sep 11 15:55:27.041902 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:27.041822 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f6c5948-c774-49b6-aaf5-026c7b5db718-service-ca-bundle\") pod \"insights-operator-5844b856db-mjh4m\" (UID: \"9f6c5948-c774-49b6-aaf5-026c7b5db718\") " pod="openshift-insights/insights-operator-5844b856db-mjh4m" Sep 11 15:55:27.041902 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:27.041862 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9f6c5948-c774-49b6-aaf5-026c7b5db718-serving-cert\") pod \"insights-operator-5844b856db-mjh4m\" (UID: \"9f6c5948-c774-49b6-aaf5-026c7b5db718\") " pod="openshift-insights/insights-operator-5844b856db-mjh4m" Sep 11 15:55:27.041902 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:27.041887 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g59z4\" (UniqueName: \"kubernetes.io/projected/9f6c5948-c774-49b6-aaf5-026c7b5db718-kube-api-access-g59z4\") pod \"insights-operator-5844b856db-mjh4m\" (UID: \"9f6c5948-c774-49b6-aaf5-026c7b5db718\") " pod="openshift-insights/insights-operator-5844b856db-mjh4m" Sep 11 15:55:27.042235 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:27.042192 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgxtc\" (UniqueName: \"kubernetes.io/projected/ddc77228-8fb6-473e-9940-7eae0c5457aa-kube-api-access-jgxtc\") pod \"volume-data-source-validator-86695b7cc7-ssz6b\" (UID: \"ddc77228-8fb6-473e-9940-7eae0c5457aa\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-ssz6b" Sep 11 15:55:27.042281 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:27.042248 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c2a34499-68a0-42b4-b77a-edc8b3b964c4-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-4hdz4\" (UID: \"c2a34499-68a0-42b4-b77a-edc8b3b964c4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-4hdz4" Sep 11 15:55:27.042315 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:27.042281 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/09f64aa8-4936-49d3-96f7-9895410e5dda-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-8q8dx\" (UID: \"09f64aa8-4936-49d3-96f7-9895410e5dda\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-8q8dx" Sep 11 15:55:27.042350 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:27.042322 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bd62\" (UniqueName: \"kubernetes.io/projected/00183511-0324-43ff-9a5f-26dfacb9401c-kube-api-access-6bd62\") pod \"network-check-source-6f8b768455-t82dg\" (UID: \"00183511-0324-43ff-9a5f-26dfacb9401c\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-t82dg" Sep 11 15:55:27.043346 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:27.042640 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4d14aeeb-9182-4334-956a-d0cc21ca6633-cert\") pod \"ingress-canary-4kl5b\" (UID: \"4d14aeeb-9182-4334-956a-d0cc21ca6633\") " pod="openshift-ingress-canary/ingress-canary-4kl5b" Sep 11 15:55:27.043346 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:27.042639 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/9f6c5948-c774-49b6-aaf5-026c7b5db718-snapshots\") pod \"insights-operator-5844b856db-mjh4m\" (UID: \"9f6c5948-c774-49b6-aaf5-026c7b5db718\") " pod="openshift-insights/insights-operator-5844b856db-mjh4m" Sep 11 15:55:27.043346 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:27.042830 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f6c5948-c774-49b6-aaf5-026c7b5db718-service-ca-bundle\") pod \"insights-operator-5844b856db-mjh4m\" (UID: \"9f6c5948-c774-49b6-aaf5-026c7b5db718\") " pod="openshift-insights/insights-operator-5844b856db-mjh4m" Sep 11 15:55:27.043346 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:27.042906 2566 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 11 15:55:27.043346 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:27.043000 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2a34499-68a0-42b4-b77a-edc8b3b964c4-samples-operator-tls podName:c2a34499-68a0-42b4-b77a-edc8b3b964c4 nodeName:}" failed. No retries permitted until 2026-09-11 15:55:27.542963056 +0000 UTC m=+34.757796955 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c2a34499-68a0-42b4-b77a-edc8b3b964c4-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-4hdz4" (UID: "c2a34499-68a0-42b4-b77a-edc8b3b964c4") : secret "samples-operator-tls" not found Sep 11 15:55:27.043346 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:27.043013 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7542ad77-c817-4370-91de-3b08db264753-trusted-ca\") pod \"console-operator-696884fb9c-xjvl5\" (UID: \"7542ad77-c817-4370-91de-3b08db264753\") " pod="openshift-console-operator/console-operator-696884fb9c-xjvl5" Sep 11 15:55:27.043346 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:27.043098 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a1e7a08b-8aa9-42e9-aa46-3225f61299e0-config-volume\") pod \"dns-default-gx9fm\" (UID: \"a1e7a08b-8aa9-42e9-aa46-3225f61299e0\") " pod="openshift-dns/dns-default-gx9fm" Sep 11 15:55:27.043346 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:27.043147 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1c358fa6-a923-4589-953f-bd49ddd4235c-original-pull-secret\") pod \"global-pull-secret-syncer-95zvh\" (UID: \"1c358fa6-a923-4589-953f-bd49ddd4235c\") " pod="kube-system/global-pull-secret-syncer-95zvh" Sep 11 15:55:27.043346 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:27.043181 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9f6c5948-c774-49b6-aaf5-026c7b5db718-tmp\") pod \"insights-operator-5844b856db-mjh4m\" (UID: \"9f6c5948-c774-49b6-aaf5-026c7b5db718\") " pod="openshift-insights/insights-operator-5844b856db-mjh4m" Sep 11 15:55:27.043346 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:27.043212 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/09f64aa8-4936-49d3-96f7-9895410e5dda-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-8q8dx\" (UID: \"09f64aa8-4936-49d3-96f7-9895410e5dda\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-8q8dx" Sep 11 15:55:27.043346 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:27.043246 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdj49\" (UniqueName: \"kubernetes.io/projected/a1e7a08b-8aa9-42e9-aa46-3225f61299e0-kube-api-access-tdj49\") pod \"dns-default-gx9fm\" (UID: \"a1e7a08b-8aa9-42e9-aa46-3225f61299e0\") " pod="openshift-dns/dns-default-gx9fm" Sep 11 15:55:27.043346 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:27.043281 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1e7a08b-8aa9-42e9-aa46-3225f61299e0-tmp-dir\") pod \"dns-default-gx9fm\" (UID: \"a1e7a08b-8aa9-42e9-aa46-3225f61299e0\") " pod="openshift-dns/dns-default-gx9fm" Sep 11 15:55:27.043346 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:27.043358 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqhgw\" (UniqueName: \"kubernetes.io/projected/7542ad77-c817-4370-91de-3b08db264753-kube-api-access-xqhgw\") pod \"console-operator-696884fb9c-xjvl5\" (UID: \"7542ad77-c817-4370-91de-3b08db264753\") " pod="openshift-console-operator/console-operator-696884fb9c-xjvl5" Sep 11 15:55:27.044056 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:27.043391 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hj7w5\" (UniqueName: \"kubernetes.io/projected/4d14aeeb-9182-4334-956a-d0cc21ca6633-kube-api-access-hj7w5\") pod \"ingress-canary-4kl5b\" (UID: \"4d14aeeb-9182-4334-956a-d0cc21ca6633\") " pod="openshift-ingress-canary/ingress-canary-4kl5b" Sep 11 15:55:27.044056 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:27.043425 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a1e7a08b-8aa9-42e9-aa46-3225f61299e0-metrics-tls\") pod \"dns-default-gx9fm\" (UID: \"a1e7a08b-8aa9-42e9-aa46-3225f61299e0\") " pod="openshift-dns/dns-default-gx9fm" Sep 11 15:55:27.044056 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:27.043496 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7542ad77-c817-4370-91de-3b08db264753-config\") pod \"console-operator-696884fb9c-xjvl5\" (UID: \"7542ad77-c817-4370-91de-3b08db264753\") " pod="openshift-console-operator/console-operator-696884fb9c-xjvl5" Sep 11 15:55:27.044056 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:27.043521 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9f6c5948-c774-49b6-aaf5-026c7b5db718-tmp\") pod \"insights-operator-5844b856db-mjh4m\" (UID: \"9f6c5948-c774-49b6-aaf5-026c7b5db718\") " pod="openshift-insights/insights-operator-5844b856db-mjh4m" Sep 11 15:55:27.044056 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:27.043536 2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 15:55:27.044056 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:27.043550 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f6c5948-c774-49b6-aaf5-026c7b5db718-trusted-ca-bundle\") pod \"insights-operator-5844b856db-mjh4m\" (UID: \"9f6c5948-c774-49b6-aaf5-026c7b5db718\") " pod="openshift-insights/insights-operator-5844b856db-mjh4m" Sep 11 15:55:27.044056 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:27.043582 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/09f64aa8-4936-49d3-96f7-9895410e5dda-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-8q8dx\" (UID: \"09f64aa8-4936-49d3-96f7-9895410e5dda\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-8q8dx" Sep 11 15:55:27.044056 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:27.043595 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c358fa6-a923-4589-953f-bd49ddd4235c-original-pull-secret podName:1c358fa6-a923-4589-953f-bd49ddd4235c nodeName:}" failed. No retries permitted until 2026-09-11 15:55:59.043580207 +0000 UTC m=+66.258414104 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1c358fa6-a923-4589-953f-bd49ddd4235c-original-pull-secret") pod "global-pull-secret-syncer-95zvh" (UID: "1c358fa6-a923-4589-953f-bd49ddd4235c") : object "kube-system"/"original-pull-secret" not registered Sep 11 15:55:27.044056 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:27.043634 2566 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 11 15:55:27.044056 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:27.043689 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/09f64aa8-4936-49d3-96f7-9895410e5dda-networking-console-plugin-cert podName:09f64aa8-4936-49d3-96f7-9895410e5dda nodeName:}" failed. No retries permitted until 2026-09-11 15:55:27.543674376 +0000 UTC m=+34.758508273 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/09f64aa8-4936-49d3-96f7-9895410e5dda-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-8q8dx" (UID: "09f64aa8-4936-49d3-96f7-9895410e5dda") : secret "networking-console-plugin-cert" not found Sep 11 15:55:27.044499 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:27.044222 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7542ad77-c817-4370-91de-3b08db264753-config\") pod \"console-operator-696884fb9c-xjvl5\" (UID: \"7542ad77-c817-4370-91de-3b08db264753\") " pod="openshift-console-operator/console-operator-696884fb9c-xjvl5" Sep 11 15:55:27.047112 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:27.046234 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f6c5948-c774-49b6-aaf5-026c7b5db718-trusted-ca-bundle\") pod \"insights-operator-5844b856db-mjh4m\" (UID: \"9f6c5948-c774-49b6-aaf5-026c7b5db718\") " pod="openshift-insights/insights-operator-5844b856db-mjh4m" Sep 11 15:55:27.048092 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:27.048064 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7542ad77-c817-4370-91de-3b08db264753-serving-cert\") pod \"console-operator-696884fb9c-xjvl5\" (UID: \"7542ad77-c817-4370-91de-3b08db264753\") " pod="openshift-console-operator/console-operator-696884fb9c-xjvl5" Sep 11 15:55:27.049329 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:27.049308 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9f6c5948-c774-49b6-aaf5-026c7b5db718-serving-cert\") pod \"insights-operator-5844b856db-mjh4m\" (UID: \"9f6c5948-c774-49b6-aaf5-026c7b5db718\") " pod="openshift-insights/insights-operator-5844b856db-mjh4m" Sep 11 15:55:27.050275 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:27.050246 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-qjtlt" Sep 11 15:55:27.052749 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:27.052728 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9g2s\" (UniqueName: \"kubernetes.io/projected/c2a34499-68a0-42b4-b77a-edc8b3b964c4-kube-api-access-b9g2s\") pod \"cluster-samples-operator-86499c89c4-4hdz4\" (UID: \"c2a34499-68a0-42b4-b77a-edc8b3b964c4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-4hdz4" Sep 11 15:55:27.059645 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:27.059622 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bd62\" (UniqueName: \"kubernetes.io/projected/00183511-0324-43ff-9a5f-26dfacb9401c-kube-api-access-6bd62\") pod \"network-check-source-6f8b768455-t82dg\" (UID: \"00183511-0324-43ff-9a5f-26dfacb9401c\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-t82dg" Sep 11 15:55:27.061832 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:27.061804 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g59z4\" (UniqueName: \"kubernetes.io/projected/9f6c5948-c774-49b6-aaf5-026c7b5db718-kube-api-access-g59z4\") pod \"insights-operator-5844b856db-mjh4m\" (UID: \"9f6c5948-c774-49b6-aaf5-026c7b5db718\") " pod="openshift-insights/insights-operator-5844b856db-mjh4m" Sep 11 15:55:27.062819 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:27.062800 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-djghf" Sep 11 15:55:27.071174 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:27.071150 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqhgw\" (UniqueName: \"kubernetes.io/projected/7542ad77-c817-4370-91de-3b08db264753-kube-api-access-xqhgw\") pod \"console-operator-696884fb9c-xjvl5\" (UID: \"7542ad77-c817-4370-91de-3b08db264753\") " pod="openshift-console-operator/console-operator-696884fb9c-xjvl5" Sep 11 15:55:27.114601 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:27.114527 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-mjh4m" Sep 11 15:55:27.130488 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:27.130450 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-t82dg" Sep 11 15:55:27.143689 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:27.143659 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-xjvl5" Sep 11 15:55:27.144065 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:27.144040 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgxtc\" (UniqueName: \"kubernetes.io/projected/ddc77228-8fb6-473e-9940-7eae0c5457aa-kube-api-access-jgxtc\") pod \"volume-data-source-validator-86695b7cc7-ssz6b\" (UID: \"ddc77228-8fb6-473e-9940-7eae0c5457aa\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-ssz6b" Sep 11 15:55:27.144200 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:27.144089 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4d14aeeb-9182-4334-956a-d0cc21ca6633-cert\") pod \"ingress-canary-4kl5b\" (UID: \"4d14aeeb-9182-4334-956a-d0cc21ca6633\") " pod="openshift-ingress-canary/ingress-canary-4kl5b" Sep 11 15:55:27.144200 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:27.144112 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a1e7a08b-8aa9-42e9-aa46-3225f61299e0-config-volume\") pod \"dns-default-gx9fm\" (UID: \"a1e7a08b-8aa9-42e9-aa46-3225f61299e0\") " pod="openshift-dns/dns-default-gx9fm" Sep 11 15:55:27.144200 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:27.144147 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdj49\" (UniqueName: \"kubernetes.io/projected/a1e7a08b-8aa9-42e9-aa46-3225f61299e0-kube-api-access-tdj49\") pod \"dns-default-gx9fm\" (UID: \"a1e7a08b-8aa9-42e9-aa46-3225f61299e0\") " pod="openshift-dns/dns-default-gx9fm" Sep 11 15:55:27.144200 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:27.144165 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1e7a08b-8aa9-42e9-aa46-3225f61299e0-tmp-dir\") pod \"dns-default-gx9fm\" (UID: \"a1e7a08b-8aa9-42e9-aa46-3225f61299e0\") " pod="openshift-dns/dns-default-gx9fm" Sep 11 15:55:27.144200 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:27.144197 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hj7w5\" (UniqueName: \"kubernetes.io/projected/4d14aeeb-9182-4334-956a-d0cc21ca6633-kube-api-access-hj7w5\") pod \"ingress-canary-4kl5b\" (UID: \"4d14aeeb-9182-4334-956a-d0cc21ca6633\") " pod="openshift-ingress-canary/ingress-canary-4kl5b" Sep 11 15:55:27.144460 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:27.144217 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a1e7a08b-8aa9-42e9-aa46-3225f61299e0-metrics-tls\") pod \"dns-default-gx9fm\" (UID: \"a1e7a08b-8aa9-42e9-aa46-3225f61299e0\") " pod="openshift-dns/dns-default-gx9fm" Sep 11 15:55:27.144460 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:27.144242 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9g7p\" (UniqueName: \"kubernetes.io/projected/f6c05fce-e71f-4a9c-b219-a77b1df84ac8-kube-api-access-h9g7p\") pod \"network-check-target-nzcjl\" (UID: \"f6c05fce-e71f-4a9c-b219-a77b1df84ac8\") " pod="openshift-network-diagnostics/network-check-target-nzcjl" Sep 11 15:55:27.144546 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:27.144492 2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 15:55:27.144546 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:27.144507 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1e7a08b-8aa9-42e9-aa46-3225f61299e0-tmp-dir\") pod \"dns-default-gx9fm\" (UID: \"a1e7a08b-8aa9-42e9-aa46-3225f61299e0\") " pod="openshift-dns/dns-default-gx9fm" Sep 11 15:55:27.144650 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:27.144579 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1e7a08b-8aa9-42e9-aa46-3225f61299e0-metrics-tls podName:a1e7a08b-8aa9-42e9-aa46-3225f61299e0 nodeName:}" failed. No retries permitted until 2026-09-11 15:55:27.644558383 +0000 UTC m=+34.859392494 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a1e7a08b-8aa9-42e9-aa46-3225f61299e0-metrics-tls") pod "dns-default-gx9fm" (UID: "a1e7a08b-8aa9-42e9-aa46-3225f61299e0") : secret "dns-default-metrics-tls" not found Sep 11 15:55:27.144718 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:27.144706 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a1e7a08b-8aa9-42e9-aa46-3225f61299e0-config-volume\") pod \"dns-default-gx9fm\" (UID: \"a1e7a08b-8aa9-42e9-aa46-3225f61299e0\") " pod="openshift-dns/dns-default-gx9fm" Sep 11 15:55:27.146869 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:27.146848 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4d14aeeb-9182-4334-956a-d0cc21ca6633-cert\") pod \"ingress-canary-4kl5b\" (UID: \"4d14aeeb-9182-4334-956a-d0cc21ca6633\") " pod="openshift-ingress-canary/ingress-canary-4kl5b" Sep 11 15:55:27.147010 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:27.146988 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9g7p\" (UniqueName: \"kubernetes.io/projected/f6c05fce-e71f-4a9c-b219-a77b1df84ac8-kube-api-access-h9g7p\") pod \"network-check-target-nzcjl\" (UID: \"f6c05fce-e71f-4a9c-b219-a77b1df84ac8\") " pod="openshift-network-diagnostics/network-check-target-nzcjl" Sep 11 15:55:27.150703 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:27.150671 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spxb6_94ab0396-7020-4ad3-b1b7-63cd05490e84/konnectivity-agent/0.log" limit=1024 Sep 11 15:55:27.153054 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:27.153029 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hj7w5\" (UniqueName: \"kubernetes.io/projected/4d14aeeb-9182-4334-956a-d0cc21ca6633-kube-api-access-hj7w5\") pod \"ingress-canary-4kl5b\" (UID: \"4d14aeeb-9182-4334-956a-d0cc21ca6633\") " pod="openshift-ingress-canary/ingress-canary-4kl5b" Sep 11 15:55:27.153693 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:27.153672 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgxtc\" (UniqueName: \"kubernetes.io/projected/ddc77228-8fb6-473e-9940-7eae0c5457aa-kube-api-access-jgxtc\") pod \"volume-data-source-validator-86695b7cc7-ssz6b\" (UID: \"ddc77228-8fb6-473e-9940-7eae0c5457aa\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-ssz6b" Sep 11 15:55:27.153776 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:27.153716 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdj49\" (UniqueName: \"kubernetes.io/projected/a1e7a08b-8aa9-42e9-aa46-3225f61299e0-kube-api-access-tdj49\") pod \"dns-default-gx9fm\" (UID: \"a1e7a08b-8aa9-42e9-aa46-3225f61299e0\") " pod="openshift-dns/dns-default-gx9fm" Sep 11 15:55:27.170904 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:27.170879 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-ssz6b" Sep 11 15:55:27.199376 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:27.199351 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-4kl5b" Sep 11 15:55:27.233195 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:27.232814 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nzcjl" Sep 11 15:55:27.233195 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:27.232822 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-95zvh" Sep 11 15:55:27.233195 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:27.233077 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-plkhx" Sep 11 15:55:27.235509 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:27.235385 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 11 15:55:27.235509 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:27.235494 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 11 15:55:27.235698 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:27.235556 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-lsspg\"" Sep 11 15:55:27.235777 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:27.235760 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zsbjs\"" Sep 11 15:55:27.244543 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:27.244517 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nzcjl" Sep 11 15:55:27.397157 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:27.397069 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spxb6_94ab0396-7020-4ad3-b1b7-63cd05490e84/konnectivity-agent/0.log" limit=1024 Sep 11 15:55:27.452109 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:27.451437 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/313238e9-79d7-45f7-8d96-beb55feb415a-metrics-certs\") pod \"router-default-844dfdb99b-766v5\" (UID: \"313238e9-79d7-45f7-8d96-beb55feb415a\") " pod="openshift-ingress/router-default-844dfdb99b-766v5" Sep 11 15:55:27.452109 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:27.451501 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5429cde4-cb80-4579-9687-e71617b15b9f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-kqtwh\" (UID: \"5429cde4-cb80-4579-9687-e71617b15b9f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-kqtwh" Sep 11 15:55:27.452109 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:27.451566 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/313238e9-79d7-45f7-8d96-beb55feb415a-service-ca-bundle\") pod \"router-default-844dfdb99b-766v5\" (UID: \"313238e9-79d7-45f7-8d96-beb55feb415a\") " pod="openshift-ingress/router-default-844dfdb99b-766v5" Sep 11 15:55:27.452109 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:27.451628 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ae9022dc-da99-4acf-af85-3cf9858737b4-registry-tls\") pod \"image-registry-56c5d9df5d-7vqnb\" (UID: \"ae9022dc-da99-4acf-af85-3cf9858737b4\") " pod="openshift-image-registry/image-registry-56c5d9df5d-7vqnb" Sep 11 15:55:27.452109 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:27.451770 2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 11 15:55:27.452109 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:27.451785 2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-56c5d9df5d-7vqnb: secret "image-registry-tls" not found Sep 11 15:55:27.452109 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:27.451856 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ae9022dc-da99-4acf-af85-3cf9858737b4-registry-tls podName:ae9022dc-da99-4acf-af85-3cf9858737b4 nodeName:}" failed. No retries permitted until 2026-09-11 15:55:28.451836845 +0000 UTC m=+35.666670742 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ae9022dc-da99-4acf-af85-3cf9858737b4-registry-tls") pod "image-registry-56c5d9df5d-7vqnb" (UID: "ae9022dc-da99-4acf-af85-3cf9858737b4") : secret "image-registry-tls" not found Sep 11 15:55:27.452109 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:27.452101 2566 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 11 15:55:27.452675 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:27.452157 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5429cde4-cb80-4579-9687-e71617b15b9f-cluster-monitoring-operator-tls podName:5429cde4-cb80-4579-9687-e71617b15b9f nodeName:}" failed. No retries permitted until 2026-09-11 15:55:28.452140778 +0000 UTC m=+35.666974670 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/5429cde4-cb80-4579-9687-e71617b15b9f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-kqtwh" (UID: "5429cde4-cb80-4579-9687-e71617b15b9f") : secret "cluster-monitoring-operator-tls" not found Sep 11 15:55:27.452675 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:27.452212 2566 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 11 15:55:27.452675 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:27.452250 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/313238e9-79d7-45f7-8d96-beb55feb415a-metrics-certs podName:313238e9-79d7-45f7-8d96-beb55feb415a nodeName:}" failed. No retries permitted until 2026-09-11 15:55:28.452234732 +0000 UTC m=+35.667068629 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/313238e9-79d7-45f7-8d96-beb55feb415a-metrics-certs") pod "router-default-844dfdb99b-766v5" (UID: "313238e9-79d7-45f7-8d96-beb55feb415a") : secret "router-metrics-certs-default" not found Sep 11 15:55:27.452675 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:27.452340 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/313238e9-79d7-45f7-8d96-beb55feb415a-service-ca-bundle podName:313238e9-79d7-45f7-8d96-beb55feb415a nodeName:}" failed. No retries permitted until 2026-09-11 15:55:28.452317258 +0000 UTC m=+35.667151156 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/313238e9-79d7-45f7-8d96-beb55feb415a-service-ca-bundle") pod "router-default-844dfdb99b-766v5" (UID: "313238e9-79d7-45f7-8d96-beb55feb415a") : configmap references non-existent config key: service-ca.crt Sep 11 15:55:27.499455 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:27.499362 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spxb6_94ab0396-7020-4ad3-b1b7-63cd05490e84/konnectivity-agent/0.log" limit=1024 Sep 11 15:55:27.552659 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:27.552623 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c2a34499-68a0-42b4-b77a-edc8b3b964c4-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-4hdz4\" (UID: \"c2a34499-68a0-42b4-b77a-edc8b3b964c4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-4hdz4" Sep 11 15:55:27.552781 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:27.552718 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/09f64aa8-4936-49d3-96f7-9895410e5dda-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-8q8dx\" (UID: \"09f64aa8-4936-49d3-96f7-9895410e5dda\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-8q8dx" Sep 11 15:55:27.552957 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:27.552934 2566 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 11 15:55:27.553057 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:27.553035 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/09f64aa8-4936-49d3-96f7-9895410e5dda-networking-console-plugin-cert podName:09f64aa8-4936-49d3-96f7-9895410e5dda nodeName:}" failed. No retries permitted until 2026-09-11 15:55:28.553014867 +0000 UTC m=+35.767848773 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/09f64aa8-4936-49d3-96f7-9895410e5dda-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-8q8dx" (UID: "09f64aa8-4936-49d3-96f7-9895410e5dda") : secret "networking-console-plugin-cert" not found Sep 11 15:55:27.553122 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:27.552935 2566 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 11 15:55:27.553175 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:27.553153 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2a34499-68a0-42b4-b77a-edc8b3b964c4-samples-operator-tls podName:c2a34499-68a0-42b4-b77a-edc8b3b964c4 nodeName:}" failed. No retries permitted until 2026-09-11 15:55:28.553138506 +0000 UTC m=+35.767972403 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c2a34499-68a0-42b4-b77a-edc8b3b964c4-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-4hdz4" (UID: "c2a34499-68a0-42b4-b77a-edc8b3b964c4") : secret "samples-operator-tls" not found Sep 11 15:55:27.653732 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:27.653652 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a1e7a08b-8aa9-42e9-aa46-3225f61299e0-metrics-tls\") pod \"dns-default-gx9fm\" (UID: \"a1e7a08b-8aa9-42e9-aa46-3225f61299e0\") " pod="openshift-dns/dns-default-gx9fm" Sep 11 15:55:27.654382 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:27.653809 2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 15:55:27.654382 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:27.653877 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1e7a08b-8aa9-42e9-aa46-3225f61299e0-metrics-tls podName:a1e7a08b-8aa9-42e9-aa46-3225f61299e0 nodeName:}" failed. No retries permitted until 2026-09-11 15:55:28.65385885 +0000 UTC m=+35.868692756 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a1e7a08b-8aa9-42e9-aa46-3225f61299e0-metrics-tls") pod "dns-default-gx9fm" (UID: "a1e7a08b-8aa9-42e9-aa46-3225f61299e0") : secret "dns-default-metrics-tls" not found Sep 11 15:55:27.790674 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:27.790542 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-nzcjl"] Sep 11 15:55:27.793386 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:27.793314 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-djghf"] Sep 11 15:55:27.794348 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:55:27.794310 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6c05fce_e71f_4a9c_b219_a77b1df84ac8.slice/crio-bbd8cc6848146c9116f858567044b4b07cf06106f4ea777893affe990e5bb66c WatchSource:0}: Error finding container bbd8cc6848146c9116f858567044b4b07cf06106f4ea777893affe990e5bb66c: Status 404 returned error can't find the container with id bbd8cc6848146c9116f858567044b4b07cf06106f4ea777893affe990e5bb66c Sep 11 15:55:27.800191 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:27.797615 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-t82dg"] Sep 11 15:55:27.800191 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:27.797671 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-qjtlt"] Sep 11 15:55:27.800191 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:27.798337 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-ssz6b"] Sep 11 15:55:27.800191 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:27.799625 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-4kl5b"] Sep 11 15:55:27.800863 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:27.800704 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-xjvl5"] Sep 11 15:55:27.802070 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:27.802049 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-mjh4m"] Sep 11 15:55:27.805179 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:55:27.803995 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d14aeeb_9182_4334_956a_d0cc21ca6633.slice/crio-4200c5d3cda37c3e26c7b84a2a615da38fc4b900031bd839f9b87b5f0b4740ad WatchSource:0}: Error finding container 4200c5d3cda37c3e26c7b84a2a615da38fc4b900031bd839f9b87b5f0b4740ad: Status 404 returned error can't find the container with id 4200c5d3cda37c3e26c7b84a2a615da38fc4b900031bd839f9b87b5f0b4740ad Sep 11 15:55:27.806703 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:55:27.806616 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7542ad77_c817_4370_91de_3b08db264753.slice/crio-1c7132fb0204599a8fa602d76601dc9eb415c11cf17c200d14e1c4cdf4059004 WatchSource:0}: Error finding container 1c7132fb0204599a8fa602d76601dc9eb415c11cf17c200d14e1c4cdf4059004: Status 404 returned error can't find the container with id 1c7132fb0204599a8fa602d76601dc9eb415c11cf17c200d14e1c4cdf4059004 Sep 11 15:55:27.807776 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:55:27.807753 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f6c5948_c774_49b6_aaf5_026c7b5db718.slice/crio-ae2054f9c21c741792ee9c098095e903a5f1698958ffdf46658ff8a55cd113ee WatchSource:0}: Error finding container ae2054f9c21c741792ee9c098095e903a5f1698958ffdf46658ff8a55cd113ee: Status 404 returned error can't find the container with id ae2054f9c21c741792ee9c098095e903a5f1698958ffdf46658ff8a55cd113ee Sep 11 15:55:27.811240 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:55:27.811216 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07d36306_7ec8_4c74_9b97_7545503a2783.slice/crio-47a5a0a3aa8cfffcb732c0674b2f3fc08effb8803655c2ad878a671e14f66cf2 WatchSource:0}: Error finding container 47a5a0a3aa8cfffcb732c0674b2f3fc08effb8803655c2ad878a671e14f66cf2: Status 404 returned error can't find the container with id 47a5a0a3aa8cfffcb732c0674b2f3fc08effb8803655c2ad878a671e14f66cf2 Sep 11 15:55:27.885470 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:27.885301 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spxb6_94ab0396-7020-4ad3-b1b7-63cd05490e84/konnectivity-agent/0.log" limit=1024 Sep 11 15:55:27.957478 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:27.957356 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-spxb6_94ab0396-7020-4ad3-b1b7-63cd05490e84/konnectivity-agent/0.log" limit=1024 Sep 11 15:55:28.396291 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:28.396254 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-t82dg" event={"ID":"00183511-0324-43ff-9a5f-26dfacb9401c","Type":"ContainerStarted","Data":"903d7e44ce59dce84ae2f6ea1ce46da36f46319d1b582a63030ad131364ffb70"} Sep 11 15:55:28.399626 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:28.399594 2566 generic.go:356] "Generic (PLEG): container finished" podID="fe6dccde-412f-483d-a914-5d46e9922b69" containerID="388eafe80f672ae39b8aa8ef40261f0f3e624abbaac1a837e36f8a495e2010bc" exitCode=0 Sep 11 15:55:28.399847 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:28.399690 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b7ftm" event={"ID":"fe6dccde-412f-483d-a914-5d46e9922b69","Type":"ContainerDied","Data":"388eafe80f672ae39b8aa8ef40261f0f3e624abbaac1a837e36f8a495e2010bc"} Sep 11 15:55:28.401192 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:28.401004 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-nzcjl" event={"ID":"f6c05fce-e71f-4a9c-b219-a77b1df84ac8","Type":"ContainerStarted","Data":"bbd8cc6848146c9116f858567044b4b07cf06106f4ea777893affe990e5bb66c"} Sep 11 15:55:28.402601 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:28.402543 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-ssz6b" event={"ID":"ddc77228-8fb6-473e-9940-7eae0c5457aa","Type":"ContainerStarted","Data":"9b1305695199867d29e58e4e56db8ff903e60644f7579b1f540cbbd825e5db94"} Sep 11 15:55:28.404024 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:28.404000 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-djghf" event={"ID":"07d36306-7ec8-4c74-9b97-7545503a2783","Type":"ContainerStarted","Data":"47a5a0a3aa8cfffcb732c0674b2f3fc08effb8803655c2ad878a671e14f66cf2"} Sep 11 15:55:28.405328 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:28.405299 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-4kl5b" event={"ID":"4d14aeeb-9182-4334-956a-d0cc21ca6633","Type":"ContainerStarted","Data":"4200c5d3cda37c3e26c7b84a2a615da38fc4b900031bd839f9b87b5f0b4740ad"} Sep 11 15:55:28.406572 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:28.406553 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-mjh4m" event={"ID":"9f6c5948-c774-49b6-aaf5-026c7b5db718","Type":"ContainerStarted","Data":"ae2054f9c21c741792ee9c098095e903a5f1698958ffdf46658ff8a55cd113ee"} Sep 11 15:55:28.408125 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:28.408100 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-xjvl5" event={"ID":"7542ad77-c817-4370-91de-3b08db264753","Type":"ContainerStarted","Data":"1c7132fb0204599a8fa602d76601dc9eb415c11cf17c200d14e1c4cdf4059004"} Sep 11 15:55:28.409328 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:28.409308 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-qjtlt" event={"ID":"26aadde0-9d1e-4368-86b0-4a2eb04656c3","Type":"ContainerStarted","Data":"acf18a7639d39735f01db89a5f97aeb84f3bb0e50593443bf0aa4c3088ba7947"} Sep 11 15:55:28.460757 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:28.460726 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/313238e9-79d7-45f7-8d96-beb55feb415a-metrics-certs\") pod \"router-default-844dfdb99b-766v5\" (UID: \"313238e9-79d7-45f7-8d96-beb55feb415a\") " pod="openshift-ingress/router-default-844dfdb99b-766v5" Sep 11 15:55:28.460876 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:28.460762 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5429cde4-cb80-4579-9687-e71617b15b9f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-kqtwh\" (UID: \"5429cde4-cb80-4579-9687-e71617b15b9f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-kqtwh" Sep 11 15:55:28.460876 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:28.460792 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/313238e9-79d7-45f7-8d96-beb55feb415a-service-ca-bundle\") pod \"router-default-844dfdb99b-766v5\" (UID: \"313238e9-79d7-45f7-8d96-beb55feb415a\") " pod="openshift-ingress/router-default-844dfdb99b-766v5" Sep 11 15:55:28.461017 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:28.460898 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/313238e9-79d7-45f7-8d96-beb55feb415a-service-ca-bundle podName:313238e9-79d7-45f7-8d96-beb55feb415a nodeName:}" failed. No retries permitted until 2026-09-11 15:55:30.460883346 +0000 UTC m=+37.675717246 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/313238e9-79d7-45f7-8d96-beb55feb415a-service-ca-bundle") pod "router-default-844dfdb99b-766v5" (UID: "313238e9-79d7-45f7-8d96-beb55feb415a") : configmap references non-existent config key: service-ca.crt Sep 11 15:55:28.461017 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:28.460896 2566 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 11 15:55:28.461017 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:28.460948 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ae9022dc-da99-4acf-af85-3cf9858737b4-registry-tls\") pod \"image-registry-56c5d9df5d-7vqnb\" (UID: \"ae9022dc-da99-4acf-af85-3cf9858737b4\") " pod="openshift-image-registry/image-registry-56c5d9df5d-7vqnb" Sep 11 15:55:28.461017 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:28.460961 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/313238e9-79d7-45f7-8d96-beb55feb415a-metrics-certs podName:313238e9-79d7-45f7-8d96-beb55feb415a nodeName:}" failed. No retries permitted until 2026-09-11 15:55:30.460943664 +0000 UTC m=+37.675777572 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/313238e9-79d7-45f7-8d96-beb55feb415a-metrics-certs") pod "router-default-844dfdb99b-766v5" (UID: "313238e9-79d7-45f7-8d96-beb55feb415a") : secret "router-metrics-certs-default" not found Sep 11 15:55:28.461218 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:28.461038 2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 11 15:55:28.461218 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:28.461052 2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-56c5d9df5d-7vqnb: secret "image-registry-tls" not found Sep 11 15:55:28.461218 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:28.460896 2566 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 11 15:55:28.461218 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:28.461089 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ae9022dc-da99-4acf-af85-3cf9858737b4-registry-tls podName:ae9022dc-da99-4acf-af85-3cf9858737b4 nodeName:}" failed. No retries permitted until 2026-09-11 15:55:30.461077951 +0000 UTC m=+37.675911849 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ae9022dc-da99-4acf-af85-3cf9858737b4-registry-tls") pod "image-registry-56c5d9df5d-7vqnb" (UID: "ae9022dc-da99-4acf-af85-3cf9858737b4") : secret "image-registry-tls" not found Sep 11 15:55:28.461218 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:28.461127 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5429cde4-cb80-4579-9687-e71617b15b9f-cluster-monitoring-operator-tls podName:5429cde4-cb80-4579-9687-e71617b15b9f nodeName:}" failed. No retries permitted until 2026-09-11 15:55:30.461115675 +0000 UTC m=+37.675949560 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/5429cde4-cb80-4579-9687-e71617b15b9f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-kqtwh" (UID: "5429cde4-cb80-4579-9687-e71617b15b9f") : secret "cluster-monitoring-operator-tls" not found Sep 11 15:55:28.562383 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:28.562341 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c2a34499-68a0-42b4-b77a-edc8b3b964c4-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-4hdz4\" (UID: \"c2a34499-68a0-42b4-b77a-edc8b3b964c4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-4hdz4" Sep 11 15:55:28.562511 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:28.562475 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/09f64aa8-4936-49d3-96f7-9895410e5dda-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-8q8dx\" (UID: \"09f64aa8-4936-49d3-96f7-9895410e5dda\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-8q8dx" Sep 11 15:55:28.562937 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:28.562897 2566 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 11 15:55:28.563046 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:28.562993 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2a34499-68a0-42b4-b77a-edc8b3b964c4-samples-operator-tls podName:c2a34499-68a0-42b4-b77a-edc8b3b964c4 nodeName:}" failed. No retries permitted until 2026-09-11 15:55:30.562956976 +0000 UTC m=+37.777790865 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c2a34499-68a0-42b4-b77a-edc8b3b964c4-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-4hdz4" (UID: "c2a34499-68a0-42b4-b77a-edc8b3b964c4") : secret "samples-operator-tls" not found Sep 11 15:55:28.563281 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:28.563259 2566 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 11 15:55:28.563355 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:28.563329 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/09f64aa8-4936-49d3-96f7-9895410e5dda-networking-console-plugin-cert podName:09f64aa8-4936-49d3-96f7-9895410e5dda nodeName:}" failed. No retries permitted until 2026-09-11 15:55:30.56330596 +0000 UTC m=+37.778139851 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/09f64aa8-4936-49d3-96f7-9895410e5dda-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-8q8dx" (UID: "09f64aa8-4936-49d3-96f7-9895410e5dda") : secret "networking-console-plugin-cert" not found Sep 11 15:55:28.664492 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:28.663889 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a1e7a08b-8aa9-42e9-aa46-3225f61299e0-metrics-tls\") pod \"dns-default-gx9fm\" (UID: \"a1e7a08b-8aa9-42e9-aa46-3225f61299e0\") " pod="openshift-dns/dns-default-gx9fm" Sep 11 15:55:28.664492 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:28.664132 2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 15:55:28.664492 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:28.664196 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1e7a08b-8aa9-42e9-aa46-3225f61299e0-metrics-tls podName:a1e7a08b-8aa9-42e9-aa46-3225f61299e0 nodeName:}" failed. No retries permitted until 2026-09-11 15:55:30.664176996 +0000 UTC m=+37.879010894 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a1e7a08b-8aa9-42e9-aa46-3225f61299e0-metrics-tls") pod "dns-default-gx9fm" (UID: "a1e7a08b-8aa9-42e9-aa46-3225f61299e0") : secret "dns-default-metrics-tls" not found Sep 11 15:55:29.430576 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:29.430537 2566 generic.go:356] "Generic (PLEG): container finished" podID="fe6dccde-412f-483d-a914-5d46e9922b69" containerID="af4c3b025645e7cd90fec176f8249eb926ac43c4f226fd079dfd03222e548ec4" exitCode=0 Sep 11 15:55:29.430760 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:29.430589 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b7ftm" event={"ID":"fe6dccde-412f-483d-a914-5d46e9922b69","Type":"ContainerDied","Data":"af4c3b025645e7cd90fec176f8249eb926ac43c4f226fd079dfd03222e548ec4"} Sep 11 15:55:30.442349 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:30.442310 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b7ftm" event={"ID":"fe6dccde-412f-483d-a914-5d46e9922b69","Type":"ContainerStarted","Data":"f3b26bd3e306d73cb595783bccea4d059716c295afddb09c22d3c729b8960eb8"} Sep 11 15:55:30.473589 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:30.473240 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-b7ftm" podStartSLOduration=7.419305686 podStartE2EDuration="37.473216811s" totalImagesPullingTime="30.053911125s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 15:54:53 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:55:30.470626881 +0000 UTC m=+37.685460782" watchObservedRunningTime="2026-09-11 15:55:30.473216811 +0000 UTC m=+37.688050716" Sep 11 15:55:30.486040 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:30.486006 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5429cde4-cb80-4579-9687-e71617b15b9f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-kqtwh\" (UID: \"5429cde4-cb80-4579-9687-e71617b15b9f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-kqtwh" Sep 11 15:55:30.486219 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:30.486198 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/313238e9-79d7-45f7-8d96-beb55feb415a-service-ca-bundle podName:313238e9-79d7-45f7-8d96-beb55feb415a nodeName:}" failed. No retries permitted until 2026-09-11 15:55:34.486177771 +0000 UTC m=+41.701011660 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/313238e9-79d7-45f7-8d96-beb55feb415a-service-ca-bundle") pod "router-default-844dfdb99b-766v5" (UID: "313238e9-79d7-45f7-8d96-beb55feb415a") : configmap references non-existent config key: service-ca.crt Sep 11 15:55:30.486738 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:30.486320 2566 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 11 15:55:30.486738 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:30.486362 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5429cde4-cb80-4579-9687-e71617b15b9f-cluster-monitoring-operator-tls podName:5429cde4-cb80-4579-9687-e71617b15b9f nodeName:}" failed. No retries permitted until 2026-09-11 15:55:34.486351032 +0000 UTC m=+41.701184921 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/5429cde4-cb80-4579-9687-e71617b15b9f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-kqtwh" (UID: "5429cde4-cb80-4579-9687-e71617b15b9f") : secret "cluster-monitoring-operator-tls" not found Sep 11 15:55:30.486738 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:30.486076 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/313238e9-79d7-45f7-8d96-beb55feb415a-service-ca-bundle\") pod \"router-default-844dfdb99b-766v5\" (UID: \"313238e9-79d7-45f7-8d96-beb55feb415a\") " pod="openshift-ingress/router-default-844dfdb99b-766v5" Sep 11 15:55:30.486738 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:30.486448 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ae9022dc-da99-4acf-af85-3cf9858737b4-registry-tls\") pod \"image-registry-56c5d9df5d-7vqnb\" (UID: \"ae9022dc-da99-4acf-af85-3cf9858737b4\") " pod="openshift-image-registry/image-registry-56c5d9df5d-7vqnb" Sep 11 15:55:30.486738 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:30.486562 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/313238e9-79d7-45f7-8d96-beb55feb415a-metrics-certs\") pod \"router-default-844dfdb99b-766v5\" (UID: \"313238e9-79d7-45f7-8d96-beb55feb415a\") " pod="openshift-ingress/router-default-844dfdb99b-766v5" Sep 11 15:55:30.486738 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:30.486582 2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 11 15:55:30.486738 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:30.486605 2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-56c5d9df5d-7vqnb: secret "image-registry-tls" not found Sep 11 15:55:30.486738 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:30.486653 2566 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 11 15:55:30.486738 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:30.486657 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ae9022dc-da99-4acf-af85-3cf9858737b4-registry-tls podName:ae9022dc-da99-4acf-af85-3cf9858737b4 nodeName:}" failed. No retries permitted until 2026-09-11 15:55:34.486638728 +0000 UTC m=+41.701472625 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ae9022dc-da99-4acf-af85-3cf9858737b4-registry-tls") pod "image-registry-56c5d9df5d-7vqnb" (UID: "ae9022dc-da99-4acf-af85-3cf9858737b4") : secret "image-registry-tls" not found Sep 11 15:55:30.486738 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:30.486706 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/313238e9-79d7-45f7-8d96-beb55feb415a-metrics-certs podName:313238e9-79d7-45f7-8d96-beb55feb415a nodeName:}" failed. No retries permitted until 2026-09-11 15:55:34.486692222 +0000 UTC m=+41.701526110 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/313238e9-79d7-45f7-8d96-beb55feb415a-metrics-certs") pod "router-default-844dfdb99b-766v5" (UID: "313238e9-79d7-45f7-8d96-beb55feb415a") : secret "router-metrics-certs-default" not found Sep 11 15:55:30.587720 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:30.587679 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c2a34499-68a0-42b4-b77a-edc8b3b964c4-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-4hdz4\" (UID: \"c2a34499-68a0-42b4-b77a-edc8b3b964c4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-4hdz4" Sep 11 15:55:30.587871 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:30.587792 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/09f64aa8-4936-49d3-96f7-9895410e5dda-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-8q8dx\" (UID: \"09f64aa8-4936-49d3-96f7-9895410e5dda\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-8q8dx" Sep 11 15:55:30.589060 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:30.588485 2566 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 11 15:55:30.589060 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:30.588571 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2a34499-68a0-42b4-b77a-edc8b3b964c4-samples-operator-tls podName:c2a34499-68a0-42b4-b77a-edc8b3b964c4 nodeName:}" failed. No retries permitted until 2026-09-11 15:55:34.588547745 +0000 UTC m=+41.803381643 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c2a34499-68a0-42b4-b77a-edc8b3b964c4-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-4hdz4" (UID: "c2a34499-68a0-42b4-b77a-edc8b3b964c4") : secret "samples-operator-tls" not found Sep 11 15:55:30.589060 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:30.588655 2566 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 11 15:55:30.589060 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:30.588698 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/09f64aa8-4936-49d3-96f7-9895410e5dda-networking-console-plugin-cert podName:09f64aa8-4936-49d3-96f7-9895410e5dda nodeName:}" failed. No retries permitted until 2026-09-11 15:55:34.588685519 +0000 UTC m=+41.803519405 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/09f64aa8-4936-49d3-96f7-9895410e5dda-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-8q8dx" (UID: "09f64aa8-4936-49d3-96f7-9895410e5dda") : secret "networking-console-plugin-cert" not found Sep 11 15:55:30.689515 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:30.689431 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a1e7a08b-8aa9-42e9-aa46-3225f61299e0-metrics-tls\") pod \"dns-default-gx9fm\" (UID: \"a1e7a08b-8aa9-42e9-aa46-3225f61299e0\") " pod="openshift-dns/dns-default-gx9fm" Sep 11 15:55:30.689714 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:30.689626 2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 15:55:30.689714 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:30.689693 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1e7a08b-8aa9-42e9-aa46-3225f61299e0-metrics-tls podName:a1e7a08b-8aa9-42e9-aa46-3225f61299e0 nodeName:}" failed. No retries permitted until 2026-09-11 15:55:34.689674328 +0000 UTC m=+41.904508297 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a1e7a08b-8aa9-42e9-aa46-3225f61299e0-metrics-tls") pod "dns-default-gx9fm" (UID: "a1e7a08b-8aa9-42e9-aa46-3225f61299e0") : secret "dns-default-metrics-tls" not found Sep 11 15:55:34.524252 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:34.524207 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ae9022dc-da99-4acf-af85-3cf9858737b4-registry-tls\") pod \"image-registry-56c5d9df5d-7vqnb\" (UID: \"ae9022dc-da99-4acf-af85-3cf9858737b4\") " pod="openshift-image-registry/image-registry-56c5d9df5d-7vqnb" Sep 11 15:55:34.524709 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:34.524325 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/313238e9-79d7-45f7-8d96-beb55feb415a-metrics-certs\") pod \"router-default-844dfdb99b-766v5\" (UID: \"313238e9-79d7-45f7-8d96-beb55feb415a\") " pod="openshift-ingress/router-default-844dfdb99b-766v5" Sep 11 15:55:34.524709 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:34.524350 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5429cde4-cb80-4579-9687-e71617b15b9f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-kqtwh\" (UID: \"5429cde4-cb80-4579-9687-e71617b15b9f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-kqtwh" Sep 11 15:55:34.524709 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:34.524384 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/313238e9-79d7-45f7-8d96-beb55feb415a-service-ca-bundle\") pod \"router-default-844dfdb99b-766v5\" (UID: \"313238e9-79d7-45f7-8d96-beb55feb415a\") " pod="openshift-ingress/router-default-844dfdb99b-766v5" Sep 11 15:55:34.524709 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:34.524413 2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 11 15:55:34.524709 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:34.524440 2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-56c5d9df5d-7vqnb: secret "image-registry-tls" not found Sep 11 15:55:34.524709 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:34.524474 2566 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 11 15:55:34.524709 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:34.524508 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ae9022dc-da99-4acf-af85-3cf9858737b4-registry-tls podName:ae9022dc-da99-4acf-af85-3cf9858737b4 nodeName:}" failed. No retries permitted until 2026-09-11 15:55:42.524486639 +0000 UTC m=+49.739320548 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ae9022dc-da99-4acf-af85-3cf9858737b4-registry-tls") pod "image-registry-56c5d9df5d-7vqnb" (UID: "ae9022dc-da99-4acf-af85-3cf9858737b4") : secret "image-registry-tls" not found Sep 11 15:55:34.524709 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:34.524508 2566 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 11 15:55:34.524709 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:34.524537 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/313238e9-79d7-45f7-8d96-beb55feb415a-service-ca-bundle podName:313238e9-79d7-45f7-8d96-beb55feb415a nodeName:}" failed. No retries permitted until 2026-09-11 15:55:42.524518592 +0000 UTC m=+49.739352477 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/313238e9-79d7-45f7-8d96-beb55feb415a-service-ca-bundle") pod "router-default-844dfdb99b-766v5" (UID: "313238e9-79d7-45f7-8d96-beb55feb415a") : configmap references non-existent config key: service-ca.crt Sep 11 15:55:34.524709 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:34.524555 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/313238e9-79d7-45f7-8d96-beb55feb415a-metrics-certs podName:313238e9-79d7-45f7-8d96-beb55feb415a nodeName:}" failed. No retries permitted until 2026-09-11 15:55:42.52454593 +0000 UTC m=+49.739379815 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/313238e9-79d7-45f7-8d96-beb55feb415a-metrics-certs") pod "router-default-844dfdb99b-766v5" (UID: "313238e9-79d7-45f7-8d96-beb55feb415a") : secret "router-metrics-certs-default" not found Sep 11 15:55:34.524709 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:34.524579 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5429cde4-cb80-4579-9687-e71617b15b9f-cluster-monitoring-operator-tls podName:5429cde4-cb80-4579-9687-e71617b15b9f nodeName:}" failed. No retries permitted until 2026-09-11 15:55:42.524560809 +0000 UTC m=+49.739394709 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/5429cde4-cb80-4579-9687-e71617b15b9f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-kqtwh" (UID: "5429cde4-cb80-4579-9687-e71617b15b9f") : secret "cluster-monitoring-operator-tls" not found Sep 11 15:55:34.625094 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:34.625056 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c2a34499-68a0-42b4-b77a-edc8b3b964c4-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-4hdz4\" (UID: \"c2a34499-68a0-42b4-b77a-edc8b3b964c4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-4hdz4" Sep 11 15:55:34.625243 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:34.625147 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/09f64aa8-4936-49d3-96f7-9895410e5dda-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-8q8dx\" (UID: \"09f64aa8-4936-49d3-96f7-9895410e5dda\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-8q8dx" Sep 11 15:55:34.625243 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:34.625204 2566 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 11 15:55:34.625310 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:34.625269 2566 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 11 15:55:34.625310 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:34.625277 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2a34499-68a0-42b4-b77a-edc8b3b964c4-samples-operator-tls podName:c2a34499-68a0-42b4-b77a-edc8b3b964c4 nodeName:}" failed. No retries permitted until 2026-09-11 15:55:42.625257046 +0000 UTC m=+49.840090944 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c2a34499-68a0-42b4-b77a-edc8b3b964c4-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-4hdz4" (UID: "c2a34499-68a0-42b4-b77a-edc8b3b964c4") : secret "samples-operator-tls" not found Sep 11 15:55:34.625374 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:34.625324 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/09f64aa8-4936-49d3-96f7-9895410e5dda-networking-console-plugin-cert podName:09f64aa8-4936-49d3-96f7-9895410e5dda nodeName:}" failed. No retries permitted until 2026-09-11 15:55:42.625308289 +0000 UTC m=+49.840142172 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/09f64aa8-4936-49d3-96f7-9895410e5dda-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-8q8dx" (UID: "09f64aa8-4936-49d3-96f7-9895410e5dda") : secret "networking-console-plugin-cert" not found Sep 11 15:55:34.725753 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:34.725715 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a1e7a08b-8aa9-42e9-aa46-3225f61299e0-metrics-tls\") pod \"dns-default-gx9fm\" (UID: \"a1e7a08b-8aa9-42e9-aa46-3225f61299e0\") " pod="openshift-dns/dns-default-gx9fm" Sep 11 15:55:34.725895 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:34.725876 2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 15:55:34.725955 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:34.725945 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1e7a08b-8aa9-42e9-aa46-3225f61299e0-metrics-tls podName:a1e7a08b-8aa9-42e9-aa46-3225f61299e0 nodeName:}" failed. No retries permitted until 2026-09-11 15:55:42.725928802 +0000 UTC m=+49.940762686 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a1e7a08b-8aa9-42e9-aa46-3225f61299e0-metrics-tls") pod "dns-default-gx9fm" (UID: "a1e7a08b-8aa9-42e9-aa46-3225f61299e0") : secret "dns-default-metrics-tls" not found Sep 11 15:55:36.469899 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:36.469860 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-qjtlt" event={"ID":"26aadde0-9d1e-4368-86b0-4a2eb04656c3","Type":"ContainerStarted","Data":"83617c4adcdb33c090adfb71161c6a7e280f2b9a4b37017ea7886d14ae3a263c"} Sep 11 15:55:36.472484 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:36.472012 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-t82dg" event={"ID":"00183511-0324-43ff-9a5f-26dfacb9401c","Type":"ContainerStarted","Data":"ea1cc373f9f5895eb2f2cddd55e4e793c5e8b68e1acc0609769a7b3d5bd4e1ee"} Sep 11 15:55:36.474753 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:36.474429 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-nzcjl" event={"ID":"f6c05fce-e71f-4a9c-b219-a77b1df84ac8","Type":"ContainerStarted","Data":"55c1dc4349360c1ff2380eb6282388b1bd28011b654b4d0258dfd8f31e90060f"} Sep 11 15:55:36.474887 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:36.474849 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-nzcjl" Sep 11 15:55:36.477822 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:36.477796 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-ssz6b" event={"ID":"ddc77228-8fb6-473e-9940-7eae0c5457aa","Type":"ContainerStarted","Data":"c82a20fe5acc1f05ad984620e61d546a575fe68a2b3c7954d4ff998847bfb832"} Sep 11 15:55:36.480286 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:36.480234 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-djghf" event={"ID":"07d36306-7ec8-4c74-9b97-7545503a2783","Type":"ContainerStarted","Data":"b5450ba9123eea42e233adb54457f73d7a14b2e1cec3fcad65b5ae24d9314e2e"} Sep 11 15:55:36.482255 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:36.482234 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-4kl5b" event={"ID":"4d14aeeb-9182-4334-956a-d0cc21ca6633","Type":"ContainerStarted","Data":"4fd1fb1935578d8e8c55a48fdbbcd1203fe2d8624fb11a95371252df0e0ca433"} Sep 11 15:55:36.484078 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:36.484057 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-mjh4m" event={"ID":"9f6c5948-c774-49b6-aaf5-026c7b5db718","Type":"ContainerStarted","Data":"bd2d55ad0c91c8b05fd10361874923987f18d568a43b128b1af2facc4523d361"} Sep 11 15:55:36.486363 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:36.486344 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-xjvl5_7542ad77-c817-4370-91de-3b08db264753/console-operator/0.log" Sep 11 15:55:36.486472 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:36.486389 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-xjvl5" event={"ID":"7542ad77-c817-4370-91de-3b08db264753","Type":"ContainerStarted","Data":"d6500136322c1c5b0c90bc5127c96febf4ae625f2ab1e2bffe272d9d5a5fa369"} Sep 11 15:55:36.486648 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:36.486633 2566 scope.go:117] "RemoveContainer" containerID="d6500136322c1c5b0c90bc5127c96febf4ae625f2ab1e2bffe272d9d5a5fa369" Sep 11 15:55:36.490033 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:36.489501 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-qjtlt" podStartSLOduration=3.094466169 podStartE2EDuration="11.489476465s" totalImagesPullingTime="8.395010296s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:55:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:55:36.488023969 +0000 UTC m=+43.702857875" watchObservedRunningTime="2026-09-11 15:55:36.489476465 +0000 UTC m=+43.704310372" Sep 11 15:55:36.505141 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:36.504546 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-nzcjl" podStartSLOduration=35.102017902 podStartE2EDuration="43.504517755s" totalImagesPullingTime="8.402499853s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:54:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:55:36.50428555 +0000 UTC m=+43.719119457" watchObservedRunningTime="2026-09-11 15:55:36.504517755 +0000 UTC m=+43.719351656" Sep 11 15:55:36.525868 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:36.525684 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-djghf" podStartSLOduration=3.137222494 podStartE2EDuration="11.525659849s" totalImagesPullingTime="8.388437355s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:55:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:55:36.524399333 +0000 UTC m=+43.739233236" watchObservedRunningTime="2026-09-11 15:55:36.525659849 +0000 UTC m=+43.740493755" Sep 11 15:55:36.548103 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:36.546834 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5844b856db-mjh4m" podStartSLOduration=3.155668837 podStartE2EDuration="11.546801754s" totalImagesPullingTime="8.391132917s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:55:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:55:36.545571245 +0000 UTC m=+43.760405153" watchObservedRunningTime="2026-09-11 15:55:36.546801754 +0000 UTC m=+43.761635652" Sep 11 15:55:36.602544 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:36.602477 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f8b768455-t82dg" podStartSLOduration=3.212439042 podStartE2EDuration="11.602454518s" totalImagesPullingTime="8.390015476s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:55:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:55:36.58294452 +0000 UTC m=+43.797778426" watchObservedRunningTime="2026-09-11 15:55:36.602454518 +0000 UTC m=+43.817288424" Sep 11 15:55:36.623200 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:36.623136 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-ssz6b" podStartSLOduration=3.465826254 podStartE2EDuration="11.623100628s" totalImagesPullingTime="8.157274374s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:55:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:55:36.601752056 +0000 UTC m=+43.816585960" watchObservedRunningTime="2026-09-11 15:55:36.623100628 +0000 UTC m=+43.837934531" Sep 11 15:55:37.144552 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:37.144502 2566 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-xjvl5" Sep 11 15:55:37.144739 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:37.144614 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-xjvl5" Sep 11 15:55:37.484936 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:37.484833 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-4kl5b" podStartSLOduration=3.081212449 podStartE2EDuration="11.484814084s" totalImagesPullingTime="8.403601635s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:55:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:55:36.624695184 +0000 UTC m=+43.839529089" watchObservedRunningTime="2026-09-11 15:55:37.484814084 +0000 UTC m=+44.699647992" Sep 11 15:55:37.485340 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:37.485323 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-qdzw9"] Sep 11 15:55:37.491383 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:37.491361 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-xjvl5_7542ad77-c817-4370-91de-3b08db264753/console-operator/1.log" Sep 11 15:55:37.491812 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:37.491798 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-xjvl5_7542ad77-c817-4370-91de-3b08db264753/console-operator/0.log" Sep 11 15:55:37.491897 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:37.491833 2566 generic.go:356] "Generic (PLEG): container finished" podID="7542ad77-c817-4370-91de-3b08db264753" containerID="d6500136322c1c5b0c90bc5127c96febf4ae625f2ab1e2bffe272d9d5a5fa369" exitCode=255 Sep 11 15:55:37.491897 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:37.491847 2566 generic.go:356] "Generic (PLEG): container finished" podID="7542ad77-c817-4370-91de-3b08db264753" containerID="c3dd6745a01fe446bf4d5209534f46c7bcbeaa19660d0c793274d6c3128ee27a" exitCode=255 Sep 11 15:55:37.493230 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:37.493210 2566 scope.go:117] "RemoveContainer" containerID="c3dd6745a01fe446bf4d5209534f46c7bcbeaa19660d0c793274d6c3128ee27a" Sep 11 15:55:37.493377 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:37.493357 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-xjvl5_openshift-console-operator(7542ad77-c817-4370-91de-3b08db264753)\"" pod="openshift-console-operator/console-operator-696884fb9c-xjvl5" podUID="7542ad77-c817-4370-91de-3b08db264753" Sep 11 15:55:37.505381 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:37.505363 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-qdzw9"] Sep 11 15:55:37.505547 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:37.505522 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-xjvl5" event={"ID":"7542ad77-c817-4370-91de-3b08db264753","Type":"ContainerDied","Data":"d6500136322c1c5b0c90bc5127c96febf4ae625f2ab1e2bffe272d9d5a5fa369"} Sep 11 15:55:37.505675 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:37.505558 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-xjvl5" event={"ID":"7542ad77-c817-4370-91de-3b08db264753","Type":"ContainerDied","Data":"c3dd6745a01fe446bf4d5209534f46c7bcbeaa19660d0c793274d6c3128ee27a"} Sep 11 15:55:37.505675 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:37.505596 2566 scope.go:117] "RemoveContainer" containerID="d6500136322c1c5b0c90bc5127c96febf4ae625f2ab1e2bffe272d9d5a5fa369" Sep 11 15:55:37.505800 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:37.505750 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-qdzw9" Sep 11 15:55:37.508721 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:37.508595 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 11 15:55:37.508721 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:37.508601 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 11 15:55:37.509073 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:37.509053 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-q4ddf\"" Sep 11 15:55:37.538690 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:37.538671 2566 scope.go:117] "RemoveContainer" containerID="d6500136322c1c5b0c90bc5127c96febf4ae625f2ab1e2bffe272d9d5a5fa369" Sep 11 15:55:37.539151 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:37.539123 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6500136322c1c5b0c90bc5127c96febf4ae625f2ab1e2bffe272d9d5a5fa369\": container with ID starting with d6500136322c1c5b0c90bc5127c96febf4ae625f2ab1e2bffe272d9d5a5fa369 not found: ID does not exist" containerID="d6500136322c1c5b0c90bc5127c96febf4ae625f2ab1e2bffe272d9d5a5fa369" Sep 11 15:55:37.539248 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:37.539162 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6500136322c1c5b0c90bc5127c96febf4ae625f2ab1e2bffe272d9d5a5fa369"} err="failed to get container status \"d6500136322c1c5b0c90bc5127c96febf4ae625f2ab1e2bffe272d9d5a5fa369\": rpc error: code = NotFound desc = could not find container \"d6500136322c1c5b0c90bc5127c96febf4ae625f2ab1e2bffe272d9d5a5fa369\": container with ID starting with d6500136322c1c5b0c90bc5127c96febf4ae625f2ab1e2bffe272d9d5a5fa369 not found: ID does not exist" Sep 11 15:55:37.552074 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:37.552050 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/715b88ae-4c15-451d-acda-67b2ec875590-data-volume\") pod \"insights-runtime-extractor-qdzw9\" (UID: \"715b88ae-4c15-451d-acda-67b2ec875590\") " pod="openshift-insights/insights-runtime-extractor-qdzw9" Sep 11 15:55:37.552424 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:37.552185 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/715b88ae-4c15-451d-acda-67b2ec875590-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qdzw9\" (UID: \"715b88ae-4c15-451d-acda-67b2ec875590\") " pod="openshift-insights/insights-runtime-extractor-qdzw9" Sep 11 15:55:37.552424 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:37.552302 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/715b88ae-4c15-451d-acda-67b2ec875590-crio-socket\") pod \"insights-runtime-extractor-qdzw9\" (UID: \"715b88ae-4c15-451d-acda-67b2ec875590\") " pod="openshift-insights/insights-runtime-extractor-qdzw9" Sep 11 15:55:37.553142 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:37.552671 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/715b88ae-4c15-451d-acda-67b2ec875590-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qdzw9\" (UID: \"715b88ae-4c15-451d-acda-67b2ec875590\") " pod="openshift-insights/insights-runtime-extractor-qdzw9" Sep 11 15:55:37.553142 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:37.552842 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwhkl\" (UniqueName: \"kubernetes.io/projected/715b88ae-4c15-451d-acda-67b2ec875590-kube-api-access-bwhkl\") pod \"insights-runtime-extractor-qdzw9\" (UID: \"715b88ae-4c15-451d-acda-67b2ec875590\") " pod="openshift-insights/insights-runtime-extractor-qdzw9" Sep 11 15:55:37.653775 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:37.653718 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/715b88ae-4c15-451d-acda-67b2ec875590-crio-socket\") pod \"insights-runtime-extractor-qdzw9\" (UID: \"715b88ae-4c15-451d-acda-67b2ec875590\") " pod="openshift-insights/insights-runtime-extractor-qdzw9" Sep 11 15:55:37.653938 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:37.653826 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/715b88ae-4c15-451d-acda-67b2ec875590-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qdzw9\" (UID: \"715b88ae-4c15-451d-acda-67b2ec875590\") " pod="openshift-insights/insights-runtime-extractor-qdzw9" Sep 11 15:55:37.653938 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:37.653845 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwhkl\" (UniqueName: \"kubernetes.io/projected/715b88ae-4c15-451d-acda-67b2ec875590-kube-api-access-bwhkl\") pod \"insights-runtime-extractor-qdzw9\" (UID: \"715b88ae-4c15-451d-acda-67b2ec875590\") " pod="openshift-insights/insights-runtime-extractor-qdzw9" Sep 11 15:55:37.653938 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:37.653865 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/715b88ae-4c15-451d-acda-67b2ec875590-crio-socket\") pod \"insights-runtime-extractor-qdzw9\" (UID: \"715b88ae-4c15-451d-acda-67b2ec875590\") " pod="openshift-insights/insights-runtime-extractor-qdzw9" Sep 11 15:55:37.653938 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:37.653924 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/715b88ae-4c15-451d-acda-67b2ec875590-data-volume\") pod \"insights-runtime-extractor-qdzw9\" (UID: \"715b88ae-4c15-451d-acda-67b2ec875590\") " pod="openshift-insights/insights-runtime-extractor-qdzw9" Sep 11 15:55:37.654129 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:37.653996 2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 11 15:55:37.654129 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:37.654014 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/715b88ae-4c15-451d-acda-67b2ec875590-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qdzw9\" (UID: \"715b88ae-4c15-451d-acda-67b2ec875590\") " pod="openshift-insights/insights-runtime-extractor-qdzw9" Sep 11 15:55:37.654129 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:37.654060 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/715b88ae-4c15-451d-acda-67b2ec875590-insights-runtime-extractor-tls podName:715b88ae-4c15-451d-acda-67b2ec875590 nodeName:}" failed. No retries permitted until 2026-09-11 15:55:38.154041124 +0000 UTC m=+45.368875027 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/715b88ae-4c15-451d-acda-67b2ec875590-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qdzw9" (UID: "715b88ae-4c15-451d-acda-67b2ec875590") : secret "insights-runtime-extractor-tls" not found Sep 11 15:55:37.654129 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:37.654075 2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 11 15:55:37.654320 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:37.654149 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/715b88ae-4c15-451d-acda-67b2ec875590-kube-rbac-proxy-cm podName:715b88ae-4c15-451d-acda-67b2ec875590 nodeName:}" failed. No retries permitted until 2026-09-11 15:55:38.154132733 +0000 UTC m=+45.368966630 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/715b88ae-4c15-451d-acda-67b2ec875590-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qdzw9" (UID: "715b88ae-4c15-451d-acda-67b2ec875590") : configmap "kube-rbac-proxy" not found Sep 11 15:55:37.654377 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:37.654316 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/715b88ae-4c15-451d-acda-67b2ec875590-data-volume\") pod \"insights-runtime-extractor-qdzw9\" (UID: \"715b88ae-4c15-451d-acda-67b2ec875590\") " pod="openshift-insights/insights-runtime-extractor-qdzw9" Sep 11 15:55:37.664151 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:37.664118 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwhkl\" (UniqueName: \"kubernetes.io/projected/715b88ae-4c15-451d-acda-67b2ec875590-kube-api-access-bwhkl\") pod \"insights-runtime-extractor-qdzw9\" (UID: \"715b88ae-4c15-451d-acda-67b2ec875590\") " pod="openshift-insights/insights-runtime-extractor-qdzw9" Sep 11 15:55:38.159382 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:38.159343 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/715b88ae-4c15-451d-acda-67b2ec875590-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qdzw9\" (UID: \"715b88ae-4c15-451d-acda-67b2ec875590\") " pod="openshift-insights/insights-runtime-extractor-qdzw9" Sep 11 15:55:38.159536 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:38.159497 2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 11 15:55:38.159536 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:38.159515 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/715b88ae-4c15-451d-acda-67b2ec875590-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qdzw9\" (UID: \"715b88ae-4c15-451d-acda-67b2ec875590\") " pod="openshift-insights/insights-runtime-extractor-qdzw9" Sep 11 15:55:38.159634 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:38.159569 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/715b88ae-4c15-451d-acda-67b2ec875590-insights-runtime-extractor-tls podName:715b88ae-4c15-451d-acda-67b2ec875590 nodeName:}" failed. No retries permitted until 2026-09-11 15:55:39.159553814 +0000 UTC m=+46.374387710 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/715b88ae-4c15-451d-acda-67b2ec875590-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qdzw9" (UID: "715b88ae-4c15-451d-acda-67b2ec875590") : secret "insights-runtime-extractor-tls" not found Sep 11 15:55:38.159634 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:38.159619 2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 11 15:55:38.159715 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:38.159678 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/715b88ae-4c15-451d-acda-67b2ec875590-kube-rbac-proxy-cm podName:715b88ae-4c15-451d-acda-67b2ec875590 nodeName:}" failed. No retries permitted until 2026-09-11 15:55:39.159665285 +0000 UTC m=+46.374499168 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/715b88ae-4c15-451d-acda-67b2ec875590-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qdzw9" (UID: "715b88ae-4c15-451d-acda-67b2ec875590") : configmap "kube-rbac-proxy" not found Sep 11 15:55:38.496644 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:38.496576 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-xjvl5_7542ad77-c817-4370-91de-3b08db264753/console-operator/1.log" Sep 11 15:55:38.500289 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:38.497322 2566 scope.go:117] "RemoveContainer" containerID="c3dd6745a01fe446bf4d5209534f46c7bcbeaa19660d0c793274d6c3128ee27a" Sep 11 15:55:38.500289 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:38.497600 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-xjvl5_openshift-console-operator(7542ad77-c817-4370-91de-3b08db264753)\"" pod="openshift-console-operator/console-operator-696884fb9c-xjvl5" podUID="7542ad77-c817-4370-91de-3b08db264753" Sep 11 15:55:39.167135 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:39.167097 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/715b88ae-4c15-451d-acda-67b2ec875590-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qdzw9\" (UID: \"715b88ae-4c15-451d-acda-67b2ec875590\") " pod="openshift-insights/insights-runtime-extractor-qdzw9" Sep 11 15:55:39.167307 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:39.167214 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/715b88ae-4c15-451d-acda-67b2ec875590-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qdzw9\" (UID: \"715b88ae-4c15-451d-acda-67b2ec875590\") " pod="openshift-insights/insights-runtime-extractor-qdzw9" Sep 11 15:55:39.167307 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:39.167254 2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 11 15:55:39.167424 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:39.167322 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/715b88ae-4c15-451d-acda-67b2ec875590-insights-runtime-extractor-tls podName:715b88ae-4c15-451d-acda-67b2ec875590 nodeName:}" failed. No retries permitted until 2026-09-11 15:55:41.167305916 +0000 UTC m=+48.382139798 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/715b88ae-4c15-451d-acda-67b2ec875590-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qdzw9" (UID: "715b88ae-4c15-451d-acda-67b2ec875590") : secret "insights-runtime-extractor-tls" not found Sep 11 15:55:39.167424 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:39.167328 2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 11 15:55:39.167424 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:39.167380 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/715b88ae-4c15-451d-acda-67b2ec875590-kube-rbac-proxy-cm podName:715b88ae-4c15-451d-acda-67b2ec875590 nodeName:}" failed. No retries permitted until 2026-09-11 15:55:41.167367118 +0000 UTC m=+48.382201013 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/715b88ae-4c15-451d-acda-67b2ec875590-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qdzw9" (UID: "715b88ae-4c15-451d-acda-67b2ec875590") : configmap "kube-rbac-proxy" not found Sep 11 15:55:39.228830 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:39.228806 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-sxqj2_ed90592a-177c-4128-b153-b51d18b8f619/dns-node-resolver/0.log" Sep 11 15:55:39.500237 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:39.500166 2566 scope.go:117] "RemoveContainer" containerID="c3dd6745a01fe446bf4d5209534f46c7bcbeaa19660d0c793274d6c3128ee27a" Sep 11 15:55:39.500601 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:39.500325 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-xjvl5_openshift-console-operator(7542ad77-c817-4370-91de-3b08db264753)\"" pod="openshift-console-operator/console-operator-696884fb9c-xjvl5" podUID="7542ad77-c817-4370-91de-3b08db264753" Sep 11 15:55:39.828210 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:39.828180 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bctd5_249a0e7e-1da6-422a-8492-1820f6437318/node-ca/0.log" Sep 11 15:55:40.628027 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:40.627998 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-4kl5b_4d14aeeb-9182-4334-956a-d0cc21ca6633/serve-healthcheck-canary/0.log" Sep 11 15:55:41.184688 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:41.184653 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/715b88ae-4c15-451d-acda-67b2ec875590-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qdzw9\" (UID: \"715b88ae-4c15-451d-acda-67b2ec875590\") " pod="openshift-insights/insights-runtime-extractor-qdzw9" Sep 11 15:55:41.184867 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:41.184743 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/715b88ae-4c15-451d-acda-67b2ec875590-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qdzw9\" (UID: \"715b88ae-4c15-451d-acda-67b2ec875590\") " pod="openshift-insights/insights-runtime-extractor-qdzw9" Sep 11 15:55:41.184867 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:41.184774 2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 11 15:55:41.184867 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:41.184839 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/715b88ae-4c15-451d-acda-67b2ec875590-kube-rbac-proxy-cm podName:715b88ae-4c15-451d-acda-67b2ec875590 nodeName:}" failed. No retries permitted until 2026-09-11 15:55:45.184824289 +0000 UTC m=+52.399658172 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/715b88ae-4c15-451d-acda-67b2ec875590-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qdzw9" (UID: "715b88ae-4c15-451d-acda-67b2ec875590") : configmap "kube-rbac-proxy" not found Sep 11 15:55:41.184867 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:41.184840 2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 11 15:55:41.185053 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:41.184884 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/715b88ae-4c15-451d-acda-67b2ec875590-insights-runtime-extractor-tls podName:715b88ae-4c15-451d-acda-67b2ec875590 nodeName:}" failed. No retries permitted until 2026-09-11 15:55:45.184872016 +0000 UTC m=+52.399705916 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/715b88ae-4c15-451d-acda-67b2ec875590-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qdzw9" (UID: "715b88ae-4c15-451d-acda-67b2ec875590") : secret "insights-runtime-extractor-tls" not found Sep 11 15:55:42.596393 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:42.596355 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5429cde4-cb80-4579-9687-e71617b15b9f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-kqtwh\" (UID: \"5429cde4-cb80-4579-9687-e71617b15b9f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-kqtwh" Sep 11 15:55:42.596791 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:42.596413 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/313238e9-79d7-45f7-8d96-beb55feb415a-service-ca-bundle\") pod \"router-default-844dfdb99b-766v5\" (UID: \"313238e9-79d7-45f7-8d96-beb55feb415a\") " pod="openshift-ingress/router-default-844dfdb99b-766v5" Sep 11 15:55:42.596791 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:42.596451 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ae9022dc-da99-4acf-af85-3cf9858737b4-registry-tls\") pod \"image-registry-56c5d9df5d-7vqnb\" (UID: \"ae9022dc-da99-4acf-af85-3cf9858737b4\") " pod="openshift-image-registry/image-registry-56c5d9df5d-7vqnb" Sep 11 15:55:42.596791 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:42.596517 2566 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 11 15:55:42.596791 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:42.596533 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/313238e9-79d7-45f7-8d96-beb55feb415a-metrics-certs\") pod \"router-default-844dfdb99b-766v5\" (UID: \"313238e9-79d7-45f7-8d96-beb55feb415a\") " pod="openshift-ingress/router-default-844dfdb99b-766v5" Sep 11 15:55:42.596791 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:42.596584 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5429cde4-cb80-4579-9687-e71617b15b9f-cluster-monitoring-operator-tls podName:5429cde4-cb80-4579-9687-e71617b15b9f nodeName:}" failed. No retries permitted until 2026-09-11 15:55:58.596569217 +0000 UTC m=+65.811403099 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/5429cde4-cb80-4579-9687-e71617b15b9f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-kqtwh" (UID: "5429cde4-cb80-4579-9687-e71617b15b9f") : secret "cluster-monitoring-operator-tls" not found Sep 11 15:55:42.596791 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:42.596599 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/313238e9-79d7-45f7-8d96-beb55feb415a-service-ca-bundle podName:313238e9-79d7-45f7-8d96-beb55feb415a nodeName:}" failed. No retries permitted until 2026-09-11 15:55:58.596593153 +0000 UTC m=+65.811427036 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/313238e9-79d7-45f7-8d96-beb55feb415a-service-ca-bundle") pod "router-default-844dfdb99b-766v5" (UID: "313238e9-79d7-45f7-8d96-beb55feb415a") : configmap references non-existent config key: service-ca.crt Sep 11 15:55:42.596791 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:42.596641 2566 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 11 15:55:42.596791 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:42.596646 2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 11 15:55:42.596791 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:42.596662 2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-56c5d9df5d-7vqnb: secret "image-registry-tls" not found Sep 11 15:55:42.596791 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:42.596695 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/313238e9-79d7-45f7-8d96-beb55feb415a-metrics-certs podName:313238e9-79d7-45f7-8d96-beb55feb415a nodeName:}" failed. No retries permitted until 2026-09-11 15:55:58.596680351 +0000 UTC m=+65.811514252 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/313238e9-79d7-45f7-8d96-beb55feb415a-metrics-certs") pod "router-default-844dfdb99b-766v5" (UID: "313238e9-79d7-45f7-8d96-beb55feb415a") : secret "router-metrics-certs-default" not found Sep 11 15:55:42.596791 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:42.596710 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ae9022dc-da99-4acf-af85-3cf9858737b4-registry-tls podName:ae9022dc-da99-4acf-af85-3cf9858737b4 nodeName:}" failed. No retries permitted until 2026-09-11 15:55:58.596703857 +0000 UTC m=+65.811537740 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ae9022dc-da99-4acf-af85-3cf9858737b4-registry-tls") pod "image-registry-56c5d9df5d-7vqnb" (UID: "ae9022dc-da99-4acf-af85-3cf9858737b4") : secret "image-registry-tls" not found Sep 11 15:55:42.697780 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:42.697743 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/09f64aa8-4936-49d3-96f7-9895410e5dda-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-8q8dx\" (UID: \"09f64aa8-4936-49d3-96f7-9895410e5dda\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-8q8dx" Sep 11 15:55:42.697932 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:42.697869 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c2a34499-68a0-42b4-b77a-edc8b3b964c4-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-4hdz4\" (UID: \"c2a34499-68a0-42b4-b77a-edc8b3b964c4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-4hdz4" Sep 11 15:55:42.697932 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:42.697908 2566 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 11 15:55:42.698036 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:42.697964 2566 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 11 15:55:42.698036 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:42.697994 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/09f64aa8-4936-49d3-96f7-9895410e5dda-networking-console-plugin-cert podName:09f64aa8-4936-49d3-96f7-9895410e5dda nodeName:}" failed. No retries permitted until 2026-09-11 15:55:58.697963348 +0000 UTC m=+65.912797251 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/09f64aa8-4936-49d3-96f7-9895410e5dda-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-8q8dx" (UID: "09f64aa8-4936-49d3-96f7-9895410e5dda") : secret "networking-console-plugin-cert" not found Sep 11 15:55:42.698036 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:42.698021 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2a34499-68a0-42b4-b77a-edc8b3b964c4-samples-operator-tls podName:c2a34499-68a0-42b4-b77a-edc8b3b964c4 nodeName:}" failed. No retries permitted until 2026-09-11 15:55:58.698010122 +0000 UTC m=+65.912844005 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c2a34499-68a0-42b4-b77a-edc8b3b964c4-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-4hdz4" (UID: "c2a34499-68a0-42b4-b77a-edc8b3b964c4") : secret "samples-operator-tls" not found Sep 11 15:55:42.798483 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:42.798446 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a1e7a08b-8aa9-42e9-aa46-3225f61299e0-metrics-tls\") pod \"dns-default-gx9fm\" (UID: \"a1e7a08b-8aa9-42e9-aa46-3225f61299e0\") " pod="openshift-dns/dns-default-gx9fm" Sep 11 15:55:42.798606 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:42.798564 2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 15:55:42.798657 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:42.798613 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1e7a08b-8aa9-42e9-aa46-3225f61299e0-metrics-tls podName:a1e7a08b-8aa9-42e9-aa46-3225f61299e0 nodeName:}" failed. No retries permitted until 2026-09-11 15:55:58.798600548 +0000 UTC m=+66.013434445 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a1e7a08b-8aa9-42e9-aa46-3225f61299e0-metrics-tls") pod "dns-default-gx9fm" (UID: "a1e7a08b-8aa9-42e9-aa46-3225f61299e0") : secret "dns-default-metrics-tls" not found Sep 11 15:55:45.221306 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:45.221265 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/715b88ae-4c15-451d-acda-67b2ec875590-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qdzw9\" (UID: \"715b88ae-4c15-451d-acda-67b2ec875590\") " pod="openshift-insights/insights-runtime-extractor-qdzw9" Sep 11 15:55:45.221678 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:45.221367 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/715b88ae-4c15-451d-acda-67b2ec875590-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qdzw9\" (UID: \"715b88ae-4c15-451d-acda-67b2ec875590\") " pod="openshift-insights/insights-runtime-extractor-qdzw9" Sep 11 15:55:45.221678 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:45.221488 2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 11 15:55:45.221678 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:45.221555 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/715b88ae-4c15-451d-acda-67b2ec875590-insights-runtime-extractor-tls podName:715b88ae-4c15-451d-acda-67b2ec875590 nodeName:}" failed. No retries permitted until 2026-09-11 15:55:53.221537744 +0000 UTC m=+60.436371628 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/715b88ae-4c15-451d-acda-67b2ec875590-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qdzw9" (UID: "715b88ae-4c15-451d-acda-67b2ec875590") : secret "insights-runtime-extractor-tls" not found Sep 11 15:55:45.221820 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:45.221797 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/715b88ae-4c15-451d-acda-67b2ec875590-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qdzw9\" (UID: \"715b88ae-4c15-451d-acda-67b2ec875590\") " pod="openshift-insights/insights-runtime-extractor-qdzw9" Sep 11 15:55:47.144374 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:47.144335 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-xjvl5" Sep 11 15:55:47.144826 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:47.144406 2566 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-xjvl5" Sep 11 15:55:47.144912 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:47.144842 2566 scope.go:117] "RemoveContainer" containerID="c3dd6745a01fe446bf4d5209534f46c7bcbeaa19660d0c793274d6c3128ee27a" Sep 11 15:55:47.527431 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:47.527407 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-xjvl5_7542ad77-c817-4370-91de-3b08db264753/console-operator/2.log" Sep 11 15:55:47.527812 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:47.527793 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-xjvl5_7542ad77-c817-4370-91de-3b08db264753/console-operator/1.log" Sep 11 15:55:47.527904 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:47.527833 2566 generic.go:356] "Generic (PLEG): container finished" podID="7542ad77-c817-4370-91de-3b08db264753" containerID="56fd95fcc18f51f8f027556616d2de24aea35a383d6ec6817332b3d49b2e7ddf" exitCode=255 Sep 11 15:55:47.527904 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:47.527894 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-xjvl5" event={"ID":"7542ad77-c817-4370-91de-3b08db264753","Type":"ContainerDied","Data":"56fd95fcc18f51f8f027556616d2de24aea35a383d6ec6817332b3d49b2e7ddf"} Sep 11 15:55:47.528033 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:47.527952 2566 scope.go:117] "RemoveContainer" containerID="c3dd6745a01fe446bf4d5209534f46c7bcbeaa19660d0c793274d6c3128ee27a" Sep 11 15:55:47.528297 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:47.528268 2566 scope.go:117] "RemoveContainer" containerID="56fd95fcc18f51f8f027556616d2de24aea35a383d6ec6817332b3d49b2e7ddf" Sep 11 15:55:47.528457 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:47.528438 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-696884fb9c-xjvl5_openshift-console-operator(7542ad77-c817-4370-91de-3b08db264753)\"" pod="openshift-console-operator/console-operator-696884fb9c-xjvl5" podUID="7542ad77-c817-4370-91de-3b08db264753" Sep 11 15:55:48.533119 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:48.533086 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-xjvl5_7542ad77-c817-4370-91de-3b08db264753/console-operator/2.log" Sep 11 15:55:53.290773 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:53.290739 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/715b88ae-4c15-451d-acda-67b2ec875590-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qdzw9\" (UID: \"715b88ae-4c15-451d-acda-67b2ec875590\") " pod="openshift-insights/insights-runtime-extractor-qdzw9" Sep 11 15:55:53.293020 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:53.292996 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/715b88ae-4c15-451d-acda-67b2ec875590-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qdzw9\" (UID: \"715b88ae-4c15-451d-acda-67b2ec875590\") " pod="openshift-insights/insights-runtime-extractor-qdzw9" Sep 11 15:55:53.394764 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:53.394736 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tgq5c" Sep 11 15:55:53.425328 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:53.425300 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-q4ddf\"" Sep 11 15:55:53.434297 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:53.434279 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-qdzw9" Sep 11 15:55:53.705339 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:53.705253 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-qdzw9"] Sep 11 15:55:53.707993 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:55:53.707947 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod715b88ae_4c15_451d_acda_67b2ec875590.slice/crio-933bdee6dc71a0fd9dd01f82c8fac329fa1aff018b1618b19b7a698bfafe74c4 WatchSource:0}: Error finding container 933bdee6dc71a0fd9dd01f82c8fac329fa1aff018b1618b19b7a698bfafe74c4: Status 404 returned error can't find the container with id 933bdee6dc71a0fd9dd01f82c8fac329fa1aff018b1618b19b7a698bfafe74c4 Sep 11 15:55:54.552618 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:54.552586 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qdzw9" event={"ID":"715b88ae-4c15-451d-acda-67b2ec875590","Type":"ContainerStarted","Data":"70c6f5b28d9f49d8b76b9c127b03966376c39995c0be2831ebc93f22f3f15740"} Sep 11 15:55:54.553005 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:54.552648 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qdzw9" event={"ID":"715b88ae-4c15-451d-acda-67b2ec875590","Type":"ContainerStarted","Data":"933bdee6dc71a0fd9dd01f82c8fac329fa1aff018b1618b19b7a698bfafe74c4"} Sep 11 15:55:55.557910 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:55.557872 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qdzw9" event={"ID":"715b88ae-4c15-451d-acda-67b2ec875590","Type":"ContainerStarted","Data":"88c9697bbfa6d93f47b98d93ed1996aa5ec517cc522f9d4b8b67fb00ef1f83d9"} Sep 11 15:55:56.563821 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:56.563778 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qdzw9" event={"ID":"715b88ae-4c15-451d-acda-67b2ec875590","Type":"ContainerStarted","Data":"0744863abe0e395aef5a4954470f0a24476144b26036af6bcc15a70d1dcb9538"} Sep 11 15:55:56.582408 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:56.582358 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-qdzw9" podStartSLOduration=16.973093745 podStartE2EDuration="19.582343539s" totalImagesPullingTime="2.609249794s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 15:55:37 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:55:56.580741547 +0000 UTC m=+63.795575453" watchObservedRunningTime="2026-09-11 15:55:56.582343539 +0000 UTC m=+63.797177443" Sep 11 15:55:57.144620 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:57.144584 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-xjvl5" Sep 11 15:55:57.144814 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:57.144652 2566 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-xjvl5" Sep 11 15:55:57.145053 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:57.145036 2566 scope.go:117] "RemoveContainer" containerID="56fd95fcc18f51f8f027556616d2de24aea35a383d6ec6817332b3d49b2e7ddf" Sep 11 15:55:57.145234 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:55:57.145212 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-696884fb9c-xjvl5_openshift-console-operator(7542ad77-c817-4370-91de-3b08db264753)\"" pod="openshift-console-operator/console-operator-696884fb9c-xjvl5" podUID="7542ad77-c817-4370-91de-3b08db264753" Sep 11 15:55:58.636565 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:58.636528 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ae9022dc-da99-4acf-af85-3cf9858737b4-registry-tls\") pod \"image-registry-56c5d9df5d-7vqnb\" (UID: \"ae9022dc-da99-4acf-af85-3cf9858737b4\") " pod="openshift-image-registry/image-registry-56c5d9df5d-7vqnb" Sep 11 15:55:58.636957 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:58.636728 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/313238e9-79d7-45f7-8d96-beb55feb415a-metrics-certs\") pod \"router-default-844dfdb99b-766v5\" (UID: \"313238e9-79d7-45f7-8d96-beb55feb415a\") " pod="openshift-ingress/router-default-844dfdb99b-766v5" Sep 11 15:55:58.636957 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:58.636764 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5429cde4-cb80-4579-9687-e71617b15b9f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-kqtwh\" (UID: \"5429cde4-cb80-4579-9687-e71617b15b9f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-kqtwh" Sep 11 15:55:58.636957 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:58.636906 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/313238e9-79d7-45f7-8d96-beb55feb415a-service-ca-bundle\") pod \"router-default-844dfdb99b-766v5\" (UID: \"313238e9-79d7-45f7-8d96-beb55feb415a\") " pod="openshift-ingress/router-default-844dfdb99b-766v5" Sep 11 15:55:58.637550 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:58.637524 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/313238e9-79d7-45f7-8d96-beb55feb415a-service-ca-bundle\") pod \"router-default-844dfdb99b-766v5\" (UID: \"313238e9-79d7-45f7-8d96-beb55feb415a\") " pod="openshift-ingress/router-default-844dfdb99b-766v5" Sep 11 15:55:58.639053 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:58.639030 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ae9022dc-da99-4acf-af85-3cf9858737b4-registry-tls\") pod \"image-registry-56c5d9df5d-7vqnb\" (UID: \"ae9022dc-da99-4acf-af85-3cf9858737b4\") " pod="openshift-image-registry/image-registry-56c5d9df5d-7vqnb" Sep 11 15:55:58.639140 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:58.639062 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/313238e9-79d7-45f7-8d96-beb55feb415a-metrics-certs\") pod \"router-default-844dfdb99b-766v5\" (UID: \"313238e9-79d7-45f7-8d96-beb55feb415a\") " pod="openshift-ingress/router-default-844dfdb99b-766v5" Sep 11 15:55:58.639140 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:58.639087 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5429cde4-cb80-4579-9687-e71617b15b9f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-kqtwh\" (UID: \"5429cde4-cb80-4579-9687-e71617b15b9f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-kqtwh" Sep 11 15:55:58.737606 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:58.737571 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c2a34499-68a0-42b4-b77a-edc8b3b964c4-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-4hdz4\" (UID: \"c2a34499-68a0-42b4-b77a-edc8b3b964c4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-4hdz4" Sep 11 15:55:58.737797 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:58.737624 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/09f64aa8-4936-49d3-96f7-9895410e5dda-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-8q8dx\" (UID: \"09f64aa8-4936-49d3-96f7-9895410e5dda\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-8q8dx" Sep 11 15:55:58.739965 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:58.739929 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/09f64aa8-4936-49d3-96f7-9895410e5dda-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-8q8dx\" (UID: \"09f64aa8-4936-49d3-96f7-9895410e5dda\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-8q8dx" Sep 11 15:55:58.740134 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:58.740111 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c2a34499-68a0-42b4-b77a-edc8b3b964c4-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-4hdz4\" (UID: \"c2a34499-68a0-42b4-b77a-edc8b3b964c4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-4hdz4" Sep 11 15:55:58.796473 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:58.796444 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-n6wjl\"" Sep 11 15:55:58.805185 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:58.805164 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-56c5d9df5d-7vqnb" Sep 11 15:55:58.815373 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:58.815352 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-zk78d\"" Sep 11 15:55:58.823994 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:58.823965 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-844dfdb99b-766v5" Sep 11 15:55:58.831606 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:58.831591 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-gpzqq\"" Sep 11 15:55:58.838424 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:58.838395 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a1e7a08b-8aa9-42e9-aa46-3225f61299e0-metrics-tls\") pod \"dns-default-gx9fm\" (UID: \"a1e7a08b-8aa9-42e9-aa46-3225f61299e0\") " pod="openshift-dns/dns-default-gx9fm" Sep 11 15:55:58.839780 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:58.839761 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-kqtwh" Sep 11 15:55:58.840578 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:58.840554 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a1e7a08b-8aa9-42e9-aa46-3225f61299e0-metrics-tls\") pod \"dns-default-gx9fm\" (UID: \"a1e7a08b-8aa9-42e9-aa46-3225f61299e0\") " pod="openshift-dns/dns-default-gx9fm" Sep 11 15:55:58.889917 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:58.889856 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-m665j\"" Sep 11 15:55:58.903152 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:58.902417 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-4hdz4" Sep 11 15:55:58.961531 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:58.960632 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-lhmcj\"" Sep 11 15:55:58.968213 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:58.968165 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-8q8dx" Sep 11 15:55:58.991022 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:58.991000 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-jvv4c\"" Sep 11 15:55:58.999713 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:58.999684 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gx9fm" Sep 11 15:55:59.039657 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:59.039598 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/331632d0-fa59-4462-805e-bce0e18b95bc-metrics-certs\") pod \"network-metrics-daemon-plkhx\" (UID: \"331632d0-fa59-4462-805e-bce0e18b95bc\") " pod="openshift-multus/network-metrics-daemon-plkhx" Sep 11 15:55:59.042299 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:59.042278 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 11 15:55:59.052575 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:59.052492 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/331632d0-fa59-4462-805e-bce0e18b95bc-metrics-certs\") pod \"network-metrics-daemon-plkhx\" (UID: \"331632d0-fa59-4462-805e-bce0e18b95bc\") " pod="openshift-multus/network-metrics-daemon-plkhx" Sep 11 15:55:59.059637 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:59.059619 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zsbjs\"" Sep 11 15:55:59.067581 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:59.067561 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-plkhx" Sep 11 15:55:59.112010 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:59.111951 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-56c5d9df5d-7vqnb"] Sep 11 15:55:59.114401 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:55:59.114379 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae9022dc_da99_4acf_af85_3cf9858737b4.slice/crio-3f4337e42e3663b9b51a15f30d9673a26fdcc74394e00b05ad99395b4bd1999b WatchSource:0}: Error finding container 3f4337e42e3663b9b51a15f30d9673a26fdcc74394e00b05ad99395b4bd1999b: Status 404 returned error can't find the container with id 3f4337e42e3663b9b51a15f30d9673a26fdcc74394e00b05ad99395b4bd1999b Sep 11 15:55:59.140553 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:59.140479 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1c358fa6-a923-4589-953f-bd49ddd4235c-original-pull-secret\") pod \"global-pull-secret-syncer-95zvh\" (UID: \"1c358fa6-a923-4589-953f-bd49ddd4235c\") " pod="kube-system/global-pull-secret-syncer-95zvh" Sep 11 15:55:59.143306 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:59.143282 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 11 15:55:59.155025 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:59.155002 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1c358fa6-a923-4589-953f-bd49ddd4235c-original-pull-secret\") pod \"global-pull-secret-syncer-95zvh\" (UID: \"1c358fa6-a923-4589-953f-bd49ddd4235c\") " pod="kube-system/global-pull-secret-syncer-95zvh" Sep 11 15:55:59.184816 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:59.184788 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-844dfdb99b-766v5"] Sep 11 15:55:59.187251 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:55:59.187226 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod313238e9_79d7_45f7_8d96_beb55feb415a.slice/crio-e7d251b534b4c10f1699fb89c5bb4264cd5b185b2949e1a81e683f7723efd837 WatchSource:0}: Error finding container e7d251b534b4c10f1699fb89c5bb4264cd5b185b2949e1a81e683f7723efd837: Status 404 returned error can't find the container with id e7d251b534b4c10f1699fb89c5bb4264cd5b185b2949e1a81e683f7723efd837 Sep 11 15:55:59.196526 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:59.196506 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-kqtwh"] Sep 11 15:55:59.199137 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:55:59.199117 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5429cde4_cb80_4579_9687_e71617b15b9f.slice/crio-45122a4866f2ddbc4a0dc766be66ee37c97a437dbcd36c7ad7adb701ae5964fb WatchSource:0}: Error finding container 45122a4866f2ddbc4a0dc766be66ee37c97a437dbcd36c7ad7adb701ae5964fb: Status 404 returned error can't find the container with id 45122a4866f2ddbc4a0dc766be66ee37c97a437dbcd36c7ad7adb701ae5964fb Sep 11 15:55:59.211806 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:59.211781 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-4hdz4"] Sep 11 15:55:59.269525 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:59.269498 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-8q8dx"] Sep 11 15:55:59.272414 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:55:59.272252 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09f64aa8_4936_49d3_96f7_9895410e5dda.slice/crio-d14fe79db2992b4a603269b1ad21b3bcc31c9016df68ad33896cfa880e262557 WatchSource:0}: Error finding container d14fe79db2992b4a603269b1ad21b3bcc31c9016df68ad33896cfa880e262557: Status 404 returned error can't find the container with id d14fe79db2992b4a603269b1ad21b3bcc31c9016df68ad33896cfa880e262557 Sep 11 15:55:59.306348 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:59.306311 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-gx9fm"] Sep 11 15:55:59.310464 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:55:59.310392 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1e7a08b_8aa9_42e9_aa46_3225f61299e0.slice/crio-d85ad4f056363cd8a51ae419aa5ab9149775a28788c4529dcb3d452709d9f72d WatchSource:0}: Error finding container d85ad4f056363cd8a51ae419aa5ab9149775a28788c4529dcb3d452709d9f72d: Status 404 returned error can't find the container with id d85ad4f056363cd8a51ae419aa5ab9149775a28788c4529dcb3d452709d9f72d Sep 11 15:55:59.352071 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:59.352031 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-95zvh" Sep 11 15:55:59.364689 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:59.364658 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-plkhx"] Sep 11 15:55:59.367582 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:55:59.367541 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod331632d0_fa59_4462_805e_bce0e18b95bc.slice/crio-079354ccd9b6cf319e0265f128914f3a189f79236b3497a0b389d5d2597a5109 WatchSource:0}: Error finding container 079354ccd9b6cf319e0265f128914f3a189f79236b3497a0b389d5d2597a5109: Status 404 returned error can't find the container with id 079354ccd9b6cf319e0265f128914f3a189f79236b3497a0b389d5d2597a5109 Sep 11 15:55:59.575522 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:59.575487 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-plkhx" event={"ID":"331632d0-fa59-4462-805e-bce0e18b95bc","Type":"ContainerStarted","Data":"079354ccd9b6cf319e0265f128914f3a189f79236b3497a0b389d5d2597a5109"} Sep 11 15:55:59.576922 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:59.576873 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-8q8dx" event={"ID":"09f64aa8-4936-49d3-96f7-9895410e5dda","Type":"ContainerStarted","Data":"d14fe79db2992b4a603269b1ad21b3bcc31c9016df68ad33896cfa880e262557"} Sep 11 15:55:59.578285 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:59.578257 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-4hdz4" event={"ID":"c2a34499-68a0-42b4-b77a-edc8b3b964c4","Type":"ContainerStarted","Data":"347d38e5fcb85aac14f86a81f120b934a65b5a67c532eb1793bf71c2d6f7ccb3"} Sep 11 15:55:59.579533 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:59.579507 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-kqtwh" event={"ID":"5429cde4-cb80-4579-9687-e71617b15b9f","Type":"ContainerStarted","Data":"45122a4866f2ddbc4a0dc766be66ee37c97a437dbcd36c7ad7adb701ae5964fb"} Sep 11 15:55:59.581090 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:59.581069 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-844dfdb99b-766v5" event={"ID":"313238e9-79d7-45f7-8d96-beb55feb415a","Type":"ContainerStarted","Data":"758e065900b74aa8c0c7b1ed9ea7299b3ed234584dc852126dfec57025cbec9d"} Sep 11 15:55:59.581183 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:59.581103 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-844dfdb99b-766v5" event={"ID":"313238e9-79d7-45f7-8d96-beb55feb415a","Type":"ContainerStarted","Data":"e7d251b534b4c10f1699fb89c5bb4264cd5b185b2949e1a81e683f7723efd837"} Sep 11 15:55:59.582858 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:59.582831 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-56c5d9df5d-7vqnb" event={"ID":"ae9022dc-da99-4acf-af85-3cf9858737b4","Type":"ContainerStarted","Data":"09e14c00a6b5b9039b882f090c099281b1ab2c4e1032f060c2cd71aa23707eb5"} Sep 11 15:55:59.583002 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:59.582875 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-56c5d9df5d-7vqnb" event={"ID":"ae9022dc-da99-4acf-af85-3cf9858737b4","Type":"ContainerStarted","Data":"3f4337e42e3663b9b51a15f30d9673a26fdcc74394e00b05ad99395b4bd1999b"} Sep 11 15:55:59.583002 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:59.582903 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-56c5d9df5d-7vqnb" Sep 11 15:55:59.584067 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:59.584040 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gx9fm" event={"ID":"a1e7a08b-8aa9-42e9-aa46-3225f61299e0","Type":"ContainerStarted","Data":"d85ad4f056363cd8a51ae419aa5ab9149775a28788c4529dcb3d452709d9f72d"} Sep 11 15:55:59.613668 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:59.613607 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-844dfdb99b-766v5" podStartSLOduration=34.613582771 podStartE2EDuration="34.613582771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:55:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:55:59.612505334 +0000 UTC m=+66.827339237" watchObservedRunningTime="2026-09-11 15:55:59.613582771 +0000 UTC m=+66.828416670" Sep 11 15:55:59.655674 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:59.655646 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-95zvh"] Sep 11 15:55:59.659914 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:55:59.659886 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c358fa6_a923_4589_953f_bd49ddd4235c.slice/crio-92535e65dcd00979c2d950f0acf81a91309014f9c301e8b46850a5dea700b51a WatchSource:0}: Error finding container 92535e65dcd00979c2d950f0acf81a91309014f9c301e8b46850a5dea700b51a: Status 404 returned error can't find the container with id 92535e65dcd00979c2d950f0acf81a91309014f9c301e8b46850a5dea700b51a Sep 11 15:55:59.663087 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:59.663010 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-56c5d9df5d-7vqnb" podStartSLOduration=66.662992629 podStartE2EDuration="1m6.662992629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:54:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:55:59.661439691 +0000 UTC m=+66.876273600" watchObservedRunningTime="2026-09-11 15:55:59.662992629 +0000 UTC m=+66.877826528" Sep 11 15:55:59.825101 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:59.825061 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-844dfdb99b-766v5" Sep 11 15:55:59.828280 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:55:59.828251 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-844dfdb99b-766v5" Sep 11 15:56:00.604085 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:00.603458 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-95zvh" event={"ID":"1c358fa6-a923-4589-953f-bd49ddd4235c","Type":"ContainerStarted","Data":"92535e65dcd00979c2d950f0acf81a91309014f9c301e8b46850a5dea700b51a"} Sep 11 15:56:00.604085 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:00.603528 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-844dfdb99b-766v5" Sep 11 15:56:00.605983 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:00.605819 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-844dfdb99b-766v5" Sep 11 15:56:03.618296 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:03.618249 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-plkhx" event={"ID":"331632d0-fa59-4462-805e-bce0e18b95bc","Type":"ContainerStarted","Data":"2cf424f3d9e8a3b31396d52f493571c04c62a431f862808beac014fd2d785cb1"} Sep 11 15:56:03.618781 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:03.618320 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-plkhx" event={"ID":"331632d0-fa59-4462-805e-bce0e18b95bc","Type":"ContainerStarted","Data":"d6753aa98d8252b61cd7920d27f45bf84f0626e12dd49bb6452fcd5bda105289"} Sep 11 15:56:03.620483 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:03.620448 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-8q8dx" event={"ID":"09f64aa8-4936-49d3-96f7-9895410e5dda","Type":"ContainerStarted","Data":"b97bfba35853ce59952e279383b87126fa7307a27cf70cc42293131c0690027a"} Sep 11 15:56:03.622504 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:03.622465 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-4hdz4" event={"ID":"c2a34499-68a0-42b4-b77a-edc8b3b964c4","Type":"ContainerStarted","Data":"9551129810a91f1d3da4d848e1a43bd1d7b50ff7f3a62721f5e914a19c5bb15e"} Sep 11 15:56:03.622627 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:03.622517 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-4hdz4" event={"ID":"c2a34499-68a0-42b4-b77a-edc8b3b964c4","Type":"ContainerStarted","Data":"8f3e9dea8729a670d8f0a4e4786bc8d102bb57efbf31078c9233845e47566905"} Sep 11 15:56:03.624184 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:03.624159 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-kqtwh" event={"ID":"5429cde4-cb80-4579-9687-e71617b15b9f","Type":"ContainerStarted","Data":"96382c0d250633ffd81cdfba0707a245b05c2b23a5953ac7f80c39353577ffc8"} Sep 11 15:56:03.626076 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:03.626045 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gx9fm" event={"ID":"a1e7a08b-8aa9-42e9-aa46-3225f61299e0","Type":"ContainerStarted","Data":"37bc6e922c08e22772b8956f388ece93844f462d4f3f2fb30dcbcabd34561514"} Sep 11 15:56:03.626192 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:03.626096 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gx9fm" event={"ID":"a1e7a08b-8aa9-42e9-aa46-3225f61299e0","Type":"ContainerStarted","Data":"a376d6a5c383991c91da81e84531073691cee65b1181f995b5ecf785e4512371"} Sep 11 15:56:03.626192 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:03.626142 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-gx9fm" Sep 11 15:56:03.635302 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:03.635253 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-plkhx" podStartSLOduration=67.265842041 podStartE2EDuration="1m10.635235189s" totalImagesPullingTime="3.369393148s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:54:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:56:03.633316023 +0000 UTC m=+70.848149939" watchObservedRunningTime="2026-09-11 15:56:03.635235189 +0000 UTC m=+70.850069097" Sep 11 15:56:03.650840 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:03.650793 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-8q8dx" podStartSLOduration=35.185694178 podStartE2EDuration="38.65077721s" totalImagesPullingTime="3.465083032s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:55:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:56:03.649448858 +0000 UTC m=+70.864282763" watchObservedRunningTime="2026-09-11 15:56:03.65077721 +0000 UTC m=+70.865611115" Sep 11 15:56:03.668713 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:03.668659 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-4hdz4" podStartSLOduration=35.191465837 podStartE2EDuration="38.668636187s" totalImagesPullingTime="3.47717035s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:55:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:56:03.667074587 +0000 UTC m=+70.881908492" watchObservedRunningTime="2026-09-11 15:56:03.668636187 +0000 UTC m=+70.883470093" Sep 11 15:56:03.689330 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:03.689263 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-gx9fm" podStartSLOduration=34.262526695 podStartE2EDuration="37.689240674s" totalImagesPullingTime="3.426713979s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:55:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:56:03.688623264 +0000 UTC m=+70.903457172" watchObservedRunningTime="2026-09-11 15:56:03.689240674 +0000 UTC m=+70.904074581" Sep 11 15:56:03.708992 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:03.708932 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-kqtwh" podStartSLOduration=35.169540199 podStartE2EDuration="38.708911223s" totalImagesPullingTime="3.539371024s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:55:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:56:03.707684542 +0000 UTC m=+70.922518446" watchObservedRunningTime="2026-09-11 15:56:03.708911223 +0000 UTC m=+70.923745129" Sep 11 15:56:05.635214 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:05.635173 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-95zvh" event={"ID":"1c358fa6-a923-4589-953f-bd49ddd4235c","Type":"ContainerStarted","Data":"f1b8f984bea2bbd7719d4be9ab69cfa213fb35e93635c7dc5c2480d1f5507b98"} Sep 11 15:56:05.655237 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:05.655185 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-95zvh" podStartSLOduration=67.484110046 podStartE2EDuration="1m12.655167842s" totalImagesPullingTime="5.171057796s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 15:54:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:56:05.654253223 +0000 UTC m=+72.869087127" watchObservedRunningTime="2026-09-11 15:56:05.655167842 +0000 UTC m=+72.870001747" Sep 11 15:56:07.494450 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:07.494420 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-nzcjl" Sep 11 15:56:09.232686 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:09.232655 2566 scope.go:117] "RemoveContainer" containerID="56fd95fcc18f51f8f027556616d2de24aea35a383d6ec6817332b3d49b2e7ddf" Sep 11 15:56:09.651841 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:09.651814 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-xjvl5_7542ad77-c817-4370-91de-3b08db264753/console-operator/2.log" Sep 11 15:56:09.652023 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:09.651877 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-xjvl5" event={"ID":"7542ad77-c817-4370-91de-3b08db264753","Type":"ContainerStarted","Data":"422fd702bbb5da1f350be2193607e077075279cf0ba786c3ae0f32de8fd9a798"} Sep 11 15:56:09.652174 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:09.652156 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-xjvl5" Sep 11 15:56:09.671322 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:09.671261 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-696884fb9c-xjvl5" podStartSLOduration=36.278706785 podStartE2EDuration="44.671240029s" totalImagesPullingTime="8.392533244s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:55:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:56:09.669563889 +0000 UTC m=+76.884397795" watchObservedRunningTime="2026-09-11 15:56:09.671240029 +0000 UTC m=+76.886073935" Sep 11 15:56:09.908333 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:09.908257 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-696884fb9c-xjvl5" Sep 11 15:56:10.084381 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:10.084345 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-76f99584f4-2f2j4"] Sep 11 15:56:10.104513 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:10.104479 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-2f2j4"] Sep 11 15:56:10.104720 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:10.104607 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-2f2j4" Sep 11 15:56:10.106995 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:10.106946 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-5bgng\"" Sep 11 15:56:10.107137 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:10.107080 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 11 15:56:10.107245 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:10.107225 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 11 15:56:10.240298 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:10.240212 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gzt4\" (UniqueName: \"kubernetes.io/projected/6e65377d-74a2-4a00-9752-ce734ca7d357-kube-api-access-2gzt4\") pod \"downloads-76f99584f4-2f2j4\" (UID: \"6e65377d-74a2-4a00-9752-ce734ca7d357\") " pod="openshift-console/downloads-76f99584f4-2f2j4" Sep 11 15:56:10.340772 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:10.340738 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gzt4\" (UniqueName: \"kubernetes.io/projected/6e65377d-74a2-4a00-9752-ce734ca7d357-kube-api-access-2gzt4\") pod \"downloads-76f99584f4-2f2j4\" (UID: \"6e65377d-74a2-4a00-9752-ce734ca7d357\") " pod="openshift-console/downloads-76f99584f4-2f2j4" Sep 11 15:56:10.350288 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:10.350264 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gzt4\" (UniqueName: \"kubernetes.io/projected/6e65377d-74a2-4a00-9752-ce734ca7d357-kube-api-access-2gzt4\") pod \"downloads-76f99584f4-2f2j4\" (UID: \"6e65377d-74a2-4a00-9752-ce734ca7d357\") " pod="openshift-console/downloads-76f99584f4-2f2j4" Sep 11 15:56:10.414620 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:10.414586 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-2f2j4" Sep 11 15:56:10.709659 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:10.709625 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-2f2j4"] Sep 11 15:56:10.712625 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:56:10.712592 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e65377d_74a2_4a00_9752_ce734ca7d357.slice/crio-ff1640dffe141706939868b5289cfa2b166eabe3878f5d099a0eba33e6fd9eb5 WatchSource:0}: Error finding container ff1640dffe141706939868b5289cfa2b166eabe3878f5d099a0eba33e6fd9eb5: Status 404 returned error can't find the container with id ff1640dffe141706939868b5289cfa2b166eabe3878f5d099a0eba33e6fd9eb5 Sep 11 15:56:11.661948 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:11.661906 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-2f2j4" event={"ID":"6e65377d-74a2-4a00-9752-ce734ca7d357","Type":"ContainerStarted","Data":"ff1640dffe141706939868b5289cfa2b166eabe3878f5d099a0eba33e6fd9eb5"} Sep 11 15:56:11.814340 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:11.814307 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-wxn2r"] Sep 11 15:56:11.817046 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:11.817018 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-wxn2r" Sep 11 15:56:11.819029 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:11.819002 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Sep 11 15:56:11.819159 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:11.819059 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Sep 11 15:56:11.819739 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:11.819719 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-lzp7m\"" Sep 11 15:56:11.819823 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:11.819720 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 11 15:56:11.819922 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:11.819818 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Sep 11 15:56:11.834160 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:11.834134 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-wxn2r"] Sep 11 15:56:11.836070 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:11.836046 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-2thkm"] Sep 11 15:56:11.838907 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:11.838879 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2thkm" Sep 11 15:56:11.840937 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:11.840916 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 11 15:56:11.841359 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:11.841339 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-9j5tf\"" Sep 11 15:56:11.841461 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:11.841389 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 11 15:56:11.841714 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:11.841685 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 11 15:56:11.953743 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:11.953651 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8edb9d38-7d93-4bdd-bb6d-c0c566471af0-node-exporter-accelerators-collector-config\") pod \"node-exporter-2thkm\" (UID: \"8edb9d38-7d93-4bdd-bb6d-c0c566471af0\") " pod="openshift-monitoring/node-exporter-2thkm" Sep 11 15:56:11.953743 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:11.953705 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7cr7\" (UniqueName: \"kubernetes.io/projected/8edb9d38-7d93-4bdd-bb6d-c0c566471af0-kube-api-access-m7cr7\") pod \"node-exporter-2thkm\" (UID: \"8edb9d38-7d93-4bdd-bb6d-c0c566471af0\") " pod="openshift-monitoring/node-exporter-2thkm" Sep 11 15:56:11.953743 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:11.953739 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/562f8af8-8658-4994-a330-2bbfb42e8339-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-wxn2r\" (UID: \"562f8af8-8658-4994-a330-2bbfb42e8339\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-wxn2r" Sep 11 15:56:11.954018 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:11.953822 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8edb9d38-7d93-4bdd-bb6d-c0c566471af0-node-exporter-textfile\") pod \"node-exporter-2thkm\" (UID: \"8edb9d38-7d93-4bdd-bb6d-c0c566471af0\") " pod="openshift-monitoring/node-exporter-2thkm" Sep 11 15:56:11.954018 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:11.953860 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8edb9d38-7d93-4bdd-bb6d-c0c566471af0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2thkm\" (UID: \"8edb9d38-7d93-4bdd-bb6d-c0c566471af0\") " pod="openshift-monitoring/node-exporter-2thkm" Sep 11 15:56:11.954018 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:11.953891 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8edb9d38-7d93-4bdd-bb6d-c0c566471af0-node-exporter-tls\") pod \"node-exporter-2thkm\" (UID: \"8edb9d38-7d93-4bdd-bb6d-c0c566471af0\") " pod="openshift-monitoring/node-exporter-2thkm" Sep 11 15:56:11.954018 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:11.953939 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8edb9d38-7d93-4bdd-bb6d-c0c566471af0-node-exporter-wtmp\") pod \"node-exporter-2thkm\" (UID: \"8edb9d38-7d93-4bdd-bb6d-c0c566471af0\") " pod="openshift-monitoring/node-exporter-2thkm" Sep 11 15:56:11.954189 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:11.954083 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/562f8af8-8658-4994-a330-2bbfb42e8339-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-wxn2r\" (UID: \"562f8af8-8658-4994-a330-2bbfb42e8339\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-wxn2r" Sep 11 15:56:11.954189 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:11.954127 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/562f8af8-8658-4994-a330-2bbfb42e8339-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-wxn2r\" (UID: \"562f8af8-8658-4994-a330-2bbfb42e8339\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-wxn2r" Sep 11 15:56:11.954189 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:11.954165 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8edb9d38-7d93-4bdd-bb6d-c0c566471af0-metrics-client-ca\") pod \"node-exporter-2thkm\" (UID: \"8edb9d38-7d93-4bdd-bb6d-c0c566471af0\") " pod="openshift-monitoring/node-exporter-2thkm" Sep 11 15:56:11.954189 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:11.954185 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8edb9d38-7d93-4bdd-bb6d-c0c566471af0-root\") pod \"node-exporter-2thkm\" (UID: \"8edb9d38-7d93-4bdd-bb6d-c0c566471af0\") " pod="openshift-monitoring/node-exporter-2thkm" Sep 11 15:56:11.954376 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:11.954213 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/562f8af8-8658-4994-a330-2bbfb42e8339-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-wxn2r\" (UID: \"562f8af8-8658-4994-a330-2bbfb42e8339\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-wxn2r" Sep 11 15:56:11.954376 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:11.954245 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/562f8af8-8658-4994-a330-2bbfb42e8339-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-wxn2r\" (UID: \"562f8af8-8658-4994-a330-2bbfb42e8339\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-wxn2r" Sep 11 15:56:11.954376 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:11.954292 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8edb9d38-7d93-4bdd-bb6d-c0c566471af0-sys\") pod \"node-exporter-2thkm\" (UID: \"8edb9d38-7d93-4bdd-bb6d-c0c566471af0\") " pod="openshift-monitoring/node-exporter-2thkm" Sep 11 15:56:11.954376 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:11.954338 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsk4b\" (UniqueName: \"kubernetes.io/projected/562f8af8-8658-4994-a330-2bbfb42e8339-kube-api-access-bsk4b\") pod \"kube-state-metrics-59fbdd77d-wxn2r\" (UID: \"562f8af8-8658-4994-a330-2bbfb42e8339\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-wxn2r" Sep 11 15:56:12.055390 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:12.055354 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8edb9d38-7d93-4bdd-bb6d-c0c566471af0-metrics-client-ca\") pod \"node-exporter-2thkm\" (UID: \"8edb9d38-7d93-4bdd-bb6d-c0c566471af0\") " pod="openshift-monitoring/node-exporter-2thkm" Sep 11 15:56:12.055390 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:12.055395 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8edb9d38-7d93-4bdd-bb6d-c0c566471af0-root\") pod \"node-exporter-2thkm\" (UID: \"8edb9d38-7d93-4bdd-bb6d-c0c566471af0\") " pod="openshift-monitoring/node-exporter-2thkm" Sep 11 15:56:12.055647 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:12.055423 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/562f8af8-8658-4994-a330-2bbfb42e8339-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-wxn2r\" (UID: \"562f8af8-8658-4994-a330-2bbfb42e8339\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-wxn2r" Sep 11 15:56:12.055647 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:12.055447 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/562f8af8-8658-4994-a330-2bbfb42e8339-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-wxn2r\" (UID: \"562f8af8-8658-4994-a330-2bbfb42e8339\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-wxn2r" Sep 11 15:56:12.055647 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:12.055487 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8edb9d38-7d93-4bdd-bb6d-c0c566471af0-sys\") pod \"node-exporter-2thkm\" (UID: \"8edb9d38-7d93-4bdd-bb6d-c0c566471af0\") " pod="openshift-monitoring/node-exporter-2thkm" Sep 11 15:56:12.055647 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:12.055512 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsk4b\" (UniqueName: \"kubernetes.io/projected/562f8af8-8658-4994-a330-2bbfb42e8339-kube-api-access-bsk4b\") pod \"kube-state-metrics-59fbdd77d-wxn2r\" (UID: \"562f8af8-8658-4994-a330-2bbfb42e8339\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-wxn2r" Sep 11 15:56:12.055647 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:12.055543 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8edb9d38-7d93-4bdd-bb6d-c0c566471af0-node-exporter-accelerators-collector-config\") pod \"node-exporter-2thkm\" (UID: \"8edb9d38-7d93-4bdd-bb6d-c0c566471af0\") " pod="openshift-monitoring/node-exporter-2thkm" Sep 11 15:56:12.055647 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:12.055575 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7cr7\" (UniqueName: \"kubernetes.io/projected/8edb9d38-7d93-4bdd-bb6d-c0c566471af0-kube-api-access-m7cr7\") pod \"node-exporter-2thkm\" (UID: \"8edb9d38-7d93-4bdd-bb6d-c0c566471af0\") " pod="openshift-monitoring/node-exporter-2thkm" Sep 11 15:56:12.055647 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:12.055601 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/562f8af8-8658-4994-a330-2bbfb42e8339-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-wxn2r\" (UID: \"562f8af8-8658-4994-a330-2bbfb42e8339\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-wxn2r" Sep 11 15:56:12.055647 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:12.055635 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8edb9d38-7d93-4bdd-bb6d-c0c566471af0-node-exporter-textfile\") pod \"node-exporter-2thkm\" (UID: \"8edb9d38-7d93-4bdd-bb6d-c0c566471af0\") " pod="openshift-monitoring/node-exporter-2thkm" Sep 11 15:56:12.056067 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:12.055677 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8edb9d38-7d93-4bdd-bb6d-c0c566471af0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2thkm\" (UID: \"8edb9d38-7d93-4bdd-bb6d-c0c566471af0\") " pod="openshift-monitoring/node-exporter-2thkm" Sep 11 15:56:12.056067 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:12.055707 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8edb9d38-7d93-4bdd-bb6d-c0c566471af0-node-exporter-tls\") pod \"node-exporter-2thkm\" (UID: \"8edb9d38-7d93-4bdd-bb6d-c0c566471af0\") " pod="openshift-monitoring/node-exporter-2thkm" Sep 11 15:56:12.056067 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:12.055729 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8edb9d38-7d93-4bdd-bb6d-c0c566471af0-node-exporter-wtmp\") pod \"node-exporter-2thkm\" (UID: \"8edb9d38-7d93-4bdd-bb6d-c0c566471af0\") " pod="openshift-monitoring/node-exporter-2thkm" Sep 11 15:56:12.056067 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:12.055764 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/562f8af8-8658-4994-a330-2bbfb42e8339-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-wxn2r\" (UID: \"562f8af8-8658-4994-a330-2bbfb42e8339\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-wxn2r" Sep 11 15:56:12.056067 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:12.055800 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/562f8af8-8658-4994-a330-2bbfb42e8339-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-wxn2r\" (UID: \"562f8af8-8658-4994-a330-2bbfb42e8339\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-wxn2r" Sep 11 15:56:12.056067 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:56:12.055919 2566 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Sep 11 15:56:12.056067 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:56:12.056009 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/562f8af8-8658-4994-a330-2bbfb42e8339-kube-state-metrics-tls podName:562f8af8-8658-4994-a330-2bbfb42e8339 nodeName:}" failed. No retries permitted until 2026-09-11 15:56:12.555961964 +0000 UTC m=+79.770795864 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/562f8af8-8658-4994-a330-2bbfb42e8339-kube-state-metrics-tls") pod "kube-state-metrics-59fbdd77d-wxn2r" (UID: "562f8af8-8658-4994-a330-2bbfb42e8339") : secret "kube-state-metrics-tls" not found Sep 11 15:56:12.056067 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:12.055507 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8edb9d38-7d93-4bdd-bb6d-c0c566471af0-root\") pod \"node-exporter-2thkm\" (UID: \"8edb9d38-7d93-4bdd-bb6d-c0c566471af0\") " pod="openshift-monitoring/node-exporter-2thkm" Sep 11 15:56:12.056466 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:12.056080 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8edb9d38-7d93-4bdd-bb6d-c0c566471af0-metrics-client-ca\") pod \"node-exporter-2thkm\" (UID: \"8edb9d38-7d93-4bdd-bb6d-c0c566471af0\") " pod="openshift-monitoring/node-exporter-2thkm" Sep 11 15:56:12.056466 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:12.056107 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8edb9d38-7d93-4bdd-bb6d-c0c566471af0-sys\") pod \"node-exporter-2thkm\" (UID: \"8edb9d38-7d93-4bdd-bb6d-c0c566471af0\") " pod="openshift-monitoring/node-exporter-2thkm" Sep 11 15:56:12.056466 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:56:12.056168 2566 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Sep 11 15:56:12.056466 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:56:12.056209 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8edb9d38-7d93-4bdd-bb6d-c0c566471af0-node-exporter-tls podName:8edb9d38-7d93-4bdd-bb6d-c0c566471af0 nodeName:}" failed. No retries permitted until 2026-09-11 15:56:12.556194983 +0000 UTC m=+79.771028887 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/8edb9d38-7d93-4bdd-bb6d-c0c566471af0-node-exporter-tls") pod "node-exporter-2thkm" (UID: "8edb9d38-7d93-4bdd-bb6d-c0c566471af0") : secret "node-exporter-tls" not found Sep 11 15:56:12.056466 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:12.056267 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/562f8af8-8658-4994-a330-2bbfb42e8339-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-wxn2r\" (UID: \"562f8af8-8658-4994-a330-2bbfb42e8339\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-wxn2r" Sep 11 15:56:12.056466 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:12.056309 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8edb9d38-7d93-4bdd-bb6d-c0c566471af0-node-exporter-wtmp\") pod \"node-exporter-2thkm\" (UID: \"8edb9d38-7d93-4bdd-bb6d-c0c566471af0\") " pod="openshift-monitoring/node-exporter-2thkm" Sep 11 15:56:12.056466 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:12.056316 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/562f8af8-8658-4994-a330-2bbfb42e8339-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-wxn2r\" (UID: \"562f8af8-8658-4994-a330-2bbfb42e8339\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-wxn2r" Sep 11 15:56:12.056771 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:12.056476 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8edb9d38-7d93-4bdd-bb6d-c0c566471af0-node-exporter-textfile\") pod \"node-exporter-2thkm\" (UID: \"8edb9d38-7d93-4bdd-bb6d-c0c566471af0\") " pod="openshift-monitoring/node-exporter-2thkm" Sep 11 15:56:12.056771 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:12.056651 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/562f8af8-8658-4994-a330-2bbfb42e8339-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-wxn2r\" (UID: \"562f8af8-8658-4994-a330-2bbfb42e8339\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-wxn2r" Sep 11 15:56:12.056771 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:12.056689 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8edb9d38-7d93-4bdd-bb6d-c0c566471af0-node-exporter-accelerators-collector-config\") pod \"node-exporter-2thkm\" (UID: \"8edb9d38-7d93-4bdd-bb6d-c0c566471af0\") " pod="openshift-monitoring/node-exporter-2thkm" Sep 11 15:56:12.058785 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:12.058744 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/562f8af8-8658-4994-a330-2bbfb42e8339-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-wxn2r\" (UID: \"562f8af8-8658-4994-a330-2bbfb42e8339\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-wxn2r" Sep 11 15:56:12.058891 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:12.058870 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8edb9d38-7d93-4bdd-bb6d-c0c566471af0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2thkm\" (UID: \"8edb9d38-7d93-4bdd-bb6d-c0c566471af0\") " pod="openshift-monitoring/node-exporter-2thkm" Sep 11 15:56:12.064407 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:12.064380 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsk4b\" (UniqueName: \"kubernetes.io/projected/562f8af8-8658-4994-a330-2bbfb42e8339-kube-api-access-bsk4b\") pod \"kube-state-metrics-59fbdd77d-wxn2r\" (UID: \"562f8af8-8658-4994-a330-2bbfb42e8339\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-wxn2r" Sep 11 15:56:12.064538 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:12.064417 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7cr7\" (UniqueName: \"kubernetes.io/projected/8edb9d38-7d93-4bdd-bb6d-c0c566471af0-kube-api-access-m7cr7\") pod \"node-exporter-2thkm\" (UID: \"8edb9d38-7d93-4bdd-bb6d-c0c566471af0\") " pod="openshift-monitoring/node-exporter-2thkm" Sep 11 15:56:12.560525 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:12.560492 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8edb9d38-7d93-4bdd-bb6d-c0c566471af0-node-exporter-tls\") pod \"node-exporter-2thkm\" (UID: \"8edb9d38-7d93-4bdd-bb6d-c0c566471af0\") " pod="openshift-monitoring/node-exporter-2thkm" Sep 11 15:56:12.560697 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:12.560537 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/562f8af8-8658-4994-a330-2bbfb42e8339-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-wxn2r\" (UID: \"562f8af8-8658-4994-a330-2bbfb42e8339\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-wxn2r" Sep 11 15:56:12.563299 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:12.563270 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8edb9d38-7d93-4bdd-bb6d-c0c566471af0-node-exporter-tls\") pod \"node-exporter-2thkm\" (UID: \"8edb9d38-7d93-4bdd-bb6d-c0c566471af0\") " pod="openshift-monitoring/node-exporter-2thkm" Sep 11 15:56:12.563299 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:12.563289 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/562f8af8-8658-4994-a330-2bbfb42e8339-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-wxn2r\" (UID: \"562f8af8-8658-4994-a330-2bbfb42e8339\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-wxn2r" Sep 11 15:56:12.728176 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:12.727891 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-wxn2r" Sep 11 15:56:12.750148 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:12.750110 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2thkm" Sep 11 15:56:12.763516 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:56:12.763482 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8edb9d38_7d93_4bdd_bb6d_c0c566471af0.slice/crio-f849ec27c3fcb3c9530b2cf48da551311b66c568429b8ce74e9b7e2c0ada428e WatchSource:0}: Error finding container f849ec27c3fcb3c9530b2cf48da551311b66c568429b8ce74e9b7e2c0ada428e: Status 404 returned error can't find the container with id f849ec27c3fcb3c9530b2cf48da551311b66c568429b8ce74e9b7e2c0ada428e Sep 11 15:56:12.854148 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:12.853981 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 11 15:56:12.858007 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:12.857965 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 11 15:56:12.861862 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:12.861686 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Sep 11 15:56:12.861862 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:12.861686 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Sep 11 15:56:12.861862 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:12.861696 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Sep 11 15:56:12.861862 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:12.861837 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Sep 11 15:56:12.862238 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:12.862219 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Sep 11 15:56:12.862330 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:12.862299 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Sep 11 15:56:12.862706 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:12.862388 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Sep 11 15:56:12.862706 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:12.862510 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Sep 11 15:56:12.862706 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:12.862512 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-s2gtr\"" Sep 11 15:56:12.862919 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:12.862821 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Sep 11 15:56:12.874937 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:12.874905 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 11 15:56:12.963313 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:12.963278 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 15:56:12.963505 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:12.963334 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 15:56:12.963505 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:12.963357 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e-config-volume\") pod \"alertmanager-main-0\" (UID: \"4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 15:56:12.963505 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:12.963382 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 15:56:12.963505 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:12.963425 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 15:56:12.963658 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:12.963500 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 15:56:12.963658 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:12.963572 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 15:56:12.963658 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:12.963606 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e-web-config\") pod \"alertmanager-main-0\" (UID: \"4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 15:56:12.963658 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:12.963631 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ws64g\" (UniqueName: \"kubernetes.io/projected/4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e-kube-api-access-ws64g\") pod \"alertmanager-main-0\" (UID: \"4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 15:56:12.963820 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:12.963673 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 15:56:12.963820 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:12.963734 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 15:56:12.963820 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:12.963779 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e-config-out\") pod \"alertmanager-main-0\" (UID: \"4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 15:56:12.963820 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:12.963808 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 15:56:13.028777 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:13.028742 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-wxn2r"] Sep 11 15:56:13.031757 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:56:13.031726 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod562f8af8_8658_4994_a330_2bbfb42e8339.slice/crio-ea301096e94cd787329ff1a2da52017ceb43861889f52bdd60a52dcb169ede12 WatchSource:0}: Error finding container ea301096e94cd787329ff1a2da52017ceb43861889f52bdd60a52dcb169ede12: Status 404 returned error can't find the container with id ea301096e94cd787329ff1a2da52017ceb43861889f52bdd60a52dcb169ede12 Sep 11 15:56:13.064736 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:13.064703 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 15:56:13.064891 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:13.064745 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e-web-config\") pod \"alertmanager-main-0\" (UID: \"4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 15:56:13.064891 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:13.064764 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ws64g\" (UniqueName: \"kubernetes.io/projected/4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e-kube-api-access-ws64g\") pod \"alertmanager-main-0\" (UID: \"4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 15:56:13.064891 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:13.064786 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 15:56:13.064891 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:13.064809 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 15:56:13.067006 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:13.065237 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 15:56:13.067006 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:13.065237 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e-config-out\") pod \"alertmanager-main-0\" (UID: \"4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 15:56:13.067006 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:13.065302 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 15:56:13.067006 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:13.065347 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 15:56:13.067006 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:13.065393 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 15:56:13.067006 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:13.065420 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e-config-volume\") pod \"alertmanager-main-0\" (UID: \"4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 15:56:13.067006 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:13.065444 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 15:56:13.067006 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:13.065472 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 15:56:13.067006 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:13.065498 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 15:56:13.067006 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:13.066909 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 15:56:13.067701 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:13.067671 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 15:56:13.068469 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:13.068434 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 15:56:13.071247 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:13.071201 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 15:56:13.071776 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:13.071499 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 15:56:13.071776 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:13.071504 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 15:56:13.071776 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:13.071749 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e-config-out\") pod \"alertmanager-main-0\" (UID: \"4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 15:56:13.071989 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:13.071820 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 15:56:13.072231 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:13.072209 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e-web-config\") pod \"alertmanager-main-0\" (UID: \"4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 15:56:13.072756 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:13.072735 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 15:56:13.074846 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:13.074821 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e-config-volume\") pod \"alertmanager-main-0\" (UID: \"4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 15:56:13.074984 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:13.074948 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ws64g\" (UniqueName: \"kubernetes.io/projected/4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e-kube-api-access-ws64g\") pod \"alertmanager-main-0\" (UID: \"4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 15:56:13.172841 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:13.172753 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 11 15:56:13.477663 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:13.477627 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 11 15:56:13.480896 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:56:13.480859 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b0b3e5a_ac62_4fa8_b9f1_9b57e00d528e.slice/crio-12d1a487e9a70cb3a313e4abf47fc168e7963439685c4ebae9521463a2c38800 WatchSource:0}: Error finding container 12d1a487e9a70cb3a313e4abf47fc168e7963439685c4ebae9521463a2c38800: Status 404 returned error can't find the container with id 12d1a487e9a70cb3a313e4abf47fc168e7963439685c4ebae9521463a2c38800 Sep 11 15:56:13.633202 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:13.633167 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-gx9fm" Sep 11 15:56:13.675745 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:13.675539 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e","Type":"ContainerStarted","Data":"12d1a487e9a70cb3a313e4abf47fc168e7963439685c4ebae9521463a2c38800"} Sep 11 15:56:13.677430 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:13.677373 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2thkm" event={"ID":"8edb9d38-7d93-4bdd-bb6d-c0c566471af0","Type":"ContainerStarted","Data":"f849ec27c3fcb3c9530b2cf48da551311b66c568429b8ce74e9b7e2c0ada428e"} Sep 11 15:56:13.683638 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:13.683210 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-wxn2r" event={"ID":"562f8af8-8658-4994-a330-2bbfb42e8339","Type":"ContainerStarted","Data":"ea301096e94cd787329ff1a2da52017ceb43861889f52bdd60a52dcb169ede12"} Sep 11 15:56:14.690287 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:14.690251 2566 generic.go:356] "Generic (PLEG): container finished" podID="8edb9d38-7d93-4bdd-bb6d-c0c566471af0" containerID="52d74317fbbbdcf3a59ac1baae540f333ecc4a84a6d0fd747d69c366b45bd45e" exitCode=0 Sep 11 15:56:14.690728 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:14.690352 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2thkm" event={"ID":"8edb9d38-7d93-4bdd-bb6d-c0c566471af0","Type":"ContainerDied","Data":"52d74317fbbbdcf3a59ac1baae540f333ecc4a84a6d0fd747d69c366b45bd45e"} Sep 11 15:56:15.698266 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:15.698118 2566 generic.go:356] "Generic (PLEG): container finished" podID="4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e" containerID="ccaf640ff68701612e58cd96151db2f80ed789801f744d5ea514d349bb1d9b52" exitCode=0 Sep 11 15:56:15.698266 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:15.698215 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e","Type":"ContainerDied","Data":"ccaf640ff68701612e58cd96151db2f80ed789801f744d5ea514d349bb1d9b52"} Sep 11 15:56:15.701900 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:15.701872 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2thkm" event={"ID":"8edb9d38-7d93-4bdd-bb6d-c0c566471af0","Type":"ContainerStarted","Data":"7d83fdfbb2bce3b8b4b9cf7f354c4b2eb57d5ef864b62750b68d941929d98bc6"} Sep 11 15:56:15.702026 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:15.701933 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2thkm" event={"ID":"8edb9d38-7d93-4bdd-bb6d-c0c566471af0","Type":"ContainerStarted","Data":"38e97a88bbe590eff63617c27e833df469702f9ee97aad033e4b6530c25356fc"} Sep 11 15:56:15.704606 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:15.704578 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-wxn2r" event={"ID":"562f8af8-8658-4994-a330-2bbfb42e8339","Type":"ContainerStarted","Data":"c2c3869efaceba3b2c6df17894396c6af7f6ddc4a737b199ed1a1fb58f5ed399"} Sep 11 15:56:15.704722 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:15.704646 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-wxn2r" event={"ID":"562f8af8-8658-4994-a330-2bbfb42e8339","Type":"ContainerStarted","Data":"00eb5ed34bd0f58512dbc6feb67339186435f93ae49d328eb194953c04f40eec"} Sep 11 15:56:15.704722 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:15.704680 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-wxn2r" event={"ID":"562f8af8-8658-4994-a330-2bbfb42e8339","Type":"ContainerStarted","Data":"e2e66f88074e9d5a6efc8a6b05f405c6d98cbf8a7b3461b54ad513fa222327b7"} Sep 11 15:56:15.744361 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:15.744298 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-wxn2r" podStartSLOduration=2.894371565 podStartE2EDuration="4.744282011s" totalImagesPullingTime="1.849910446s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:56:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:56:15.743336617 +0000 UTC m=+82.958170523" watchObservedRunningTime="2026-09-11 15:56:15.744282011 +0000 UTC m=+82.959115915" Sep 11 15:56:15.762143 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:15.762083 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-2thkm" podStartSLOduration=3.415602445 podStartE2EDuration="4.762065965s" totalImagesPullingTime="1.266671148s" totalInitContainerRuntime="79.792372ms" isStatefulPod=true podCreationTimestamp="2026-09-11 15:56:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:56:15.760423205 +0000 UTC m=+82.975257113" watchObservedRunningTime="2026-09-11 15:56:15.762065965 +0000 UTC m=+82.976899863" Sep 11 15:56:17.718730 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:17.718611 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e","Type":"ContainerStarted","Data":"0e3c4adc5c97f22adf03fc3eac0e6dc7532feee94523cf9844720b38a0a155eb"} Sep 11 15:56:17.718730 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:17.718698 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e","Type":"ContainerStarted","Data":"fb6a81fde4b2618df228016db131e5c0b0a13ff947344cebc687e5385dd47f76"} Sep 11 15:56:18.728347 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:18.728315 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e","Type":"ContainerStarted","Data":"00528d1348836b690cfbaec38fa1113f608911bfbe67b2078644847c56587987"} Sep 11 15:56:18.728822 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:18.728393 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e","Type":"ContainerStarted","Data":"52238d9341f63161169e082f4bc1d3b237f431909b8dfd7bd8ad22aebebad77c"} Sep 11 15:56:18.728822 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:18.728436 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e","Type":"ContainerStarted","Data":"73de8c544a4364477c0476052ff4f678b0f381aaa022a218cd41ecde0972824e"} Sep 11 15:56:18.728822 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:18.728476 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e","Type":"ContainerStarted","Data":"da377bde8e965d6c15a6ed75fcccb48f858bc1b1a64924817b33f2e22cf49c89"} Sep 11 15:56:18.758172 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:18.758117 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.748574528 podStartE2EDuration="6.758099909s" totalImagesPullingTime="3.984137554s" totalInitContainerRuntime="25.387827ms" isStatefulPod=false podCreationTimestamp="2026-09-11 15:56:12 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:56:18.755348138 +0000 UTC m=+85.970182043" watchObservedRunningTime="2026-09-11 15:56:18.758099909 +0000 UTC m=+85.972933814" Sep 11 15:56:19.143565 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:19.143527 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6bbcf8987c-4z9gw"] Sep 11 15:56:19.145829 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:19.145806 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bbcf8987c-4z9gw" Sep 11 15:56:19.148328 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:19.148303 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 11 15:56:19.148580 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:19.148559 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 11 15:56:19.148702 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:19.148619 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 11 15:56:19.148812 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:19.148789 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 11 15:56:19.148910 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:19.148884 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 11 15:56:19.149490 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:19.149473 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-zh674\"" Sep 11 15:56:19.170925 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:19.170900 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6bbcf8987c-4z9gw"] Sep 11 15:56:19.228317 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:19.228280 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4800e361-38ce-40a5-8bf7-dfb46395f3cb-console-config\") pod \"console-6bbcf8987c-4z9gw\" (UID: \"4800e361-38ce-40a5-8bf7-dfb46395f3cb\") " pod="openshift-console/console-6bbcf8987c-4z9gw" Sep 11 15:56:19.228488 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:19.228335 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lldgb\" (UniqueName: \"kubernetes.io/projected/4800e361-38ce-40a5-8bf7-dfb46395f3cb-kube-api-access-lldgb\") pod \"console-6bbcf8987c-4z9gw\" (UID: \"4800e361-38ce-40a5-8bf7-dfb46395f3cb\") " pod="openshift-console/console-6bbcf8987c-4z9gw" Sep 11 15:56:19.228488 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:19.228377 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4800e361-38ce-40a5-8bf7-dfb46395f3cb-oauth-serving-cert\") pod \"console-6bbcf8987c-4z9gw\" (UID: \"4800e361-38ce-40a5-8bf7-dfb46395f3cb\") " pod="openshift-console/console-6bbcf8987c-4z9gw" Sep 11 15:56:19.228488 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:19.228435 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4800e361-38ce-40a5-8bf7-dfb46395f3cb-service-ca\") pod \"console-6bbcf8987c-4z9gw\" (UID: \"4800e361-38ce-40a5-8bf7-dfb46395f3cb\") " pod="openshift-console/console-6bbcf8987c-4z9gw" Sep 11 15:56:19.228488 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:19.228475 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4800e361-38ce-40a5-8bf7-dfb46395f3cb-console-serving-cert\") pod \"console-6bbcf8987c-4z9gw\" (UID: \"4800e361-38ce-40a5-8bf7-dfb46395f3cb\") " pod="openshift-console/console-6bbcf8987c-4z9gw" Sep 11 15:56:19.228669 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:19.228552 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4800e361-38ce-40a5-8bf7-dfb46395f3cb-console-oauth-config\") pod \"console-6bbcf8987c-4z9gw\" (UID: \"4800e361-38ce-40a5-8bf7-dfb46395f3cb\") " pod="openshift-console/console-6bbcf8987c-4z9gw" Sep 11 15:56:19.329560 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:19.329523 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4800e361-38ce-40a5-8bf7-dfb46395f3cb-oauth-serving-cert\") pod \"console-6bbcf8987c-4z9gw\" (UID: \"4800e361-38ce-40a5-8bf7-dfb46395f3cb\") " pod="openshift-console/console-6bbcf8987c-4z9gw" Sep 11 15:56:19.329720 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:19.329568 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4800e361-38ce-40a5-8bf7-dfb46395f3cb-service-ca\") pod \"console-6bbcf8987c-4z9gw\" (UID: \"4800e361-38ce-40a5-8bf7-dfb46395f3cb\") " pod="openshift-console/console-6bbcf8987c-4z9gw" Sep 11 15:56:19.329720 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:19.329601 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4800e361-38ce-40a5-8bf7-dfb46395f3cb-console-serving-cert\") pod \"console-6bbcf8987c-4z9gw\" (UID: \"4800e361-38ce-40a5-8bf7-dfb46395f3cb\") " pod="openshift-console/console-6bbcf8987c-4z9gw" Sep 11 15:56:19.329720 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:19.329651 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4800e361-38ce-40a5-8bf7-dfb46395f3cb-console-oauth-config\") pod \"console-6bbcf8987c-4z9gw\" (UID: \"4800e361-38ce-40a5-8bf7-dfb46395f3cb\") " pod="openshift-console/console-6bbcf8987c-4z9gw" Sep 11 15:56:19.329720 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:19.329709 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4800e361-38ce-40a5-8bf7-dfb46395f3cb-console-config\") pod \"console-6bbcf8987c-4z9gw\" (UID: \"4800e361-38ce-40a5-8bf7-dfb46395f3cb\") " pod="openshift-console/console-6bbcf8987c-4z9gw" Sep 11 15:56:19.329944 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:19.329750 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lldgb\" (UniqueName: \"kubernetes.io/projected/4800e361-38ce-40a5-8bf7-dfb46395f3cb-kube-api-access-lldgb\") pod \"console-6bbcf8987c-4z9gw\" (UID: \"4800e361-38ce-40a5-8bf7-dfb46395f3cb\") " pod="openshift-console/console-6bbcf8987c-4z9gw" Sep 11 15:56:19.330436 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:19.330410 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4800e361-38ce-40a5-8bf7-dfb46395f3cb-oauth-serving-cert\") pod \"console-6bbcf8987c-4z9gw\" (UID: \"4800e361-38ce-40a5-8bf7-dfb46395f3cb\") " pod="openshift-console/console-6bbcf8987c-4z9gw" Sep 11 15:56:19.330532 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:19.330444 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4800e361-38ce-40a5-8bf7-dfb46395f3cb-service-ca\") pod \"console-6bbcf8987c-4z9gw\" (UID: \"4800e361-38ce-40a5-8bf7-dfb46395f3cb\") " pod="openshift-console/console-6bbcf8987c-4z9gw" Sep 11 15:56:19.330532 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:19.330445 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4800e361-38ce-40a5-8bf7-dfb46395f3cb-console-config\") pod \"console-6bbcf8987c-4z9gw\" (UID: \"4800e361-38ce-40a5-8bf7-dfb46395f3cb\") " pod="openshift-console/console-6bbcf8987c-4z9gw" Sep 11 15:56:19.332583 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:19.332557 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4800e361-38ce-40a5-8bf7-dfb46395f3cb-console-serving-cert\") pod \"console-6bbcf8987c-4z9gw\" (UID: \"4800e361-38ce-40a5-8bf7-dfb46395f3cb\") " pod="openshift-console/console-6bbcf8987c-4z9gw" Sep 11 15:56:19.332817 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:19.332800 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4800e361-38ce-40a5-8bf7-dfb46395f3cb-console-oauth-config\") pod \"console-6bbcf8987c-4z9gw\" (UID: \"4800e361-38ce-40a5-8bf7-dfb46395f3cb\") " pod="openshift-console/console-6bbcf8987c-4z9gw" Sep 11 15:56:19.340185 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:19.340164 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lldgb\" (UniqueName: \"kubernetes.io/projected/4800e361-38ce-40a5-8bf7-dfb46395f3cb-kube-api-access-lldgb\") pod \"console-6bbcf8987c-4z9gw\" (UID: \"4800e361-38ce-40a5-8bf7-dfb46395f3cb\") " pod="openshift-console/console-6bbcf8987c-4z9gw" Sep 11 15:56:19.457384 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:19.457290 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bbcf8987c-4z9gw" Sep 11 15:56:19.738373 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:19.738334 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6bbcf8987c-4z9gw"] Sep 11 15:56:19.742947 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:56:19.741574 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4800e361_38ce_40a5_8bf7_dfb46395f3cb.slice/crio-0a43e8e0105da327cb1f93705b069a790bfc64e7f4fc5aac7df05f6cf51d8f13 WatchSource:0}: Error finding container 0a43e8e0105da327cb1f93705b069a790bfc64e7f4fc5aac7df05f6cf51d8f13: Status 404 returned error can't find the container with id 0a43e8e0105da327cb1f93705b069a790bfc64e7f4fc5aac7df05f6cf51d8f13 Sep 11 15:56:20.609434 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:20.609398 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-56c5d9df5d-7vqnb" Sep 11 15:56:20.741437 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:20.741396 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bbcf8987c-4z9gw" event={"ID":"4800e361-38ce-40a5-8bf7-dfb46395f3cb","Type":"ContainerStarted","Data":"0a43e8e0105da327cb1f93705b069a790bfc64e7f4fc5aac7df05f6cf51d8f13"} Sep 11 15:56:21.539440 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:21.539380 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-56c5d9df5d-7vqnb"] Sep 11 15:56:27.787687 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:27.787648 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-85c44c5bc8-mfchx"] Sep 11 15:56:27.827572 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:27.827538 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-85c44c5bc8-mfchx"] Sep 11 15:56:27.827723 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:27.827648 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85c44c5bc8-mfchx" Sep 11 15:56:27.834877 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:27.834853 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Sep 11 15:56:27.903083 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:27.903043 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cbdb6554-ce1f-4bb1-a3bd-1d746da28bc4-console-oauth-config\") pod \"console-85c44c5bc8-mfchx\" (UID: \"cbdb6554-ce1f-4bb1-a3bd-1d746da28bc4\") " pod="openshift-console/console-85c44c5bc8-mfchx" Sep 11 15:56:27.903255 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:27.903094 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cbdb6554-ce1f-4bb1-a3bd-1d746da28bc4-oauth-serving-cert\") pod \"console-85c44c5bc8-mfchx\" (UID: \"cbdb6554-ce1f-4bb1-a3bd-1d746da28bc4\") " pod="openshift-console/console-85c44c5bc8-mfchx" Sep 11 15:56:27.903255 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:27.903152 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cbdb6554-ce1f-4bb1-a3bd-1d746da28bc4-service-ca\") pod \"console-85c44c5bc8-mfchx\" (UID: \"cbdb6554-ce1f-4bb1-a3bd-1d746da28bc4\") " pod="openshift-console/console-85c44c5bc8-mfchx" Sep 11 15:56:27.903255 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:27.903185 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmfg7\" (UniqueName: \"kubernetes.io/projected/cbdb6554-ce1f-4bb1-a3bd-1d746da28bc4-kube-api-access-pmfg7\") pod \"console-85c44c5bc8-mfchx\" (UID: \"cbdb6554-ce1f-4bb1-a3bd-1d746da28bc4\") " pod="openshift-console/console-85c44c5bc8-mfchx" Sep 11 15:56:27.903255 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:27.903249 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cbdb6554-ce1f-4bb1-a3bd-1d746da28bc4-trusted-ca-bundle\") pod \"console-85c44c5bc8-mfchx\" (UID: \"cbdb6554-ce1f-4bb1-a3bd-1d746da28bc4\") " pod="openshift-console/console-85c44c5bc8-mfchx" Sep 11 15:56:27.903456 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:27.903278 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cbdb6554-ce1f-4bb1-a3bd-1d746da28bc4-console-serving-cert\") pod \"console-85c44c5bc8-mfchx\" (UID: \"cbdb6554-ce1f-4bb1-a3bd-1d746da28bc4\") " pod="openshift-console/console-85c44c5bc8-mfchx" Sep 11 15:56:27.903456 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:27.903381 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cbdb6554-ce1f-4bb1-a3bd-1d746da28bc4-console-config\") pod \"console-85c44c5bc8-mfchx\" (UID: \"cbdb6554-ce1f-4bb1-a3bd-1d746da28bc4\") " pod="openshift-console/console-85c44c5bc8-mfchx" Sep 11 15:56:28.004015 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:28.003953 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cbdb6554-ce1f-4bb1-a3bd-1d746da28bc4-trusted-ca-bundle\") pod \"console-85c44c5bc8-mfchx\" (UID: \"cbdb6554-ce1f-4bb1-a3bd-1d746da28bc4\") " pod="openshift-console/console-85c44c5bc8-mfchx" Sep 11 15:56:28.004015 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:28.004015 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cbdb6554-ce1f-4bb1-a3bd-1d746da28bc4-console-serving-cert\") pod \"console-85c44c5bc8-mfchx\" (UID: \"cbdb6554-ce1f-4bb1-a3bd-1d746da28bc4\") " pod="openshift-console/console-85c44c5bc8-mfchx" Sep 11 15:56:28.004283 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:28.004050 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cbdb6554-ce1f-4bb1-a3bd-1d746da28bc4-console-config\") pod \"console-85c44c5bc8-mfchx\" (UID: \"cbdb6554-ce1f-4bb1-a3bd-1d746da28bc4\") " pod="openshift-console/console-85c44c5bc8-mfchx" Sep 11 15:56:28.004283 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:28.004076 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cbdb6554-ce1f-4bb1-a3bd-1d746da28bc4-console-oauth-config\") pod \"console-85c44c5bc8-mfchx\" (UID: \"cbdb6554-ce1f-4bb1-a3bd-1d746da28bc4\") " pod="openshift-console/console-85c44c5bc8-mfchx" Sep 11 15:56:28.004283 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:28.004094 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cbdb6554-ce1f-4bb1-a3bd-1d746da28bc4-oauth-serving-cert\") pod \"console-85c44c5bc8-mfchx\" (UID: \"cbdb6554-ce1f-4bb1-a3bd-1d746da28bc4\") " pod="openshift-console/console-85c44c5bc8-mfchx" Sep 11 15:56:28.004283 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:28.004111 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cbdb6554-ce1f-4bb1-a3bd-1d746da28bc4-service-ca\") pod \"console-85c44c5bc8-mfchx\" (UID: \"cbdb6554-ce1f-4bb1-a3bd-1d746da28bc4\") " pod="openshift-console/console-85c44c5bc8-mfchx" Sep 11 15:56:28.004283 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:28.004128 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmfg7\" (UniqueName: \"kubernetes.io/projected/cbdb6554-ce1f-4bb1-a3bd-1d746da28bc4-kube-api-access-pmfg7\") pod \"console-85c44c5bc8-mfchx\" (UID: \"cbdb6554-ce1f-4bb1-a3bd-1d746da28bc4\") " pod="openshift-console/console-85c44c5bc8-mfchx" Sep 11 15:56:28.004854 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:28.004823 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cbdb6554-ce1f-4bb1-a3bd-1d746da28bc4-service-ca\") pod \"console-85c44c5bc8-mfchx\" (UID: \"cbdb6554-ce1f-4bb1-a3bd-1d746da28bc4\") " pod="openshift-console/console-85c44c5bc8-mfchx" Sep 11 15:56:28.005006 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:28.004895 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cbdb6554-ce1f-4bb1-a3bd-1d746da28bc4-console-config\") pod \"console-85c44c5bc8-mfchx\" (UID: \"cbdb6554-ce1f-4bb1-a3bd-1d746da28bc4\") " pod="openshift-console/console-85c44c5bc8-mfchx" Sep 11 15:56:28.005006 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:28.004921 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cbdb6554-ce1f-4bb1-a3bd-1d746da28bc4-trusted-ca-bundle\") pod \"console-85c44c5bc8-mfchx\" (UID: \"cbdb6554-ce1f-4bb1-a3bd-1d746da28bc4\") " pod="openshift-console/console-85c44c5bc8-mfchx" Sep 11 15:56:28.005006 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:28.004935 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cbdb6554-ce1f-4bb1-a3bd-1d746da28bc4-oauth-serving-cert\") pod \"console-85c44c5bc8-mfchx\" (UID: \"cbdb6554-ce1f-4bb1-a3bd-1d746da28bc4\") " pod="openshift-console/console-85c44c5bc8-mfchx" Sep 11 15:56:28.006727 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:28.006694 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cbdb6554-ce1f-4bb1-a3bd-1d746da28bc4-console-oauth-config\") pod \"console-85c44c5bc8-mfchx\" (UID: \"cbdb6554-ce1f-4bb1-a3bd-1d746da28bc4\") " pod="openshift-console/console-85c44c5bc8-mfchx" Sep 11 15:56:28.006727 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:28.006714 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cbdb6554-ce1f-4bb1-a3bd-1d746da28bc4-console-serving-cert\") pod \"console-85c44c5bc8-mfchx\" (UID: \"cbdb6554-ce1f-4bb1-a3bd-1d746da28bc4\") " pod="openshift-console/console-85c44c5bc8-mfchx" Sep 11 15:56:28.012624 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:28.012598 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmfg7\" (UniqueName: \"kubernetes.io/projected/cbdb6554-ce1f-4bb1-a3bd-1d746da28bc4-kube-api-access-pmfg7\") pod \"console-85c44c5bc8-mfchx\" (UID: \"cbdb6554-ce1f-4bb1-a3bd-1d746da28bc4\") " pod="openshift-console/console-85c44c5bc8-mfchx" Sep 11 15:56:28.139667 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:28.139584 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85c44c5bc8-mfchx" Sep 11 15:56:29.715202 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:29.715139 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-85c44c5bc8-mfchx"] Sep 11 15:56:29.718057 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:56:29.718028 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbdb6554_ce1f_4bb1_a3bd_1d746da28bc4.slice/crio-b14ddb4d59191910e02b5f2433acc9fd8e3186fadd6e93a141e9336c63e467df WatchSource:0}: Error finding container b14ddb4d59191910e02b5f2433acc9fd8e3186fadd6e93a141e9336c63e467df: Status 404 returned error can't find the container with id b14ddb4d59191910e02b5f2433acc9fd8e3186fadd6e93a141e9336c63e467df Sep 11 15:56:29.785380 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:29.785344 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-2f2j4" event={"ID":"6e65377d-74a2-4a00-9752-ce734ca7d357","Type":"ContainerStarted","Data":"0117320a880acc6271447ad2eedf7462ec9e5a95c34a3a5c1f3d41c437e73a35"} Sep 11 15:56:29.785509 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:29.785418 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-76f99584f4-2f2j4" Sep 11 15:56:29.786876 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:29.786852 2566 patch_prober.go:28] interesting pod/downloads-76f99584f4-2f2j4 container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.132.0.20:8080/\": dial tcp 10.132.0.20:8080: connect: connection refused" start-of-body= Sep 11 15:56:29.786993 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:29.786898 2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-76f99584f4-2f2j4" podUID="6e65377d-74a2-4a00-9752-ce734ca7d357" containerName="download-server" probeResult="failure" output="Get \"http://10.132.0.20:8080/\": dial tcp 10.132.0.20:8080: connect: connection refused" Sep 11 15:56:29.787292 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:29.787263 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bbcf8987c-4z9gw" event={"ID":"4800e361-38ce-40a5-8bf7-dfb46395f3cb","Type":"ContainerStarted","Data":"aa312fb182185e7210e32e797c81488a30872e5c18a0f21f8151e5710c34b0fa"} Sep 11 15:56:29.788853 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:29.788813 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85c44c5bc8-mfchx" event={"ID":"cbdb6554-ce1f-4bb1-a3bd-1d746da28bc4","Type":"ContainerStarted","Data":"b14ddb4d59191910e02b5f2433acc9fd8e3186fadd6e93a141e9336c63e467df"} Sep 11 15:56:29.803662 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:29.803619 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-76f99584f4-2f2j4" podStartSLOduration=1.054409135 podStartE2EDuration="19.803603387s" totalImagesPullingTime="18.749194252s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:56:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:56:29.801857977 +0000 UTC m=+97.016691881" watchObservedRunningTime="2026-09-11 15:56:29.803603387 +0000 UTC m=+97.018437292" Sep 11 15:56:29.819025 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:29.818866 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6bbcf8987c-4z9gw" podStartSLOduration=1.113650352 podStartE2EDuration="10.818849555s" totalImagesPullingTime="9.705199203s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:56:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:56:29.81851834 +0000 UTC m=+97.033352245" watchObservedRunningTime="2026-09-11 15:56:29.818849555 +0000 UTC m=+97.033683460" Sep 11 15:56:30.796307 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:30.796261 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85c44c5bc8-mfchx" event={"ID":"cbdb6554-ce1f-4bb1-a3bd-1d746da28bc4","Type":"ContainerStarted","Data":"e98df785efcba8dd657c55a8d888c84fe084c2a5f835a60f1890a6ea470a151e"} Sep 11 15:56:30.799667 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:30.799639 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-76f99584f4-2f2j4" Sep 11 15:56:30.813383 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:30.813324 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-85c44c5bc8-mfchx" podStartSLOduration=3.8133046569999998 podStartE2EDuration="3.813304657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:56:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:56:30.812127977 +0000 UTC m=+98.026961882" watchObservedRunningTime="2026-09-11 15:56:30.813304657 +0000 UTC m=+98.028138565" Sep 11 15:56:38.139799 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:38.139749 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-85c44c5bc8-mfchx" Sep 11 15:56:38.140369 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:38.139838 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-85c44c5bc8-mfchx" Sep 11 15:56:38.145321 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:38.145287 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-85c44c5bc8-mfchx" Sep 11 15:56:38.841161 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:38.841124 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-85c44c5bc8-mfchx" Sep 11 15:56:38.898879 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:38.898844 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6bbcf8987c-4z9gw"] Sep 11 15:56:39.457435 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:39.457400 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6bbcf8987c-4z9gw" Sep 11 15:56:46.564095 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:46.564053 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-56c5d9df5d-7vqnb" podUID="ae9022dc-da99-4acf-af85-3cf9858737b4" containerName="registry" containerID="cri-o://09e14c00a6b5b9039b882f090c099281b1ab2c4e1032f060c2cd71aa23707eb5" gracePeriod=30 Sep 11 15:56:46.848268 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:46.848243 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-56c5d9df5d-7vqnb" Sep 11 15:56:46.868370 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:46.868341 2566 generic.go:356] "Generic (PLEG): container finished" podID="ae9022dc-da99-4acf-af85-3cf9858737b4" containerID="09e14c00a6b5b9039b882f090c099281b1ab2c4e1032f060c2cd71aa23707eb5" exitCode=0 Sep 11 15:56:46.868502 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:46.868379 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-56c5d9df5d-7vqnb" Sep 11 15:56:46.868502 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:46.868413 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-56c5d9df5d-7vqnb" event={"ID":"ae9022dc-da99-4acf-af85-3cf9858737b4","Type":"ContainerDied","Data":"09e14c00a6b5b9039b882f090c099281b1ab2c4e1032f060c2cd71aa23707eb5"} Sep 11 15:56:46.868502 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:46.868473 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-56c5d9df5d-7vqnb" event={"ID":"ae9022dc-da99-4acf-af85-3cf9858737b4","Type":"ContainerDied","Data":"3f4337e42e3663b9b51a15f30d9673a26fdcc74394e00b05ad99395b4bd1999b"} Sep 11 15:56:46.868502 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:46.868493 2566 scope.go:117] "RemoveContainer" containerID="09e14c00a6b5b9039b882f090c099281b1ab2c4e1032f060c2cd71aa23707eb5" Sep 11 15:56:46.879033 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:46.879006 2566 scope.go:117] "RemoveContainer" containerID="09e14c00a6b5b9039b882f090c099281b1ab2c4e1032f060c2cd71aa23707eb5" Sep 11 15:56:46.879333 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:56:46.879313 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09e14c00a6b5b9039b882f090c099281b1ab2c4e1032f060c2cd71aa23707eb5\": container with ID starting with 09e14c00a6b5b9039b882f090c099281b1ab2c4e1032f060c2cd71aa23707eb5 not found: ID does not exist" containerID="09e14c00a6b5b9039b882f090c099281b1ab2c4e1032f060c2cd71aa23707eb5" Sep 11 15:56:46.879396 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:46.879345 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09e14c00a6b5b9039b882f090c099281b1ab2c4e1032f060c2cd71aa23707eb5"} err="failed to get container status \"09e14c00a6b5b9039b882f090c099281b1ab2c4e1032f060c2cd71aa23707eb5\": rpc error: code = NotFound desc = could not find container \"09e14c00a6b5b9039b882f090c099281b1ab2c4e1032f060c2cd71aa23707eb5\": container with ID starting with 09e14c00a6b5b9039b882f090c099281b1ab2c4e1032f060c2cd71aa23707eb5 not found: ID does not exist" Sep 11 15:56:46.977768 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:46.977733 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jxrcg\" (UniqueName: \"kubernetes.io/projected/ae9022dc-da99-4acf-af85-3cf9858737b4-kube-api-access-jxrcg\") pod \"ae9022dc-da99-4acf-af85-3cf9858737b4\" (UID: \"ae9022dc-da99-4acf-af85-3cf9858737b4\") " Sep 11 15:56:46.977904 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:46.977778 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ae9022dc-da99-4acf-af85-3cf9858737b4-ca-trust-extracted\") pod \"ae9022dc-da99-4acf-af85-3cf9858737b4\" (UID: \"ae9022dc-da99-4acf-af85-3cf9858737b4\") " Sep 11 15:56:46.977904 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:46.977801 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ae9022dc-da99-4acf-af85-3cf9858737b4-installation-pull-secrets\") pod \"ae9022dc-da99-4acf-af85-3cf9858737b4\" (UID: \"ae9022dc-da99-4acf-af85-3cf9858737b4\") " Sep 11 15:56:46.977904 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:46.977834 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ae9022dc-da99-4acf-af85-3cf9858737b4-trusted-ca\") pod \"ae9022dc-da99-4acf-af85-3cf9858737b4\" (UID: \"ae9022dc-da99-4acf-af85-3cf9858737b4\") " Sep 11 15:56:46.978074 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:46.977909 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ae9022dc-da99-4acf-af85-3cf9858737b4-registry-certificates\") pod \"ae9022dc-da99-4acf-af85-3cf9858737b4\" (UID: \"ae9022dc-da99-4acf-af85-3cf9858737b4\") " Sep 11 15:56:46.978074 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:46.977931 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ae9022dc-da99-4acf-af85-3cf9858737b4-image-registry-private-configuration\") pod \"ae9022dc-da99-4acf-af85-3cf9858737b4\" (UID: \"ae9022dc-da99-4acf-af85-3cf9858737b4\") " Sep 11 15:56:46.978074 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:46.977950 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ae9022dc-da99-4acf-af85-3cf9858737b4-registry-tls\") pod \"ae9022dc-da99-4acf-af85-3cf9858737b4\" (UID: \"ae9022dc-da99-4acf-af85-3cf9858737b4\") " Sep 11 15:56:46.978074 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:46.978000 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ae9022dc-da99-4acf-af85-3cf9858737b4-bound-sa-token\") pod \"ae9022dc-da99-4acf-af85-3cf9858737b4\" (UID: \"ae9022dc-da99-4acf-af85-3cf9858737b4\") " Sep 11 15:56:46.978609 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:46.978512 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae9022dc-da99-4acf-af85-3cf9858737b4-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "ae9022dc-da99-4acf-af85-3cf9858737b4" (UID: "ae9022dc-da99-4acf-af85-3cf9858737b4"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:56:46.978609 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:46.978589 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae9022dc-da99-4acf-af85-3cf9858737b4-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "ae9022dc-da99-4acf-af85-3cf9858737b4" (UID: "ae9022dc-da99-4acf-af85-3cf9858737b4"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:56:46.980550 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:46.980513 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae9022dc-da99-4acf-af85-3cf9858737b4-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "ae9022dc-da99-4acf-af85-3cf9858737b4" (UID: "ae9022dc-da99-4acf-af85-3cf9858737b4"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 15:56:46.980783 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:46.980737 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae9022dc-da99-4acf-af85-3cf9858737b4-kube-api-access-jxrcg" (OuterVolumeSpecName: "kube-api-access-jxrcg") pod "ae9022dc-da99-4acf-af85-3cf9858737b4" (UID: "ae9022dc-da99-4acf-af85-3cf9858737b4"). InnerVolumeSpecName "kube-api-access-jxrcg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:56:46.980863 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:46.980813 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae9022dc-da99-4acf-af85-3cf9858737b4-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "ae9022dc-da99-4acf-af85-3cf9858737b4" (UID: "ae9022dc-da99-4acf-af85-3cf9858737b4"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:56:46.980905 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:46.980862 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae9022dc-da99-4acf-af85-3cf9858737b4-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "ae9022dc-da99-4acf-af85-3cf9858737b4" (UID: "ae9022dc-da99-4acf-af85-3cf9858737b4"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 15:56:46.980943 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:46.980892 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae9022dc-da99-4acf-af85-3cf9858737b4-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "ae9022dc-da99-4acf-af85-3cf9858737b4" (UID: "ae9022dc-da99-4acf-af85-3cf9858737b4"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:56:46.986218 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:46.986195 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae9022dc-da99-4acf-af85-3cf9858737b4-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "ae9022dc-da99-4acf-af85-3cf9858737b4" (UID: "ae9022dc-da99-4acf-af85-3cf9858737b4"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:56:47.079660 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:47.079577 2566 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ae9022dc-da99-4acf-af85-3cf9858737b4-trusted-ca\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 15:56:47.079660 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:47.079608 2566 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ae9022dc-da99-4acf-af85-3cf9858737b4-registry-certificates\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 15:56:47.079660 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:47.079620 2566 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ae9022dc-da99-4acf-af85-3cf9858737b4-image-registry-private-configuration\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 15:56:47.079660 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:47.079629 2566 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ae9022dc-da99-4acf-af85-3cf9858737b4-registry-tls\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 15:56:47.079660 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:47.079636 2566 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ae9022dc-da99-4acf-af85-3cf9858737b4-bound-sa-token\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 15:56:47.079660 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:47.079644 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxrcg\" (UniqueName: \"kubernetes.io/projected/ae9022dc-da99-4acf-af85-3cf9858737b4-kube-api-access-jxrcg\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 15:56:47.079660 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:47.079652 2566 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ae9022dc-da99-4acf-af85-3cf9858737b4-ca-trust-extracted\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 15:56:47.079660 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:47.079660 2566 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ae9022dc-da99-4acf-af85-3cf9858737b4-installation-pull-secrets\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 15:56:47.189848 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:47.189817 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-56c5d9df5d-7vqnb"] Sep 11 15:56:47.195181 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:47.194800 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-56c5d9df5d-7vqnb"] Sep 11 15:56:47.237877 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:47.237839 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae9022dc-da99-4acf-af85-3cf9858737b4" path="/var/lib/kubelet/pods/ae9022dc-da99-4acf-af85-3cf9858737b4/volumes" Sep 11 15:56:47.875029 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:47.874993 2566 generic.go:356] "Generic (PLEG): container finished" podID="9f6c5948-c774-49b6-aaf5-026c7b5db718" containerID="bd2d55ad0c91c8b05fd10361874923987f18d568a43b128b1af2facc4523d361" exitCode=0 Sep 11 15:56:47.875464 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:47.875064 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-mjh4m" event={"ID":"9f6c5948-c774-49b6-aaf5-026c7b5db718","Type":"ContainerDied","Data":"bd2d55ad0c91c8b05fd10361874923987f18d568a43b128b1af2facc4523d361"} Sep 11 15:56:47.875464 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:47.875426 2566 scope.go:117] "RemoveContainer" containerID="bd2d55ad0c91c8b05fd10361874923987f18d568a43b128b1af2facc4523d361" Sep 11 15:56:47.876661 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:47.876640 2566 generic.go:356] "Generic (PLEG): container finished" podID="26aadde0-9d1e-4368-86b0-4a2eb04656c3" containerID="83617c4adcdb33c090adfb71161c6a7e280f2b9a4b37017ea7886d14ae3a263c" exitCode=0 Sep 11 15:56:47.876747 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:47.876669 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-qjtlt" event={"ID":"26aadde0-9d1e-4368-86b0-4a2eb04656c3","Type":"ContainerDied","Data":"83617c4adcdb33c090adfb71161c6a7e280f2b9a4b37017ea7886d14ae3a263c"} Sep 11 15:56:47.876929 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:47.876916 2566 scope.go:117] "RemoveContainer" containerID="83617c4adcdb33c090adfb71161c6a7e280f2b9a4b37017ea7886d14ae3a263c" Sep 11 15:56:48.882427 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:48.882392 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-mjh4m" event={"ID":"9f6c5948-c774-49b6-aaf5-026c7b5db718","Type":"ContainerStarted","Data":"af2afcf61f706da7e2cada6cae3dd64f694bf281979470100bea59392c559bf6"} Sep 11 15:56:48.884376 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:56:48.884348 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-qjtlt" event={"ID":"26aadde0-9d1e-4368-86b0-4a2eb04656c3","Type":"ContainerStarted","Data":"0d82f3cfc3e987b0070724ccb032fb30188ecc99aceedd3dee4ee0e9f258b412"} Sep 11 15:57:03.927847 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:03.927798 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6bbcf8987c-4z9gw" podUID="4800e361-38ce-40a5-8bf7-dfb46395f3cb" containerName="console" containerID="cri-o://aa312fb182185e7210e32e797c81488a30872e5c18a0f21f8151e5710c34b0fa" gracePeriod=15 Sep 11 15:57:04.191702 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:04.191680 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bbcf8987c-4z9gw" Sep 11 15:57:04.326164 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:04.326127 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4800e361-38ce-40a5-8bf7-dfb46395f3cb-console-config\") pod \"4800e361-38ce-40a5-8bf7-dfb46395f3cb\" (UID: \"4800e361-38ce-40a5-8bf7-dfb46395f3cb\") " Sep 11 15:57:04.326349 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:04.326230 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4800e361-38ce-40a5-8bf7-dfb46395f3cb-oauth-serving-cert\") pod \"4800e361-38ce-40a5-8bf7-dfb46395f3cb\" (UID: \"4800e361-38ce-40a5-8bf7-dfb46395f3cb\") " Sep 11 15:57:04.326349 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:04.326259 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4800e361-38ce-40a5-8bf7-dfb46395f3cb-console-oauth-config\") pod \"4800e361-38ce-40a5-8bf7-dfb46395f3cb\" (UID: \"4800e361-38ce-40a5-8bf7-dfb46395f3cb\") " Sep 11 15:57:04.326349 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:04.326285 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4800e361-38ce-40a5-8bf7-dfb46395f3cb-service-ca\") pod \"4800e361-38ce-40a5-8bf7-dfb46395f3cb\" (UID: \"4800e361-38ce-40a5-8bf7-dfb46395f3cb\") " Sep 11 15:57:04.326349 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:04.326313 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4800e361-38ce-40a5-8bf7-dfb46395f3cb-console-serving-cert\") pod \"4800e361-38ce-40a5-8bf7-dfb46395f3cb\" (UID: \"4800e361-38ce-40a5-8bf7-dfb46395f3cb\") " Sep 11 15:57:04.326553 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:04.326361 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lldgb\" (UniqueName: \"kubernetes.io/projected/4800e361-38ce-40a5-8bf7-dfb46395f3cb-kube-api-access-lldgb\") pod \"4800e361-38ce-40a5-8bf7-dfb46395f3cb\" (UID: \"4800e361-38ce-40a5-8bf7-dfb46395f3cb\") " Sep 11 15:57:04.326648 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:04.326608 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4800e361-38ce-40a5-8bf7-dfb46395f3cb-console-config" (OuterVolumeSpecName: "console-config") pod "4800e361-38ce-40a5-8bf7-dfb46395f3cb" (UID: "4800e361-38ce-40a5-8bf7-dfb46395f3cb"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:57:04.326727 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:04.326661 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4800e361-38ce-40a5-8bf7-dfb46395f3cb-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "4800e361-38ce-40a5-8bf7-dfb46395f3cb" (UID: "4800e361-38ce-40a5-8bf7-dfb46395f3cb"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:57:04.326727 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:04.326714 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4800e361-38ce-40a5-8bf7-dfb46395f3cb-service-ca" (OuterVolumeSpecName: "service-ca") pod "4800e361-38ce-40a5-8bf7-dfb46395f3cb" (UID: "4800e361-38ce-40a5-8bf7-dfb46395f3cb"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:57:04.327051 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:04.326839 2566 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4800e361-38ce-40a5-8bf7-dfb46395f3cb-oauth-serving-cert\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 15:57:04.327051 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:04.326865 2566 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4800e361-38ce-40a5-8bf7-dfb46395f3cb-service-ca\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 15:57:04.327051 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:04.326882 2566 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4800e361-38ce-40a5-8bf7-dfb46395f3cb-console-config\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 15:57:04.328585 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:04.328560 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4800e361-38ce-40a5-8bf7-dfb46395f3cb-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "4800e361-38ce-40a5-8bf7-dfb46395f3cb" (UID: "4800e361-38ce-40a5-8bf7-dfb46395f3cb"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 15:57:04.328954 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:04.328932 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4800e361-38ce-40a5-8bf7-dfb46395f3cb-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "4800e361-38ce-40a5-8bf7-dfb46395f3cb" (UID: "4800e361-38ce-40a5-8bf7-dfb46395f3cb"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 15:57:04.329045 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:04.328933 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4800e361-38ce-40a5-8bf7-dfb46395f3cb-kube-api-access-lldgb" (OuterVolumeSpecName: "kube-api-access-lldgb") pod "4800e361-38ce-40a5-8bf7-dfb46395f3cb" (UID: "4800e361-38ce-40a5-8bf7-dfb46395f3cb"). InnerVolumeSpecName "kube-api-access-lldgb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:57:04.427465 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:04.427427 2566 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4800e361-38ce-40a5-8bf7-dfb46395f3cb-console-oauth-config\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 15:57:04.427465 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:04.427459 2566 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4800e361-38ce-40a5-8bf7-dfb46395f3cb-console-serving-cert\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 15:57:04.427697 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:04.427489 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lldgb\" (UniqueName: \"kubernetes.io/projected/4800e361-38ce-40a5-8bf7-dfb46395f3cb-kube-api-access-lldgb\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 15:57:04.954229 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:04.954192 2566 generic.go:356] "Generic (PLEG): container finished" podID="4800e361-38ce-40a5-8bf7-dfb46395f3cb" containerID="aa312fb182185e7210e32e797c81488a30872e5c18a0f21f8151e5710c34b0fa" exitCode=0 Sep 11 15:57:04.954623 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:04.954237 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bbcf8987c-4z9gw" Sep 11 15:57:04.954623 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:04.954276 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bbcf8987c-4z9gw" event={"ID":"4800e361-38ce-40a5-8bf7-dfb46395f3cb","Type":"ContainerDied","Data":"aa312fb182185e7210e32e797c81488a30872e5c18a0f21f8151e5710c34b0fa"} Sep 11 15:57:04.954623 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:04.954327 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bbcf8987c-4z9gw" event={"ID":"4800e361-38ce-40a5-8bf7-dfb46395f3cb","Type":"ContainerDied","Data":"0a43e8e0105da327cb1f93705b069a790bfc64e7f4fc5aac7df05f6cf51d8f13"} Sep 11 15:57:04.954623 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:04.954348 2566 scope.go:117] "RemoveContainer" containerID="aa312fb182185e7210e32e797c81488a30872e5c18a0f21f8151e5710c34b0fa" Sep 11 15:57:04.963325 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:04.963309 2566 scope.go:117] "RemoveContainer" containerID="aa312fb182185e7210e32e797c81488a30872e5c18a0f21f8151e5710c34b0fa" Sep 11 15:57:04.963603 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:57:04.963583 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa312fb182185e7210e32e797c81488a30872e5c18a0f21f8151e5710c34b0fa\": container with ID starting with aa312fb182185e7210e32e797c81488a30872e5c18a0f21f8151e5710c34b0fa not found: ID does not exist" containerID="aa312fb182185e7210e32e797c81488a30872e5c18a0f21f8151e5710c34b0fa" Sep 11 15:57:04.963673 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:04.963609 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa312fb182185e7210e32e797c81488a30872e5c18a0f21f8151e5710c34b0fa"} err="failed to get container status \"aa312fb182185e7210e32e797c81488a30872e5c18a0f21f8151e5710c34b0fa\": rpc error: code = NotFound desc = could not find container \"aa312fb182185e7210e32e797c81488a30872e5c18a0f21f8151e5710c34b0fa\": container with ID starting with aa312fb182185e7210e32e797c81488a30872e5c18a0f21f8151e5710c34b0fa not found: ID does not exist" Sep 11 15:57:04.975381 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:04.975354 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6bbcf8987c-4z9gw"] Sep 11 15:57:04.979371 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:04.979347 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6bbcf8987c-4z9gw"] Sep 11 15:57:05.237955 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:05.237861 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4800e361-38ce-40a5-8bf7-dfb46395f3cb" path="/var/lib/kubelet/pods/4800e361-38ce-40a5-8bf7-dfb46395f3cb/volumes" Sep 11 15:57:06.964557 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:06.964527 2566 generic.go:356] "Generic (PLEG): container finished" podID="07d36306-7ec8-4c74-9b97-7545503a2783" containerID="b5450ba9123eea42e233adb54457f73d7a14b2e1cec3fcad65b5ae24d9314e2e" exitCode=0 Sep 11 15:57:06.964917 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:06.964601 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-djghf" event={"ID":"07d36306-7ec8-4c74-9b97-7545503a2783","Type":"ContainerDied","Data":"b5450ba9123eea42e233adb54457f73d7a14b2e1cec3fcad65b5ae24d9314e2e"} Sep 11 15:57:06.965000 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:06.964989 2566 scope.go:117] "RemoveContainer" containerID="b5450ba9123eea42e233adb54457f73d7a14b2e1cec3fcad65b5ae24d9314e2e" Sep 11 15:57:07.970570 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:07.970537 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-djghf" event={"ID":"07d36306-7ec8-4c74-9b97-7545503a2783","Type":"ContainerStarted","Data":"c6c1e7932c1e942225a55ef0086321d508b34b8f202e6b8af5e93e3b589f0888"} Sep 11 15:57:32.084884 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:32.084844 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 11 15:57:32.085350 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:32.085165 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e" containerName="alertmanager" containerID="cri-o://fb6a81fde4b2618df228016db131e5c0b0a13ff947344cebc687e5385dd47f76" gracePeriod=120 Sep 11 15:57:32.085350 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:32.085222 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e" containerName="kube-rbac-proxy-metric" containerID="cri-o://52238d9341f63161169e082f4bc1d3b237f431909b8dfd7bd8ad22aebebad77c" gracePeriod=120 Sep 11 15:57:32.085350 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:32.085243 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e" containerName="kube-rbac-proxy-web" containerID="cri-o://da377bde8e965d6c15a6ed75fcccb48f858bc1b1a64924817b33f2e22cf49c89" gracePeriod=120 Sep 11 15:57:32.085350 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:32.085247 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e" containerName="config-reloader" containerID="cri-o://0e3c4adc5c97f22adf03fc3eac0e6dc7532feee94523cf9844720b38a0a155eb" gracePeriod=120 Sep 11 15:57:32.085350 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:32.085303 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e" containerName="kube-rbac-proxy" containerID="cri-o://73de8c544a4364477c0476052ff4f678b0f381aaa022a218cd41ecde0972824e" gracePeriod=120 Sep 11 15:57:32.085350 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:32.085340 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e" containerName="prom-label-proxy" containerID="cri-o://00528d1348836b690cfbaec38fa1113f608911bfbe67b2078644847c56587987" gracePeriod=120 Sep 11 15:57:33.078410 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:33.078375 2566 generic.go:356] "Generic (PLEG): container finished" podID="4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e" containerID="00528d1348836b690cfbaec38fa1113f608911bfbe67b2078644847c56587987" exitCode=0 Sep 11 15:57:33.078410 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:33.078401 2566 generic.go:356] "Generic (PLEG): container finished" podID="4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e" containerID="73de8c544a4364477c0476052ff4f678b0f381aaa022a218cd41ecde0972824e" exitCode=0 Sep 11 15:57:33.078410 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:33.078408 2566 generic.go:356] "Generic (PLEG): container finished" podID="4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e" containerID="0e3c4adc5c97f22adf03fc3eac0e6dc7532feee94523cf9844720b38a0a155eb" exitCode=0 Sep 11 15:57:33.078410 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:33.078413 2566 generic.go:356] "Generic (PLEG): container finished" podID="4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e" containerID="fb6a81fde4b2618df228016db131e5c0b0a13ff947344cebc687e5385dd47f76" exitCode=0 Sep 11 15:57:33.078725 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:33.078446 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e","Type":"ContainerDied","Data":"00528d1348836b690cfbaec38fa1113f608911bfbe67b2078644847c56587987"} Sep 11 15:57:33.078725 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:33.078529 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e","Type":"ContainerDied","Data":"73de8c544a4364477c0476052ff4f678b0f381aaa022a218cd41ecde0972824e"} Sep 11 15:57:33.078725 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:33.078571 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e","Type":"ContainerDied","Data":"0e3c4adc5c97f22adf03fc3eac0e6dc7532feee94523cf9844720b38a0a155eb"} Sep 11 15:57:33.078725 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:33.078605 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e","Type":"ContainerDied","Data":"fb6a81fde4b2618df228016db131e5c0b0a13ff947344cebc687e5385dd47f76"} Sep 11 15:57:33.324678 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:33.324654 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 11 15:57:33.367664 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:33.367581 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e-alertmanager-trusted-ca-bundle\") pod \"4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e\" (UID: \"4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e\") " Sep 11 15:57:33.367664 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:33.367623 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e-metrics-client-ca\") pod \"4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e\" (UID: \"4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e\") " Sep 11 15:57:33.367873 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:33.367669 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e-config-volume\") pod \"4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e\" (UID: \"4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e\") " Sep 11 15:57:33.367873 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:33.367701 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e\" (UID: \"4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e\") " Sep 11 15:57:33.368086 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:33.368049 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e-web-config\") pod \"4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e\" (UID: \"4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e\") " Sep 11 15:57:33.368185 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:33.368162 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e" (UID: "4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:57:33.368250 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:33.368191 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ws64g\" (UniqueName: \"kubernetes.io/projected/4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e-kube-api-access-ws64g\") pod \"4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e\" (UID: \"4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e\") " Sep 11 15:57:33.368250 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:33.368237 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e-secret-alertmanager-kube-rbac-proxy-web\") pod \"4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e\" (UID: \"4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e\") " Sep 11 15:57:33.368350 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:33.368263 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e-config-out\") pod \"4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e\" (UID: \"4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e\") " Sep 11 15:57:33.368350 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:33.368295 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e-tls-assets\") pod \"4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e\" (UID: \"4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e\") " Sep 11 15:57:33.368350 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:33.368292 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e" (UID: "4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:57:33.368350 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:33.368325 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e-alertmanager-main-db\") pod \"4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e\" (UID: \"4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e\") " Sep 11 15:57:33.368538 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:33.368354 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e-secret-alertmanager-kube-rbac-proxy\") pod \"4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e\" (UID: \"4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e\") " Sep 11 15:57:33.368538 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:33.368423 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e-cluster-tls-config\") pod \"4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e\" (UID: \"4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e\") " Sep 11 15:57:33.369288 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:33.369019 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e" (UID: "4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:57:33.369961 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:33.369612 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e-secret-alertmanager-main-tls\") pod \"4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e\" (UID: \"4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e\") " Sep 11 15:57:33.369961 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:33.369894 2566 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 15:57:33.369961 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:33.369918 2566 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e-metrics-client-ca\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 15:57:33.369961 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:33.369935 2566 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e-alertmanager-main-db\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 15:57:33.371906 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:33.371838 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e" (UID: "4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 15:57:33.373777 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:33.373747 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e-config-volume" (OuterVolumeSpecName: "config-volume") pod "4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e" (UID: "4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 15:57:33.373777 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:33.373756 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e-kube-api-access-ws64g" (OuterVolumeSpecName: "kube-api-access-ws64g") pod "4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e" (UID: "4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e"). InnerVolumeSpecName "kube-api-access-ws64g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:57:33.374619 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:33.374587 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e" (UID: "4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 15:57:33.374904 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:33.374873 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e" (UID: "4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:57:33.375037 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:33.374907 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e" (UID: "4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 15:57:33.375037 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:33.374995 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e-config-out" (OuterVolumeSpecName: "config-out") pod "4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e" (UID: "4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 15:57:33.375234 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:33.375214 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e" (UID: "4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 15:57:33.378931 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:33.378907 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e" (UID: "4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 15:57:33.386157 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:33.386135 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e-web-config" (OuterVolumeSpecName: "web-config") pod "4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e" (UID: "4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 15:57:33.470451 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:33.470410 2566 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e-cluster-tls-config\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 15:57:33.470451 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:33.470444 2566 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e-secret-alertmanager-main-tls\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 15:57:33.470451 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:33.470457 2566 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e-config-volume\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 15:57:33.470687 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:33.470470 2566 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 15:57:33.470687 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:33.470485 2566 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e-web-config\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 15:57:33.470687 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:33.470497 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ws64g\" (UniqueName: \"kubernetes.io/projected/4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e-kube-api-access-ws64g\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 15:57:33.470687 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:33.470507 2566 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 15:57:33.470687 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:33.470519 2566 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e-config-out\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 15:57:33.470687 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:33.470531 2566 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e-tls-assets\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 15:57:33.470687 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:33.470541 2566 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 15:57:34.085800 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:34.085759 2566 generic.go:356] "Generic (PLEG): container finished" podID="4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e" containerID="52238d9341f63161169e082f4bc1d3b237f431909b8dfd7bd8ad22aebebad77c" exitCode=0 Sep 11 15:57:34.085800 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:34.085794 2566 generic.go:356] "Generic (PLEG): container finished" podID="4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e" containerID="da377bde8e965d6c15a6ed75fcccb48f858bc1b1a64924817b33f2e22cf49c89" exitCode=0 Sep 11 15:57:34.086026 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:34.085836 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e","Type":"ContainerDied","Data":"52238d9341f63161169e082f4bc1d3b237f431909b8dfd7bd8ad22aebebad77c"} Sep 11 15:57:34.086026 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:34.085858 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 11 15:57:34.086026 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:34.085906 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e","Type":"ContainerDied","Data":"da377bde8e965d6c15a6ed75fcccb48f858bc1b1a64924817b33f2e22cf49c89"} Sep 11 15:57:34.086026 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:34.085934 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e","Type":"ContainerDied","Data":"12d1a487e9a70cb3a313e4abf47fc168e7963439685c4ebae9521463a2c38800"} Sep 11 15:57:34.086026 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:34.085979 2566 scope.go:117] "RemoveContainer" containerID="00528d1348836b690cfbaec38fa1113f608911bfbe67b2078644847c56587987" Sep 11 15:57:34.094029 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:34.094004 2566 scope.go:117] "RemoveContainer" containerID="52238d9341f63161169e082f4bc1d3b237f431909b8dfd7bd8ad22aebebad77c" Sep 11 15:57:34.100801 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:34.100785 2566 scope.go:117] "RemoveContainer" containerID="73de8c544a4364477c0476052ff4f678b0f381aaa022a218cd41ecde0972824e" Sep 11 15:57:34.107488 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:34.107469 2566 scope.go:117] "RemoveContainer" containerID="da377bde8e965d6c15a6ed75fcccb48f858bc1b1a64924817b33f2e22cf49c89" Sep 11 15:57:34.110702 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:34.110681 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 11 15:57:34.114633 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:34.114615 2566 scope.go:117] "RemoveContainer" containerID="0e3c4adc5c97f22adf03fc3eac0e6dc7532feee94523cf9844720b38a0a155eb" Sep 11 15:57:34.121186 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:34.121159 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 11 15:57:34.127410 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:34.127377 2566 scope.go:117] "RemoveContainer" containerID="fb6a81fde4b2618df228016db131e5c0b0a13ff947344cebc687e5385dd47f76" Sep 11 15:57:34.134708 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:34.134685 2566 scope.go:117] "RemoveContainer" containerID="ccaf640ff68701612e58cd96151db2f80ed789801f744d5ea514d349bb1d9b52" Sep 11 15:57:34.141880 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:34.141865 2566 scope.go:117] "RemoveContainer" containerID="00528d1348836b690cfbaec38fa1113f608911bfbe67b2078644847c56587987" Sep 11 15:57:34.142256 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:57:34.142233 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00528d1348836b690cfbaec38fa1113f608911bfbe67b2078644847c56587987\": container with ID starting with 00528d1348836b690cfbaec38fa1113f608911bfbe67b2078644847c56587987 not found: ID does not exist" containerID="00528d1348836b690cfbaec38fa1113f608911bfbe67b2078644847c56587987" Sep 11 15:57:34.142349 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:34.142267 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00528d1348836b690cfbaec38fa1113f608911bfbe67b2078644847c56587987"} err="failed to get container status \"00528d1348836b690cfbaec38fa1113f608911bfbe67b2078644847c56587987\": rpc error: code = NotFound desc = could not find container \"00528d1348836b690cfbaec38fa1113f608911bfbe67b2078644847c56587987\": container with ID starting with 00528d1348836b690cfbaec38fa1113f608911bfbe67b2078644847c56587987 not found: ID does not exist" Sep 11 15:57:34.142349 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:34.142293 2566 scope.go:117] "RemoveContainer" containerID="52238d9341f63161169e082f4bc1d3b237f431909b8dfd7bd8ad22aebebad77c" Sep 11 15:57:34.142599 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:57:34.142581 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52238d9341f63161169e082f4bc1d3b237f431909b8dfd7bd8ad22aebebad77c\": container with ID starting with 52238d9341f63161169e082f4bc1d3b237f431909b8dfd7bd8ad22aebebad77c not found: ID does not exist" containerID="52238d9341f63161169e082f4bc1d3b237f431909b8dfd7bd8ad22aebebad77c" Sep 11 15:57:34.142639 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:34.142603 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52238d9341f63161169e082f4bc1d3b237f431909b8dfd7bd8ad22aebebad77c"} err="failed to get container status \"52238d9341f63161169e082f4bc1d3b237f431909b8dfd7bd8ad22aebebad77c\": rpc error: code = NotFound desc = could not find container \"52238d9341f63161169e082f4bc1d3b237f431909b8dfd7bd8ad22aebebad77c\": container with ID starting with 52238d9341f63161169e082f4bc1d3b237f431909b8dfd7bd8ad22aebebad77c not found: ID does not exist" Sep 11 15:57:34.142639 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:34.142618 2566 scope.go:117] "RemoveContainer" containerID="73de8c544a4364477c0476052ff4f678b0f381aaa022a218cd41ecde0972824e" Sep 11 15:57:34.142856 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:57:34.142836 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73de8c544a4364477c0476052ff4f678b0f381aaa022a218cd41ecde0972824e\": container with ID starting with 73de8c544a4364477c0476052ff4f678b0f381aaa022a218cd41ecde0972824e not found: ID does not exist" containerID="73de8c544a4364477c0476052ff4f678b0f381aaa022a218cd41ecde0972824e" Sep 11 15:57:34.142922 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:34.142862 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73de8c544a4364477c0476052ff4f678b0f381aaa022a218cd41ecde0972824e"} err="failed to get container status \"73de8c544a4364477c0476052ff4f678b0f381aaa022a218cd41ecde0972824e\": rpc error: code = NotFound desc = could not find container \"73de8c544a4364477c0476052ff4f678b0f381aaa022a218cd41ecde0972824e\": container with ID starting with 73de8c544a4364477c0476052ff4f678b0f381aaa022a218cd41ecde0972824e not found: ID does not exist" Sep 11 15:57:34.142922 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:34.142884 2566 scope.go:117] "RemoveContainer" containerID="da377bde8e965d6c15a6ed75fcccb48f858bc1b1a64924817b33f2e22cf49c89" Sep 11 15:57:34.143159 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:57:34.143141 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da377bde8e965d6c15a6ed75fcccb48f858bc1b1a64924817b33f2e22cf49c89\": container with ID starting with da377bde8e965d6c15a6ed75fcccb48f858bc1b1a64924817b33f2e22cf49c89 not found: ID does not exist" containerID="da377bde8e965d6c15a6ed75fcccb48f858bc1b1a64924817b33f2e22cf49c89" Sep 11 15:57:34.143209 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:34.143164 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da377bde8e965d6c15a6ed75fcccb48f858bc1b1a64924817b33f2e22cf49c89"} err="failed to get container status \"da377bde8e965d6c15a6ed75fcccb48f858bc1b1a64924817b33f2e22cf49c89\": rpc error: code = NotFound desc = could not find container \"da377bde8e965d6c15a6ed75fcccb48f858bc1b1a64924817b33f2e22cf49c89\": container with ID starting with da377bde8e965d6c15a6ed75fcccb48f858bc1b1a64924817b33f2e22cf49c89 not found: ID does not exist" Sep 11 15:57:34.143209 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:34.143176 2566 scope.go:117] "RemoveContainer" containerID="0e3c4adc5c97f22adf03fc3eac0e6dc7532feee94523cf9844720b38a0a155eb" Sep 11 15:57:34.143422 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:57:34.143407 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e3c4adc5c97f22adf03fc3eac0e6dc7532feee94523cf9844720b38a0a155eb\": container with ID starting with 0e3c4adc5c97f22adf03fc3eac0e6dc7532feee94523cf9844720b38a0a155eb not found: ID does not exist" containerID="0e3c4adc5c97f22adf03fc3eac0e6dc7532feee94523cf9844720b38a0a155eb" Sep 11 15:57:34.143482 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:34.143426 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e3c4adc5c97f22adf03fc3eac0e6dc7532feee94523cf9844720b38a0a155eb"} err="failed to get container status \"0e3c4adc5c97f22adf03fc3eac0e6dc7532feee94523cf9844720b38a0a155eb\": rpc error: code = NotFound desc = could not find container \"0e3c4adc5c97f22adf03fc3eac0e6dc7532feee94523cf9844720b38a0a155eb\": container with ID starting with 0e3c4adc5c97f22adf03fc3eac0e6dc7532feee94523cf9844720b38a0a155eb not found: ID does not exist" Sep 11 15:57:34.143482 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:34.143438 2566 scope.go:117] "RemoveContainer" containerID="fb6a81fde4b2618df228016db131e5c0b0a13ff947344cebc687e5385dd47f76" Sep 11 15:57:34.143657 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:57:34.143641 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb6a81fde4b2618df228016db131e5c0b0a13ff947344cebc687e5385dd47f76\": container with ID starting with fb6a81fde4b2618df228016db131e5c0b0a13ff947344cebc687e5385dd47f76 not found: ID does not exist" containerID="fb6a81fde4b2618df228016db131e5c0b0a13ff947344cebc687e5385dd47f76" Sep 11 15:57:34.143697 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:34.143658 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb6a81fde4b2618df228016db131e5c0b0a13ff947344cebc687e5385dd47f76"} err="failed to get container status \"fb6a81fde4b2618df228016db131e5c0b0a13ff947344cebc687e5385dd47f76\": rpc error: code = NotFound desc = could not find container \"fb6a81fde4b2618df228016db131e5c0b0a13ff947344cebc687e5385dd47f76\": container with ID starting with fb6a81fde4b2618df228016db131e5c0b0a13ff947344cebc687e5385dd47f76 not found: ID does not exist" Sep 11 15:57:34.143697 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:34.143669 2566 scope.go:117] "RemoveContainer" containerID="ccaf640ff68701612e58cd96151db2f80ed789801f744d5ea514d349bb1d9b52" Sep 11 15:57:34.143917 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:57:34.143903 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccaf640ff68701612e58cd96151db2f80ed789801f744d5ea514d349bb1d9b52\": container with ID starting with ccaf640ff68701612e58cd96151db2f80ed789801f744d5ea514d349bb1d9b52 not found: ID does not exist" containerID="ccaf640ff68701612e58cd96151db2f80ed789801f744d5ea514d349bb1d9b52" Sep 11 15:57:34.143980 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:34.143918 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccaf640ff68701612e58cd96151db2f80ed789801f744d5ea514d349bb1d9b52"} err="failed to get container status \"ccaf640ff68701612e58cd96151db2f80ed789801f744d5ea514d349bb1d9b52\": rpc error: code = NotFound desc = could not find container \"ccaf640ff68701612e58cd96151db2f80ed789801f744d5ea514d349bb1d9b52\": container with ID starting with ccaf640ff68701612e58cd96151db2f80ed789801f744d5ea514d349bb1d9b52 not found: ID does not exist" Sep 11 15:57:34.143980 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:34.143930 2566 scope.go:117] "RemoveContainer" containerID="00528d1348836b690cfbaec38fa1113f608911bfbe67b2078644847c56587987" Sep 11 15:57:34.144221 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:34.144197 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00528d1348836b690cfbaec38fa1113f608911bfbe67b2078644847c56587987"} err="failed to get container status \"00528d1348836b690cfbaec38fa1113f608911bfbe67b2078644847c56587987\": rpc error: code = NotFound desc = could not find container \"00528d1348836b690cfbaec38fa1113f608911bfbe67b2078644847c56587987\": container with ID starting with 00528d1348836b690cfbaec38fa1113f608911bfbe67b2078644847c56587987 not found: ID does not exist" Sep 11 15:57:34.144263 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:34.144224 2566 scope.go:117] "RemoveContainer" containerID="52238d9341f63161169e082f4bc1d3b237f431909b8dfd7bd8ad22aebebad77c" Sep 11 15:57:34.144465 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:34.144448 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52238d9341f63161169e082f4bc1d3b237f431909b8dfd7bd8ad22aebebad77c"} err="failed to get container status \"52238d9341f63161169e082f4bc1d3b237f431909b8dfd7bd8ad22aebebad77c\": rpc error: code = NotFound desc = could not find container \"52238d9341f63161169e082f4bc1d3b237f431909b8dfd7bd8ad22aebebad77c\": container with ID starting with 52238d9341f63161169e082f4bc1d3b237f431909b8dfd7bd8ad22aebebad77c not found: ID does not exist" Sep 11 15:57:34.144508 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:34.144465 2566 scope.go:117] "RemoveContainer" containerID="73de8c544a4364477c0476052ff4f678b0f381aaa022a218cd41ecde0972824e" Sep 11 15:57:34.144668 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:34.144651 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73de8c544a4364477c0476052ff4f678b0f381aaa022a218cd41ecde0972824e"} err="failed to get container status \"73de8c544a4364477c0476052ff4f678b0f381aaa022a218cd41ecde0972824e\": rpc error: code = NotFound desc = could not find container \"73de8c544a4364477c0476052ff4f678b0f381aaa022a218cd41ecde0972824e\": container with ID starting with 73de8c544a4364477c0476052ff4f678b0f381aaa022a218cd41ecde0972824e not found: ID does not exist" Sep 11 15:57:34.144717 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:34.144668 2566 scope.go:117] "RemoveContainer" containerID="da377bde8e965d6c15a6ed75fcccb48f858bc1b1a64924817b33f2e22cf49c89" Sep 11 15:57:34.144871 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:34.144856 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da377bde8e965d6c15a6ed75fcccb48f858bc1b1a64924817b33f2e22cf49c89"} err="failed to get container status \"da377bde8e965d6c15a6ed75fcccb48f858bc1b1a64924817b33f2e22cf49c89\": rpc error: code = NotFound desc = could not find container \"da377bde8e965d6c15a6ed75fcccb48f858bc1b1a64924817b33f2e22cf49c89\": container with ID starting with da377bde8e965d6c15a6ed75fcccb48f858bc1b1a64924817b33f2e22cf49c89 not found: ID does not exist" Sep 11 15:57:34.144912 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:34.144871 2566 scope.go:117] "RemoveContainer" containerID="0e3c4adc5c97f22adf03fc3eac0e6dc7532feee94523cf9844720b38a0a155eb" Sep 11 15:57:34.145097 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:34.145081 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e3c4adc5c97f22adf03fc3eac0e6dc7532feee94523cf9844720b38a0a155eb"} err="failed to get container status \"0e3c4adc5c97f22adf03fc3eac0e6dc7532feee94523cf9844720b38a0a155eb\": rpc error: code = NotFound desc = could not find container \"0e3c4adc5c97f22adf03fc3eac0e6dc7532feee94523cf9844720b38a0a155eb\": container with ID starting with 0e3c4adc5c97f22adf03fc3eac0e6dc7532feee94523cf9844720b38a0a155eb not found: ID does not exist" Sep 11 15:57:34.145097 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:34.145096 2566 scope.go:117] "RemoveContainer" containerID="fb6a81fde4b2618df228016db131e5c0b0a13ff947344cebc687e5385dd47f76" Sep 11 15:57:34.145337 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:34.145319 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb6a81fde4b2618df228016db131e5c0b0a13ff947344cebc687e5385dd47f76"} err="failed to get container status \"fb6a81fde4b2618df228016db131e5c0b0a13ff947344cebc687e5385dd47f76\": rpc error: code = NotFound desc = could not find container \"fb6a81fde4b2618df228016db131e5c0b0a13ff947344cebc687e5385dd47f76\": container with ID starting with fb6a81fde4b2618df228016db131e5c0b0a13ff947344cebc687e5385dd47f76 not found: ID does not exist" Sep 11 15:57:34.145384 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:34.145338 2566 scope.go:117] "RemoveContainer" containerID="ccaf640ff68701612e58cd96151db2f80ed789801f744d5ea514d349bb1d9b52" Sep 11 15:57:34.145623 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:34.145607 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccaf640ff68701612e58cd96151db2f80ed789801f744d5ea514d349bb1d9b52"} err="failed to get container status \"ccaf640ff68701612e58cd96151db2f80ed789801f744d5ea514d349bb1d9b52\": rpc error: code = NotFound desc = could not find container \"ccaf640ff68701612e58cd96151db2f80ed789801f744d5ea514d349bb1d9b52\": container with ID starting with ccaf640ff68701612e58cd96151db2f80ed789801f744d5ea514d349bb1d9b52 not found: ID does not exist" Sep 11 15:57:34.148814 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:34.148795 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 11 15:57:34.149132 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:34.149118 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e" containerName="init-config-reloader" Sep 11 15:57:34.149179 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:34.149133 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e" containerName="init-config-reloader" Sep 11 15:57:34.149179 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:34.149145 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4800e361-38ce-40a5-8bf7-dfb46395f3cb" containerName="console" Sep 11 15:57:34.149179 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:34.149151 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="4800e361-38ce-40a5-8bf7-dfb46395f3cb" containerName="console" Sep 11 15:57:34.149179 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:34.149163 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e" containerName="kube-rbac-proxy-web" Sep 11 15:57:34.149179 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:34.149169 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e" containerName="kube-rbac-proxy-web" Sep 11 15:57:34.149179 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:34.149176 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e" containerName="prom-label-proxy" Sep 11 15:57:34.149179 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:34.149181 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e" containerName="prom-label-proxy" Sep 11 15:57:34.149387 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:34.149187 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e" containerName="kube-rbac-proxy-metric" Sep 11 15:57:34.149387 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:34.149192 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e" containerName="kube-rbac-proxy-metric" Sep 11 15:57:34.149387 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:34.149200 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e" containerName="alertmanager" Sep 11 15:57:34.149387 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:34.149205 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e" containerName="alertmanager" Sep 11 15:57:34.149387 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:34.149210 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e" containerName="config-reloader" Sep 11 15:57:34.149387 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:34.149215 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e" containerName="config-reloader" Sep 11 15:57:34.149387 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:34.149222 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e" containerName="kube-rbac-proxy" Sep 11 15:57:34.149387 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:34.149226 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e" containerName="kube-rbac-proxy" Sep 11 15:57:34.149387 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:34.149240 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae9022dc-da99-4acf-af85-3cf9858737b4" containerName="registry" Sep 11 15:57:34.149387 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:34.149248 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae9022dc-da99-4acf-af85-3cf9858737b4" containerName="registry" Sep 11 15:57:34.149387 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:34.149313 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e" containerName="kube-rbac-proxy" Sep 11 15:57:34.149387 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:34.149322 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="4800e361-38ce-40a5-8bf7-dfb46395f3cb" containerName="console" Sep 11 15:57:34.149387 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:34.149327 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e" containerName="prom-label-proxy" Sep 11 15:57:34.149387 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:34.149333 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e" containerName="kube-rbac-proxy-metric" Sep 11 15:57:34.149387 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:34.149338 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e" containerName="kube-rbac-proxy-web" Sep 11 15:57:34.149387 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:34.149343 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae9022dc-da99-4acf-af85-3cf9858737b4" containerName="registry" Sep 11 15:57:34.149387 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:34.149348 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e" containerName="config-reloader" Sep 11 15:57:34.149387 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:34.149353 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e" containerName="alertmanager" Sep 11 15:57:34.152669 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:34.152654 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 11 15:57:34.155850 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:34.155828 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Sep 11 15:57:34.155924 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:34.155850 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Sep 11 15:57:34.155990 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:34.155944 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Sep 11 15:57:34.156297 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:34.156278 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Sep 11 15:57:34.156378 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:34.156315 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Sep 11 15:57:34.156847 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:34.156828 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Sep 11 15:57:34.157451 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:34.157426 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-s2gtr\"" Sep 11 15:57:34.157525 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:34.157426 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Sep 11 15:57:34.158368 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:34.158346 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Sep 11 15:57:34.162397 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:34.162380 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Sep 11 15:57:34.172473 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:34.172446 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 11 15:57:34.276268 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:34.276230 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvczb\" (UniqueName: \"kubernetes.io/projected/3722b1cf-6259-4249-91ca-50ef6fac0a5b-kube-api-access-lvczb\") pod \"alertmanager-main-0\" (UID: \"3722b1cf-6259-4249-91ca-50ef6fac0a5b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 15:57:34.276268 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:34.276267 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3722b1cf-6259-4249-91ca-50ef6fac0a5b-web-config\") pod \"alertmanager-main-0\" (UID: \"3722b1cf-6259-4249-91ca-50ef6fac0a5b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 15:57:34.276458 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:34.276296 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3722b1cf-6259-4249-91ca-50ef6fac0a5b-config-volume\") pod \"alertmanager-main-0\" (UID: \"3722b1cf-6259-4249-91ca-50ef6fac0a5b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 15:57:34.276458 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:34.276328 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3722b1cf-6259-4249-91ca-50ef6fac0a5b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"3722b1cf-6259-4249-91ca-50ef6fac0a5b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 15:57:34.276458 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:34.276363 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3722b1cf-6259-4249-91ca-50ef6fac0a5b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"3722b1cf-6259-4249-91ca-50ef6fac0a5b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 15:57:34.276458 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:34.276434 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3722b1cf-6259-4249-91ca-50ef6fac0a5b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"3722b1cf-6259-4249-91ca-50ef6fac0a5b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 15:57:34.276588 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:34.276472 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3722b1cf-6259-4249-91ca-50ef6fac0a5b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"3722b1cf-6259-4249-91ca-50ef6fac0a5b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 15:57:34.276588 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:34.276486 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3722b1cf-6259-4249-91ca-50ef6fac0a5b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"3722b1cf-6259-4249-91ca-50ef6fac0a5b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 15:57:34.276588 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:34.276518 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3722b1cf-6259-4249-91ca-50ef6fac0a5b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"3722b1cf-6259-4249-91ca-50ef6fac0a5b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 15:57:34.276588 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:34.276565 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3722b1cf-6259-4249-91ca-50ef6fac0a5b-config-out\") pod \"alertmanager-main-0\" (UID: \"3722b1cf-6259-4249-91ca-50ef6fac0a5b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 15:57:34.276588 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:34.276584 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3722b1cf-6259-4249-91ca-50ef6fac0a5b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"3722b1cf-6259-4249-91ca-50ef6fac0a5b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 15:57:34.276783 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:34.276627 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3722b1cf-6259-4249-91ca-50ef6fac0a5b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"3722b1cf-6259-4249-91ca-50ef6fac0a5b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 15:57:34.276783 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:34.276673 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3722b1cf-6259-4249-91ca-50ef6fac0a5b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"3722b1cf-6259-4249-91ca-50ef6fac0a5b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 15:57:34.377423 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:34.377340 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3722b1cf-6259-4249-91ca-50ef6fac0a5b-config-volume\") pod \"alertmanager-main-0\" (UID: \"3722b1cf-6259-4249-91ca-50ef6fac0a5b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 15:57:34.377423 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:34.377377 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3722b1cf-6259-4249-91ca-50ef6fac0a5b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"3722b1cf-6259-4249-91ca-50ef6fac0a5b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 15:57:34.377423 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:34.377393 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3722b1cf-6259-4249-91ca-50ef6fac0a5b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"3722b1cf-6259-4249-91ca-50ef6fac0a5b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 15:57:34.377423 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:34.377409 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3722b1cf-6259-4249-91ca-50ef6fac0a5b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"3722b1cf-6259-4249-91ca-50ef6fac0a5b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 15:57:34.378036 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:34.377429 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3722b1cf-6259-4249-91ca-50ef6fac0a5b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"3722b1cf-6259-4249-91ca-50ef6fac0a5b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 15:57:34.378036 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:34.377453 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3722b1cf-6259-4249-91ca-50ef6fac0a5b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"3722b1cf-6259-4249-91ca-50ef6fac0a5b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 15:57:34.378036 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:34.377495 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3722b1cf-6259-4249-91ca-50ef6fac0a5b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"3722b1cf-6259-4249-91ca-50ef6fac0a5b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 15:57:34.378036 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:34.377532 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3722b1cf-6259-4249-91ca-50ef6fac0a5b-config-out\") pod \"alertmanager-main-0\" (UID: \"3722b1cf-6259-4249-91ca-50ef6fac0a5b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 15:57:34.378036 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:34.377556 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3722b1cf-6259-4249-91ca-50ef6fac0a5b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"3722b1cf-6259-4249-91ca-50ef6fac0a5b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 15:57:34.378036 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:34.377580 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3722b1cf-6259-4249-91ca-50ef6fac0a5b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"3722b1cf-6259-4249-91ca-50ef6fac0a5b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 15:57:34.378036 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:34.377614 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3722b1cf-6259-4249-91ca-50ef6fac0a5b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"3722b1cf-6259-4249-91ca-50ef6fac0a5b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 15:57:34.378036 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:34.377646 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvczb\" (UniqueName: \"kubernetes.io/projected/3722b1cf-6259-4249-91ca-50ef6fac0a5b-kube-api-access-lvczb\") pod \"alertmanager-main-0\" (UID: \"3722b1cf-6259-4249-91ca-50ef6fac0a5b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 15:57:34.378036 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:34.377677 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3722b1cf-6259-4249-91ca-50ef6fac0a5b-web-config\") pod \"alertmanager-main-0\" (UID: \"3722b1cf-6259-4249-91ca-50ef6fac0a5b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 15:57:34.378489 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:34.378193 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3722b1cf-6259-4249-91ca-50ef6fac0a5b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"3722b1cf-6259-4249-91ca-50ef6fac0a5b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 15:57:34.378489 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:34.378425 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3722b1cf-6259-4249-91ca-50ef6fac0a5b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"3722b1cf-6259-4249-91ca-50ef6fac0a5b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 15:57:34.378702 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:34.378675 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3722b1cf-6259-4249-91ca-50ef6fac0a5b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"3722b1cf-6259-4249-91ca-50ef6fac0a5b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 15:57:34.380629 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:34.380597 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3722b1cf-6259-4249-91ca-50ef6fac0a5b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"3722b1cf-6259-4249-91ca-50ef6fac0a5b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 15:57:34.380753 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:34.380670 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3722b1cf-6259-4249-91ca-50ef6fac0a5b-config-out\") pod \"alertmanager-main-0\" (UID: \"3722b1cf-6259-4249-91ca-50ef6fac0a5b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 15:57:34.380816 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:34.380749 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3722b1cf-6259-4249-91ca-50ef6fac0a5b-web-config\") pod \"alertmanager-main-0\" (UID: \"3722b1cf-6259-4249-91ca-50ef6fac0a5b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 15:57:34.380878 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:34.380818 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3722b1cf-6259-4249-91ca-50ef6fac0a5b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"3722b1cf-6259-4249-91ca-50ef6fac0a5b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 15:57:34.381098 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:34.381066 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3722b1cf-6259-4249-91ca-50ef6fac0a5b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"3722b1cf-6259-4249-91ca-50ef6fac0a5b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 15:57:34.381278 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:34.381256 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3722b1cf-6259-4249-91ca-50ef6fac0a5b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"3722b1cf-6259-4249-91ca-50ef6fac0a5b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 15:57:34.381339 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:34.381298 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3722b1cf-6259-4249-91ca-50ef6fac0a5b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"3722b1cf-6259-4249-91ca-50ef6fac0a5b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 15:57:34.381382 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:34.381354 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3722b1cf-6259-4249-91ca-50ef6fac0a5b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"3722b1cf-6259-4249-91ca-50ef6fac0a5b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 15:57:34.382681 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:34.382659 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3722b1cf-6259-4249-91ca-50ef6fac0a5b-config-volume\") pod \"alertmanager-main-0\" (UID: \"3722b1cf-6259-4249-91ca-50ef6fac0a5b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 15:57:34.390126 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:34.390101 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvczb\" (UniqueName: \"kubernetes.io/projected/3722b1cf-6259-4249-91ca-50ef6fac0a5b-kube-api-access-lvczb\") pod \"alertmanager-main-0\" (UID: \"3722b1cf-6259-4249-91ca-50ef6fac0a5b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 15:57:34.462166 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:34.462121 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 11 15:57:34.750398 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:34.750367 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 11 15:57:34.752675 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:57:34.752645 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3722b1cf_6259_4249_91ca_50ef6fac0a5b.slice/crio-8007c71abb3549cfe292159c3706a3861cb8343ce0ac4ba3c40f9c559be00a31 WatchSource:0}: Error finding container 8007c71abb3549cfe292159c3706a3861cb8343ce0ac4ba3c40f9c559be00a31: Status 404 returned error can't find the container with id 8007c71abb3549cfe292159c3706a3861cb8343ce0ac4ba3c40f9c559be00a31 Sep 11 15:57:35.091634 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:35.091602 2566 generic.go:356] "Generic (PLEG): container finished" podID="3722b1cf-6259-4249-91ca-50ef6fac0a5b" containerID="df4fe4253a5eb8f0375d8aae761ef976180fdb3bcb2ea9eb4baafa408f6fdbb0" exitCode=0 Sep 11 15:57:35.091841 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:35.091696 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3722b1cf-6259-4249-91ca-50ef6fac0a5b","Type":"ContainerDied","Data":"df4fe4253a5eb8f0375d8aae761ef976180fdb3bcb2ea9eb4baafa408f6fdbb0"} Sep 11 15:57:35.091841 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:35.091821 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3722b1cf-6259-4249-91ca-50ef6fac0a5b","Type":"ContainerStarted","Data":"8007c71abb3549cfe292159c3706a3861cb8343ce0ac4ba3c40f9c559be00a31"} Sep 11 15:57:35.239867 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:35.239836 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e" path="/var/lib/kubelet/pods/4b0b3e5a-ac62-4fa8-b9f1-9b57e00d528e/volumes" Sep 11 15:57:36.089410 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:36.089377 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-77c6977cfd-hxwhg"] Sep 11 15:57:36.091764 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:36.091743 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-77c6977cfd-hxwhg" Sep 11 15:57:36.094284 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:36.094240 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Sep 11 15:57:36.094385 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:36.094298 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-vr85z\"" Sep 11 15:57:36.094571 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:36.094508 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Sep 11 15:57:36.094571 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:36.094527 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Sep 11 15:57:36.094751 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:36.094632 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Sep 11 15:57:36.094995 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:36.094962 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Sep 11 15:57:36.099126 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:36.099095 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Sep 11 15:57:36.103550 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:36.103487 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3722b1cf-6259-4249-91ca-50ef6fac0a5b","Type":"ContainerStarted","Data":"fcb9b348d20539e652b41fb3805887d3b2f01169a983c06aa237f606cf104bf7"} Sep 11 15:57:36.103672 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:36.103556 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3722b1cf-6259-4249-91ca-50ef6fac0a5b","Type":"ContainerStarted","Data":"252323758de4d87af499f58c0611572b67623b4b0621cd56e9391b75c5feb3a7"} Sep 11 15:57:36.103672 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:36.103600 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3722b1cf-6259-4249-91ca-50ef6fac0a5b","Type":"ContainerStarted","Data":"a1190b2700035f4209bb87dd0bbc3bb8ce2dbcec4d9a19354551c51d575b1c09"} Sep 11 15:57:36.103672 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:36.103652 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3722b1cf-6259-4249-91ca-50ef6fac0a5b","Type":"ContainerStarted","Data":"7b0af4782cd7d05919f7b7abb107867d4cf4455f538fcf1c01fbcac79d30bbd7"} Sep 11 15:57:36.103840 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:36.103687 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3722b1cf-6259-4249-91ca-50ef6fac0a5b","Type":"ContainerStarted","Data":"9dca80ad292fe57a8f6bd62bcb10f7c8ad2d90420d4769ffa104ac11b4801bf5"} Sep 11 15:57:36.103840 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:36.103728 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3722b1cf-6259-4249-91ca-50ef6fac0a5b","Type":"ContainerStarted","Data":"167c12ab8eb4537805aca2584e9e22f3f1288760594a59eb724b2b2759f9cfbb"} Sep 11 15:57:36.105756 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:36.105731 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-77c6977cfd-hxwhg"] Sep 11 15:57:36.165772 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:36.165716 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.147387533 podStartE2EDuration="2.165699587s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.312054ms" isStatefulPod=false podCreationTimestamp="2026-09-11 15:57:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:57:36.164554325 +0000 UTC m=+163.379388229" watchObservedRunningTime="2026-09-11 15:57:36.165699587 +0000 UTC m=+163.380533492" Sep 11 15:57:36.192820 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:36.192784 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de3f52af-8138-47b2-9096-74d6d4e421a2-telemeter-trusted-ca-bundle\") pod \"telemeter-client-77c6977cfd-hxwhg\" (UID: \"de3f52af-8138-47b2-9096-74d6d4e421a2\") " pod="openshift-monitoring/telemeter-client-77c6977cfd-hxwhg" Sep 11 15:57:36.192820 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:36.192822 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/de3f52af-8138-47b2-9096-74d6d4e421a2-metrics-client-ca\") pod \"telemeter-client-77c6977cfd-hxwhg\" (UID: \"de3f52af-8138-47b2-9096-74d6d4e421a2\") " pod="openshift-monitoring/telemeter-client-77c6977cfd-hxwhg" Sep 11 15:57:36.193158 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:36.193124 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de3f52af-8138-47b2-9096-74d6d4e421a2-serving-certs-ca-bundle\") pod \"telemeter-client-77c6977cfd-hxwhg\" (UID: \"de3f52af-8138-47b2-9096-74d6d4e421a2\") " pod="openshift-monitoring/telemeter-client-77c6977cfd-hxwhg" Sep 11 15:57:36.193255 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:36.193168 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/de3f52af-8138-47b2-9096-74d6d4e421a2-secret-telemeter-client\") pod \"telemeter-client-77c6977cfd-hxwhg\" (UID: \"de3f52af-8138-47b2-9096-74d6d4e421a2\") " pod="openshift-monitoring/telemeter-client-77c6977cfd-hxwhg" Sep 11 15:57:36.193255 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:36.193207 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/de3f52af-8138-47b2-9096-74d6d4e421a2-telemeter-client-tls\") pod \"telemeter-client-77c6977cfd-hxwhg\" (UID: \"de3f52af-8138-47b2-9096-74d6d4e421a2\") " pod="openshift-monitoring/telemeter-client-77c6977cfd-hxwhg" Sep 11 15:57:36.193347 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:36.193322 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/de3f52af-8138-47b2-9096-74d6d4e421a2-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-77c6977cfd-hxwhg\" (UID: \"de3f52af-8138-47b2-9096-74d6d4e421a2\") " pod="openshift-monitoring/telemeter-client-77c6977cfd-hxwhg" Sep 11 15:57:36.193409 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:36.193395 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/de3f52af-8138-47b2-9096-74d6d4e421a2-federate-client-tls\") pod \"telemeter-client-77c6977cfd-hxwhg\" (UID: \"de3f52af-8138-47b2-9096-74d6d4e421a2\") " pod="openshift-monitoring/telemeter-client-77c6977cfd-hxwhg" Sep 11 15:57:36.193448 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:36.193428 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h54cb\" (UniqueName: \"kubernetes.io/projected/de3f52af-8138-47b2-9096-74d6d4e421a2-kube-api-access-h54cb\") pod \"telemeter-client-77c6977cfd-hxwhg\" (UID: \"de3f52af-8138-47b2-9096-74d6d4e421a2\") " pod="openshift-monitoring/telemeter-client-77c6977cfd-hxwhg" Sep 11 15:57:36.294056 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:36.294019 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/de3f52af-8138-47b2-9096-74d6d4e421a2-metrics-client-ca\") pod \"telemeter-client-77c6977cfd-hxwhg\" (UID: \"de3f52af-8138-47b2-9096-74d6d4e421a2\") " pod="openshift-monitoring/telemeter-client-77c6977cfd-hxwhg" Sep 11 15:57:36.294331 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:36.294082 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de3f52af-8138-47b2-9096-74d6d4e421a2-serving-certs-ca-bundle\") pod \"telemeter-client-77c6977cfd-hxwhg\" (UID: \"de3f52af-8138-47b2-9096-74d6d4e421a2\") " pod="openshift-monitoring/telemeter-client-77c6977cfd-hxwhg" Sep 11 15:57:36.294331 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:36.294099 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/de3f52af-8138-47b2-9096-74d6d4e421a2-secret-telemeter-client\") pod \"telemeter-client-77c6977cfd-hxwhg\" (UID: \"de3f52af-8138-47b2-9096-74d6d4e421a2\") " pod="openshift-monitoring/telemeter-client-77c6977cfd-hxwhg" Sep 11 15:57:36.294331 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:36.294114 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/de3f52af-8138-47b2-9096-74d6d4e421a2-telemeter-client-tls\") pod \"telemeter-client-77c6977cfd-hxwhg\" (UID: \"de3f52af-8138-47b2-9096-74d6d4e421a2\") " pod="openshift-monitoring/telemeter-client-77c6977cfd-hxwhg" Sep 11 15:57:36.294331 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:36.294133 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/de3f52af-8138-47b2-9096-74d6d4e421a2-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-77c6977cfd-hxwhg\" (UID: \"de3f52af-8138-47b2-9096-74d6d4e421a2\") " pod="openshift-monitoring/telemeter-client-77c6977cfd-hxwhg" Sep 11 15:57:36.294331 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:36.294171 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/de3f52af-8138-47b2-9096-74d6d4e421a2-federate-client-tls\") pod \"telemeter-client-77c6977cfd-hxwhg\" (UID: \"de3f52af-8138-47b2-9096-74d6d4e421a2\") " pod="openshift-monitoring/telemeter-client-77c6977cfd-hxwhg" Sep 11 15:57:36.294331 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:36.294195 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h54cb\" (UniqueName: \"kubernetes.io/projected/de3f52af-8138-47b2-9096-74d6d4e421a2-kube-api-access-h54cb\") pod \"telemeter-client-77c6977cfd-hxwhg\" (UID: \"de3f52af-8138-47b2-9096-74d6d4e421a2\") " pod="openshift-monitoring/telemeter-client-77c6977cfd-hxwhg" Sep 11 15:57:36.294331 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:36.294224 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de3f52af-8138-47b2-9096-74d6d4e421a2-telemeter-trusted-ca-bundle\") pod \"telemeter-client-77c6977cfd-hxwhg\" (UID: \"de3f52af-8138-47b2-9096-74d6d4e421a2\") " pod="openshift-monitoring/telemeter-client-77c6977cfd-hxwhg" Sep 11 15:57:36.294877 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:36.294842 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de3f52af-8138-47b2-9096-74d6d4e421a2-serving-certs-ca-bundle\") pod \"telemeter-client-77c6977cfd-hxwhg\" (UID: \"de3f52af-8138-47b2-9096-74d6d4e421a2\") " pod="openshift-monitoring/telemeter-client-77c6977cfd-hxwhg" Sep 11 15:57:36.295015 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:36.294908 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/de3f52af-8138-47b2-9096-74d6d4e421a2-metrics-client-ca\") pod \"telemeter-client-77c6977cfd-hxwhg\" (UID: \"de3f52af-8138-47b2-9096-74d6d4e421a2\") " pod="openshift-monitoring/telemeter-client-77c6977cfd-hxwhg" Sep 11 15:57:36.295139 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:36.295116 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de3f52af-8138-47b2-9096-74d6d4e421a2-telemeter-trusted-ca-bundle\") pod \"telemeter-client-77c6977cfd-hxwhg\" (UID: \"de3f52af-8138-47b2-9096-74d6d4e421a2\") " pod="openshift-monitoring/telemeter-client-77c6977cfd-hxwhg" Sep 11 15:57:36.296874 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:36.296842 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/de3f52af-8138-47b2-9096-74d6d4e421a2-secret-telemeter-client\") pod \"telemeter-client-77c6977cfd-hxwhg\" (UID: \"de3f52af-8138-47b2-9096-74d6d4e421a2\") " pod="openshift-monitoring/telemeter-client-77c6977cfd-hxwhg" Sep 11 15:57:36.297001 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:36.296887 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/de3f52af-8138-47b2-9096-74d6d4e421a2-federate-client-tls\") pod \"telemeter-client-77c6977cfd-hxwhg\" (UID: \"de3f52af-8138-47b2-9096-74d6d4e421a2\") " pod="openshift-monitoring/telemeter-client-77c6977cfd-hxwhg" Sep 11 15:57:36.297086 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:36.297063 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/de3f52af-8138-47b2-9096-74d6d4e421a2-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-77c6977cfd-hxwhg\" (UID: \"de3f52af-8138-47b2-9096-74d6d4e421a2\") " pod="openshift-monitoring/telemeter-client-77c6977cfd-hxwhg" Sep 11 15:57:36.297205 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:36.297183 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/de3f52af-8138-47b2-9096-74d6d4e421a2-telemeter-client-tls\") pod \"telemeter-client-77c6977cfd-hxwhg\" (UID: \"de3f52af-8138-47b2-9096-74d6d4e421a2\") " pod="openshift-monitoring/telemeter-client-77c6977cfd-hxwhg" Sep 11 15:57:36.302450 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:36.302431 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h54cb\" (UniqueName: \"kubernetes.io/projected/de3f52af-8138-47b2-9096-74d6d4e421a2-kube-api-access-h54cb\") pod \"telemeter-client-77c6977cfd-hxwhg\" (UID: \"de3f52af-8138-47b2-9096-74d6d4e421a2\") " pod="openshift-monitoring/telemeter-client-77c6977cfd-hxwhg" Sep 11 15:57:36.403605 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:36.403525 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-77c6977cfd-hxwhg" Sep 11 15:57:36.679396 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:36.679310 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-77c6977cfd-hxwhg"] Sep 11 15:57:36.682723 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:57:36.682695 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde3f52af_8138_47b2_9096_74d6d4e421a2.slice/crio-32a635d31b14d30868f061cc276f3e13dd76e7b18a3dc5892b51e9e07b05dbfc WatchSource:0}: Error finding container 32a635d31b14d30868f061cc276f3e13dd76e7b18a3dc5892b51e9e07b05dbfc: Status 404 returned error can't find the container with id 32a635d31b14d30868f061cc276f3e13dd76e7b18a3dc5892b51e9e07b05dbfc Sep 11 15:57:37.110062 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:37.110022 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-77c6977cfd-hxwhg" event={"ID":"de3f52af-8138-47b2-9096-74d6d4e421a2","Type":"ContainerStarted","Data":"32a635d31b14d30868f061cc276f3e13dd76e7b18a3dc5892b51e9e07b05dbfc"} Sep 11 15:57:39.123390 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:39.123351 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-77c6977cfd-hxwhg" event={"ID":"de3f52af-8138-47b2-9096-74d6d4e421a2","Type":"ContainerStarted","Data":"f3ea2c6a154a7278541854fdb940d37a1c55686319bcda132f09e1342e2d9821"} Sep 11 15:57:39.123849 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:39.123435 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-77c6977cfd-hxwhg" event={"ID":"de3f52af-8138-47b2-9096-74d6d4e421a2","Type":"ContainerStarted","Data":"7ddaf3e9982f9dab3836f682e0c805f4f365aacafb6a0fc66f81547e118c7ce2"} Sep 11 15:57:39.123849 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:39.123457 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-77c6977cfd-hxwhg" event={"ID":"de3f52af-8138-47b2-9096-74d6d4e421a2","Type":"ContainerStarted","Data":"9cda11578b7d68c50b0ca2e7f9a7eaed3ec7e456aa77dd42db313e5c240b9e68"} Sep 11 15:57:39.151542 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:39.151477 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-77c6977cfd-hxwhg" podStartSLOduration=1.337824304 podStartE2EDuration="3.151458983s" totalImagesPullingTime="1.813634679s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:57:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:57:39.14981421 +0000 UTC m=+166.364648115" watchObservedRunningTime="2026-09-11 15:57:39.151458983 +0000 UTC m=+166.366292888" Sep 11 15:57:39.252502 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:39.252472 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5bfb5f587b-cnf5x"] Sep 11 15:57:39.254768 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:39.254747 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bfb5f587b-cnf5x" Sep 11 15:57:39.270668 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:39.270639 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5bfb5f587b-cnf5x"] Sep 11 15:57:39.320172 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:39.320134 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2ae0f8d9-4863-4a81-ae57-759db754a994-console-config\") pod \"console-5bfb5f587b-cnf5x\" (UID: \"2ae0f8d9-4863-4a81-ae57-759db754a994\") " pod="openshift-console/console-5bfb5f587b-cnf5x" Sep 11 15:57:39.320172 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:39.320169 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2ae0f8d9-4863-4a81-ae57-759db754a994-trusted-ca-bundle\") pod \"console-5bfb5f587b-cnf5x\" (UID: \"2ae0f8d9-4863-4a81-ae57-759db754a994\") " pod="openshift-console/console-5bfb5f587b-cnf5x" Sep 11 15:57:39.320382 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:39.320192 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2ae0f8d9-4863-4a81-ae57-759db754a994-oauth-serving-cert\") pod \"console-5bfb5f587b-cnf5x\" (UID: \"2ae0f8d9-4863-4a81-ae57-759db754a994\") " pod="openshift-console/console-5bfb5f587b-cnf5x" Sep 11 15:57:39.320382 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:39.320282 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2ae0f8d9-4863-4a81-ae57-759db754a994-console-serving-cert\") pod \"console-5bfb5f587b-cnf5x\" (UID: \"2ae0f8d9-4863-4a81-ae57-759db754a994\") " pod="openshift-console/console-5bfb5f587b-cnf5x" Sep 11 15:57:39.320382 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:39.320305 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2ae0f8d9-4863-4a81-ae57-759db754a994-service-ca\") pod \"console-5bfb5f587b-cnf5x\" (UID: \"2ae0f8d9-4863-4a81-ae57-759db754a994\") " pod="openshift-console/console-5bfb5f587b-cnf5x" Sep 11 15:57:39.320382 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:39.320323 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2ae0f8d9-4863-4a81-ae57-759db754a994-console-oauth-config\") pod \"console-5bfb5f587b-cnf5x\" (UID: \"2ae0f8d9-4863-4a81-ae57-759db754a994\") " pod="openshift-console/console-5bfb5f587b-cnf5x" Sep 11 15:57:39.320382 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:39.320339 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q668w\" (UniqueName: \"kubernetes.io/projected/2ae0f8d9-4863-4a81-ae57-759db754a994-kube-api-access-q668w\") pod \"console-5bfb5f587b-cnf5x\" (UID: \"2ae0f8d9-4863-4a81-ae57-759db754a994\") " pod="openshift-console/console-5bfb5f587b-cnf5x" Sep 11 15:57:39.421647 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:39.421549 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2ae0f8d9-4863-4a81-ae57-759db754a994-console-serving-cert\") pod \"console-5bfb5f587b-cnf5x\" (UID: \"2ae0f8d9-4863-4a81-ae57-759db754a994\") " pod="openshift-console/console-5bfb5f587b-cnf5x" Sep 11 15:57:39.421647 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:39.421605 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2ae0f8d9-4863-4a81-ae57-759db754a994-service-ca\") pod \"console-5bfb5f587b-cnf5x\" (UID: \"2ae0f8d9-4863-4a81-ae57-759db754a994\") " pod="openshift-console/console-5bfb5f587b-cnf5x" Sep 11 15:57:39.421862 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:39.421726 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2ae0f8d9-4863-4a81-ae57-759db754a994-console-oauth-config\") pod \"console-5bfb5f587b-cnf5x\" (UID: \"2ae0f8d9-4863-4a81-ae57-759db754a994\") " pod="openshift-console/console-5bfb5f587b-cnf5x" Sep 11 15:57:39.421862 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:39.421763 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q668w\" (UniqueName: \"kubernetes.io/projected/2ae0f8d9-4863-4a81-ae57-759db754a994-kube-api-access-q668w\") pod \"console-5bfb5f587b-cnf5x\" (UID: \"2ae0f8d9-4863-4a81-ae57-759db754a994\") " pod="openshift-console/console-5bfb5f587b-cnf5x" Sep 11 15:57:39.421862 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:39.421828 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2ae0f8d9-4863-4a81-ae57-759db754a994-console-config\") pod \"console-5bfb5f587b-cnf5x\" (UID: \"2ae0f8d9-4863-4a81-ae57-759db754a994\") " pod="openshift-console/console-5bfb5f587b-cnf5x" Sep 11 15:57:39.421862 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:39.421844 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2ae0f8d9-4863-4a81-ae57-759db754a994-trusted-ca-bundle\") pod \"console-5bfb5f587b-cnf5x\" (UID: \"2ae0f8d9-4863-4a81-ae57-759db754a994\") " pod="openshift-console/console-5bfb5f587b-cnf5x" Sep 11 15:57:39.422106 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:39.421865 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2ae0f8d9-4863-4a81-ae57-759db754a994-oauth-serving-cert\") pod \"console-5bfb5f587b-cnf5x\" (UID: \"2ae0f8d9-4863-4a81-ae57-759db754a994\") " pod="openshift-console/console-5bfb5f587b-cnf5x" Sep 11 15:57:39.422438 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:39.422410 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2ae0f8d9-4863-4a81-ae57-759db754a994-service-ca\") pod \"console-5bfb5f587b-cnf5x\" (UID: \"2ae0f8d9-4863-4a81-ae57-759db754a994\") " pod="openshift-console/console-5bfb5f587b-cnf5x" Sep 11 15:57:39.422575 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:39.422479 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2ae0f8d9-4863-4a81-ae57-759db754a994-console-config\") pod \"console-5bfb5f587b-cnf5x\" (UID: \"2ae0f8d9-4863-4a81-ae57-759db754a994\") " pod="openshift-console/console-5bfb5f587b-cnf5x" Sep 11 15:57:39.422648 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:39.422609 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2ae0f8d9-4863-4a81-ae57-759db754a994-oauth-serving-cert\") pod \"console-5bfb5f587b-cnf5x\" (UID: \"2ae0f8d9-4863-4a81-ae57-759db754a994\") " pod="openshift-console/console-5bfb5f587b-cnf5x" Sep 11 15:57:39.422999 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:39.422947 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2ae0f8d9-4863-4a81-ae57-759db754a994-trusted-ca-bundle\") pod \"console-5bfb5f587b-cnf5x\" (UID: \"2ae0f8d9-4863-4a81-ae57-759db754a994\") " pod="openshift-console/console-5bfb5f587b-cnf5x" Sep 11 15:57:39.424247 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:39.424226 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2ae0f8d9-4863-4a81-ae57-759db754a994-console-serving-cert\") pod \"console-5bfb5f587b-cnf5x\" (UID: \"2ae0f8d9-4863-4a81-ae57-759db754a994\") " pod="openshift-console/console-5bfb5f587b-cnf5x" Sep 11 15:57:39.424330 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:39.424265 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2ae0f8d9-4863-4a81-ae57-759db754a994-console-oauth-config\") pod \"console-5bfb5f587b-cnf5x\" (UID: \"2ae0f8d9-4863-4a81-ae57-759db754a994\") " pod="openshift-console/console-5bfb5f587b-cnf5x" Sep 11 15:57:39.431623 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:39.431603 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q668w\" (UniqueName: \"kubernetes.io/projected/2ae0f8d9-4863-4a81-ae57-759db754a994-kube-api-access-q668w\") pod \"console-5bfb5f587b-cnf5x\" (UID: \"2ae0f8d9-4863-4a81-ae57-759db754a994\") " pod="openshift-console/console-5bfb5f587b-cnf5x" Sep 11 15:57:39.564357 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:39.564322 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bfb5f587b-cnf5x" Sep 11 15:57:39.836783 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:39.836748 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5bfb5f587b-cnf5x"] Sep 11 15:57:39.839783 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:57:39.839749 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ae0f8d9_4863_4a81_ae57_759db754a994.slice/crio-33d7deab2ae526bff9e5d959a8a18481aaf188c53f5ff043e40104c6fb4aa42d WatchSource:0}: Error finding container 33d7deab2ae526bff9e5d959a8a18481aaf188c53f5ff043e40104c6fb4aa42d: Status 404 returned error can't find the container with id 33d7deab2ae526bff9e5d959a8a18481aaf188c53f5ff043e40104c6fb4aa42d Sep 11 15:57:40.129840 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:40.129738 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bfb5f587b-cnf5x" event={"ID":"2ae0f8d9-4863-4a81-ae57-759db754a994","Type":"ContainerStarted","Data":"ea45633fb37beab1eb26a5db19d3415c45bd0eb9ecbd42595b514a38db33387a"} Sep 11 15:57:40.129840 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:40.129793 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bfb5f587b-cnf5x" event={"ID":"2ae0f8d9-4863-4a81-ae57-759db754a994","Type":"ContainerStarted","Data":"33d7deab2ae526bff9e5d959a8a18481aaf188c53f5ff043e40104c6fb4aa42d"} Sep 11 15:57:40.147480 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:40.147410 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5bfb5f587b-cnf5x" podStartSLOduration=1.147392382 podStartE2EDuration="1.147392382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:57:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:57:40.146689696 +0000 UTC m=+167.361523638" watchObservedRunningTime="2026-09-11 15:57:40.147392382 +0000 UTC m=+167.362226288" Sep 11 15:57:49.565543 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:49.565452 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5bfb5f587b-cnf5x" Sep 11 15:57:49.565543 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:49.565530 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5bfb5f587b-cnf5x" Sep 11 15:57:49.570167 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:49.570137 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5bfb5f587b-cnf5x" Sep 11 15:57:50.178548 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:50.178521 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5bfb5f587b-cnf5x" Sep 11 15:57:50.228463 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:57:50.228427 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-85c44c5bc8-mfchx"] Sep 11 15:58:15.253491 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:58:15.253442 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-85c44c5bc8-mfchx" podUID="cbdb6554-ce1f-4bb1-a3bd-1d746da28bc4" containerName="console" containerID="cri-o://e98df785efcba8dd657c55a8d888c84fe084c2a5f835a60f1890a6ea470a151e" gracePeriod=15 Sep 11 15:58:15.496423 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:58:15.496396 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85c44c5bc8-mfchx" Sep 11 15:58:15.530238 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:58:15.530200 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pmfg7\" (UniqueName: \"kubernetes.io/projected/cbdb6554-ce1f-4bb1-a3bd-1d746da28bc4-kube-api-access-pmfg7\") pod \"cbdb6554-ce1f-4bb1-a3bd-1d746da28bc4\" (UID: \"cbdb6554-ce1f-4bb1-a3bd-1d746da28bc4\") " Sep 11 15:58:15.530238 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:58:15.530239 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cbdb6554-ce1f-4bb1-a3bd-1d746da28bc4-console-serving-cert\") pod \"cbdb6554-ce1f-4bb1-a3bd-1d746da28bc4\" (UID: \"cbdb6554-ce1f-4bb1-a3bd-1d746da28bc4\") " Sep 11 15:58:15.530502 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:58:15.530287 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cbdb6554-ce1f-4bb1-a3bd-1d746da28bc4-oauth-serving-cert\") pod \"cbdb6554-ce1f-4bb1-a3bd-1d746da28bc4\" (UID: \"cbdb6554-ce1f-4bb1-a3bd-1d746da28bc4\") " Sep 11 15:58:15.530502 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:58:15.530364 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cbdb6554-ce1f-4bb1-a3bd-1d746da28bc4-service-ca\") pod \"cbdb6554-ce1f-4bb1-a3bd-1d746da28bc4\" (UID: \"cbdb6554-ce1f-4bb1-a3bd-1d746da28bc4\") " Sep 11 15:58:15.530502 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:58:15.530415 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cbdb6554-ce1f-4bb1-a3bd-1d746da28bc4-trusted-ca-bundle\") pod \"cbdb6554-ce1f-4bb1-a3bd-1d746da28bc4\" (UID: \"cbdb6554-ce1f-4bb1-a3bd-1d746da28bc4\") " Sep 11 15:58:15.530502 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:58:15.530444 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cbdb6554-ce1f-4bb1-a3bd-1d746da28bc4-console-config\") pod \"cbdb6554-ce1f-4bb1-a3bd-1d746da28bc4\" (UID: \"cbdb6554-ce1f-4bb1-a3bd-1d746da28bc4\") " Sep 11 15:58:15.530502 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:58:15.530487 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cbdb6554-ce1f-4bb1-a3bd-1d746da28bc4-console-oauth-config\") pod \"cbdb6554-ce1f-4bb1-a3bd-1d746da28bc4\" (UID: \"cbdb6554-ce1f-4bb1-a3bd-1d746da28bc4\") " Sep 11 15:58:15.530894 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:58:15.530865 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbdb6554-ce1f-4bb1-a3bd-1d746da28bc4-service-ca" (OuterVolumeSpecName: "service-ca") pod "cbdb6554-ce1f-4bb1-a3bd-1d746da28bc4" (UID: "cbdb6554-ce1f-4bb1-a3bd-1d746da28bc4"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:58:15.531019 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:58:15.530894 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbdb6554-ce1f-4bb1-a3bd-1d746da28bc4-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "cbdb6554-ce1f-4bb1-a3bd-1d746da28bc4" (UID: "cbdb6554-ce1f-4bb1-a3bd-1d746da28bc4"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:58:15.531019 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:58:15.530939 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbdb6554-ce1f-4bb1-a3bd-1d746da28bc4-console-config" (OuterVolumeSpecName: "console-config") pod "cbdb6554-ce1f-4bb1-a3bd-1d746da28bc4" (UID: "cbdb6554-ce1f-4bb1-a3bd-1d746da28bc4"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:58:15.531019 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:58:15.530951 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbdb6554-ce1f-4bb1-a3bd-1d746da28bc4-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "cbdb6554-ce1f-4bb1-a3bd-1d746da28bc4" (UID: "cbdb6554-ce1f-4bb1-a3bd-1d746da28bc4"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:58:15.532596 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:58:15.532560 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbdb6554-ce1f-4bb1-a3bd-1d746da28bc4-kube-api-access-pmfg7" (OuterVolumeSpecName: "kube-api-access-pmfg7") pod "cbdb6554-ce1f-4bb1-a3bd-1d746da28bc4" (UID: "cbdb6554-ce1f-4bb1-a3bd-1d746da28bc4"). InnerVolumeSpecName "kube-api-access-pmfg7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:58:15.532738 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:58:15.532716 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbdb6554-ce1f-4bb1-a3bd-1d746da28bc4-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "cbdb6554-ce1f-4bb1-a3bd-1d746da28bc4" (UID: "cbdb6554-ce1f-4bb1-a3bd-1d746da28bc4"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 15:58:15.532810 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:58:15.532796 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbdb6554-ce1f-4bb1-a3bd-1d746da28bc4-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "cbdb6554-ce1f-4bb1-a3bd-1d746da28bc4" (UID: "cbdb6554-ce1f-4bb1-a3bd-1d746da28bc4"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 15:58:15.631348 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:58:15.631296 2566 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cbdb6554-ce1f-4bb1-a3bd-1d746da28bc4-service-ca\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 15:58:15.631348 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:58:15.631343 2566 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cbdb6554-ce1f-4bb1-a3bd-1d746da28bc4-trusted-ca-bundle\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 15:58:15.631348 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:58:15.631355 2566 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cbdb6554-ce1f-4bb1-a3bd-1d746da28bc4-console-config\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 15:58:15.631348 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:58:15.631365 2566 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cbdb6554-ce1f-4bb1-a3bd-1d746da28bc4-console-oauth-config\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 15:58:15.631617 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:58:15.631374 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmfg7\" (UniqueName: \"kubernetes.io/projected/cbdb6554-ce1f-4bb1-a3bd-1d746da28bc4-kube-api-access-pmfg7\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 15:58:15.631617 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:58:15.631383 2566 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cbdb6554-ce1f-4bb1-a3bd-1d746da28bc4-console-serving-cert\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 15:58:15.631617 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:58:15.631391 2566 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cbdb6554-ce1f-4bb1-a3bd-1d746da28bc4-oauth-serving-cert\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 15:58:16.293758 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:58:16.293723 2566 generic.go:356] "Generic (PLEG): container finished" podID="cbdb6554-ce1f-4bb1-a3bd-1d746da28bc4" containerID="e98df785efcba8dd657c55a8d888c84fe084c2a5f835a60f1890a6ea470a151e" exitCode=0 Sep 11 15:58:16.293758 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:58:16.293762 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85c44c5bc8-mfchx" Sep 11 15:58:16.294209 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:58:16.293763 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85c44c5bc8-mfchx" event={"ID":"cbdb6554-ce1f-4bb1-a3bd-1d746da28bc4","Type":"ContainerDied","Data":"e98df785efcba8dd657c55a8d888c84fe084c2a5f835a60f1890a6ea470a151e"} Sep 11 15:58:16.294209 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:58:16.293888 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85c44c5bc8-mfchx" event={"ID":"cbdb6554-ce1f-4bb1-a3bd-1d746da28bc4","Type":"ContainerDied","Data":"b14ddb4d59191910e02b5f2433acc9fd8e3186fadd6e93a141e9336c63e467df"} Sep 11 15:58:16.294209 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:58:16.293910 2566 scope.go:117] "RemoveContainer" containerID="e98df785efcba8dd657c55a8d888c84fe084c2a5f835a60f1890a6ea470a151e" Sep 11 15:58:16.302762 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:58:16.302740 2566 scope.go:117] "RemoveContainer" containerID="e98df785efcba8dd657c55a8d888c84fe084c2a5f835a60f1890a6ea470a151e" Sep 11 15:58:16.303090 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:58:16.303071 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e98df785efcba8dd657c55a8d888c84fe084c2a5f835a60f1890a6ea470a151e\": container with ID starting with e98df785efcba8dd657c55a8d888c84fe084c2a5f835a60f1890a6ea470a151e not found: ID does not exist" containerID="e98df785efcba8dd657c55a8d888c84fe084c2a5f835a60f1890a6ea470a151e" Sep 11 15:58:16.303147 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:58:16.303100 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e98df785efcba8dd657c55a8d888c84fe084c2a5f835a60f1890a6ea470a151e"} err="failed to get container status \"e98df785efcba8dd657c55a8d888c84fe084c2a5f835a60f1890a6ea470a151e\": rpc error: code = NotFound desc = could not find container \"e98df785efcba8dd657c55a8d888c84fe084c2a5f835a60f1890a6ea470a151e\": container with ID starting with e98df785efcba8dd657c55a8d888c84fe084c2a5f835a60f1890a6ea470a151e not found: ID does not exist" Sep 11 15:58:16.321224 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:58:16.321193 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-85c44c5bc8-mfchx"] Sep 11 15:58:16.326395 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:58:16.326372 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-85c44c5bc8-mfchx"] Sep 11 15:58:17.238554 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:58:17.238523 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cbdb6554-ce1f-4bb1-a3bd-1d746da28bc4" path="/var/lib/kubelet/pods/cbdb6554-ce1f-4bb1-a3bd-1d746da28bc4/volumes" Sep 11 15:58:49.521247 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:58:49.521210 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-c5ff7cbc8-wcdds"] Sep 11 15:58:49.521816 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:58:49.521680 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbdb6554-ce1f-4bb1-a3bd-1d746da28bc4" containerName="console" Sep 11 15:58:49.521816 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:58:49.521700 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbdb6554-ce1f-4bb1-a3bd-1d746da28bc4" containerName="console" Sep 11 15:58:49.521816 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:58:49.521779 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="cbdb6554-ce1f-4bb1-a3bd-1d746da28bc4" containerName="console" Sep 11 15:58:49.524903 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:58:49.524879 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c5ff7cbc8-wcdds" Sep 11 15:58:49.544548 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:58:49.544523 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-c5ff7cbc8-wcdds"] Sep 11 15:58:49.617014 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:58:49.616957 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/29a58f59-f953-4969-8c21-24c4ab762364-oauth-serving-cert\") pod \"console-c5ff7cbc8-wcdds\" (UID: \"29a58f59-f953-4969-8c21-24c4ab762364\") " pod="openshift-console/console-c5ff7cbc8-wcdds" Sep 11 15:58:49.617178 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:58:49.617026 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/29a58f59-f953-4969-8c21-24c4ab762364-console-serving-cert\") pod \"console-c5ff7cbc8-wcdds\" (UID: \"29a58f59-f953-4969-8c21-24c4ab762364\") " pod="openshift-console/console-c5ff7cbc8-wcdds" Sep 11 15:58:49.617178 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:58:49.617065 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/29a58f59-f953-4969-8c21-24c4ab762364-console-oauth-config\") pod \"console-c5ff7cbc8-wcdds\" (UID: \"29a58f59-f953-4969-8c21-24c4ab762364\") " pod="openshift-console/console-c5ff7cbc8-wcdds" Sep 11 15:58:49.617178 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:58:49.617096 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/29a58f59-f953-4969-8c21-24c4ab762364-console-config\") pod \"console-c5ff7cbc8-wcdds\" (UID: \"29a58f59-f953-4969-8c21-24c4ab762364\") " pod="openshift-console/console-c5ff7cbc8-wcdds" Sep 11 15:58:49.617178 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:58:49.617142 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/29a58f59-f953-4969-8c21-24c4ab762364-service-ca\") pod \"console-c5ff7cbc8-wcdds\" (UID: \"29a58f59-f953-4969-8c21-24c4ab762364\") " pod="openshift-console/console-c5ff7cbc8-wcdds" Sep 11 15:58:49.617178 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:58:49.617172 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdcjz\" (UniqueName: \"kubernetes.io/projected/29a58f59-f953-4969-8c21-24c4ab762364-kube-api-access-jdcjz\") pod \"console-c5ff7cbc8-wcdds\" (UID: \"29a58f59-f953-4969-8c21-24c4ab762364\") " pod="openshift-console/console-c5ff7cbc8-wcdds" Sep 11 15:58:49.617341 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:58:49.617193 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29a58f59-f953-4969-8c21-24c4ab762364-trusted-ca-bundle\") pod \"console-c5ff7cbc8-wcdds\" (UID: \"29a58f59-f953-4969-8c21-24c4ab762364\") " pod="openshift-console/console-c5ff7cbc8-wcdds" Sep 11 15:58:49.717878 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:58:49.717842 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdcjz\" (UniqueName: \"kubernetes.io/projected/29a58f59-f953-4969-8c21-24c4ab762364-kube-api-access-jdcjz\") pod \"console-c5ff7cbc8-wcdds\" (UID: \"29a58f59-f953-4969-8c21-24c4ab762364\") " pod="openshift-console/console-c5ff7cbc8-wcdds" Sep 11 15:58:49.717878 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:58:49.717882 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29a58f59-f953-4969-8c21-24c4ab762364-trusted-ca-bundle\") pod \"console-c5ff7cbc8-wcdds\" (UID: \"29a58f59-f953-4969-8c21-24c4ab762364\") " pod="openshift-console/console-c5ff7cbc8-wcdds" Sep 11 15:58:49.718124 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:58:49.717922 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/29a58f59-f953-4969-8c21-24c4ab762364-oauth-serving-cert\") pod \"console-c5ff7cbc8-wcdds\" (UID: \"29a58f59-f953-4969-8c21-24c4ab762364\") " pod="openshift-console/console-c5ff7cbc8-wcdds" Sep 11 15:58:49.718124 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:58:49.717950 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/29a58f59-f953-4969-8c21-24c4ab762364-console-serving-cert\") pod \"console-c5ff7cbc8-wcdds\" (UID: \"29a58f59-f953-4969-8c21-24c4ab762364\") " pod="openshift-console/console-c5ff7cbc8-wcdds" Sep 11 15:58:49.718124 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:58:49.718009 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/29a58f59-f953-4969-8c21-24c4ab762364-console-oauth-config\") pod \"console-c5ff7cbc8-wcdds\" (UID: \"29a58f59-f953-4969-8c21-24c4ab762364\") " pod="openshift-console/console-c5ff7cbc8-wcdds" Sep 11 15:58:49.718124 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:58:49.718052 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/29a58f59-f953-4969-8c21-24c4ab762364-console-config\") pod \"console-c5ff7cbc8-wcdds\" (UID: \"29a58f59-f953-4969-8c21-24c4ab762364\") " pod="openshift-console/console-c5ff7cbc8-wcdds" Sep 11 15:58:49.718124 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:58:49.718077 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/29a58f59-f953-4969-8c21-24c4ab762364-service-ca\") pod \"console-c5ff7cbc8-wcdds\" (UID: \"29a58f59-f953-4969-8c21-24c4ab762364\") " pod="openshift-console/console-c5ff7cbc8-wcdds" Sep 11 15:58:49.718833 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:58:49.718799 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/29a58f59-f953-4969-8c21-24c4ab762364-oauth-serving-cert\") pod \"console-c5ff7cbc8-wcdds\" (UID: \"29a58f59-f953-4969-8c21-24c4ab762364\") " pod="openshift-console/console-c5ff7cbc8-wcdds" Sep 11 15:58:49.718956 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:58:49.718830 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/29a58f59-f953-4969-8c21-24c4ab762364-console-config\") pod \"console-c5ff7cbc8-wcdds\" (UID: \"29a58f59-f953-4969-8c21-24c4ab762364\") " pod="openshift-console/console-c5ff7cbc8-wcdds" Sep 11 15:58:49.718956 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:58:49.718842 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/29a58f59-f953-4969-8c21-24c4ab762364-service-ca\") pod \"console-c5ff7cbc8-wcdds\" (UID: \"29a58f59-f953-4969-8c21-24c4ab762364\") " pod="openshift-console/console-c5ff7cbc8-wcdds" Sep 11 15:58:49.719376 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:58:49.719349 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29a58f59-f953-4969-8c21-24c4ab762364-trusted-ca-bundle\") pod \"console-c5ff7cbc8-wcdds\" (UID: \"29a58f59-f953-4969-8c21-24c4ab762364\") " pod="openshift-console/console-c5ff7cbc8-wcdds" Sep 11 15:58:49.720587 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:58:49.720555 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/29a58f59-f953-4969-8c21-24c4ab762364-console-serving-cert\") pod \"console-c5ff7cbc8-wcdds\" (UID: \"29a58f59-f953-4969-8c21-24c4ab762364\") " pod="openshift-console/console-c5ff7cbc8-wcdds" Sep 11 15:58:49.720699 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:58:49.720676 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/29a58f59-f953-4969-8c21-24c4ab762364-console-oauth-config\") pod \"console-c5ff7cbc8-wcdds\" (UID: \"29a58f59-f953-4969-8c21-24c4ab762364\") " pod="openshift-console/console-c5ff7cbc8-wcdds" Sep 11 15:58:49.726138 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:58:49.726119 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdcjz\" (UniqueName: \"kubernetes.io/projected/29a58f59-f953-4969-8c21-24c4ab762364-kube-api-access-jdcjz\") pod \"console-c5ff7cbc8-wcdds\" (UID: \"29a58f59-f953-4969-8c21-24c4ab762364\") " pod="openshift-console/console-c5ff7cbc8-wcdds" Sep 11 15:58:49.837155 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:58:49.837063 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c5ff7cbc8-wcdds" Sep 11 15:58:50.120774 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:58:50.120683 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-c5ff7cbc8-wcdds"] Sep 11 15:58:50.123878 ip-10-0-136-96 kubenswrapper[2566]: W0911 15:58:50.123851 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29a58f59_f953_4969_8c21_24c4ab762364.slice/crio-d4f4f38d42d86c8005b6b2d9a986b798933f340a6834dcb35777a2a7208ec8f7 WatchSource:0}: Error finding container d4f4f38d42d86c8005b6b2d9a986b798933f340a6834dcb35777a2a7208ec8f7: Status 404 returned error can't find the container with id d4f4f38d42d86c8005b6b2d9a986b798933f340a6834dcb35777a2a7208ec8f7 Sep 11 15:58:50.436298 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:58:50.436213 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c5ff7cbc8-wcdds" event={"ID":"29a58f59-f953-4969-8c21-24c4ab762364","Type":"ContainerStarted","Data":"2206202e0d916c0743b492ffe0022bba22a184e33d7178b16ffaa4aaac9432cc"} Sep 11 15:58:50.436298 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:58:50.436273 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c5ff7cbc8-wcdds" event={"ID":"29a58f59-f953-4969-8c21-24c4ab762364","Type":"ContainerStarted","Data":"d4f4f38d42d86c8005b6b2d9a986b798933f340a6834dcb35777a2a7208ec8f7"} Sep 11 15:58:50.474408 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:58:50.474355 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-c5ff7cbc8-wcdds" podStartSLOduration=1.474337955 podStartE2EDuration="1.474337955s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:58:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:58:50.472404219 +0000 UTC m=+237.687238124" watchObservedRunningTime="2026-09-11 15:58:50.474337955 +0000 UTC m=+237.689171860" Sep 11 15:58:59.838187 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:58:59.838130 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-c5ff7cbc8-wcdds" Sep 11 15:58:59.838635 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:58:59.838211 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-c5ff7cbc8-wcdds" Sep 11 15:58:59.843025 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:58:59.843001 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-c5ff7cbc8-wcdds" Sep 11 15:59:00.482731 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:59:00.482701 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-c5ff7cbc8-wcdds" Sep 11 15:59:00.546990 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:59:00.546935 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5bfb5f587b-cnf5x"] Sep 11 15:59:25.566933 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:59:25.566887 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5bfb5f587b-cnf5x" podUID="2ae0f8d9-4863-4a81-ae57-759db754a994" containerName="console" containerID="cri-o://ea45633fb37beab1eb26a5db19d3415c45bd0eb9ecbd42595b514a38db33387a" gracePeriod=15 Sep 11 15:59:25.806843 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:59:25.806816 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bfb5f587b-cnf5x" Sep 11 15:59:25.906482 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:59:25.906391 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2ae0f8d9-4863-4a81-ae57-759db754a994-service-ca\") pod \"2ae0f8d9-4863-4a81-ae57-759db754a994\" (UID: \"2ae0f8d9-4863-4a81-ae57-759db754a994\") " Sep 11 15:59:25.906482 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:59:25.906453 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2ae0f8d9-4863-4a81-ae57-759db754a994-console-config\") pod \"2ae0f8d9-4863-4a81-ae57-759db754a994\" (UID: \"2ae0f8d9-4863-4a81-ae57-759db754a994\") " Sep 11 15:59:25.906703 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:59:25.906502 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2ae0f8d9-4863-4a81-ae57-759db754a994-trusted-ca-bundle\") pod \"2ae0f8d9-4863-4a81-ae57-759db754a994\" (UID: \"2ae0f8d9-4863-4a81-ae57-759db754a994\") " Sep 11 15:59:25.906703 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:59:25.906527 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2ae0f8d9-4863-4a81-ae57-759db754a994-console-oauth-config\") pod \"2ae0f8d9-4863-4a81-ae57-759db754a994\" (UID: \"2ae0f8d9-4863-4a81-ae57-759db754a994\") " Sep 11 15:59:25.906703 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:59:25.906569 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2ae0f8d9-4863-4a81-ae57-759db754a994-oauth-serving-cert\") pod \"2ae0f8d9-4863-4a81-ae57-759db754a994\" (UID: \"2ae0f8d9-4863-4a81-ae57-759db754a994\") " Sep 11 15:59:25.906862 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:59:25.906714 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2ae0f8d9-4863-4a81-ae57-759db754a994-console-serving-cert\") pod \"2ae0f8d9-4863-4a81-ae57-759db754a994\" (UID: \"2ae0f8d9-4863-4a81-ae57-759db754a994\") " Sep 11 15:59:25.906862 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:59:25.906771 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q668w\" (UniqueName: \"kubernetes.io/projected/2ae0f8d9-4863-4a81-ae57-759db754a994-kube-api-access-q668w\") pod \"2ae0f8d9-4863-4a81-ae57-759db754a994\" (UID: \"2ae0f8d9-4863-4a81-ae57-759db754a994\") " Sep 11 15:59:25.907001 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:59:25.906868 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ae0f8d9-4863-4a81-ae57-759db754a994-console-config" (OuterVolumeSpecName: "console-config") pod "2ae0f8d9-4863-4a81-ae57-759db754a994" (UID: "2ae0f8d9-4863-4a81-ae57-759db754a994"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:59:25.907071 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:59:25.906996 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ae0f8d9-4863-4a81-ae57-759db754a994-service-ca" (OuterVolumeSpecName: "service-ca") pod "2ae0f8d9-4863-4a81-ae57-759db754a994" (UID: "2ae0f8d9-4863-4a81-ae57-759db754a994"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:59:25.907071 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:59:25.907018 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ae0f8d9-4863-4a81-ae57-759db754a994-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "2ae0f8d9-4863-4a81-ae57-759db754a994" (UID: "2ae0f8d9-4863-4a81-ae57-759db754a994"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:59:25.907071 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:59:25.907028 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ae0f8d9-4863-4a81-ae57-759db754a994-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "2ae0f8d9-4863-4a81-ae57-759db754a994" (UID: "2ae0f8d9-4863-4a81-ae57-759db754a994"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 15:59:25.907187 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:59:25.907107 2566 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2ae0f8d9-4863-4a81-ae57-759db754a994-oauth-serving-cert\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 15:59:25.907187 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:59:25.907129 2566 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2ae0f8d9-4863-4a81-ae57-759db754a994-service-ca\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 15:59:25.907187 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:59:25.907144 2566 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2ae0f8d9-4863-4a81-ae57-759db754a994-console-config\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 15:59:25.907187 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:59:25.907156 2566 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2ae0f8d9-4863-4a81-ae57-759db754a994-trusted-ca-bundle\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 15:59:25.908637 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:59:25.908609 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ae0f8d9-4863-4a81-ae57-759db754a994-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "2ae0f8d9-4863-4a81-ae57-759db754a994" (UID: "2ae0f8d9-4863-4a81-ae57-759db754a994"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 15:59:25.908840 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:59:25.908823 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ae0f8d9-4863-4a81-ae57-759db754a994-kube-api-access-q668w" (OuterVolumeSpecName: "kube-api-access-q668w") pod "2ae0f8d9-4863-4a81-ae57-759db754a994" (UID: "2ae0f8d9-4863-4a81-ae57-759db754a994"). InnerVolumeSpecName "kube-api-access-q668w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 15:59:25.908881 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:59:25.908855 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ae0f8d9-4863-4a81-ae57-759db754a994-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "2ae0f8d9-4863-4a81-ae57-759db754a994" (UID: "2ae0f8d9-4863-4a81-ae57-759db754a994"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 15:59:26.007641 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:59:26.007606 2566 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2ae0f8d9-4863-4a81-ae57-759db754a994-console-oauth-config\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 15:59:26.007641 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:59:26.007635 2566 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2ae0f8d9-4863-4a81-ae57-759db754a994-console-serving-cert\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 15:59:26.007641 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:59:26.007646 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q668w\" (UniqueName: \"kubernetes.io/projected/2ae0f8d9-4863-4a81-ae57-759db754a994-kube-api-access-q668w\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 15:59:26.583079 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:59:26.583043 2566 generic.go:356] "Generic (PLEG): container finished" podID="2ae0f8d9-4863-4a81-ae57-759db754a994" containerID="ea45633fb37beab1eb26a5db19d3415c45bd0eb9ecbd42595b514a38db33387a" exitCode=0 Sep 11 15:59:26.583456 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:59:26.583092 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bfb5f587b-cnf5x" Sep 11 15:59:26.583456 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:59:26.583129 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bfb5f587b-cnf5x" event={"ID":"2ae0f8d9-4863-4a81-ae57-759db754a994","Type":"ContainerDied","Data":"ea45633fb37beab1eb26a5db19d3415c45bd0eb9ecbd42595b514a38db33387a"} Sep 11 15:59:26.583456 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:59:26.583183 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bfb5f587b-cnf5x" event={"ID":"2ae0f8d9-4863-4a81-ae57-759db754a994","Type":"ContainerDied","Data":"33d7deab2ae526bff9e5d959a8a18481aaf188c53f5ff043e40104c6fb4aa42d"} Sep 11 15:59:26.583456 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:59:26.583210 2566 scope.go:117] "RemoveContainer" containerID="ea45633fb37beab1eb26a5db19d3415c45bd0eb9ecbd42595b514a38db33387a" Sep 11 15:59:26.591743 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:59:26.591726 2566 scope.go:117] "RemoveContainer" containerID="ea45633fb37beab1eb26a5db19d3415c45bd0eb9ecbd42595b514a38db33387a" Sep 11 15:59:26.592027 ip-10-0-136-96 kubenswrapper[2566]: E0911 15:59:26.592008 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea45633fb37beab1eb26a5db19d3415c45bd0eb9ecbd42595b514a38db33387a\": container with ID starting with ea45633fb37beab1eb26a5db19d3415c45bd0eb9ecbd42595b514a38db33387a not found: ID does not exist" containerID="ea45633fb37beab1eb26a5db19d3415c45bd0eb9ecbd42595b514a38db33387a" Sep 11 15:59:26.592096 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:59:26.592038 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea45633fb37beab1eb26a5db19d3415c45bd0eb9ecbd42595b514a38db33387a"} err="failed to get container status \"ea45633fb37beab1eb26a5db19d3415c45bd0eb9ecbd42595b514a38db33387a\": rpc error: code = NotFound desc = could not find container \"ea45633fb37beab1eb26a5db19d3415c45bd0eb9ecbd42595b514a38db33387a\": container with ID starting with ea45633fb37beab1eb26a5db19d3415c45bd0eb9ecbd42595b514a38db33387a not found: ID does not exist" Sep 11 15:59:26.614665 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:59:26.614637 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5bfb5f587b-cnf5x"] Sep 11 15:59:26.622891 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:59:26.622863 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5bfb5f587b-cnf5x"] Sep 11 15:59:27.237715 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:59:27.237682 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ae0f8d9-4863-4a81-ae57-759db754a994" path="/var/lib/kubelet/pods/2ae0f8d9-4863-4a81-ae57-759db754a994/volumes" Sep 11 15:59:53.190733 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:59:53.190707 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-xjvl5_7542ad77-c817-4370-91de-3b08db264753/console-operator/2.log" Sep 11 15:59:53.191202 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:59:53.190928 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-xjvl5_7542ad77-c817-4370-91de-3b08db264753/console-operator/2.log" Sep 11 15:59:53.198090 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:59:53.198067 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tgq5c_981d7d1e-f81f-48fc-8962-7831bc5c2b27/ovn-acl-logging/0.log" Sep 11 15:59:53.198378 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:59:53.198361 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tgq5c_981d7d1e-f81f-48fc-8962-7831bc5c2b27/ovn-acl-logging/0.log" Sep 11 15:59:53.201348 ip-10-0-136-96 kubenswrapper[2566]: I0911 15:59:53.201318 2566 kubelet.go:1648] "Image garbage collection succeeded" Sep 11 16:00:03.607729 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:03.607691 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jbf59"] Sep 11 16:00:03.610162 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:03.608044 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ae0f8d9-4863-4a81-ae57-759db754a994" containerName="console" Sep 11 16:00:03.610162 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:03.608056 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ae0f8d9-4863-4a81-ae57-759db754a994" containerName="console" Sep 11 16:00:03.610162 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:03.608123 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ae0f8d9-4863-4a81-ae57-759db754a994" containerName="console" Sep 11 16:00:03.611086 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:03.611069 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jbf59" Sep 11 16:00:03.613418 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:03.613393 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 11 16:00:03.613607 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:03.613499 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-z84gm\"" Sep 11 16:00:03.614124 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:03.614107 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 11 16:00:03.622080 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:03.622056 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jbf59"] Sep 11 16:00:03.626739 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:03.626712 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1606d845-d8ec-4ab3-a642-c97f8035207f-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jbf59\" (UID: \"1606d845-d8ec-4ab3-a642-c97f8035207f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jbf59" Sep 11 16:00:03.626877 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:03.626771 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1606d845-d8ec-4ab3-a642-c97f8035207f-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jbf59\" (UID: \"1606d845-d8ec-4ab3-a642-c97f8035207f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jbf59" Sep 11 16:00:03.626877 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:03.626813 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6ndb\" (UniqueName: \"kubernetes.io/projected/1606d845-d8ec-4ab3-a642-c97f8035207f-kube-api-access-f6ndb\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jbf59\" (UID: \"1606d845-d8ec-4ab3-a642-c97f8035207f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jbf59" Sep 11 16:00:03.727862 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:03.727825 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6ndb\" (UniqueName: \"kubernetes.io/projected/1606d845-d8ec-4ab3-a642-c97f8035207f-kube-api-access-f6ndb\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jbf59\" (UID: \"1606d845-d8ec-4ab3-a642-c97f8035207f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jbf59" Sep 11 16:00:03.728107 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:03.727881 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1606d845-d8ec-4ab3-a642-c97f8035207f-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jbf59\" (UID: \"1606d845-d8ec-4ab3-a642-c97f8035207f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jbf59" Sep 11 16:00:03.728107 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:03.727921 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1606d845-d8ec-4ab3-a642-c97f8035207f-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jbf59\" (UID: \"1606d845-d8ec-4ab3-a642-c97f8035207f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jbf59" Sep 11 16:00:03.728301 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:03.728282 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1606d845-d8ec-4ab3-a642-c97f8035207f-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jbf59\" (UID: \"1606d845-d8ec-4ab3-a642-c97f8035207f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jbf59" Sep 11 16:00:03.728369 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:03.728327 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1606d845-d8ec-4ab3-a642-c97f8035207f-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jbf59\" (UID: \"1606d845-d8ec-4ab3-a642-c97f8035207f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jbf59" Sep 11 16:00:03.737610 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:03.737582 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6ndb\" (UniqueName: \"kubernetes.io/projected/1606d845-d8ec-4ab3-a642-c97f8035207f-kube-api-access-f6ndb\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jbf59\" (UID: \"1606d845-d8ec-4ab3-a642-c97f8035207f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jbf59" Sep 11 16:00:03.919904 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:03.919812 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jbf59" Sep 11 16:00:04.203066 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:04.202963 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jbf59"] Sep 11 16:00:04.206812 ip-10-0-136-96 kubenswrapper[2566]: W0911 16:00:04.206786 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1606d845_d8ec_4ab3_a642_c97f8035207f.slice/crio-6e015151a5e45a18d1692f7d4d0bd6e03a433b626ed117af6622fb1bc4ea7dfa WatchSource:0}: Error finding container 6e015151a5e45a18d1692f7d4d0bd6e03a433b626ed117af6622fb1bc4ea7dfa: Status 404 returned error can't find the container with id 6e015151a5e45a18d1692f7d4d0bd6e03a433b626ed117af6622fb1bc4ea7dfa Sep 11 16:00:04.209346 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:04.209325 2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 16:00:04.747673 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:04.747631 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jbf59" event={"ID":"1606d845-d8ec-4ab3-a642-c97f8035207f","Type":"ContainerStarted","Data":"6e015151a5e45a18d1692f7d4d0bd6e03a433b626ed117af6622fb1bc4ea7dfa"} Sep 11 16:00:11.780814 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:11.780779 2566 generic.go:356] "Generic (PLEG): container finished" podID="1606d845-d8ec-4ab3-a642-c97f8035207f" containerID="e2d5b47c2080db90f87dbbadbff3b0628e9fc5f4c13918b6533184f1f26a8a7e" exitCode=0 Sep 11 16:00:11.781190 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:11.780861 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jbf59" event={"ID":"1606d845-d8ec-4ab3-a642-c97f8035207f","Type":"ContainerDied","Data":"e2d5b47c2080db90f87dbbadbff3b0628e9fc5f4c13918b6533184f1f26a8a7e"} Sep 11 16:00:14.058151 ip-10-0-136-96 kubenswrapper[2566]: E0911 16:00:14.058121 2566 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1606d845_d8ec_4ab3_a642_c97f8035207f.slice/crio-00ceaa74706700db58a23344f0623b5edc10a34eca05ad27e4e8f187f5e733b8.scope\": RecentStats: unable to find data in memory cache]" Sep 11 16:00:14.795320 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:14.795281 2566 generic.go:356] "Generic (PLEG): container finished" podID="1606d845-d8ec-4ab3-a642-c97f8035207f" containerID="00ceaa74706700db58a23344f0623b5edc10a34eca05ad27e4e8f187f5e733b8" exitCode=0 Sep 11 16:00:14.795489 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:14.795367 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jbf59" event={"ID":"1606d845-d8ec-4ab3-a642-c97f8035207f","Type":"ContainerDied","Data":"00ceaa74706700db58a23344f0623b5edc10a34eca05ad27e4e8f187f5e733b8"} Sep 11 16:00:22.832935 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:22.832897 2566 generic.go:356] "Generic (PLEG): container finished" podID="1606d845-d8ec-4ab3-a642-c97f8035207f" containerID="2e937444adb601bb215efd9085ff3eb8a55472430f03284c4806d4945be24832" exitCode=0 Sep 11 16:00:22.833390 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:22.832991 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jbf59" event={"ID":"1606d845-d8ec-4ab3-a642-c97f8035207f","Type":"ContainerDied","Data":"2e937444adb601bb215efd9085ff3eb8a55472430f03284c4806d4945be24832"} Sep 11 16:00:23.960119 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:23.960090 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jbf59" Sep 11 16:00:24.106424 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:24.106328 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1606d845-d8ec-4ab3-a642-c97f8035207f-util\") pod \"1606d845-d8ec-4ab3-a642-c97f8035207f\" (UID: \"1606d845-d8ec-4ab3-a642-c97f8035207f\") " Sep 11 16:00:24.106424 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:24.106400 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f6ndb\" (UniqueName: \"kubernetes.io/projected/1606d845-d8ec-4ab3-a642-c97f8035207f-kube-api-access-f6ndb\") pod \"1606d845-d8ec-4ab3-a642-c97f8035207f\" (UID: \"1606d845-d8ec-4ab3-a642-c97f8035207f\") " Sep 11 16:00:24.106691 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:24.106431 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1606d845-d8ec-4ab3-a642-c97f8035207f-bundle\") pod \"1606d845-d8ec-4ab3-a642-c97f8035207f\" (UID: \"1606d845-d8ec-4ab3-a642-c97f8035207f\") " Sep 11 16:00:24.107128 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:24.107101 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1606d845-d8ec-4ab3-a642-c97f8035207f-bundle" (OuterVolumeSpecName: "bundle") pod "1606d845-d8ec-4ab3-a642-c97f8035207f" (UID: "1606d845-d8ec-4ab3-a642-c97f8035207f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 16:00:24.108624 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:24.108581 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1606d845-d8ec-4ab3-a642-c97f8035207f-kube-api-access-f6ndb" (OuterVolumeSpecName: "kube-api-access-f6ndb") pod "1606d845-d8ec-4ab3-a642-c97f8035207f" (UID: "1606d845-d8ec-4ab3-a642-c97f8035207f"). InnerVolumeSpecName "kube-api-access-f6ndb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 16:00:24.111252 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:24.111232 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1606d845-d8ec-4ab3-a642-c97f8035207f-util" (OuterVolumeSpecName: "util") pod "1606d845-d8ec-4ab3-a642-c97f8035207f" (UID: "1606d845-d8ec-4ab3-a642-c97f8035207f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 16:00:24.207323 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:24.207277 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6ndb\" (UniqueName: \"kubernetes.io/projected/1606d845-d8ec-4ab3-a642-c97f8035207f-kube-api-access-f6ndb\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:00:24.207323 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:24.207318 2566 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1606d845-d8ec-4ab3-a642-c97f8035207f-bundle\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:00:24.207323 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:24.207327 2566 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1606d845-d8ec-4ab3-a642-c97f8035207f-util\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:00:24.842941 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:24.842902 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jbf59" event={"ID":"1606d845-d8ec-4ab3-a642-c97f8035207f","Type":"ContainerDied","Data":"6e015151a5e45a18d1692f7d4d0bd6e03a433b626ed117af6622fb1bc4ea7dfa"} Sep 11 16:00:24.843116 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:24.842961 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6e015151a5e45a18d1692f7d4d0bd6e03a433b626ed117af6622fb1bc4ea7dfa" Sep 11 16:00:24.843116 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:24.842920 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jbf59" Sep 11 16:00:31.754135 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:31.754097 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-g7ctv"] Sep 11 16:00:31.754517 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:31.754438 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1606d845-d8ec-4ab3-a642-c97f8035207f" containerName="extract" Sep 11 16:00:31.754517 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:31.754449 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="1606d845-d8ec-4ab3-a642-c97f8035207f" containerName="extract" Sep 11 16:00:31.754517 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:31.754474 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1606d845-d8ec-4ab3-a642-c97f8035207f" containerName="util" Sep 11 16:00:31.754517 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:31.754480 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="1606d845-d8ec-4ab3-a642-c97f8035207f" containerName="util" Sep 11 16:00:31.754517 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:31.754487 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1606d845-d8ec-4ab3-a642-c97f8035207f" containerName="pull" Sep 11 16:00:31.754517 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:31.754491 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="1606d845-d8ec-4ab3-a642-c97f8035207f" containerName="pull" Sep 11 16:00:31.754698 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:31.754544 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="1606d845-d8ec-4ab3-a642-c97f8035207f" containerName="extract" Sep 11 16:00:31.757579 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:31.757562 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-g7ctv" Sep 11 16:00:31.760822 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:31.760799 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Sep 11 16:00:31.760946 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:31.760830 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Sep 11 16:00:31.763722 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:31.763701 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-2r65r\"" Sep 11 16:00:31.782047 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:31.782016 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-g7ctv"] Sep 11 16:00:31.871852 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:31.871811 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrg6q\" (UniqueName: \"kubernetes.io/projected/90c00dec-cbb7-4a1f-825d-1db890f29321-kube-api-access-hrg6q\") pod \"cert-manager-operator-controller-manager-55c444bf86-g7ctv\" (UID: \"90c00dec-cbb7-4a1f-825d-1db890f29321\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-g7ctv" Sep 11 16:00:31.872102 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:31.871866 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/90c00dec-cbb7-4a1f-825d-1db890f29321-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-g7ctv\" (UID: \"90c00dec-cbb7-4a1f-825d-1db890f29321\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-g7ctv" Sep 11 16:00:31.972650 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:31.972610 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrg6q\" (UniqueName: \"kubernetes.io/projected/90c00dec-cbb7-4a1f-825d-1db890f29321-kube-api-access-hrg6q\") pod \"cert-manager-operator-controller-manager-55c444bf86-g7ctv\" (UID: \"90c00dec-cbb7-4a1f-825d-1db890f29321\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-g7ctv" Sep 11 16:00:31.972825 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:31.972663 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/90c00dec-cbb7-4a1f-825d-1db890f29321-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-g7ctv\" (UID: \"90c00dec-cbb7-4a1f-825d-1db890f29321\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-g7ctv" Sep 11 16:00:31.973071 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:31.973054 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/90c00dec-cbb7-4a1f-825d-1db890f29321-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-g7ctv\" (UID: \"90c00dec-cbb7-4a1f-825d-1db890f29321\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-g7ctv" Sep 11 16:00:31.982455 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:31.982428 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrg6q\" (UniqueName: \"kubernetes.io/projected/90c00dec-cbb7-4a1f-825d-1db890f29321-kube-api-access-hrg6q\") pod \"cert-manager-operator-controller-manager-55c444bf86-g7ctv\" (UID: \"90c00dec-cbb7-4a1f-825d-1db890f29321\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-g7ctv" Sep 11 16:00:32.066226 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:32.066187 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-g7ctv" Sep 11 16:00:32.358904 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:32.358828 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-g7ctv"] Sep 11 16:00:32.362512 ip-10-0-136-96 kubenswrapper[2566]: W0911 16:00:32.362478 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90c00dec_cbb7_4a1f_825d_1db890f29321.slice/crio-e7006d8c0e39d6b1841a43eed8399c8429098413a947f08d97ad00201d21b970 WatchSource:0}: Error finding container e7006d8c0e39d6b1841a43eed8399c8429098413a947f08d97ad00201d21b970: Status 404 returned error can't find the container with id e7006d8c0e39d6b1841a43eed8399c8429098413a947f08d97ad00201d21b970 Sep 11 16:00:32.884108 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:32.884074 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-g7ctv" event={"ID":"90c00dec-cbb7-4a1f-825d-1db890f29321","Type":"ContainerStarted","Data":"e7006d8c0e39d6b1841a43eed8399c8429098413a947f08d97ad00201d21b970"} Sep 11 16:00:34.898032 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:34.897991 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-g7ctv" event={"ID":"90c00dec-cbb7-4a1f-825d-1db890f29321","Type":"ContainerStarted","Data":"149edae96957298b711917c919e4004fee5538750b0e7edb87902f46aaa630bc"} Sep 11 16:00:34.920670 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:34.920619 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-g7ctv" podStartSLOduration=2.2941726989999998 podStartE2EDuration="3.920602669s" totalImagesPullingTime="1.62642997s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:00:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:00:34.918805233 +0000 UTC m=+342.133639145" watchObservedRunningTime="2026-09-11 16:00:34.920602669 +0000 UTC m=+342.135436552" Sep 11 16:00:35.547027 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:35.546989 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdgzj6"] Sep 11 16:00:35.549475 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:35.549458 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdgzj6" Sep 11 16:00:35.552321 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:35.552299 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 11 16:00:35.552997 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:35.552960 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 11 16:00:35.554106 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:35.554091 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-z84gm\"" Sep 11 16:00:35.568834 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:35.568806 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdgzj6"] Sep 11 16:00:35.705223 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:35.705185 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3cecdf7c-a651-420e-a7b0-17a5655c348c-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdgzj6\" (UID: \"3cecdf7c-a651-420e-a7b0-17a5655c348c\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdgzj6" Sep 11 16:00:35.705223 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:35.705230 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3cecdf7c-a651-420e-a7b0-17a5655c348c-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdgzj6\" (UID: \"3cecdf7c-a651-420e-a7b0-17a5655c348c\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdgzj6" Sep 11 16:00:35.705457 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:35.705250 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hblmj\" (UniqueName: \"kubernetes.io/projected/3cecdf7c-a651-420e-a7b0-17a5655c348c-kube-api-access-hblmj\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdgzj6\" (UID: \"3cecdf7c-a651-420e-a7b0-17a5655c348c\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdgzj6" Sep 11 16:00:35.806710 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:35.806610 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3cecdf7c-a651-420e-a7b0-17a5655c348c-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdgzj6\" (UID: \"3cecdf7c-a651-420e-a7b0-17a5655c348c\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdgzj6" Sep 11 16:00:35.806710 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:35.806667 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3cecdf7c-a651-420e-a7b0-17a5655c348c-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdgzj6\" (UID: \"3cecdf7c-a651-420e-a7b0-17a5655c348c\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdgzj6" Sep 11 16:00:35.806710 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:35.806698 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hblmj\" (UniqueName: \"kubernetes.io/projected/3cecdf7c-a651-420e-a7b0-17a5655c348c-kube-api-access-hblmj\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdgzj6\" (UID: \"3cecdf7c-a651-420e-a7b0-17a5655c348c\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdgzj6" Sep 11 16:00:35.807083 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:35.807061 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3cecdf7c-a651-420e-a7b0-17a5655c348c-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdgzj6\" (UID: \"3cecdf7c-a651-420e-a7b0-17a5655c348c\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdgzj6" Sep 11 16:00:35.807142 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:35.807077 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3cecdf7c-a651-420e-a7b0-17a5655c348c-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdgzj6\" (UID: \"3cecdf7c-a651-420e-a7b0-17a5655c348c\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdgzj6" Sep 11 16:00:35.818574 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:35.818547 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hblmj\" (UniqueName: \"kubernetes.io/projected/3cecdf7c-a651-420e-a7b0-17a5655c348c-kube-api-access-hblmj\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdgzj6\" (UID: \"3cecdf7c-a651-420e-a7b0-17a5655c348c\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdgzj6" Sep 11 16:00:35.859569 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:35.859542 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdgzj6" Sep 11 16:00:35.970919 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:35.970865 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-j84z7"] Sep 11 16:00:35.974529 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:35.974498 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-j84z7" Sep 11 16:00:35.976854 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:35.976831 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Sep 11 16:00:35.977001 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:35.976835 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-6skw7\"" Sep 11 16:00:35.977001 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:35.976857 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Sep 11 16:00:35.986957 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:35.986933 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-j84z7"] Sep 11 16:00:36.108764 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:36.108641 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npc5t\" (UniqueName: \"kubernetes.io/projected/4929a5f6-34a9-449f-a9e0-4c1b8edb820e-kube-api-access-npc5t\") pod \"cert-manager-webhook-6486ffd997-j84z7\" (UID: \"4929a5f6-34a9-449f-a9e0-4c1b8edb820e\") " pod="cert-manager/cert-manager-webhook-6486ffd997-j84z7" Sep 11 16:00:36.108764 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:36.108719 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4929a5f6-34a9-449f-a9e0-4c1b8edb820e-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-j84z7\" (UID: \"4929a5f6-34a9-449f-a9e0-4c1b8edb820e\") " pod="cert-manager/cert-manager-webhook-6486ffd997-j84z7" Sep 11 16:00:36.134106 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:36.134073 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdgzj6"] Sep 11 16:00:36.137675 ip-10-0-136-96 kubenswrapper[2566]: W0911 16:00:36.137648 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cecdf7c_a651_420e_a7b0_17a5655c348c.slice/crio-14739fc31ffcf66b106793d2ac5a74cbe1147daf21483cb646132595e8dc2168 WatchSource:0}: Error finding container 14739fc31ffcf66b106793d2ac5a74cbe1147daf21483cb646132595e8dc2168: Status 404 returned error can't find the container with id 14739fc31ffcf66b106793d2ac5a74cbe1147daf21483cb646132595e8dc2168 Sep 11 16:00:36.209688 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:36.209655 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npc5t\" (UniqueName: \"kubernetes.io/projected/4929a5f6-34a9-449f-a9e0-4c1b8edb820e-kube-api-access-npc5t\") pod \"cert-manager-webhook-6486ffd997-j84z7\" (UID: \"4929a5f6-34a9-449f-a9e0-4c1b8edb820e\") " pod="cert-manager/cert-manager-webhook-6486ffd997-j84z7" Sep 11 16:00:36.209826 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:36.209694 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4929a5f6-34a9-449f-a9e0-4c1b8edb820e-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-j84z7\" (UID: \"4929a5f6-34a9-449f-a9e0-4c1b8edb820e\") " pod="cert-manager/cert-manager-webhook-6486ffd997-j84z7" Sep 11 16:00:36.218466 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:36.218443 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4929a5f6-34a9-449f-a9e0-4c1b8edb820e-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-j84z7\" (UID: \"4929a5f6-34a9-449f-a9e0-4c1b8edb820e\") " pod="cert-manager/cert-manager-webhook-6486ffd997-j84z7" Sep 11 16:00:36.218605 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:36.218585 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-npc5t\" (UniqueName: \"kubernetes.io/projected/4929a5f6-34a9-449f-a9e0-4c1b8edb820e-kube-api-access-npc5t\") pod \"cert-manager-webhook-6486ffd997-j84z7\" (UID: \"4929a5f6-34a9-449f-a9e0-4c1b8edb820e\") " pod="cert-manager/cert-manager-webhook-6486ffd997-j84z7" Sep 11 16:00:36.295086 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:36.295055 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-j84z7" Sep 11 16:00:36.571862 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:36.571832 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-j84z7"] Sep 11 16:00:36.574150 ip-10-0-136-96 kubenswrapper[2566]: W0911 16:00:36.574118 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4929a5f6_34a9_449f_a9e0_4c1b8edb820e.slice/crio-043f0dd92278cbc561b57fc409aa6107ba41cf459b616d35630b5fd0d08c3513 WatchSource:0}: Error finding container 043f0dd92278cbc561b57fc409aa6107ba41cf459b616d35630b5fd0d08c3513: Status 404 returned error can't find the container with id 043f0dd92278cbc561b57fc409aa6107ba41cf459b616d35630b5fd0d08c3513 Sep 11 16:00:36.909538 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:36.909505 2566 generic.go:356] "Generic (PLEG): container finished" podID="3cecdf7c-a651-420e-a7b0-17a5655c348c" containerID="57331668d3f4558f470e8daecb3b09357bf620bbd5be13aa39972596d56810e7" exitCode=0 Sep 11 16:00:36.909726 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:36.909585 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdgzj6" event={"ID":"3cecdf7c-a651-420e-a7b0-17a5655c348c","Type":"ContainerDied","Data":"57331668d3f4558f470e8daecb3b09357bf620bbd5be13aa39972596d56810e7"} Sep 11 16:00:36.909726 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:36.909647 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdgzj6" event={"ID":"3cecdf7c-a651-420e-a7b0-17a5655c348c","Type":"ContainerStarted","Data":"14739fc31ffcf66b106793d2ac5a74cbe1147daf21483cb646132595e8dc2168"} Sep 11 16:00:36.910717 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:36.910697 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-j84z7" event={"ID":"4929a5f6-34a9-449f-a9e0-4c1b8edb820e","Type":"ContainerStarted","Data":"043f0dd92278cbc561b57fc409aa6107ba41cf459b616d35630b5fd0d08c3513"} Sep 11 16:00:39.930270 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:39.930236 2566 generic.go:356] "Generic (PLEG): container finished" podID="3cecdf7c-a651-420e-a7b0-17a5655c348c" containerID="784631fac0c6fb8a18d9b73b35bd275725241c9888f7f62810da50ce144debdd" exitCode=0 Sep 11 16:00:39.930694 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:39.930316 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdgzj6" event={"ID":"3cecdf7c-a651-420e-a7b0-17a5655c348c","Type":"ContainerDied","Data":"784631fac0c6fb8a18d9b73b35bd275725241c9888f7f62810da50ce144debdd"} Sep 11 16:00:39.931992 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:39.931946 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-j84z7" event={"ID":"4929a5f6-34a9-449f-a9e0-4c1b8edb820e","Type":"ContainerStarted","Data":"88d2a6feeb04c9da8fd24ce620cc7afa55bdff68b7e605e7a1118e87bd34f5c7"} Sep 11 16:00:39.932111 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:39.932091 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-j84z7" Sep 11 16:00:39.968258 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:39.968201 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-j84z7" podStartSLOduration=1.775180027 podStartE2EDuration="4.968187511s" totalImagesPullingTime="3.193007484s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:00:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:00:39.966708826 +0000 UTC m=+347.181542731" watchObservedRunningTime="2026-09-11 16:00:39.968187511 +0000 UTC m=+347.183021415" Sep 11 16:00:40.939075 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:40.939041 2566 generic.go:356] "Generic (PLEG): container finished" podID="3cecdf7c-a651-420e-a7b0-17a5655c348c" containerID="a6755de7f9ec869d693ba74a8e7d565584a8c36eef9ad3d0e41d02a6c96495ee" exitCode=0 Sep 11 16:00:40.939541 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:40.939101 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdgzj6" event={"ID":"3cecdf7c-a651-420e-a7b0-17a5655c348c","Type":"ContainerDied","Data":"a6755de7f9ec869d693ba74a8e7d565584a8c36eef9ad3d0e41d02a6c96495ee"} Sep 11 16:00:42.074269 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:42.074239 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdgzj6" Sep 11 16:00:42.163787 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:42.163752 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3cecdf7c-a651-420e-a7b0-17a5655c348c-bundle\") pod \"3cecdf7c-a651-420e-a7b0-17a5655c348c\" (UID: \"3cecdf7c-a651-420e-a7b0-17a5655c348c\") " Sep 11 16:00:42.163997 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:42.163829 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hblmj\" (UniqueName: \"kubernetes.io/projected/3cecdf7c-a651-420e-a7b0-17a5655c348c-kube-api-access-hblmj\") pod \"3cecdf7c-a651-420e-a7b0-17a5655c348c\" (UID: \"3cecdf7c-a651-420e-a7b0-17a5655c348c\") " Sep 11 16:00:42.163997 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:42.163853 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3cecdf7c-a651-420e-a7b0-17a5655c348c-util\") pod \"3cecdf7c-a651-420e-a7b0-17a5655c348c\" (UID: \"3cecdf7c-a651-420e-a7b0-17a5655c348c\") " Sep 11 16:00:42.164295 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:42.164267 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cecdf7c-a651-420e-a7b0-17a5655c348c-bundle" (OuterVolumeSpecName: "bundle") pod "3cecdf7c-a651-420e-a7b0-17a5655c348c" (UID: "3cecdf7c-a651-420e-a7b0-17a5655c348c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 16:00:42.166023 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:42.166002 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cecdf7c-a651-420e-a7b0-17a5655c348c-kube-api-access-hblmj" (OuterVolumeSpecName: "kube-api-access-hblmj") pod "3cecdf7c-a651-420e-a7b0-17a5655c348c" (UID: "3cecdf7c-a651-420e-a7b0-17a5655c348c"). InnerVolumeSpecName "kube-api-access-hblmj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 16:00:42.168680 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:42.168652 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cecdf7c-a651-420e-a7b0-17a5655c348c-util" (OuterVolumeSpecName: "util") pod "3cecdf7c-a651-420e-a7b0-17a5655c348c" (UID: "3cecdf7c-a651-420e-a7b0-17a5655c348c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 16:00:42.264664 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:42.264629 2566 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3cecdf7c-a651-420e-a7b0-17a5655c348c-util\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:00:42.264664 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:42.264662 2566 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3cecdf7c-a651-420e-a7b0-17a5655c348c-bundle\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:00:42.264844 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:42.264672 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hblmj\" (UniqueName: \"kubernetes.io/projected/3cecdf7c-a651-420e-a7b0-17a5655c348c-kube-api-access-hblmj\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:00:42.949130 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:42.949091 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdgzj6" event={"ID":"3cecdf7c-a651-420e-a7b0-17a5655c348c","Type":"ContainerDied","Data":"14739fc31ffcf66b106793d2ac5a74cbe1147daf21483cb646132595e8dc2168"} Sep 11 16:00:42.949302 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:42.949169 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="14739fc31ffcf66b106793d2ac5a74cbe1147daf21483cb646132595e8dc2168" Sep 11 16:00:42.949302 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:42.949108 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fdgzj6" Sep 11 16:00:45.941438 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:45.941359 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-j84z7" Sep 11 16:00:47.966498 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:47.966465 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-vn9h2"] Sep 11 16:00:47.966873 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:47.966809 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3cecdf7c-a651-420e-a7b0-17a5655c348c" containerName="pull" Sep 11 16:00:47.966873 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:47.966822 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="3cecdf7c-a651-420e-a7b0-17a5655c348c" containerName="pull" Sep 11 16:00:47.966873 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:47.966837 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3cecdf7c-a651-420e-a7b0-17a5655c348c" containerName="util" Sep 11 16:00:47.966873 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:47.966842 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="3cecdf7c-a651-420e-a7b0-17a5655c348c" containerName="util" Sep 11 16:00:47.966873 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:47.966851 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3cecdf7c-a651-420e-a7b0-17a5655c348c" containerName="extract" Sep 11 16:00:47.966873 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:47.966856 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="3cecdf7c-a651-420e-a7b0-17a5655c348c" containerName="extract" Sep 11 16:00:47.967080 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:47.966912 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="3cecdf7c-a651-420e-a7b0-17a5655c348c" containerName="extract" Sep 11 16:00:47.970004 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:47.969965 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-vn9h2" Sep 11 16:00:47.972004 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:47.971984 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-4nszp\"" Sep 11 16:00:47.976466 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:47.976440 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-vn9h2"] Sep 11 16:00:48.012644 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:48.012604 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jg69\" (UniqueName: \"kubernetes.io/projected/7d007481-6c59-4feb-9a25-3b4954c554a7-kube-api-access-4jg69\") pod \"cert-manager-55b4d954d5-vn9h2\" (UID: \"7d007481-6c59-4feb-9a25-3b4954c554a7\") " pod="cert-manager/cert-manager-55b4d954d5-vn9h2" Sep 11 16:00:48.012800 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:48.012652 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7d007481-6c59-4feb-9a25-3b4954c554a7-bound-sa-token\") pod \"cert-manager-55b4d954d5-vn9h2\" (UID: \"7d007481-6c59-4feb-9a25-3b4954c554a7\") " pod="cert-manager/cert-manager-55b4d954d5-vn9h2" Sep 11 16:00:48.113843 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:48.113798 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jg69\" (UniqueName: \"kubernetes.io/projected/7d007481-6c59-4feb-9a25-3b4954c554a7-kube-api-access-4jg69\") pod \"cert-manager-55b4d954d5-vn9h2\" (UID: \"7d007481-6c59-4feb-9a25-3b4954c554a7\") " pod="cert-manager/cert-manager-55b4d954d5-vn9h2" Sep 11 16:00:48.114054 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:48.113956 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7d007481-6c59-4feb-9a25-3b4954c554a7-bound-sa-token\") pod \"cert-manager-55b4d954d5-vn9h2\" (UID: \"7d007481-6c59-4feb-9a25-3b4954c554a7\") " pod="cert-manager/cert-manager-55b4d954d5-vn9h2" Sep 11 16:00:48.121923 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:48.121899 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7d007481-6c59-4feb-9a25-3b4954c554a7-bound-sa-token\") pod \"cert-manager-55b4d954d5-vn9h2\" (UID: \"7d007481-6c59-4feb-9a25-3b4954c554a7\") " pod="cert-manager/cert-manager-55b4d954d5-vn9h2" Sep 11 16:00:48.122084 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:48.122061 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jg69\" (UniqueName: \"kubernetes.io/projected/7d007481-6c59-4feb-9a25-3b4954c554a7-kube-api-access-4jg69\") pod \"cert-manager-55b4d954d5-vn9h2\" (UID: \"7d007481-6c59-4feb-9a25-3b4954c554a7\") " pod="cert-manager/cert-manager-55b4d954d5-vn9h2" Sep 11 16:00:48.279638 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:48.279589 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-vn9h2" Sep 11 16:00:48.554408 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:48.554361 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-vn9h2"] Sep 11 16:00:48.556640 ip-10-0-136-96 kubenswrapper[2566]: W0911 16:00:48.556607 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7d007481_6c59_4feb_9a25_3b4954c554a7.slice/crio-617dd85d4fc5ecaf5575483157dda3d1d20feafc029e9043c914412f85f967b9 WatchSource:0}: Error finding container 617dd85d4fc5ecaf5575483157dda3d1d20feafc029e9043c914412f85f967b9: Status 404 returned error can't find the container with id 617dd85d4fc5ecaf5575483157dda3d1d20feafc029e9043c914412f85f967b9 Sep 11 16:00:48.977740 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:48.977656 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-vn9h2" event={"ID":"7d007481-6c59-4feb-9a25-3b4954c554a7","Type":"ContainerStarted","Data":"77071f7b69d4a6cbfae44cab32e003a363bf6689e02905f367536262e60715bc"} Sep 11 16:00:48.977740 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:48.977704 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-vn9h2" event={"ID":"7d007481-6c59-4feb-9a25-3b4954c554a7","Type":"ContainerStarted","Data":"617dd85d4fc5ecaf5575483157dda3d1d20feafc029e9043c914412f85f967b9"} Sep 11 16:00:49.000183 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:49.000126 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-vn9h2" podStartSLOduration=2.000107844 podStartE2EDuration="2.000107844s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:00:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:00:48.999050436 +0000 UTC m=+356.213884340" watchObservedRunningTime="2026-09-11 16:00:49.000107844 +0000 UTC m=+356.214941768" Sep 11 16:00:55.947940 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:55.947900 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgl8c6"] Sep 11 16:00:55.952817 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:55.952796 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgl8c6" Sep 11 16:00:55.955241 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:55.955217 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 11 16:00:55.955346 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:55.955225 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 11 16:00:55.955641 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:55.955627 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-z84gm\"" Sep 11 16:00:55.960245 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:55.960149 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgl8c6"] Sep 11 16:00:55.974104 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:55.974078 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bfhl\" (UniqueName: \"kubernetes.io/projected/a7c4379a-115d-4d02-a717-292844ead6b6-kube-api-access-8bfhl\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgl8c6\" (UID: \"a7c4379a-115d-4d02-a717-292844ead6b6\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgl8c6" Sep 11 16:00:55.974242 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:55.974189 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a7c4379a-115d-4d02-a717-292844ead6b6-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgl8c6\" (UID: \"a7c4379a-115d-4d02-a717-292844ead6b6\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgl8c6" Sep 11 16:00:55.974242 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:55.974220 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a7c4379a-115d-4d02-a717-292844ead6b6-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgl8c6\" (UID: \"a7c4379a-115d-4d02-a717-292844ead6b6\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgl8c6" Sep 11 16:00:56.075183 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:56.075145 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bfhl\" (UniqueName: \"kubernetes.io/projected/a7c4379a-115d-4d02-a717-292844ead6b6-kube-api-access-8bfhl\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgl8c6\" (UID: \"a7c4379a-115d-4d02-a717-292844ead6b6\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgl8c6" Sep 11 16:00:56.075388 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:56.075260 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a7c4379a-115d-4d02-a717-292844ead6b6-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgl8c6\" (UID: \"a7c4379a-115d-4d02-a717-292844ead6b6\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgl8c6" Sep 11 16:00:56.075388 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:56.075286 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a7c4379a-115d-4d02-a717-292844ead6b6-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgl8c6\" (UID: \"a7c4379a-115d-4d02-a717-292844ead6b6\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgl8c6" Sep 11 16:00:56.075640 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:56.075619 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a7c4379a-115d-4d02-a717-292844ead6b6-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgl8c6\" (UID: \"a7c4379a-115d-4d02-a717-292844ead6b6\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgl8c6" Sep 11 16:00:56.075705 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:56.075655 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a7c4379a-115d-4d02-a717-292844ead6b6-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgl8c6\" (UID: \"a7c4379a-115d-4d02-a717-292844ead6b6\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgl8c6" Sep 11 16:00:56.083092 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:56.083060 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bfhl\" (UniqueName: \"kubernetes.io/projected/a7c4379a-115d-4d02-a717-292844ead6b6-kube-api-access-8bfhl\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgl8c6\" (UID: \"a7c4379a-115d-4d02-a717-292844ead6b6\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgl8c6" Sep 11 16:00:56.263145 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:56.263110 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgl8c6" Sep 11 16:00:56.536323 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:56.536297 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgl8c6"] Sep 11 16:00:56.538760 ip-10-0-136-96 kubenswrapper[2566]: W0911 16:00:56.538732 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7c4379a_115d_4d02_a717_292844ead6b6.slice/crio-9ad790f13aedf0fb781528f4199f64f768dd580088186667a1110181fe3a34c6 WatchSource:0}: Error finding container 9ad790f13aedf0fb781528f4199f64f768dd580088186667a1110181fe3a34c6: Status 404 returned error can't find the container with id 9ad790f13aedf0fb781528f4199f64f768dd580088186667a1110181fe3a34c6 Sep 11 16:00:57.013432 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:57.013397 2566 generic.go:356] "Generic (PLEG): container finished" podID="a7c4379a-115d-4d02-a717-292844ead6b6" containerID="507fd21c48522f86818c60d5bbb40a57aaa610173a5b56f117a678ec73a01212" exitCode=0 Sep 11 16:00:57.013814 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:57.013476 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgl8c6" event={"ID":"a7c4379a-115d-4d02-a717-292844ead6b6","Type":"ContainerDied","Data":"507fd21c48522f86818c60d5bbb40a57aaa610173a5b56f117a678ec73a01212"} Sep 11 16:00:57.013814 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:57.013532 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgl8c6" event={"ID":"a7c4379a-115d-4d02-a717-292844ead6b6","Type":"ContainerStarted","Data":"9ad790f13aedf0fb781528f4199f64f768dd580088186667a1110181fe3a34c6"} Sep 11 16:00:58.020004 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:58.019958 2566 generic.go:356] "Generic (PLEG): container finished" podID="a7c4379a-115d-4d02-a717-292844ead6b6" containerID="d87e3176db8447a3f587399d1e69e373421cc76b1223e22fb5d1b48bb4d04d74" exitCode=0 Sep 11 16:00:58.020368 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:58.020018 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgl8c6" event={"ID":"a7c4379a-115d-4d02-a717-292844ead6b6","Type":"ContainerDied","Data":"d87e3176db8447a3f587399d1e69e373421cc76b1223e22fb5d1b48bb4d04d74"} Sep 11 16:00:59.026990 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:59.026945 2566 generic.go:356] "Generic (PLEG): container finished" podID="a7c4379a-115d-4d02-a717-292844ead6b6" containerID="1527a601a23057e26f02b78bc6f1c0d423cafd9c419ab40723301e38ac24aa66" exitCode=0 Sep 11 16:00:59.027358 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:00:59.027007 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgl8c6" event={"ID":"a7c4379a-115d-4d02-a717-292844ead6b6","Type":"ContainerDied","Data":"1527a601a23057e26f02b78bc6f1c0d423cafd9c419ab40723301e38ac24aa66"} Sep 11 16:01:00.163172 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:00.163147 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgl8c6" Sep 11 16:01:00.211670 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:00.211637 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8bfhl\" (UniqueName: \"kubernetes.io/projected/a7c4379a-115d-4d02-a717-292844ead6b6-kube-api-access-8bfhl\") pod \"a7c4379a-115d-4d02-a717-292844ead6b6\" (UID: \"a7c4379a-115d-4d02-a717-292844ead6b6\") " Sep 11 16:01:00.211858 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:00.211687 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a7c4379a-115d-4d02-a717-292844ead6b6-util\") pod \"a7c4379a-115d-4d02-a717-292844ead6b6\" (UID: \"a7c4379a-115d-4d02-a717-292844ead6b6\") " Sep 11 16:01:00.211858 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:00.211721 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a7c4379a-115d-4d02-a717-292844ead6b6-bundle\") pod \"a7c4379a-115d-4d02-a717-292844ead6b6\" (UID: \"a7c4379a-115d-4d02-a717-292844ead6b6\") " Sep 11 16:01:00.212480 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:00.212433 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7c4379a-115d-4d02-a717-292844ead6b6-bundle" (OuterVolumeSpecName: "bundle") pod "a7c4379a-115d-4d02-a717-292844ead6b6" (UID: "a7c4379a-115d-4d02-a717-292844ead6b6"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 16:01:00.213771 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:00.213747 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7c4379a-115d-4d02-a717-292844ead6b6-kube-api-access-8bfhl" (OuterVolumeSpecName: "kube-api-access-8bfhl") pod "a7c4379a-115d-4d02-a717-292844ead6b6" (UID: "a7c4379a-115d-4d02-a717-292844ead6b6"). InnerVolumeSpecName "kube-api-access-8bfhl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 16:01:00.220532 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:00.220504 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7c4379a-115d-4d02-a717-292844ead6b6-util" (OuterVolumeSpecName: "util") pod "a7c4379a-115d-4d02-a717-292844ead6b6" (UID: "a7c4379a-115d-4d02-a717-292844ead6b6"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 16:01:00.312991 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:00.312887 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bfhl\" (UniqueName: \"kubernetes.io/projected/a7c4379a-115d-4d02-a717-292844ead6b6-kube-api-access-8bfhl\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:01:00.312991 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:00.312916 2566 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a7c4379a-115d-4d02-a717-292844ead6b6-util\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:01:00.312991 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:00.312926 2566 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a7c4379a-115d-4d02-a717-292844ead6b6-bundle\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:01:01.037506 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:01.037480 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgl8c6" Sep 11 16:01:01.037506 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:01.037502 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afgl8c6" event={"ID":"a7c4379a-115d-4d02-a717-292844ead6b6","Type":"ContainerDied","Data":"9ad790f13aedf0fb781528f4199f64f768dd580088186667a1110181fe3a34c6"} Sep 11 16:01:01.037754 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:01.037539 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9ad790f13aedf0fb781528f4199f64f768dd580088186667a1110181fe3a34c6" Sep 11 16:01:10.162942 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:10.162902 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tgr4d"] Sep 11 16:01:10.163348 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:10.163241 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7c4379a-115d-4d02-a717-292844ead6b6" containerName="extract" Sep 11 16:01:10.163348 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:10.163252 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7c4379a-115d-4d02-a717-292844ead6b6" containerName="extract" Sep 11 16:01:10.163348 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:10.163265 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7c4379a-115d-4d02-a717-292844ead6b6" containerName="pull" Sep 11 16:01:10.163348 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:10.163271 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7c4379a-115d-4d02-a717-292844ead6b6" containerName="pull" Sep 11 16:01:10.163348 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:10.163286 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7c4379a-115d-4d02-a717-292844ead6b6" containerName="util" Sep 11 16:01:10.163348 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:10.163292 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7c4379a-115d-4d02-a717-292844ead6b6" containerName="util" Sep 11 16:01:10.163348 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:10.163348 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7c4379a-115d-4d02-a717-292844ead6b6" containerName="extract" Sep 11 16:01:10.166238 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:10.166215 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tgr4d" Sep 11 16:01:10.168798 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:10.168772 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 11 16:01:10.168798 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:10.168782 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-z84gm\"" Sep 11 16:01:10.169333 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:10.169314 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 11 16:01:10.176826 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:10.176801 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tgr4d"] Sep 11 16:01:10.196189 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:10.196152 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0866e140-3413-47b3-be58-6bfbcc43c487-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tgr4d\" (UID: \"0866e140-3413-47b3-be58-6bfbcc43c487\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tgr4d" Sep 11 16:01:10.196354 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:10.196212 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0866e140-3413-47b3-be58-6bfbcc43c487-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tgr4d\" (UID: \"0866e140-3413-47b3-be58-6bfbcc43c487\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tgr4d" Sep 11 16:01:10.196354 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:10.196257 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gp5p4\" (UniqueName: \"kubernetes.io/projected/0866e140-3413-47b3-be58-6bfbcc43c487-kube-api-access-gp5p4\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tgr4d\" (UID: \"0866e140-3413-47b3-be58-6bfbcc43c487\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tgr4d" Sep 11 16:01:10.297045 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:10.297013 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0866e140-3413-47b3-be58-6bfbcc43c487-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tgr4d\" (UID: \"0866e140-3413-47b3-be58-6bfbcc43c487\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tgr4d" Sep 11 16:01:10.297223 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:10.297064 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0866e140-3413-47b3-be58-6bfbcc43c487-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tgr4d\" (UID: \"0866e140-3413-47b3-be58-6bfbcc43c487\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tgr4d" Sep 11 16:01:10.297223 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:10.297084 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gp5p4\" (UniqueName: \"kubernetes.io/projected/0866e140-3413-47b3-be58-6bfbcc43c487-kube-api-access-gp5p4\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tgr4d\" (UID: \"0866e140-3413-47b3-be58-6bfbcc43c487\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tgr4d" Sep 11 16:01:10.297404 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:10.297383 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0866e140-3413-47b3-be58-6bfbcc43c487-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tgr4d\" (UID: \"0866e140-3413-47b3-be58-6bfbcc43c487\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tgr4d" Sep 11 16:01:10.297479 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:10.297439 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0866e140-3413-47b3-be58-6bfbcc43c487-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tgr4d\" (UID: \"0866e140-3413-47b3-be58-6bfbcc43c487\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tgr4d" Sep 11 16:01:10.310080 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:10.310045 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gp5p4\" (UniqueName: \"kubernetes.io/projected/0866e140-3413-47b3-be58-6bfbcc43c487-kube-api-access-gp5p4\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tgr4d\" (UID: \"0866e140-3413-47b3-be58-6bfbcc43c487\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tgr4d" Sep 11 16:01:10.475854 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:10.475766 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tgr4d" Sep 11 16:01:10.861690 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:10.861661 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tgr4d"] Sep 11 16:01:10.864336 ip-10-0-136-96 kubenswrapper[2566]: W0911 16:01:10.864308 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0866e140_3413_47b3_be58_6bfbcc43c487.slice/crio-40935b4e3b65569aa5e4eab1bda4eaa6cd4ac8557adc47212a4e6a4d93705381 WatchSource:0}: Error finding container 40935b4e3b65569aa5e4eab1bda4eaa6cd4ac8557adc47212a4e6a4d93705381: Status 404 returned error can't find the container with id 40935b4e3b65569aa5e4eab1bda4eaa6cd4ac8557adc47212a4e6a4d93705381 Sep 11 16:01:11.032333 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:11.032298 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6848676c-8zbl6"] Sep 11 16:01:11.034566 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:11.034549 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6848676c-8zbl6" Sep 11 16:01:11.038221 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:11.038161 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Sep 11 16:01:11.038221 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:11.038193 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Sep 11 16:01:11.038221 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:11.038210 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Sep 11 16:01:11.038452 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:11.038241 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Sep 11 16:01:11.038452 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:11.038193 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-tm5qj\"" Sep 11 16:01:11.038452 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:11.038194 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Sep 11 16:01:11.063552 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:11.063519 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6848676c-8zbl6"] Sep 11 16:01:11.085054 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:11.085023 2566 generic.go:356] "Generic (PLEG): container finished" podID="0866e140-3413-47b3-be58-6bfbcc43c487" containerID="8e3758f3b7d173f32e31f8d7e63fd9a596f5ecfd25d3d86e4f242c9171be0c59" exitCode=0 Sep 11 16:01:11.085209 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:11.085114 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tgr4d" event={"ID":"0866e140-3413-47b3-be58-6bfbcc43c487","Type":"ContainerDied","Data":"8e3758f3b7d173f32e31f8d7e63fd9a596f5ecfd25d3d86e4f242c9171be0c59"} Sep 11 16:01:11.085209 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:11.085160 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tgr4d" event={"ID":"0866e140-3413-47b3-be58-6bfbcc43c487","Type":"ContainerStarted","Data":"40935b4e3b65569aa5e4eab1bda4eaa6cd4ac8557adc47212a4e6a4d93705381"} Sep 11 16:01:11.102632 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:11.102604 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/4291750d-8a58-4dfe-b6a7-6d959f0f63cd-metrics-cert\") pod \"lws-controller-manager-6848676c-8zbl6\" (UID: \"4291750d-8a58-4dfe-b6a7-6d959f0f63cd\") " pod="openshift-lws-operator/lws-controller-manager-6848676c-8zbl6" Sep 11 16:01:11.102763 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:11.102665 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4291750d-8a58-4dfe-b6a7-6d959f0f63cd-cert\") pod \"lws-controller-manager-6848676c-8zbl6\" (UID: \"4291750d-8a58-4dfe-b6a7-6d959f0f63cd\") " pod="openshift-lws-operator/lws-controller-manager-6848676c-8zbl6" Sep 11 16:01:11.102763 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:11.102736 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/4291750d-8a58-4dfe-b6a7-6d959f0f63cd-manager-config\") pod \"lws-controller-manager-6848676c-8zbl6\" (UID: \"4291750d-8a58-4dfe-b6a7-6d959f0f63cd\") " pod="openshift-lws-operator/lws-controller-manager-6848676c-8zbl6" Sep 11 16:01:11.102902 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:11.102778 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7g6wb\" (UniqueName: \"kubernetes.io/projected/4291750d-8a58-4dfe-b6a7-6d959f0f63cd-kube-api-access-7g6wb\") pod \"lws-controller-manager-6848676c-8zbl6\" (UID: \"4291750d-8a58-4dfe-b6a7-6d959f0f63cd\") " pod="openshift-lws-operator/lws-controller-manager-6848676c-8zbl6" Sep 11 16:01:11.207317 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:11.207206 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4291750d-8a58-4dfe-b6a7-6d959f0f63cd-cert\") pod \"lws-controller-manager-6848676c-8zbl6\" (UID: \"4291750d-8a58-4dfe-b6a7-6d959f0f63cd\") " pod="openshift-lws-operator/lws-controller-manager-6848676c-8zbl6" Sep 11 16:01:11.207317 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:11.207287 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/4291750d-8a58-4dfe-b6a7-6d959f0f63cd-manager-config\") pod \"lws-controller-manager-6848676c-8zbl6\" (UID: \"4291750d-8a58-4dfe-b6a7-6d959f0f63cd\") " pod="openshift-lws-operator/lws-controller-manager-6848676c-8zbl6" Sep 11 16:01:11.207317 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:11.207315 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7g6wb\" (UniqueName: \"kubernetes.io/projected/4291750d-8a58-4dfe-b6a7-6d959f0f63cd-kube-api-access-7g6wb\") pod \"lws-controller-manager-6848676c-8zbl6\" (UID: \"4291750d-8a58-4dfe-b6a7-6d959f0f63cd\") " pod="openshift-lws-operator/lws-controller-manager-6848676c-8zbl6" Sep 11 16:01:11.207909 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:11.207397 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/4291750d-8a58-4dfe-b6a7-6d959f0f63cd-metrics-cert\") pod \"lws-controller-manager-6848676c-8zbl6\" (UID: \"4291750d-8a58-4dfe-b6a7-6d959f0f63cd\") " pod="openshift-lws-operator/lws-controller-manager-6848676c-8zbl6" Sep 11 16:01:11.208083 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:11.208061 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/4291750d-8a58-4dfe-b6a7-6d959f0f63cd-manager-config\") pod \"lws-controller-manager-6848676c-8zbl6\" (UID: \"4291750d-8a58-4dfe-b6a7-6d959f0f63cd\") " pod="openshift-lws-operator/lws-controller-manager-6848676c-8zbl6" Sep 11 16:01:11.209886 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:11.209862 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4291750d-8a58-4dfe-b6a7-6d959f0f63cd-cert\") pod \"lws-controller-manager-6848676c-8zbl6\" (UID: \"4291750d-8a58-4dfe-b6a7-6d959f0f63cd\") " pod="openshift-lws-operator/lws-controller-manager-6848676c-8zbl6" Sep 11 16:01:11.209989 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:11.209895 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/4291750d-8a58-4dfe-b6a7-6d959f0f63cd-metrics-cert\") pod \"lws-controller-manager-6848676c-8zbl6\" (UID: \"4291750d-8a58-4dfe-b6a7-6d959f0f63cd\") " pod="openshift-lws-operator/lws-controller-manager-6848676c-8zbl6" Sep 11 16:01:11.229064 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:11.229030 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7g6wb\" (UniqueName: \"kubernetes.io/projected/4291750d-8a58-4dfe-b6a7-6d959f0f63cd-kube-api-access-7g6wb\") pod \"lws-controller-manager-6848676c-8zbl6\" (UID: \"4291750d-8a58-4dfe-b6a7-6d959f0f63cd\") " pod="openshift-lws-operator/lws-controller-manager-6848676c-8zbl6" Sep 11 16:01:11.344105 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:11.344068 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6848676c-8zbl6" Sep 11 16:01:11.629895 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:11.629835 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6848676c-8zbl6"] Sep 11 16:01:11.633169 ip-10-0-136-96 kubenswrapper[2566]: W0911 16:01:11.633142 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4291750d_8a58_4dfe_b6a7_6d959f0f63cd.slice/crio-e0884bc423cda31e2529bf529695c2dd2d9f19a852f010d02ba90474f134656f WatchSource:0}: Error finding container e0884bc423cda31e2529bf529695c2dd2d9f19a852f010d02ba90474f134656f: Status 404 returned error can't find the container with id e0884bc423cda31e2529bf529695c2dd2d9f19a852f010d02ba90474f134656f Sep 11 16:01:12.093848 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:12.093809 2566 generic.go:356] "Generic (PLEG): container finished" podID="0866e140-3413-47b3-be58-6bfbcc43c487" containerID="0bb38184e9b5d75d8dc4d07437e97a418a62869bad647c91f083f415c6f4c051" exitCode=0 Sep 11 16:01:12.094037 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:12.093887 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tgr4d" event={"ID":"0866e140-3413-47b3-be58-6bfbcc43c487","Type":"ContainerDied","Data":"0bb38184e9b5d75d8dc4d07437e97a418a62869bad647c91f083f415c6f4c051"} Sep 11 16:01:12.095492 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:12.095443 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6848676c-8zbl6" event={"ID":"4291750d-8a58-4dfe-b6a7-6d959f0f63cd","Type":"ContainerStarted","Data":"e0884bc423cda31e2529bf529695c2dd2d9f19a852f010d02ba90474f134656f"} Sep 11 16:01:12.220709 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:12.220672 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-58647b66f4-kj5l9"] Sep 11 16:01:12.223171 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:12.223147 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-58647b66f4-kj5l9" Sep 11 16:01:12.226332 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:12.226301 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 11 16:01:12.227217 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:12.227106 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 11 16:01:12.227408 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:12.227386 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-wl6q7\"" Sep 11 16:01:12.227627 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:12.227610 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 11 16:01:12.232546 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:12.232514 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 11 16:01:12.260316 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:12.260278 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-58647b66f4-kj5l9"] Sep 11 16:01:12.319267 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:12.319232 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4a19c7c5-5d8e-48b1-977a-08b22dbfa94d-webhook-cert\") pod \"opendatahub-operator-controller-manager-58647b66f4-kj5l9\" (UID: \"4a19c7c5-5d8e-48b1-977a-08b22dbfa94d\") " pod="opendatahub/opendatahub-operator-controller-manager-58647b66f4-kj5l9" Sep 11 16:01:12.319429 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:12.319363 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mg76t\" (UniqueName: \"kubernetes.io/projected/4a19c7c5-5d8e-48b1-977a-08b22dbfa94d-kube-api-access-mg76t\") pod \"opendatahub-operator-controller-manager-58647b66f4-kj5l9\" (UID: \"4a19c7c5-5d8e-48b1-977a-08b22dbfa94d\") " pod="opendatahub/opendatahub-operator-controller-manager-58647b66f4-kj5l9" Sep 11 16:01:12.319496 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:12.319477 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4a19c7c5-5d8e-48b1-977a-08b22dbfa94d-manifests\") pod \"opendatahub-operator-controller-manager-58647b66f4-kj5l9\" (UID: \"4a19c7c5-5d8e-48b1-977a-08b22dbfa94d\") " pod="opendatahub/opendatahub-operator-controller-manager-58647b66f4-kj5l9" Sep 11 16:01:12.319541 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:12.319524 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4a19c7c5-5d8e-48b1-977a-08b22dbfa94d-tmp\") pod \"opendatahub-operator-controller-manager-58647b66f4-kj5l9\" (UID: \"4a19c7c5-5d8e-48b1-977a-08b22dbfa94d\") " pod="opendatahub/opendatahub-operator-controller-manager-58647b66f4-kj5l9" Sep 11 16:01:12.319616 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:12.319598 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4a19c7c5-5d8e-48b1-977a-08b22dbfa94d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-58647b66f4-kj5l9\" (UID: \"4a19c7c5-5d8e-48b1-977a-08b22dbfa94d\") " pod="opendatahub/opendatahub-operator-controller-manager-58647b66f4-kj5l9" Sep 11 16:01:12.420667 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:12.420630 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4a19c7c5-5d8e-48b1-977a-08b22dbfa94d-manifests\") pod \"opendatahub-operator-controller-manager-58647b66f4-kj5l9\" (UID: \"4a19c7c5-5d8e-48b1-977a-08b22dbfa94d\") " pod="opendatahub/opendatahub-operator-controller-manager-58647b66f4-kj5l9" Sep 11 16:01:12.420667 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:12.420667 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4a19c7c5-5d8e-48b1-977a-08b22dbfa94d-tmp\") pod \"opendatahub-operator-controller-manager-58647b66f4-kj5l9\" (UID: \"4a19c7c5-5d8e-48b1-977a-08b22dbfa94d\") " pod="opendatahub/opendatahub-operator-controller-manager-58647b66f4-kj5l9" Sep 11 16:01:12.420854 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:12.420688 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4a19c7c5-5d8e-48b1-977a-08b22dbfa94d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-58647b66f4-kj5l9\" (UID: \"4a19c7c5-5d8e-48b1-977a-08b22dbfa94d\") " pod="opendatahub/opendatahub-operator-controller-manager-58647b66f4-kj5l9" Sep 11 16:01:12.420854 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:12.420727 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4a19c7c5-5d8e-48b1-977a-08b22dbfa94d-webhook-cert\") pod \"opendatahub-operator-controller-manager-58647b66f4-kj5l9\" (UID: \"4a19c7c5-5d8e-48b1-977a-08b22dbfa94d\") " pod="opendatahub/opendatahub-operator-controller-manager-58647b66f4-kj5l9" Sep 11 16:01:12.420854 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:12.420757 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mg76t\" (UniqueName: \"kubernetes.io/projected/4a19c7c5-5d8e-48b1-977a-08b22dbfa94d-kube-api-access-mg76t\") pod \"opendatahub-operator-controller-manager-58647b66f4-kj5l9\" (UID: \"4a19c7c5-5d8e-48b1-977a-08b22dbfa94d\") " pod="opendatahub/opendatahub-operator-controller-manager-58647b66f4-kj5l9" Sep 11 16:01:12.421137 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:12.421114 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4a19c7c5-5d8e-48b1-977a-08b22dbfa94d-manifests\") pod \"opendatahub-operator-controller-manager-58647b66f4-kj5l9\" (UID: \"4a19c7c5-5d8e-48b1-977a-08b22dbfa94d\") " pod="opendatahub/opendatahub-operator-controller-manager-58647b66f4-kj5l9" Sep 11 16:01:12.423217 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:12.423190 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4a19c7c5-5d8e-48b1-977a-08b22dbfa94d-tmp\") pod \"opendatahub-operator-controller-manager-58647b66f4-kj5l9\" (UID: \"4a19c7c5-5d8e-48b1-977a-08b22dbfa94d\") " pod="opendatahub/opendatahub-operator-controller-manager-58647b66f4-kj5l9" Sep 11 16:01:12.423330 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:12.423261 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4a19c7c5-5d8e-48b1-977a-08b22dbfa94d-webhook-cert\") pod \"opendatahub-operator-controller-manager-58647b66f4-kj5l9\" (UID: \"4a19c7c5-5d8e-48b1-977a-08b22dbfa94d\") " pod="opendatahub/opendatahub-operator-controller-manager-58647b66f4-kj5l9" Sep 11 16:01:12.423330 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:12.423298 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4a19c7c5-5d8e-48b1-977a-08b22dbfa94d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-58647b66f4-kj5l9\" (UID: \"4a19c7c5-5d8e-48b1-977a-08b22dbfa94d\") " pod="opendatahub/opendatahub-operator-controller-manager-58647b66f4-kj5l9" Sep 11 16:01:12.432239 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:12.432216 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mg76t\" (UniqueName: \"kubernetes.io/projected/4a19c7c5-5d8e-48b1-977a-08b22dbfa94d-kube-api-access-mg76t\") pod \"opendatahub-operator-controller-manager-58647b66f4-kj5l9\" (UID: \"4a19c7c5-5d8e-48b1-977a-08b22dbfa94d\") " pod="opendatahub/opendatahub-operator-controller-manager-58647b66f4-kj5l9" Sep 11 16:01:12.536928 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:12.536888 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-58647b66f4-kj5l9" Sep 11 16:01:12.816219 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:12.816192 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-58647b66f4-kj5l9"] Sep 11 16:01:12.818541 ip-10-0-136-96 kubenswrapper[2566]: W0911 16:01:12.818517 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a19c7c5_5d8e_48b1_977a_08b22dbfa94d.slice/crio-2046e7c0ce0acca9d4367225fe4cad0888d141dadae04737291e4e1eede6b35c WatchSource:0}: Error finding container 2046e7c0ce0acca9d4367225fe4cad0888d141dadae04737291e4e1eede6b35c: Status 404 returned error can't find the container with id 2046e7c0ce0acca9d4367225fe4cad0888d141dadae04737291e4e1eede6b35c Sep 11 16:01:13.103215 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:13.103119 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-58647b66f4-kj5l9" event={"ID":"4a19c7c5-5d8e-48b1-977a-08b22dbfa94d","Type":"ContainerStarted","Data":"2046e7c0ce0acca9d4367225fe4cad0888d141dadae04737291e4e1eede6b35c"} Sep 11 16:01:13.105808 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:13.105773 2566 generic.go:356] "Generic (PLEG): container finished" podID="0866e140-3413-47b3-be58-6bfbcc43c487" containerID="9dbc55ea54f0549bf568cdfb4051041910b478b0f5b4f80c21930d8d50e4d3f8" exitCode=0 Sep 11 16:01:13.105935 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:13.105809 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tgr4d" event={"ID":"0866e140-3413-47b3-be58-6bfbcc43c487","Type":"ContainerDied","Data":"9dbc55ea54f0549bf568cdfb4051041910b478b0f5b4f80c21930d8d50e4d3f8"} Sep 11 16:01:14.115859 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:14.115817 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6848676c-8zbl6" event={"ID":"4291750d-8a58-4dfe-b6a7-6d959f0f63cd","Type":"ContainerStarted","Data":"958a3accff6cb240a9145ac3e72ad35259cbd64f6811f32f3bea894f81a6867c"} Sep 11 16:01:14.116350 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:14.115945 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6848676c-8zbl6" Sep 11 16:01:14.271141 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:14.271112 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tgr4d" Sep 11 16:01:14.293503 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:14.293445 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6848676c-8zbl6" podStartSLOduration=2.019513957 podStartE2EDuration="4.29342436s" totalImagesPullingTime="2.273910403s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:01:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:01:14.163675049 +0000 UTC m=+381.378508955" watchObservedRunningTime="2026-09-11 16:01:14.29342436 +0000 UTC m=+381.508258266" Sep 11 16:01:14.338265 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:14.338231 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0866e140-3413-47b3-be58-6bfbcc43c487-util\") pod \"0866e140-3413-47b3-be58-6bfbcc43c487\" (UID: \"0866e140-3413-47b3-be58-6bfbcc43c487\") " Sep 11 16:01:14.338442 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:14.338295 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gp5p4\" (UniqueName: \"kubernetes.io/projected/0866e140-3413-47b3-be58-6bfbcc43c487-kube-api-access-gp5p4\") pod \"0866e140-3413-47b3-be58-6bfbcc43c487\" (UID: \"0866e140-3413-47b3-be58-6bfbcc43c487\") " Sep 11 16:01:14.338442 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:14.338337 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0866e140-3413-47b3-be58-6bfbcc43c487-bundle\") pod \"0866e140-3413-47b3-be58-6bfbcc43c487\" (UID: \"0866e140-3413-47b3-be58-6bfbcc43c487\") " Sep 11 16:01:14.339458 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:14.339426 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0866e140-3413-47b3-be58-6bfbcc43c487-bundle" (OuterVolumeSpecName: "bundle") pod "0866e140-3413-47b3-be58-6bfbcc43c487" (UID: "0866e140-3413-47b3-be58-6bfbcc43c487"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 16:01:14.340890 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:14.340857 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0866e140-3413-47b3-be58-6bfbcc43c487-kube-api-access-gp5p4" (OuterVolumeSpecName: "kube-api-access-gp5p4") pod "0866e140-3413-47b3-be58-6bfbcc43c487" (UID: "0866e140-3413-47b3-be58-6bfbcc43c487"). InnerVolumeSpecName "kube-api-access-gp5p4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 16:01:14.345456 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:14.345429 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0866e140-3413-47b3-be58-6bfbcc43c487-util" (OuterVolumeSpecName: "util") pod "0866e140-3413-47b3-be58-6bfbcc43c487" (UID: "0866e140-3413-47b3-be58-6bfbcc43c487"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 16:01:14.439305 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:14.439211 2566 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0866e140-3413-47b3-be58-6bfbcc43c487-util\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:01:14.439305 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:14.439251 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gp5p4\" (UniqueName: \"kubernetes.io/projected/0866e140-3413-47b3-be58-6bfbcc43c487-kube-api-access-gp5p4\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:01:14.439678 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:14.439269 2566 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0866e140-3413-47b3-be58-6bfbcc43c487-bundle\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:01:15.123731 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:15.123699 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tgr4d" Sep 11 16:01:15.123731 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:15.123705 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0tgr4d" event={"ID":"0866e140-3413-47b3-be58-6bfbcc43c487","Type":"ContainerDied","Data":"40935b4e3b65569aa5e4eab1bda4eaa6cd4ac8557adc47212a4e6a4d93705381"} Sep 11 16:01:15.124277 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:15.123797 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="40935b4e3b65569aa5e4eab1bda4eaa6cd4ac8557adc47212a4e6a4d93705381" Sep 11 16:01:16.131071 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:16.131034 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-58647b66f4-kj5l9" event={"ID":"4a19c7c5-5d8e-48b1-977a-08b22dbfa94d","Type":"ContainerStarted","Data":"744571d5d7b8cee437ab4c79a6e60af9c978200bcebe55b883588f9099a435b3"} Sep 11 16:01:17.138634 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:17.138603 2566 generic.go:356] "Generic (PLEG): container finished" podID="4a19c7c5-5d8e-48b1-977a-08b22dbfa94d" containerID="744571d5d7b8cee437ab4c79a6e60af9c978200bcebe55b883588f9099a435b3" exitCode=0 Sep 11 16:01:17.139050 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:17.138697 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-58647b66f4-kj5l9" event={"ID":"4a19c7c5-5d8e-48b1-977a-08b22dbfa94d","Type":"ContainerDied","Data":"744571d5d7b8cee437ab4c79a6e60af9c978200bcebe55b883588f9099a435b3"} Sep 11 16:01:18.145813 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:18.145709 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-58647b66f4-kj5l9" event={"ID":"4a19c7c5-5d8e-48b1-977a-08b22dbfa94d","Type":"ContainerStarted","Data":"d92d00210ccedceeecf87e9ea60068779335c2e56ab1e4121130149fb3b84490"} Sep 11 16:01:18.146222 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:18.145888 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-58647b66f4-kj5l9" Sep 11 16:01:18.171637 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:18.171584 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-58647b66f4-kj5l9" podStartSLOduration=1.359220291 podStartE2EDuration="6.17156475s" totalImagesPullingTime="3.780289152s" totalInitContainerRuntime="1.032055307s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:01:12 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:01:18.169345541 +0000 UTC m=+385.384179460" watchObservedRunningTime="2026-09-11 16:01:18.17156475 +0000 UTC m=+385.386398680" Sep 11 16:01:25.126219 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:25.126184 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6848676c-8zbl6" Sep 11 16:01:25.266903 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:25.266869 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-84cd566d5d-njpzr"] Sep 11 16:01:25.267262 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:25.267247 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0866e140-3413-47b3-be58-6bfbcc43c487" containerName="extract" Sep 11 16:01:25.267340 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:25.267264 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="0866e140-3413-47b3-be58-6bfbcc43c487" containerName="extract" Sep 11 16:01:25.267340 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:25.267277 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0866e140-3413-47b3-be58-6bfbcc43c487" containerName="pull" Sep 11 16:01:25.267340 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:25.267285 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="0866e140-3413-47b3-be58-6bfbcc43c487" containerName="pull" Sep 11 16:01:25.267340 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:25.267313 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0866e140-3413-47b3-be58-6bfbcc43c487" containerName="util" Sep 11 16:01:25.267340 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:25.267321 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="0866e140-3413-47b3-be58-6bfbcc43c487" containerName="util" Sep 11 16:01:25.267578 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:25.267399 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="0866e140-3413-47b3-be58-6bfbcc43c487" containerName="extract" Sep 11 16:01:25.275179 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:25.275154 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84cd566d5d-njpzr" Sep 11 16:01:25.285702 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:25.285669 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-84cd566d5d-njpzr"] Sep 11 16:01:25.336673 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:25.336635 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5ca9fbab-d9ba-4396-a734-ea2785cfc13d-console-serving-cert\") pod \"console-84cd566d5d-njpzr\" (UID: \"5ca9fbab-d9ba-4396-a734-ea2785cfc13d\") " pod="openshift-console/console-84cd566d5d-njpzr" Sep 11 16:01:25.336835 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:25.336700 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5ca9fbab-d9ba-4396-a734-ea2785cfc13d-console-config\") pod \"console-84cd566d5d-njpzr\" (UID: \"5ca9fbab-d9ba-4396-a734-ea2785cfc13d\") " pod="openshift-console/console-84cd566d5d-njpzr" Sep 11 16:01:25.336835 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:25.336784 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ca9fbab-d9ba-4396-a734-ea2785cfc13d-trusted-ca-bundle\") pod \"console-84cd566d5d-njpzr\" (UID: \"5ca9fbab-d9ba-4396-a734-ea2785cfc13d\") " pod="openshift-console/console-84cd566d5d-njpzr" Sep 11 16:01:25.336990 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:25.336952 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5ca9fbab-d9ba-4396-a734-ea2785cfc13d-console-oauth-config\") pod \"console-84cd566d5d-njpzr\" (UID: \"5ca9fbab-d9ba-4396-a734-ea2785cfc13d\") " pod="openshift-console/console-84cd566d5d-njpzr" Sep 11 16:01:25.337041 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:25.337019 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltv2p\" (UniqueName: \"kubernetes.io/projected/5ca9fbab-d9ba-4396-a734-ea2785cfc13d-kube-api-access-ltv2p\") pod \"console-84cd566d5d-njpzr\" (UID: \"5ca9fbab-d9ba-4396-a734-ea2785cfc13d\") " pod="openshift-console/console-84cd566d5d-njpzr" Sep 11 16:01:25.337118 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:25.337099 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5ca9fbab-d9ba-4396-a734-ea2785cfc13d-oauth-serving-cert\") pod \"console-84cd566d5d-njpzr\" (UID: \"5ca9fbab-d9ba-4396-a734-ea2785cfc13d\") " pod="openshift-console/console-84cd566d5d-njpzr" Sep 11 16:01:25.337189 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:25.337150 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5ca9fbab-d9ba-4396-a734-ea2785cfc13d-service-ca\") pod \"console-84cd566d5d-njpzr\" (UID: \"5ca9fbab-d9ba-4396-a734-ea2785cfc13d\") " pod="openshift-console/console-84cd566d5d-njpzr" Sep 11 16:01:25.438162 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:25.438071 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5ca9fbab-d9ba-4396-a734-ea2785cfc13d-service-ca\") pod \"console-84cd566d5d-njpzr\" (UID: \"5ca9fbab-d9ba-4396-a734-ea2785cfc13d\") " pod="openshift-console/console-84cd566d5d-njpzr" Sep 11 16:01:25.438162 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:25.438128 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5ca9fbab-d9ba-4396-a734-ea2785cfc13d-console-serving-cert\") pod \"console-84cd566d5d-njpzr\" (UID: \"5ca9fbab-d9ba-4396-a734-ea2785cfc13d\") " pod="openshift-console/console-84cd566d5d-njpzr" Sep 11 16:01:25.438387 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:25.438171 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5ca9fbab-d9ba-4396-a734-ea2785cfc13d-console-config\") pod \"console-84cd566d5d-njpzr\" (UID: \"5ca9fbab-d9ba-4396-a734-ea2785cfc13d\") " pod="openshift-console/console-84cd566d5d-njpzr" Sep 11 16:01:25.438387 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:25.438204 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ca9fbab-d9ba-4396-a734-ea2785cfc13d-trusted-ca-bundle\") pod \"console-84cd566d5d-njpzr\" (UID: \"5ca9fbab-d9ba-4396-a734-ea2785cfc13d\") " pod="openshift-console/console-84cd566d5d-njpzr" Sep 11 16:01:25.438387 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:25.438261 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5ca9fbab-d9ba-4396-a734-ea2785cfc13d-console-oauth-config\") pod \"console-84cd566d5d-njpzr\" (UID: \"5ca9fbab-d9ba-4396-a734-ea2785cfc13d\") " pod="openshift-console/console-84cd566d5d-njpzr" Sep 11 16:01:25.438387 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:25.438291 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltv2p\" (UniqueName: \"kubernetes.io/projected/5ca9fbab-d9ba-4396-a734-ea2785cfc13d-kube-api-access-ltv2p\") pod \"console-84cd566d5d-njpzr\" (UID: \"5ca9fbab-d9ba-4396-a734-ea2785cfc13d\") " pod="openshift-console/console-84cd566d5d-njpzr" Sep 11 16:01:25.438387 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:25.438322 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5ca9fbab-d9ba-4396-a734-ea2785cfc13d-oauth-serving-cert\") pod \"console-84cd566d5d-njpzr\" (UID: \"5ca9fbab-d9ba-4396-a734-ea2785cfc13d\") " pod="openshift-console/console-84cd566d5d-njpzr" Sep 11 16:01:25.438961 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:25.438933 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5ca9fbab-d9ba-4396-a734-ea2785cfc13d-service-ca\") pod \"console-84cd566d5d-njpzr\" (UID: \"5ca9fbab-d9ba-4396-a734-ea2785cfc13d\") " pod="openshift-console/console-84cd566d5d-njpzr" Sep 11 16:01:25.439099 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:25.438945 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5ca9fbab-d9ba-4396-a734-ea2785cfc13d-oauth-serving-cert\") pod \"console-84cd566d5d-njpzr\" (UID: \"5ca9fbab-d9ba-4396-a734-ea2785cfc13d\") " pod="openshift-console/console-84cd566d5d-njpzr" Sep 11 16:01:25.439099 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:25.439021 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5ca9fbab-d9ba-4396-a734-ea2785cfc13d-console-config\") pod \"console-84cd566d5d-njpzr\" (UID: \"5ca9fbab-d9ba-4396-a734-ea2785cfc13d\") " pod="openshift-console/console-84cd566d5d-njpzr" Sep 11 16:01:25.443008 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:25.439435 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ca9fbab-d9ba-4396-a734-ea2785cfc13d-trusted-ca-bundle\") pod \"console-84cd566d5d-njpzr\" (UID: \"5ca9fbab-d9ba-4396-a734-ea2785cfc13d\") " pod="openshift-console/console-84cd566d5d-njpzr" Sep 11 16:01:25.444984 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:25.443470 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5ca9fbab-d9ba-4396-a734-ea2785cfc13d-console-oauth-config\") pod \"console-84cd566d5d-njpzr\" (UID: \"5ca9fbab-d9ba-4396-a734-ea2785cfc13d\") " pod="openshift-console/console-84cd566d5d-njpzr" Sep 11 16:01:25.447415 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:25.447391 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5ca9fbab-d9ba-4396-a734-ea2785cfc13d-console-serving-cert\") pod \"console-84cd566d5d-njpzr\" (UID: \"5ca9fbab-d9ba-4396-a734-ea2785cfc13d\") " pod="openshift-console/console-84cd566d5d-njpzr" Sep 11 16:01:25.447745 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:25.447724 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltv2p\" (UniqueName: \"kubernetes.io/projected/5ca9fbab-d9ba-4396-a734-ea2785cfc13d-kube-api-access-ltv2p\") pod \"console-84cd566d5d-njpzr\" (UID: \"5ca9fbab-d9ba-4396-a734-ea2785cfc13d\") " pod="openshift-console/console-84cd566d5d-njpzr" Sep 11 16:01:25.586645 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:25.586601 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84cd566d5d-njpzr" Sep 11 16:01:25.865958 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:25.865927 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-84cd566d5d-njpzr"] Sep 11 16:01:25.867763 ip-10-0-136-96 kubenswrapper[2566]: W0911 16:01:25.867735 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ca9fbab_d9ba_4396_a734_ea2785cfc13d.slice/crio-5ff13280780976ff93000743f150d1c0c7895ad253733ebfe6ab435b72c3ecc9 WatchSource:0}: Error finding container 5ff13280780976ff93000743f150d1c0c7895ad253733ebfe6ab435b72c3ecc9: Status 404 returned error can't find the container with id 5ff13280780976ff93000743f150d1c0c7895ad253733ebfe6ab435b72c3ecc9 Sep 11 16:01:26.186459 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:26.186366 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84cd566d5d-njpzr" event={"ID":"5ca9fbab-d9ba-4396-a734-ea2785cfc13d","Type":"ContainerStarted","Data":"43d961296ec71b669363b06e848d1b9fea02bdba826ca9d479b0be4c1cefee44"} Sep 11 16:01:26.186459 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:26.186417 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84cd566d5d-njpzr" event={"ID":"5ca9fbab-d9ba-4396-a734-ea2785cfc13d","Type":"ContainerStarted","Data":"5ff13280780976ff93000743f150d1c0c7895ad253733ebfe6ab435b72c3ecc9"} Sep 11 16:01:26.204636 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:26.204586 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-84cd566d5d-njpzr" podStartSLOduration=1.204569811 podStartE2EDuration="1.204569811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:01:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:01:26.203467608 +0000 UTC m=+393.418301525" watchObservedRunningTime="2026-09-11 16:01:26.204569811 +0000 UTC m=+393.419403716" Sep 11 16:01:29.153712 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:29.153679 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-58647b66f4-kj5l9" Sep 11 16:01:35.587171 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:35.587130 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-84cd566d5d-njpzr" Sep 11 16:01:35.587626 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:35.587213 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-84cd566d5d-njpzr" Sep 11 16:01:35.592207 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:35.592185 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-84cd566d5d-njpzr" Sep 11 16:01:36.240392 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:36.240362 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-84cd566d5d-njpzr" Sep 11 16:01:36.289458 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:36.289418 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-c5ff7cbc8-wcdds"] Sep 11 16:01:39.637627 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:39.637592 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zdlss"] Sep 11 16:01:39.644259 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:39.644238 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zdlss" Sep 11 16:01:39.647194 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:39.647171 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-49kqn\"" Sep 11 16:01:39.647517 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:39.647503 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 11 16:01:39.670997 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:39.670951 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zdlss"] Sep 11 16:01:39.769258 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:39.769214 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cd096b0-a02a-4cc7-accd-7986120b8a76-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zdlss\" (UID: \"1cd096b0-a02a-4cc7-accd-7986120b8a76\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zdlss" Sep 11 16:01:39.769258 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:39.769258 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cd096b0-a02a-4cc7-accd-7986120b8a76-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zdlss\" (UID: \"1cd096b0-a02a-4cc7-accd-7986120b8a76\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zdlss" Sep 11 16:01:39.769481 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:39.769281 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cd096b0-a02a-4cc7-accd-7986120b8a76-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zdlss\" (UID: \"1cd096b0-a02a-4cc7-accd-7986120b8a76\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zdlss" Sep 11 16:01:39.769481 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:39.769335 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9fld\" (UniqueName: \"kubernetes.io/projected/1cd096b0-a02a-4cc7-accd-7986120b8a76-kube-api-access-m9fld\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zdlss\" (UID: \"1cd096b0-a02a-4cc7-accd-7986120b8a76\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zdlss" Sep 11 16:01:39.769481 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:39.769387 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cd096b0-a02a-4cc7-accd-7986120b8a76-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zdlss\" (UID: \"1cd096b0-a02a-4cc7-accd-7986120b8a76\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zdlss" Sep 11 16:01:39.769481 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:39.769407 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cd096b0-a02a-4cc7-accd-7986120b8a76-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zdlss\" (UID: \"1cd096b0-a02a-4cc7-accd-7986120b8a76\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zdlss" Sep 11 16:01:39.769481 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:39.769479 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cd096b0-a02a-4cc7-accd-7986120b8a76-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zdlss\" (UID: \"1cd096b0-a02a-4cc7-accd-7986120b8a76\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zdlss" Sep 11 16:01:39.769661 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:39.769500 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cd096b0-a02a-4cc7-accd-7986120b8a76-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zdlss\" (UID: \"1cd096b0-a02a-4cc7-accd-7986120b8a76\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zdlss" Sep 11 16:01:39.769661 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:39.769520 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cd096b0-a02a-4cc7-accd-7986120b8a76-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zdlss\" (UID: \"1cd096b0-a02a-4cc7-accd-7986120b8a76\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zdlss" Sep 11 16:01:39.870157 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:39.870122 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cd096b0-a02a-4cc7-accd-7986120b8a76-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zdlss\" (UID: \"1cd096b0-a02a-4cc7-accd-7986120b8a76\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zdlss" Sep 11 16:01:39.870157 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:39.870158 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cd096b0-a02a-4cc7-accd-7986120b8a76-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zdlss\" (UID: \"1cd096b0-a02a-4cc7-accd-7986120b8a76\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zdlss" Sep 11 16:01:39.870414 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:39.870179 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cd096b0-a02a-4cc7-accd-7986120b8a76-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zdlss\" (UID: \"1cd096b0-a02a-4cc7-accd-7986120b8a76\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zdlss" Sep 11 16:01:39.870414 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:39.870204 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9fld\" (UniqueName: \"kubernetes.io/projected/1cd096b0-a02a-4cc7-accd-7986120b8a76-kube-api-access-m9fld\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zdlss\" (UID: \"1cd096b0-a02a-4cc7-accd-7986120b8a76\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zdlss" Sep 11 16:01:39.870414 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:39.870255 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cd096b0-a02a-4cc7-accd-7986120b8a76-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zdlss\" (UID: \"1cd096b0-a02a-4cc7-accd-7986120b8a76\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zdlss" Sep 11 16:01:39.870414 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:39.870284 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cd096b0-a02a-4cc7-accd-7986120b8a76-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zdlss\" (UID: \"1cd096b0-a02a-4cc7-accd-7986120b8a76\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zdlss" Sep 11 16:01:39.870414 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:39.870353 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cd096b0-a02a-4cc7-accd-7986120b8a76-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zdlss\" (UID: \"1cd096b0-a02a-4cc7-accd-7986120b8a76\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zdlss" Sep 11 16:01:39.870414 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:39.870379 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cd096b0-a02a-4cc7-accd-7986120b8a76-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zdlss\" (UID: \"1cd096b0-a02a-4cc7-accd-7986120b8a76\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zdlss" Sep 11 16:01:39.870414 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:39.870404 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cd096b0-a02a-4cc7-accd-7986120b8a76-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zdlss\" (UID: \"1cd096b0-a02a-4cc7-accd-7986120b8a76\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zdlss" Sep 11 16:01:39.870903 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:39.870707 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cd096b0-a02a-4cc7-accd-7986120b8a76-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zdlss\" (UID: \"1cd096b0-a02a-4cc7-accd-7986120b8a76\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zdlss" Sep 11 16:01:39.870903 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:39.870779 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cd096b0-a02a-4cc7-accd-7986120b8a76-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zdlss\" (UID: \"1cd096b0-a02a-4cc7-accd-7986120b8a76\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zdlss" Sep 11 16:01:39.870903 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:39.870879 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cd096b0-a02a-4cc7-accd-7986120b8a76-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zdlss\" (UID: \"1cd096b0-a02a-4cc7-accd-7986120b8a76\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zdlss" Sep 11 16:01:39.871052 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:39.870946 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cd096b0-a02a-4cc7-accd-7986120b8a76-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zdlss\" (UID: \"1cd096b0-a02a-4cc7-accd-7986120b8a76\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zdlss" Sep 11 16:01:39.871115 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:39.871097 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cd096b0-a02a-4cc7-accd-7986120b8a76-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zdlss\" (UID: \"1cd096b0-a02a-4cc7-accd-7986120b8a76\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zdlss" Sep 11 16:01:39.872545 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:39.872528 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cd096b0-a02a-4cc7-accd-7986120b8a76-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zdlss\" (UID: \"1cd096b0-a02a-4cc7-accd-7986120b8a76\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zdlss" Sep 11 16:01:39.872812 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:39.872793 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cd096b0-a02a-4cc7-accd-7986120b8a76-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zdlss\" (UID: \"1cd096b0-a02a-4cc7-accd-7986120b8a76\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zdlss" Sep 11 16:01:39.878200 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:39.878178 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9fld\" (UniqueName: \"kubernetes.io/projected/1cd096b0-a02a-4cc7-accd-7986120b8a76-kube-api-access-m9fld\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zdlss\" (UID: \"1cd096b0-a02a-4cc7-accd-7986120b8a76\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zdlss" Sep 11 16:01:39.880141 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:39.880110 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cd096b0-a02a-4cc7-accd-7986120b8a76-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-zdlss\" (UID: \"1cd096b0-a02a-4cc7-accd-7986120b8a76\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zdlss" Sep 11 16:01:39.955438 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:39.955349 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zdlss" Sep 11 16:01:40.235925 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:40.235899 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zdlss"] Sep 11 16:01:40.237503 ip-10-0-136-96 kubenswrapper[2566]: W0911 16:01:40.237474 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cd096b0_a02a_4cc7_accd_7986120b8a76.slice/crio-aa0b894f8307f35c1da3ee06b896c66e3e133bf1275047393a9d6abc6b7bfbfe WatchSource:0}: Error finding container aa0b894f8307f35c1da3ee06b896c66e3e133bf1275047393a9d6abc6b7bfbfe: Status 404 returned error can't find the container with id aa0b894f8307f35c1da3ee06b896c66e3e133bf1275047393a9d6abc6b7bfbfe Sep 11 16:01:40.259111 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:40.259081 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zdlss" event={"ID":"1cd096b0-a02a-4cc7-accd-7986120b8a76","Type":"ContainerStarted","Data":"aa0b894f8307f35c1da3ee06b896c66e3e133bf1275047393a9d6abc6b7bfbfe"} Sep 11 16:01:42.932514 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:42.932474 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 16:01:42.932784 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:42.932547 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 16:01:42.932784 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:42.932576 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 16:01:43.241470 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:43.241382 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-rmzgg"] Sep 11 16:01:43.244024 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:43.244003 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-rmzgg" Sep 11 16:01:43.248068 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:43.248041 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Sep 11 16:01:43.256941 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:43.256909 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-rmzgg"] Sep 11 16:01:43.278323 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:43.278284 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zdlss" event={"ID":"1cd096b0-a02a-4cc7-accd-7986120b8a76","Type":"ContainerStarted","Data":"01aa5d1ab1299632834dc1fb3eba4d985b44a111fcb878058ca1538f0bc58100"} Sep 11 16:01:43.307872 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:43.307796 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zdlss" podStartSLOduration=1.6150557 podStartE2EDuration="4.307773472s" totalImagesPullingTime="2.692717772s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:01:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:01:43.304457769 +0000 UTC m=+410.519291674" watchObservedRunningTime="2026-09-11 16:01:43.307773472 +0000 UTC m=+410.522607379" Sep 11 16:01:43.404826 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:43.404788 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25f4149c-81d0-4345-94cf-2d650d114a4e-tls-certs\") pod \"model-serving-api-54d89bd5fc-rmzgg\" (UID: \"25f4149c-81d0-4345-94cf-2d650d114a4e\") " pod="opendatahub/model-serving-api-54d89bd5fc-rmzgg" Sep 11 16:01:43.405022 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:43.404871 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sknfz\" (UniqueName: \"kubernetes.io/projected/25f4149c-81d0-4345-94cf-2d650d114a4e-kube-api-access-sknfz\") pod \"model-serving-api-54d89bd5fc-rmzgg\" (UID: \"25f4149c-81d0-4345-94cf-2d650d114a4e\") " pod="opendatahub/model-serving-api-54d89bd5fc-rmzgg" Sep 11 16:01:43.505639 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:43.505548 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25f4149c-81d0-4345-94cf-2d650d114a4e-tls-certs\") pod \"model-serving-api-54d89bd5fc-rmzgg\" (UID: \"25f4149c-81d0-4345-94cf-2d650d114a4e\") " pod="opendatahub/model-serving-api-54d89bd5fc-rmzgg" Sep 11 16:01:43.505639 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:43.505622 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sknfz\" (UniqueName: \"kubernetes.io/projected/25f4149c-81d0-4345-94cf-2d650d114a4e-kube-api-access-sknfz\") pod \"model-serving-api-54d89bd5fc-rmzgg\" (UID: \"25f4149c-81d0-4345-94cf-2d650d114a4e\") " pod="opendatahub/model-serving-api-54d89bd5fc-rmzgg" Sep 11 16:01:43.508144 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:43.508124 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25f4149c-81d0-4345-94cf-2d650d114a4e-tls-certs\") pod \"model-serving-api-54d89bd5fc-rmzgg\" (UID: \"25f4149c-81d0-4345-94cf-2d650d114a4e\") " pod="opendatahub/model-serving-api-54d89bd5fc-rmzgg" Sep 11 16:01:43.515264 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:43.515238 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sknfz\" (UniqueName: \"kubernetes.io/projected/25f4149c-81d0-4345-94cf-2d650d114a4e-kube-api-access-sknfz\") pod \"model-serving-api-54d89bd5fc-rmzgg\" (UID: \"25f4149c-81d0-4345-94cf-2d650d114a4e\") " pod="opendatahub/model-serving-api-54d89bd5fc-rmzgg" Sep 11 16:01:43.554482 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:43.554450 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-rmzgg" Sep 11 16:01:43.829660 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:43.829629 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-rmzgg"] Sep 11 16:01:43.832925 ip-10-0-136-96 kubenswrapper[2566]: W0911 16:01:43.832899 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25f4149c_81d0_4345_94cf_2d650d114a4e.slice/crio-0dea7f7a7f80c03ba75a54a0974d265e3515338a3c20d4e9dd676c6bf79cae11 WatchSource:0}: Error finding container 0dea7f7a7f80c03ba75a54a0974d265e3515338a3c20d4e9dd676c6bf79cae11: Status 404 returned error can't find the container with id 0dea7f7a7f80c03ba75a54a0974d265e3515338a3c20d4e9dd676c6bf79cae11 Sep 11 16:01:43.956042 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:43.956007 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zdlss" Sep 11 16:01:43.960703 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:43.960678 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zdlss" Sep 11 16:01:44.285167 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:44.285134 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-rmzgg" event={"ID":"25f4149c-81d0-4345-94cf-2d650d114a4e","Type":"ContainerStarted","Data":"0dea7f7a7f80c03ba75a54a0974d265e3515338a3c20d4e9dd676c6bf79cae11"} Sep 11 16:01:44.285434 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:44.285416 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zdlss" Sep 11 16:01:44.286342 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:44.286326 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-zdlss" Sep 11 16:01:47.305252 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:47.305211 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-rmzgg" event={"ID":"25f4149c-81d0-4345-94cf-2d650d114a4e","Type":"ContainerStarted","Data":"aeb85f78c63b1fa56b0a57bf2837c39eb211c762a4f426ca38de95ef4aca4fa3"} Sep 11 16:01:47.305629 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:47.305291 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-rmzgg" Sep 11 16:01:47.324697 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:47.324644 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-rmzgg" podStartSLOduration=1.5841103410000001 podStartE2EDuration="4.324621676s" totalImagesPullingTime="2.740511335s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:01:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:01:47.322918845 +0000 UTC m=+414.537752751" watchObservedRunningTime="2026-09-11 16:01:47.324621676 +0000 UTC m=+414.539455581" Sep 11 16:01:52.885663 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:52.885619 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-48gfx"] Sep 11 16:01:52.888325 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:52.888305 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-48gfx" Sep 11 16:01:52.893108 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:52.893085 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Sep 11 16:01:52.893564 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:52.893544 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-s9668\"" Sep 11 16:01:52.900333 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:52.900310 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-48gfx"] Sep 11 16:01:52.981856 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:52.981821 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tbbk\" (UniqueName: \"kubernetes.io/projected/bfd28efb-548a-4ed8-a6c5-2bd79df593ab-kube-api-access-8tbbk\") pod \"llmisvc-controller-manager-777c6bd7d5-48gfx\" (UID: \"bfd28efb-548a-4ed8-a6c5-2bd79df593ab\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-48gfx" Sep 11 16:01:52.982070 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:52.981892 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bfd28efb-548a-4ed8-a6c5-2bd79df593ab-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-48gfx\" (UID: \"bfd28efb-548a-4ed8-a6c5-2bd79df593ab\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-48gfx" Sep 11 16:01:53.083256 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:53.083203 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tbbk\" (UniqueName: \"kubernetes.io/projected/bfd28efb-548a-4ed8-a6c5-2bd79df593ab-kube-api-access-8tbbk\") pod \"llmisvc-controller-manager-777c6bd7d5-48gfx\" (UID: \"bfd28efb-548a-4ed8-a6c5-2bd79df593ab\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-48gfx" Sep 11 16:01:53.083455 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:53.083315 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bfd28efb-548a-4ed8-a6c5-2bd79df593ab-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-48gfx\" (UID: \"bfd28efb-548a-4ed8-a6c5-2bd79df593ab\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-48gfx" Sep 11 16:01:53.085749 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:53.085717 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bfd28efb-548a-4ed8-a6c5-2bd79df593ab-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-48gfx\" (UID: \"bfd28efb-548a-4ed8-a6c5-2bd79df593ab\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-48gfx" Sep 11 16:01:53.094328 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:53.094301 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tbbk\" (UniqueName: \"kubernetes.io/projected/bfd28efb-548a-4ed8-a6c5-2bd79df593ab-kube-api-access-8tbbk\") pod \"llmisvc-controller-manager-777c6bd7d5-48gfx\" (UID: \"bfd28efb-548a-4ed8-a6c5-2bd79df593ab\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-48gfx" Sep 11 16:01:53.202449 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:53.202377 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-s9668\"" Sep 11 16:01:53.210297 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:53.210271 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-48gfx" Sep 11 16:01:53.486557 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:53.486471 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-48gfx"] Sep 11 16:01:53.489521 ip-10-0-136-96 kubenswrapper[2566]: W0911 16:01:53.489497 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podbfd28efb_548a_4ed8_a6c5_2bd79df593ab.slice/crio-556a9d7fe5db6134880a480a1eeae807044162196e16ac82085bf8fb1c949996 WatchSource:0}: Error finding container 556a9d7fe5db6134880a480a1eeae807044162196e16ac82085bf8fb1c949996: Status 404 returned error can't find the container with id 556a9d7fe5db6134880a480a1eeae807044162196e16ac82085bf8fb1c949996 Sep 11 16:01:54.344674 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:54.344629 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-48gfx" event={"ID":"bfd28efb-548a-4ed8-a6c5-2bd79df593ab","Type":"ContainerStarted","Data":"556a9d7fe5db6134880a480a1eeae807044162196e16ac82085bf8fb1c949996"} Sep 11 16:01:57.363208 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:57.363169 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-48gfx" event={"ID":"bfd28efb-548a-4ed8-a6c5-2bd79df593ab","Type":"ContainerStarted","Data":"e0b4da581aaa7aad29f809b13a3364f5b24de99bc2c635df340a466802fa75a4"} Sep 11 16:01:57.363604 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:57.363302 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-48gfx" Sep 11 16:01:57.385086 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:57.385030 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-48gfx" podStartSLOduration=2.094057576 podStartE2EDuration="5.385013727s" totalImagesPullingTime="3.290956151s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:01:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:01:57.382831102 +0000 UTC m=+424.597665008" watchObservedRunningTime="2026-09-11 16:01:57.385013727 +0000 UTC m=+424.599847633" Sep 11 16:01:58.313568 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:01:58.313539 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-rmzgg" Sep 11 16:02:01.311773 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:01.311709 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-c5ff7cbc8-wcdds" podUID="29a58f59-f953-4969-8c21-24c4ab762364" containerName="console" containerID="cri-o://2206202e0d916c0743b492ffe0022bba22a184e33d7178b16ffaa4aaac9432cc" gracePeriod=15 Sep 11 16:02:01.567934 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:01.567869 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c5ff7cbc8-wcdds" Sep 11 16:02:01.670957 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:01.670916 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29a58f59-f953-4969-8c21-24c4ab762364-trusted-ca-bundle\") pod \"29a58f59-f953-4969-8c21-24c4ab762364\" (UID: \"29a58f59-f953-4969-8c21-24c4ab762364\") " Sep 11 16:02:01.671157 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:01.671029 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/29a58f59-f953-4969-8c21-24c4ab762364-oauth-serving-cert\") pod \"29a58f59-f953-4969-8c21-24c4ab762364\" (UID: \"29a58f59-f953-4969-8c21-24c4ab762364\") " Sep 11 16:02:01.671157 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:01.671073 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/29a58f59-f953-4969-8c21-24c4ab762364-console-oauth-config\") pod \"29a58f59-f953-4969-8c21-24c4ab762364\" (UID: \"29a58f59-f953-4969-8c21-24c4ab762364\") " Sep 11 16:02:01.671157 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:01.671101 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/29a58f59-f953-4969-8c21-24c4ab762364-console-config\") pod \"29a58f59-f953-4969-8c21-24c4ab762364\" (UID: \"29a58f59-f953-4969-8c21-24c4ab762364\") " Sep 11 16:02:01.671157 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:01.671125 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/29a58f59-f953-4969-8c21-24c4ab762364-service-ca\") pod \"29a58f59-f953-4969-8c21-24c4ab762364\" (UID: \"29a58f59-f953-4969-8c21-24c4ab762364\") " Sep 11 16:02:01.671157 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:01.671150 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/29a58f59-f953-4969-8c21-24c4ab762364-console-serving-cert\") pod \"29a58f59-f953-4969-8c21-24c4ab762364\" (UID: \"29a58f59-f953-4969-8c21-24c4ab762364\") " Sep 11 16:02:01.671434 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:01.671205 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jdcjz\" (UniqueName: \"kubernetes.io/projected/29a58f59-f953-4969-8c21-24c4ab762364-kube-api-access-jdcjz\") pod \"29a58f59-f953-4969-8c21-24c4ab762364\" (UID: \"29a58f59-f953-4969-8c21-24c4ab762364\") " Sep 11 16:02:01.671562 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:01.671517 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29a58f59-f953-4969-8c21-24c4ab762364-console-config" (OuterVolumeSpecName: "console-config") pod "29a58f59-f953-4969-8c21-24c4ab762364" (UID: "29a58f59-f953-4969-8c21-24c4ab762364"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 16:02:01.671562 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:01.671554 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29a58f59-f953-4969-8c21-24c4ab762364-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "29a58f59-f953-4969-8c21-24c4ab762364" (UID: "29a58f59-f953-4969-8c21-24c4ab762364"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 16:02:01.671682 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:01.671526 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29a58f59-f953-4969-8c21-24c4ab762364-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "29a58f59-f953-4969-8c21-24c4ab762364" (UID: "29a58f59-f953-4969-8c21-24c4ab762364"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 16:02:01.671682 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:01.671598 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29a58f59-f953-4969-8c21-24c4ab762364-service-ca" (OuterVolumeSpecName: "service-ca") pod "29a58f59-f953-4969-8c21-24c4ab762364" (UID: "29a58f59-f953-4969-8c21-24c4ab762364"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 16:02:01.673353 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:01.673325 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29a58f59-f953-4969-8c21-24c4ab762364-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "29a58f59-f953-4969-8c21-24c4ab762364" (UID: "29a58f59-f953-4969-8c21-24c4ab762364"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 16:02:01.673445 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:01.673405 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29a58f59-f953-4969-8c21-24c4ab762364-kube-api-access-jdcjz" (OuterVolumeSpecName: "kube-api-access-jdcjz") pod "29a58f59-f953-4969-8c21-24c4ab762364" (UID: "29a58f59-f953-4969-8c21-24c4ab762364"). InnerVolumeSpecName "kube-api-access-jdcjz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 16:02:01.673484 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:01.673446 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29a58f59-f953-4969-8c21-24c4ab762364-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "29a58f59-f953-4969-8c21-24c4ab762364" (UID: "29a58f59-f953-4969-8c21-24c4ab762364"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 16:02:01.772220 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:01.772180 2566 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/29a58f59-f953-4969-8c21-24c4ab762364-oauth-serving-cert\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:02:01.772220 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:01.772213 2566 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/29a58f59-f953-4969-8c21-24c4ab762364-console-oauth-config\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:02:01.772220 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:01.772223 2566 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/29a58f59-f953-4969-8c21-24c4ab762364-console-config\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:02:01.772220 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:01.772233 2566 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/29a58f59-f953-4969-8c21-24c4ab762364-service-ca\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:02:01.772492 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:01.772241 2566 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/29a58f59-f953-4969-8c21-24c4ab762364-console-serving-cert\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:02:01.772492 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:01.772250 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdcjz\" (UniqueName: \"kubernetes.io/projected/29a58f59-f953-4969-8c21-24c4ab762364-kube-api-access-jdcjz\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:02:01.772492 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:01.772259 2566 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29a58f59-f953-4969-8c21-24c4ab762364-trusted-ca-bundle\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:02:02.392654 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:02.392610 2566 generic.go:356] "Generic (PLEG): container finished" podID="29a58f59-f953-4969-8c21-24c4ab762364" containerID="2206202e0d916c0743b492ffe0022bba22a184e33d7178b16ffaa4aaac9432cc" exitCode=0 Sep 11 16:02:02.393133 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:02.392662 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c5ff7cbc8-wcdds" Sep 11 16:02:02.393133 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:02.392670 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c5ff7cbc8-wcdds" event={"ID":"29a58f59-f953-4969-8c21-24c4ab762364","Type":"ContainerDied","Data":"2206202e0d916c0743b492ffe0022bba22a184e33d7178b16ffaa4aaac9432cc"} Sep 11 16:02:02.393133 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:02.392716 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c5ff7cbc8-wcdds" event={"ID":"29a58f59-f953-4969-8c21-24c4ab762364","Type":"ContainerDied","Data":"d4f4f38d42d86c8005b6b2d9a986b798933f340a6834dcb35777a2a7208ec8f7"} Sep 11 16:02:02.393133 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:02.392742 2566 scope.go:117] "RemoveContainer" containerID="2206202e0d916c0743b492ffe0022bba22a184e33d7178b16ffaa4aaac9432cc" Sep 11 16:02:02.402263 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:02.402242 2566 scope.go:117] "RemoveContainer" containerID="2206202e0d916c0743b492ffe0022bba22a184e33d7178b16ffaa4aaac9432cc" Sep 11 16:02:02.402552 ip-10-0-136-96 kubenswrapper[2566]: E0911 16:02:02.402534 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2206202e0d916c0743b492ffe0022bba22a184e33d7178b16ffaa4aaac9432cc\": container with ID starting with 2206202e0d916c0743b492ffe0022bba22a184e33d7178b16ffaa4aaac9432cc not found: ID does not exist" containerID="2206202e0d916c0743b492ffe0022bba22a184e33d7178b16ffaa4aaac9432cc" Sep 11 16:02:02.402621 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:02.402558 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2206202e0d916c0743b492ffe0022bba22a184e33d7178b16ffaa4aaac9432cc"} err="failed to get container status \"2206202e0d916c0743b492ffe0022bba22a184e33d7178b16ffaa4aaac9432cc\": rpc error: code = NotFound desc = could not find container \"2206202e0d916c0743b492ffe0022bba22a184e33d7178b16ffaa4aaac9432cc\": container with ID starting with 2206202e0d916c0743b492ffe0022bba22a184e33d7178b16ffaa4aaac9432cc not found: ID does not exist" Sep 11 16:02:02.417213 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:02.417175 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-c5ff7cbc8-wcdds"] Sep 11 16:02:02.419687 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:02.419653 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-c5ff7cbc8-wcdds"] Sep 11 16:02:03.238208 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:03.238171 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="29a58f59-f953-4969-8c21-24c4ab762364" path="/var/lib/kubelet/pods/29a58f59-f953-4969-8c21-24c4ab762364/volumes" Sep 11 16:02:28.371017 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:28.370954 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-48gfx" Sep 11 16:02:48.426751 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:48.426715 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccpxn7"] Sep 11 16:02:48.427293 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:48.427093 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29a58f59-f953-4969-8c21-24c4ab762364" containerName="console" Sep 11 16:02:48.427293 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:48.427105 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="29a58f59-f953-4969-8c21-24c4ab762364" containerName="console" Sep 11 16:02:48.427293 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:48.427178 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="29a58f59-f953-4969-8c21-24c4ab762364" containerName="console" Sep 11 16:02:48.430374 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:48.430358 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccpxn7" Sep 11 16:02:48.433064 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:48.433041 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 11 16:02:48.433182 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:48.433093 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-z84gm\"" Sep 11 16:02:48.433182 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:48.433166 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 11 16:02:48.439105 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:48.439080 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccpxn7"] Sep 11 16:02:48.493038 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:48.493000 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/20564014-116a-42f9-9b4b-30bbc7cbc6a8-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccpxn7\" (UID: \"20564014-116a-42f9-9b4b-30bbc7cbc6a8\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccpxn7" Sep 11 16:02:48.493219 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:48.493047 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-466hh\" (UniqueName: \"kubernetes.io/projected/20564014-116a-42f9-9b4b-30bbc7cbc6a8-kube-api-access-466hh\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccpxn7\" (UID: \"20564014-116a-42f9-9b4b-30bbc7cbc6a8\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccpxn7" Sep 11 16:02:48.493219 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:48.493136 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/20564014-116a-42f9-9b4b-30bbc7cbc6a8-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccpxn7\" (UID: \"20564014-116a-42f9-9b4b-30bbc7cbc6a8\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccpxn7" Sep 11 16:02:48.521601 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:48.521555 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awl285"] Sep 11 16:02:48.525349 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:48.525323 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awl285" Sep 11 16:02:48.535523 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:48.535491 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awl285"] Sep 11 16:02:48.593847 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:48.593810 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/20564014-116a-42f9-9b4b-30bbc7cbc6a8-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccpxn7\" (UID: \"20564014-116a-42f9-9b4b-30bbc7cbc6a8\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccpxn7" Sep 11 16:02:48.593847 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:48.593853 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/20564014-116a-42f9-9b4b-30bbc7cbc6a8-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccpxn7\" (UID: \"20564014-116a-42f9-9b4b-30bbc7cbc6a8\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccpxn7" Sep 11 16:02:48.594127 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:48.593886 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/06078c33-7ab5-4d1a-94af-c98c31b080b2-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awl285\" (UID: \"06078c33-7ab5-4d1a-94af-c98c31b080b2\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awl285" Sep 11 16:02:48.594127 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:48.593910 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-466hh\" (UniqueName: \"kubernetes.io/projected/20564014-116a-42f9-9b4b-30bbc7cbc6a8-kube-api-access-466hh\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccpxn7\" (UID: \"20564014-116a-42f9-9b4b-30bbc7cbc6a8\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccpxn7" Sep 11 16:02:48.594127 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:48.593927 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/06078c33-7ab5-4d1a-94af-c98c31b080b2-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awl285\" (UID: \"06078c33-7ab5-4d1a-94af-c98c31b080b2\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awl285" Sep 11 16:02:48.594127 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:48.594003 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6f67\" (UniqueName: \"kubernetes.io/projected/06078c33-7ab5-4d1a-94af-c98c31b080b2-kube-api-access-k6f67\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awl285\" (UID: \"06078c33-7ab5-4d1a-94af-c98c31b080b2\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awl285" Sep 11 16:02:48.594314 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:48.594231 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/20564014-116a-42f9-9b4b-30bbc7cbc6a8-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccpxn7\" (UID: \"20564014-116a-42f9-9b4b-30bbc7cbc6a8\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccpxn7" Sep 11 16:02:48.594314 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:48.594279 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/20564014-116a-42f9-9b4b-30bbc7cbc6a8-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccpxn7\" (UID: \"20564014-116a-42f9-9b4b-30bbc7cbc6a8\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccpxn7" Sep 11 16:02:48.604337 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:48.604307 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-466hh\" (UniqueName: \"kubernetes.io/projected/20564014-116a-42f9-9b4b-30bbc7cbc6a8-kube-api-access-466hh\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccpxn7\" (UID: \"20564014-116a-42f9-9b4b-30bbc7cbc6a8\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccpxn7" Sep 11 16:02:48.630152 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:48.630114 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wcns4"] Sep 11 16:02:48.634132 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:48.634112 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wcns4" Sep 11 16:02:48.641891 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:48.641767 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wcns4"] Sep 11 16:02:48.695442 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:48.695343 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/06078c33-7ab5-4d1a-94af-c98c31b080b2-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awl285\" (UID: \"06078c33-7ab5-4d1a-94af-c98c31b080b2\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awl285" Sep 11 16:02:48.695442 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:48.695386 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfvj6\" (UniqueName: \"kubernetes.io/projected/c0ab286c-ffb5-4656-9c6c-548589cf2508-kube-api-access-vfvj6\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wcns4\" (UID: \"c0ab286c-ffb5-4656-9c6c-548589cf2508\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wcns4" Sep 11 16:02:48.695442 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:48.695409 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/06078c33-7ab5-4d1a-94af-c98c31b080b2-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awl285\" (UID: \"06078c33-7ab5-4d1a-94af-c98c31b080b2\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awl285" Sep 11 16:02:48.695442 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:48.695428 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c0ab286c-ffb5-4656-9c6c-548589cf2508-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wcns4\" (UID: \"c0ab286c-ffb5-4656-9c6c-548589cf2508\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wcns4" Sep 11 16:02:48.695800 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:48.695475 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6f67\" (UniqueName: \"kubernetes.io/projected/06078c33-7ab5-4d1a-94af-c98c31b080b2-kube-api-access-k6f67\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awl285\" (UID: \"06078c33-7ab5-4d1a-94af-c98c31b080b2\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awl285" Sep 11 16:02:48.695800 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:48.695501 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c0ab286c-ffb5-4656-9c6c-548589cf2508-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wcns4\" (UID: \"c0ab286c-ffb5-4656-9c6c-548589cf2508\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wcns4" Sep 11 16:02:48.695800 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:48.695741 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/06078c33-7ab5-4d1a-94af-c98c31b080b2-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awl285\" (UID: \"06078c33-7ab5-4d1a-94af-c98c31b080b2\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awl285" Sep 11 16:02:48.695923 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:48.695811 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/06078c33-7ab5-4d1a-94af-c98c31b080b2-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awl285\" (UID: \"06078c33-7ab5-4d1a-94af-c98c31b080b2\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awl285" Sep 11 16:02:48.713685 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:48.713654 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6f67\" (UniqueName: \"kubernetes.io/projected/06078c33-7ab5-4d1a-94af-c98c31b080b2-kube-api-access-k6f67\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awl285\" (UID: \"06078c33-7ab5-4d1a-94af-c98c31b080b2\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awl285" Sep 11 16:02:48.741487 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:48.741447 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccpxn7" Sep 11 16:02:48.747009 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:48.746964 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xz9r8"] Sep 11 16:02:48.750900 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:48.750879 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xz9r8" Sep 11 16:02:48.767162 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:48.767126 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xz9r8"] Sep 11 16:02:48.796721 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:48.796685 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3cf62ea0-4df7-436a-83d0-3d1174886195-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xz9r8\" (UID: \"3cf62ea0-4df7-436a-83d0-3d1174886195\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xz9r8" Sep 11 16:02:48.796817 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:48.796759 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3cf62ea0-4df7-436a-83d0-3d1174886195-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xz9r8\" (UID: \"3cf62ea0-4df7-436a-83d0-3d1174886195\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xz9r8" Sep 11 16:02:48.796888 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:48.796823 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfvj6\" (UniqueName: \"kubernetes.io/projected/c0ab286c-ffb5-4656-9c6c-548589cf2508-kube-api-access-vfvj6\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wcns4\" (UID: \"c0ab286c-ffb5-4656-9c6c-548589cf2508\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wcns4" Sep 11 16:02:48.796888 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:48.796855 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmzgw\" (UniqueName: \"kubernetes.io/projected/3cf62ea0-4df7-436a-83d0-3d1174886195-kube-api-access-gmzgw\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xz9r8\" (UID: \"3cf62ea0-4df7-436a-83d0-3d1174886195\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xz9r8" Sep 11 16:02:48.797068 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:48.796892 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c0ab286c-ffb5-4656-9c6c-548589cf2508-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wcns4\" (UID: \"c0ab286c-ffb5-4656-9c6c-548589cf2508\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wcns4" Sep 11 16:02:48.797068 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:48.796964 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c0ab286c-ffb5-4656-9c6c-548589cf2508-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wcns4\" (UID: \"c0ab286c-ffb5-4656-9c6c-548589cf2508\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wcns4" Sep 11 16:02:48.797511 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:48.797471 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c0ab286c-ffb5-4656-9c6c-548589cf2508-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wcns4\" (UID: \"c0ab286c-ffb5-4656-9c6c-548589cf2508\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wcns4" Sep 11 16:02:48.797760 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:48.797738 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c0ab286c-ffb5-4656-9c6c-548589cf2508-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wcns4\" (UID: \"c0ab286c-ffb5-4656-9c6c-548589cf2508\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wcns4" Sep 11 16:02:48.813235 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:48.813158 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfvj6\" (UniqueName: \"kubernetes.io/projected/c0ab286c-ffb5-4656-9c6c-548589cf2508-kube-api-access-vfvj6\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wcns4\" (UID: \"c0ab286c-ffb5-4656-9c6c-548589cf2508\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wcns4" Sep 11 16:02:48.840339 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:48.840307 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awl285" Sep 11 16:02:48.898246 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:48.898198 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3cf62ea0-4df7-436a-83d0-3d1174886195-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xz9r8\" (UID: \"3cf62ea0-4df7-436a-83d0-3d1174886195\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xz9r8" Sep 11 16:02:48.898388 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:48.898290 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmzgw\" (UniqueName: \"kubernetes.io/projected/3cf62ea0-4df7-436a-83d0-3d1174886195-kube-api-access-gmzgw\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xz9r8\" (UID: \"3cf62ea0-4df7-436a-83d0-3d1174886195\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xz9r8" Sep 11 16:02:48.898452 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:48.898402 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3cf62ea0-4df7-436a-83d0-3d1174886195-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xz9r8\" (UID: \"3cf62ea0-4df7-436a-83d0-3d1174886195\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xz9r8" Sep 11 16:02:48.898659 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:48.898638 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3cf62ea0-4df7-436a-83d0-3d1174886195-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xz9r8\" (UID: \"3cf62ea0-4df7-436a-83d0-3d1174886195\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xz9r8" Sep 11 16:02:48.898723 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:48.898690 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3cf62ea0-4df7-436a-83d0-3d1174886195-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xz9r8\" (UID: \"3cf62ea0-4df7-436a-83d0-3d1174886195\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xz9r8" Sep 11 16:02:48.908332 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:48.908304 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmzgw\" (UniqueName: \"kubernetes.io/projected/3cf62ea0-4df7-436a-83d0-3d1174886195-kube-api-access-gmzgw\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xz9r8\" (UID: \"3cf62ea0-4df7-436a-83d0-3d1174886195\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xz9r8" Sep 11 16:02:48.945419 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:48.945383 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wcns4" Sep 11 16:02:49.020851 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:49.020535 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccpxn7"] Sep 11 16:02:49.023696 ip-10-0-136-96 kubenswrapper[2566]: W0911 16:02:49.023668 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20564014_116a_42f9_9b4b_30bbc7cbc6a8.slice/crio-ac10dbd5be9da59115c00c03bcfdd868b9693882e5e625b085c80efafaaf7a08 WatchSource:0}: Error finding container ac10dbd5be9da59115c00c03bcfdd868b9693882e5e625b085c80efafaaf7a08: Status 404 returned error can't find the container with id ac10dbd5be9da59115c00c03bcfdd868b9693882e5e625b085c80efafaaf7a08 Sep 11 16:02:49.076183 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:49.076161 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xz9r8" Sep 11 16:02:49.114819 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:49.114737 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awl285"] Sep 11 16:02:49.117323 ip-10-0-136-96 kubenswrapper[2566]: W0911 16:02:49.117280 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06078c33_7ab5_4d1a_94af_c98c31b080b2.slice/crio-d3422bc9b83358af8773112d0b747eb2fbeb22464bd7e067b6821d8130f66454 WatchSource:0}: Error finding container d3422bc9b83358af8773112d0b747eb2fbeb22464bd7e067b6821d8130f66454: Status 404 returned error can't find the container with id d3422bc9b83358af8773112d0b747eb2fbeb22464bd7e067b6821d8130f66454 Sep 11 16:02:49.220790 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:49.220762 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wcns4"] Sep 11 16:02:49.222087 ip-10-0-136-96 kubenswrapper[2566]: W0911 16:02:49.222053 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0ab286c_ffb5_4656_9c6c_548589cf2508.slice/crio-763d3b6e5cf30d8a100890133762f50cccdfe50c8d6b0276c434ae9b8430868d WatchSource:0}: Error finding container 763d3b6e5cf30d8a100890133762f50cccdfe50c8d6b0276c434ae9b8430868d: Status 404 returned error can't find the container with id 763d3b6e5cf30d8a100890133762f50cccdfe50c8d6b0276c434ae9b8430868d Sep 11 16:02:49.359254 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:49.359229 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xz9r8"] Sep 11 16:02:49.389995 ip-10-0-136-96 kubenswrapper[2566]: W0911 16:02:49.389925 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cf62ea0_4df7_436a_83d0_3d1174886195.slice/crio-b39e77994833eeee31bba62c9f97b69903caaa785b74c5ed414a1205311f66ad WatchSource:0}: Error finding container b39e77994833eeee31bba62c9f97b69903caaa785b74c5ed414a1205311f66ad: Status 404 returned error can't find the container with id b39e77994833eeee31bba62c9f97b69903caaa785b74c5ed414a1205311f66ad Sep 11 16:02:49.636325 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:49.636292 2566 generic.go:356] "Generic (PLEG): container finished" podID="3cf62ea0-4df7-436a-83d0-3d1174886195" containerID="a34401f21d878ef3c8baf604de2882d7122422031b251cb83c85ceddc957d02b" exitCode=0 Sep 11 16:02:49.636807 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:49.636377 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xz9r8" event={"ID":"3cf62ea0-4df7-436a-83d0-3d1174886195","Type":"ContainerDied","Data":"a34401f21d878ef3c8baf604de2882d7122422031b251cb83c85ceddc957d02b"} Sep 11 16:02:49.636807 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:49.636434 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xz9r8" event={"ID":"3cf62ea0-4df7-436a-83d0-3d1174886195","Type":"ContainerStarted","Data":"b39e77994833eeee31bba62c9f97b69903caaa785b74c5ed414a1205311f66ad"} Sep 11 16:02:49.638397 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:49.638224 2566 generic.go:356] "Generic (PLEG): container finished" podID="c0ab286c-ffb5-4656-9c6c-548589cf2508" containerID="e4447f8becd01c877c108350ea54dd0d52504036223a0c8701d838de3a1065bd" exitCode=0 Sep 11 16:02:49.638397 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:49.638309 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wcns4" event={"ID":"c0ab286c-ffb5-4656-9c6c-548589cf2508","Type":"ContainerDied","Data":"e4447f8becd01c877c108350ea54dd0d52504036223a0c8701d838de3a1065bd"} Sep 11 16:02:49.638397 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:49.638359 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wcns4" event={"ID":"c0ab286c-ffb5-4656-9c6c-548589cf2508","Type":"ContainerStarted","Data":"763d3b6e5cf30d8a100890133762f50cccdfe50c8d6b0276c434ae9b8430868d"} Sep 11 16:02:49.640200 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:49.640176 2566 generic.go:356] "Generic (PLEG): container finished" podID="20564014-116a-42f9-9b4b-30bbc7cbc6a8" containerID="9f8e74bd9492171e93f8a107a38424ef06b15f35a6effe2905056e6c4dbf1a1f" exitCode=0 Sep 11 16:02:49.640311 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:49.640250 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccpxn7" event={"ID":"20564014-116a-42f9-9b4b-30bbc7cbc6a8","Type":"ContainerDied","Data":"9f8e74bd9492171e93f8a107a38424ef06b15f35a6effe2905056e6c4dbf1a1f"} Sep 11 16:02:49.640513 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:49.640385 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccpxn7" event={"ID":"20564014-116a-42f9-9b4b-30bbc7cbc6a8","Type":"ContainerStarted","Data":"ac10dbd5be9da59115c00c03bcfdd868b9693882e5e625b085c80efafaaf7a08"} Sep 11 16:02:49.642016 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:49.641992 2566 generic.go:356] "Generic (PLEG): container finished" podID="06078c33-7ab5-4d1a-94af-c98c31b080b2" containerID="a8eb652c5e62825695a18d993d8ee10e840cc1ebc90adb1b99a8ef7b9139f31b" exitCode=0 Sep 11 16:02:49.642098 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:49.642060 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awl285" event={"ID":"06078c33-7ab5-4d1a-94af-c98c31b080b2","Type":"ContainerDied","Data":"a8eb652c5e62825695a18d993d8ee10e840cc1ebc90adb1b99a8ef7b9139f31b"} Sep 11 16:02:49.642171 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:49.642101 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awl285" event={"ID":"06078c33-7ab5-4d1a-94af-c98c31b080b2","Type":"ContainerStarted","Data":"d3422bc9b83358af8773112d0b747eb2fbeb22464bd7e067b6821d8130f66454"} Sep 11 16:02:50.652262 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:50.652216 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccpxn7" event={"ID":"20564014-116a-42f9-9b4b-30bbc7cbc6a8","Type":"ContainerStarted","Data":"ffe79cea3f2d09b8eb8c23636be05095372b91ebdb9576cfab86cc071e8d2a24"} Sep 11 16:02:50.654513 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:50.654477 2566 generic.go:356] "Generic (PLEG): container finished" podID="c0ab286c-ffb5-4656-9c6c-548589cf2508" containerID="164475b88cc701f477e1bdc838dde1513a8edb35ad54f246ca8f727d313412c9" exitCode=0 Sep 11 16:02:50.654648 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:50.654532 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wcns4" event={"ID":"c0ab286c-ffb5-4656-9c6c-548589cf2508","Type":"ContainerDied","Data":"164475b88cc701f477e1bdc838dde1513a8edb35ad54f246ca8f727d313412c9"} Sep 11 16:02:51.662449 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:51.662412 2566 generic.go:356] "Generic (PLEG): container finished" podID="c0ab286c-ffb5-4656-9c6c-548589cf2508" containerID="c81be028a45e84415c89115150f71e01650d5dedf398a425259831381693b27a" exitCode=0 Sep 11 16:02:51.662845 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:51.662502 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wcns4" event={"ID":"c0ab286c-ffb5-4656-9c6c-548589cf2508","Type":"ContainerDied","Data":"c81be028a45e84415c89115150f71e01650d5dedf398a425259831381693b27a"} Sep 11 16:02:51.664324 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:51.664301 2566 generic.go:356] "Generic (PLEG): container finished" podID="20564014-116a-42f9-9b4b-30bbc7cbc6a8" containerID="ffe79cea3f2d09b8eb8c23636be05095372b91ebdb9576cfab86cc071e8d2a24" exitCode=0 Sep 11 16:02:51.664410 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:51.664381 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccpxn7" event={"ID":"20564014-116a-42f9-9b4b-30bbc7cbc6a8","Type":"ContainerDied","Data":"ffe79cea3f2d09b8eb8c23636be05095372b91ebdb9576cfab86cc071e8d2a24"} Sep 11 16:02:51.666560 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:51.666537 2566 generic.go:356] "Generic (PLEG): container finished" podID="06078c33-7ab5-4d1a-94af-c98c31b080b2" containerID="e682e23afd597847372f18e4be9f4425a739c1911236a9e0b54c9f81a658f15d" exitCode=0 Sep 11 16:02:51.666647 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:51.666593 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awl285" event={"ID":"06078c33-7ab5-4d1a-94af-c98c31b080b2","Type":"ContainerDied","Data":"e682e23afd597847372f18e4be9f4425a739c1911236a9e0b54c9f81a658f15d"} Sep 11 16:02:52.674288 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:52.674250 2566 generic.go:356] "Generic (PLEG): container finished" podID="06078c33-7ab5-4d1a-94af-c98c31b080b2" containerID="140b78f9b23c54b0247c00d7518e6b2f6db81ce1614c62f5fd44354a274d0a38" exitCode=0 Sep 11 16:02:52.674688 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:52.674335 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awl285" event={"ID":"06078c33-7ab5-4d1a-94af-c98c31b080b2","Type":"ContainerDied","Data":"140b78f9b23c54b0247c00d7518e6b2f6db81ce1614c62f5fd44354a274d0a38"} Sep 11 16:02:52.676598 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:52.676573 2566 generic.go:356] "Generic (PLEG): container finished" podID="20564014-116a-42f9-9b4b-30bbc7cbc6a8" containerID="36a3ab6919e9125f6909b8b18ab0109ba39fd2a009520bdf2dee26d1c04ed3e4" exitCode=0 Sep 11 16:02:52.676719 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:52.676651 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccpxn7" event={"ID":"20564014-116a-42f9-9b4b-30bbc7cbc6a8","Type":"ContainerDied","Data":"36a3ab6919e9125f6909b8b18ab0109ba39fd2a009520bdf2dee26d1c04ed3e4"} Sep 11 16:02:52.810687 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:52.810663 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wcns4" Sep 11 16:02:52.936324 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:52.936232 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vfvj6\" (UniqueName: \"kubernetes.io/projected/c0ab286c-ffb5-4656-9c6c-548589cf2508-kube-api-access-vfvj6\") pod \"c0ab286c-ffb5-4656-9c6c-548589cf2508\" (UID: \"c0ab286c-ffb5-4656-9c6c-548589cf2508\") " Sep 11 16:02:52.936324 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:52.936312 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c0ab286c-ffb5-4656-9c6c-548589cf2508-bundle\") pod \"c0ab286c-ffb5-4656-9c6c-548589cf2508\" (UID: \"c0ab286c-ffb5-4656-9c6c-548589cf2508\") " Sep 11 16:02:52.936501 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:52.936350 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c0ab286c-ffb5-4656-9c6c-548589cf2508-util\") pod \"c0ab286c-ffb5-4656-9c6c-548589cf2508\" (UID: \"c0ab286c-ffb5-4656-9c6c-548589cf2508\") " Sep 11 16:02:52.936833 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:52.936807 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0ab286c-ffb5-4656-9c6c-548589cf2508-bundle" (OuterVolumeSpecName: "bundle") pod "c0ab286c-ffb5-4656-9c6c-548589cf2508" (UID: "c0ab286c-ffb5-4656-9c6c-548589cf2508"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 16:02:52.938452 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:52.938425 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0ab286c-ffb5-4656-9c6c-548589cf2508-kube-api-access-vfvj6" (OuterVolumeSpecName: "kube-api-access-vfvj6") pod "c0ab286c-ffb5-4656-9c6c-548589cf2508" (UID: "c0ab286c-ffb5-4656-9c6c-548589cf2508"). InnerVolumeSpecName "kube-api-access-vfvj6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 16:02:52.944024 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:52.943963 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0ab286c-ffb5-4656-9c6c-548589cf2508-util" (OuterVolumeSpecName: "util") pod "c0ab286c-ffb5-4656-9c6c-548589cf2508" (UID: "c0ab286c-ffb5-4656-9c6c-548589cf2508"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 16:02:53.037788 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:53.037748 2566 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c0ab286c-ffb5-4656-9c6c-548589cf2508-util\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:02:53.037788 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:53.037782 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfvj6\" (UniqueName: \"kubernetes.io/projected/c0ab286c-ffb5-4656-9c6c-548589cf2508-kube-api-access-vfvj6\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:02:53.037788 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:53.037792 2566 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c0ab286c-ffb5-4656-9c6c-548589cf2508-bundle\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:02:53.684076 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:53.684045 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wcns4" Sep 11 16:02:53.684076 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:53.684060 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wcns4" event={"ID":"c0ab286c-ffb5-4656-9c6c-548589cf2508","Type":"ContainerDied","Data":"763d3b6e5cf30d8a100890133762f50cccdfe50c8d6b0276c434ae9b8430868d"} Sep 11 16:02:53.684484 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:53.684115 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="763d3b6e5cf30d8a100890133762f50cccdfe50c8d6b0276c434ae9b8430868d" Sep 11 16:02:53.818801 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:53.818773 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccpxn7" Sep 11 16:02:53.867251 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:53.867222 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awl285" Sep 11 16:02:53.947827 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:53.947731 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-466hh\" (UniqueName: \"kubernetes.io/projected/20564014-116a-42f9-9b4b-30bbc7cbc6a8-kube-api-access-466hh\") pod \"20564014-116a-42f9-9b4b-30bbc7cbc6a8\" (UID: \"20564014-116a-42f9-9b4b-30bbc7cbc6a8\") " Sep 11 16:02:53.947827 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:53.947778 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/20564014-116a-42f9-9b4b-30bbc7cbc6a8-bundle\") pod \"20564014-116a-42f9-9b4b-30bbc7cbc6a8\" (UID: \"20564014-116a-42f9-9b4b-30bbc7cbc6a8\") " Sep 11 16:02:53.947827 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:53.947820 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/06078c33-7ab5-4d1a-94af-c98c31b080b2-bundle\") pod \"06078c33-7ab5-4d1a-94af-c98c31b080b2\" (UID: \"06078c33-7ab5-4d1a-94af-c98c31b080b2\") " Sep 11 16:02:53.948111 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:53.947954 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k6f67\" (UniqueName: \"kubernetes.io/projected/06078c33-7ab5-4d1a-94af-c98c31b080b2-kube-api-access-k6f67\") pod \"06078c33-7ab5-4d1a-94af-c98c31b080b2\" (UID: \"06078c33-7ab5-4d1a-94af-c98c31b080b2\") " Sep 11 16:02:53.948111 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:53.948093 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/06078c33-7ab5-4d1a-94af-c98c31b080b2-util\") pod \"06078c33-7ab5-4d1a-94af-c98c31b080b2\" (UID: \"06078c33-7ab5-4d1a-94af-c98c31b080b2\") " Sep 11 16:02:53.948203 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:53.948149 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/20564014-116a-42f9-9b4b-30bbc7cbc6a8-util\") pod \"20564014-116a-42f9-9b4b-30bbc7cbc6a8\" (UID: \"20564014-116a-42f9-9b4b-30bbc7cbc6a8\") " Sep 11 16:02:53.948613 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:53.948587 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20564014-116a-42f9-9b4b-30bbc7cbc6a8-bundle" (OuterVolumeSpecName: "bundle") pod "20564014-116a-42f9-9b4b-30bbc7cbc6a8" (UID: "20564014-116a-42f9-9b4b-30bbc7cbc6a8"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 16:02:53.948823 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:53.948780 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06078c33-7ab5-4d1a-94af-c98c31b080b2-bundle" (OuterVolumeSpecName: "bundle") pod "06078c33-7ab5-4d1a-94af-c98c31b080b2" (UID: "06078c33-7ab5-4d1a-94af-c98c31b080b2"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 16:02:53.950257 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:53.950227 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06078c33-7ab5-4d1a-94af-c98c31b080b2-kube-api-access-k6f67" (OuterVolumeSpecName: "kube-api-access-k6f67") pod "06078c33-7ab5-4d1a-94af-c98c31b080b2" (UID: "06078c33-7ab5-4d1a-94af-c98c31b080b2"). InnerVolumeSpecName "kube-api-access-k6f67". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 16:02:53.950409 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:53.950265 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20564014-116a-42f9-9b4b-30bbc7cbc6a8-kube-api-access-466hh" (OuterVolumeSpecName: "kube-api-access-466hh") pod "20564014-116a-42f9-9b4b-30bbc7cbc6a8" (UID: "20564014-116a-42f9-9b4b-30bbc7cbc6a8"). InnerVolumeSpecName "kube-api-access-466hh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 16:02:53.954212 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:53.954181 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20564014-116a-42f9-9b4b-30bbc7cbc6a8-util" (OuterVolumeSpecName: "util") pod "20564014-116a-42f9-9b4b-30bbc7cbc6a8" (UID: "20564014-116a-42f9-9b4b-30bbc7cbc6a8"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 16:02:53.954294 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:53.954202 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06078c33-7ab5-4d1a-94af-c98c31b080b2-util" (OuterVolumeSpecName: "util") pod "06078c33-7ab5-4d1a-94af-c98c31b080b2" (UID: "06078c33-7ab5-4d1a-94af-c98c31b080b2"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 16:02:54.049018 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:54.048947 2566 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/06078c33-7ab5-4d1a-94af-c98c31b080b2-bundle\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:02:54.049018 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:54.049013 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6f67\" (UniqueName: \"kubernetes.io/projected/06078c33-7ab5-4d1a-94af-c98c31b080b2-kube-api-access-k6f67\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:02:54.049018 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:54.049025 2566 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/06078c33-7ab5-4d1a-94af-c98c31b080b2-util\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:02:54.049244 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:54.049034 2566 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/20564014-116a-42f9-9b4b-30bbc7cbc6a8-util\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:02:54.049244 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:54.049043 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-466hh\" (UniqueName: \"kubernetes.io/projected/20564014-116a-42f9-9b4b-30bbc7cbc6a8-kube-api-access-466hh\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:02:54.049244 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:54.049051 2566 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/20564014-116a-42f9-9b4b-30bbc7cbc6a8-bundle\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:02:54.691611 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:54.691573 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccpxn7" event={"ID":"20564014-116a-42f9-9b4b-30bbc7cbc6a8","Type":"ContainerDied","Data":"ac10dbd5be9da59115c00c03bcfdd868b9693882e5e625b085c80efafaaf7a08"} Sep 11 16:02:54.692050 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:54.691622 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccpxn7" Sep 11 16:02:54.692050 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:54.691629 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ac10dbd5be9da59115c00c03bcfdd868b9693882e5e625b085c80efafaaf7a08" Sep 11 16:02:54.693744 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:54.693719 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awl285" Sep 11 16:02:54.693890 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:54.693741 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awl285" event={"ID":"06078c33-7ab5-4d1a-94af-c98c31b080b2","Type":"ContainerDied","Data":"d3422bc9b83358af8773112d0b747eb2fbeb22464bd7e067b6821d8130f66454"} Sep 11 16:02:54.693890 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:54.693792 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d3422bc9b83358af8773112d0b747eb2fbeb22464bd7e067b6821d8130f66454" Sep 11 16:02:57.711390 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:57.711302 2566 generic.go:356] "Generic (PLEG): container finished" podID="3cf62ea0-4df7-436a-83d0-3d1174886195" containerID="3dc1de216bc4d39f28a5190645e6fdeddf76d79d9e81d865a7c1378d6e6b9bc3" exitCode=0 Sep 11 16:02:57.711737 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:57.711389 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xz9r8" event={"ID":"3cf62ea0-4df7-436a-83d0-3d1174886195","Type":"ContainerDied","Data":"3dc1de216bc4d39f28a5190645e6fdeddf76d79d9e81d865a7c1378d6e6b9bc3"} Sep 11 16:02:58.718879 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:58.718843 2566 generic.go:356] "Generic (PLEG): container finished" podID="3cf62ea0-4df7-436a-83d0-3d1174886195" containerID="e12a31ff6f962a81752fbfd1b6f8384c3df600dd42a095fbd6a8a46cfc9c21ca" exitCode=0 Sep 11 16:02:58.719287 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:58.718931 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xz9r8" event={"ID":"3cf62ea0-4df7-436a-83d0-3d1174886195","Type":"ContainerDied","Data":"e12a31ff6f962a81752fbfd1b6f8384c3df600dd42a095fbd6a8a46cfc9c21ca"} Sep 11 16:02:59.854536 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:02:59.854509 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xz9r8" Sep 11 16:03:00.004388 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:03:00.004290 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gmzgw\" (UniqueName: \"kubernetes.io/projected/3cf62ea0-4df7-436a-83d0-3d1174886195-kube-api-access-gmzgw\") pod \"3cf62ea0-4df7-436a-83d0-3d1174886195\" (UID: \"3cf62ea0-4df7-436a-83d0-3d1174886195\") " Sep 11 16:03:00.004388 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:03:00.004340 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3cf62ea0-4df7-436a-83d0-3d1174886195-bundle\") pod \"3cf62ea0-4df7-436a-83d0-3d1174886195\" (UID: \"3cf62ea0-4df7-436a-83d0-3d1174886195\") " Sep 11 16:03:00.004388 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:03:00.004376 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3cf62ea0-4df7-436a-83d0-3d1174886195-util\") pod \"3cf62ea0-4df7-436a-83d0-3d1174886195\" (UID: \"3cf62ea0-4df7-436a-83d0-3d1174886195\") " Sep 11 16:03:00.004819 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:03:00.004787 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cf62ea0-4df7-436a-83d0-3d1174886195-bundle" (OuterVolumeSpecName: "bundle") pod "3cf62ea0-4df7-436a-83d0-3d1174886195" (UID: "3cf62ea0-4df7-436a-83d0-3d1174886195"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 16:03:00.006491 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:03:00.006464 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cf62ea0-4df7-436a-83d0-3d1174886195-kube-api-access-gmzgw" (OuterVolumeSpecName: "kube-api-access-gmzgw") pod "3cf62ea0-4df7-436a-83d0-3d1174886195" (UID: "3cf62ea0-4df7-436a-83d0-3d1174886195"). InnerVolumeSpecName "kube-api-access-gmzgw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 16:03:00.009289 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:03:00.009253 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cf62ea0-4df7-436a-83d0-3d1174886195-util" (OuterVolumeSpecName: "util") pod "3cf62ea0-4df7-436a-83d0-3d1174886195" (UID: "3cf62ea0-4df7-436a-83d0-3d1174886195"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 16:03:00.105076 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:03:00.105043 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmzgw\" (UniqueName: \"kubernetes.io/projected/3cf62ea0-4df7-436a-83d0-3d1174886195-kube-api-access-gmzgw\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:03:00.105076 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:03:00.105073 2566 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3cf62ea0-4df7-436a-83d0-3d1174886195-bundle\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:03:00.105076 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:03:00.105086 2566 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3cf62ea0-4df7-436a-83d0-3d1174886195-util\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:03:00.730728 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:03:00.730690 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xz9r8" event={"ID":"3cf62ea0-4df7-436a-83d0-3d1174886195","Type":"ContainerDied","Data":"b39e77994833eeee31bba62c9f97b69903caaa785b74c5ed414a1205311f66ad"} Sep 11 16:03:00.730728 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:03:00.730720 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6xz9r8" Sep 11 16:03:00.730929 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:03:00.730739 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b39e77994833eeee31bba62c9f97b69903caaa785b74c5ed414a1205311f66ad" Sep 11 16:03:14.500805 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:03:14.500767 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-f7h2s"] Sep 11 16:03:14.501204 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:03:14.501143 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20564014-116a-42f9-9b4b-30bbc7cbc6a8" containerName="util" Sep 11 16:03:14.501204 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:03:14.501155 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="20564014-116a-42f9-9b4b-30bbc7cbc6a8" containerName="util" Sep 11 16:03:14.501204 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:03:14.501163 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20564014-116a-42f9-9b4b-30bbc7cbc6a8" containerName="extract" Sep 11 16:03:14.501204 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:03:14.501169 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="20564014-116a-42f9-9b4b-30bbc7cbc6a8" containerName="extract" Sep 11 16:03:14.501204 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:03:14.501178 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06078c33-7ab5-4d1a-94af-c98c31b080b2" containerName="util" Sep 11 16:03:14.501204 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:03:14.501183 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="06078c33-7ab5-4d1a-94af-c98c31b080b2" containerName="util" Sep 11 16:03:14.501204 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:03:14.501192 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0ab286c-ffb5-4656-9c6c-548589cf2508" containerName="pull" Sep 11 16:03:14.501204 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:03:14.501197 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0ab286c-ffb5-4656-9c6c-548589cf2508" containerName="pull" Sep 11 16:03:14.501204 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:03:14.501204 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20564014-116a-42f9-9b4b-30bbc7cbc6a8" containerName="pull" Sep 11 16:03:14.501490 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:03:14.501210 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="20564014-116a-42f9-9b4b-30bbc7cbc6a8" containerName="pull" Sep 11 16:03:14.501490 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:03:14.501217 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0ab286c-ffb5-4656-9c6c-548589cf2508" containerName="util" Sep 11 16:03:14.501490 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:03:14.501222 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0ab286c-ffb5-4656-9c6c-548589cf2508" containerName="util" Sep 11 16:03:14.501490 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:03:14.501228 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06078c33-7ab5-4d1a-94af-c98c31b080b2" containerName="extract" Sep 11 16:03:14.501490 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:03:14.501233 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="06078c33-7ab5-4d1a-94af-c98c31b080b2" containerName="extract" Sep 11 16:03:14.501490 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:03:14.501244 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0ab286c-ffb5-4656-9c6c-548589cf2508" containerName="extract" Sep 11 16:03:14.501490 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:03:14.501250 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0ab286c-ffb5-4656-9c6c-548589cf2508" containerName="extract" Sep 11 16:03:14.501490 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:03:14.501262 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06078c33-7ab5-4d1a-94af-c98c31b080b2" containerName="pull" Sep 11 16:03:14.501490 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:03:14.501267 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="06078c33-7ab5-4d1a-94af-c98c31b080b2" containerName="pull" Sep 11 16:03:14.501490 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:03:14.501273 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3cf62ea0-4df7-436a-83d0-3d1174886195" containerName="util" Sep 11 16:03:14.501490 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:03:14.501278 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="3cf62ea0-4df7-436a-83d0-3d1174886195" containerName="util" Sep 11 16:03:14.501490 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:03:14.501284 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3cf62ea0-4df7-436a-83d0-3d1174886195" containerName="pull" Sep 11 16:03:14.501490 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:03:14.501289 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="3cf62ea0-4df7-436a-83d0-3d1174886195" containerName="pull" Sep 11 16:03:14.501490 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:03:14.501296 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3cf62ea0-4df7-436a-83d0-3d1174886195" containerName="extract" Sep 11 16:03:14.501490 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:03:14.501301 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="3cf62ea0-4df7-436a-83d0-3d1174886195" containerName="extract" Sep 11 16:03:14.501490 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:03:14.501351 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0ab286c-ffb5-4656-9c6c-548589cf2508" containerName="extract" Sep 11 16:03:14.501490 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:03:14.501360 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="06078c33-7ab5-4d1a-94af-c98c31b080b2" containerName="extract" Sep 11 16:03:14.501490 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:03:14.501366 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="3cf62ea0-4df7-436a-83d0-3d1174886195" containerName="extract" Sep 11 16:03:14.501490 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:03:14.501372 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="20564014-116a-42f9-9b4b-30bbc7cbc6a8" containerName="extract" Sep 11 16:03:14.504441 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:03:14.504423 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-f7h2s" Sep 11 16:03:14.507744 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:03:14.507721 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-tr7rl\"" Sep 11 16:03:14.508018 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:03:14.508002 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Sep 11 16:03:14.508648 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:03:14.508619 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Sep 11 16:03:14.608529 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:03:14.608485 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-f7h2s"] Sep 11 16:03:14.636057 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:03:14.636018 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5xdn\" (UniqueName: \"kubernetes.io/projected/b74f48bd-ec75-400e-a13c-7953339d6f6c-kube-api-access-k5xdn\") pod \"limitador-operator-controller-manager-74495c69f8-f7h2s\" (UID: \"b74f48bd-ec75-400e-a13c-7953339d6f6c\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-f7h2s" Sep 11 16:03:14.736931 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:03:14.736887 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5xdn\" (UniqueName: \"kubernetes.io/projected/b74f48bd-ec75-400e-a13c-7953339d6f6c-kube-api-access-k5xdn\") pod \"limitador-operator-controller-manager-74495c69f8-f7h2s\" (UID: \"b74f48bd-ec75-400e-a13c-7953339d6f6c\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-f7h2s" Sep 11 16:03:14.746317 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:03:14.746283 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5xdn\" (UniqueName: \"kubernetes.io/projected/b74f48bd-ec75-400e-a13c-7953339d6f6c-kube-api-access-k5xdn\") pod \"limitador-operator-controller-manager-74495c69f8-f7h2s\" (UID: \"b74f48bd-ec75-400e-a13c-7953339d6f6c\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-f7h2s" Sep 11 16:03:14.814459 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:03:14.814426 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-f7h2s" Sep 11 16:03:15.099296 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:03:15.099194 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-f7h2s"] Sep 11 16:03:15.105066 ip-10-0-136-96 kubenswrapper[2566]: W0911 16:03:15.105040 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb74f48bd_ec75_400e_a13c_7953339d6f6c.slice/crio-6a2e5bdc6380d473952e5b4e964ce83674e7625a8745d8c1a3b93a5351cb21bc WatchSource:0}: Error finding container 6a2e5bdc6380d473952e5b4e964ce83674e7625a8745d8c1a3b93a5351cb21bc: Status 404 returned error can't find the container with id 6a2e5bdc6380d473952e5b4e964ce83674e7625a8745d8c1a3b93a5351cb21bc Sep 11 16:03:15.816872 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:03:15.816835 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-f7h2s" event={"ID":"b74f48bd-ec75-400e-a13c-7953339d6f6c","Type":"ContainerStarted","Data":"6a2e5bdc6380d473952e5b4e964ce83674e7625a8745d8c1a3b93a5351cb21bc"} Sep 11 16:03:19.844788 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:03:19.844748 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-f7h2s" event={"ID":"b74f48bd-ec75-400e-a13c-7953339d6f6c","Type":"ContainerStarted","Data":"874f881c2cb62d9587514beab7d7ab214c8ee67a47b7e7b95979cadc6d7aa1c6"} Sep 11 16:03:19.845283 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:03:19.844876 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-f7h2s" Sep 11 16:03:19.870538 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:03:19.870442 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-f7h2s" podStartSLOduration=1.5437684950000001 podStartE2EDuration="5.870425863s" totalImagesPullingTime="4.326657368s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:03:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:03:19.869814177 +0000 UTC m=+507.084648082" watchObservedRunningTime="2026-09-11 16:03:19.870425863 +0000 UTC m=+507.085259768" Sep 11 16:03:29.954463 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:03:29.954427 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-vm2dl"] Sep 11 16:03:29.959502 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:03:29.959477 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-vm2dl" Sep 11 16:03:29.966046 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:03:29.965805 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-xcpqd\"" Sep 11 16:03:29.979252 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:03:29.979225 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-vm2dl"] Sep 11 16:03:30.070791 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:03:30.070752 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ql8w\" (UniqueName: \"kubernetes.io/projected/a4a932b0-63cc-44ed-9daf-9281962cec7e-kube-api-access-2ql8w\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-vm2dl\" (UID: \"a4a932b0-63cc-44ed-9daf-9281962cec7e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-vm2dl" Sep 11 16:03:30.070960 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:03:30.070829 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a4a932b0-63cc-44ed-9daf-9281962cec7e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-vm2dl\" (UID: \"a4a932b0-63cc-44ed-9daf-9281962cec7e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-vm2dl" Sep 11 16:03:30.171415 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:03:30.171376 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a4a932b0-63cc-44ed-9daf-9281962cec7e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-vm2dl\" (UID: \"a4a932b0-63cc-44ed-9daf-9281962cec7e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-vm2dl" Sep 11 16:03:30.171585 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:03:30.171448 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ql8w\" (UniqueName: \"kubernetes.io/projected/a4a932b0-63cc-44ed-9daf-9281962cec7e-kube-api-access-2ql8w\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-vm2dl\" (UID: \"a4a932b0-63cc-44ed-9daf-9281962cec7e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-vm2dl" Sep 11 16:03:30.171780 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:03:30.171761 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a4a932b0-63cc-44ed-9daf-9281962cec7e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-vm2dl\" (UID: \"a4a932b0-63cc-44ed-9daf-9281962cec7e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-vm2dl" Sep 11 16:03:30.180995 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:03:30.180952 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ql8w\" (UniqueName: \"kubernetes.io/projected/a4a932b0-63cc-44ed-9daf-9281962cec7e-kube-api-access-2ql8w\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-vm2dl\" (UID: \"a4a932b0-63cc-44ed-9daf-9281962cec7e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-vm2dl" Sep 11 16:03:30.269856 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:03:30.269825 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-vm2dl" Sep 11 16:03:30.428311 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:03:30.428270 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-vm2dl"] Sep 11 16:03:30.441654 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:03:30.441619 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-vm2dl"] Sep 11 16:03:30.474405 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:03:30.474371 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-f7h2s"] Sep 11 16:03:30.474584 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:03:30.474563 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-f7h2s" podUID="b74f48bd-ec75-400e-a13c-7953339d6f6c" containerName="manager" containerID="cri-o://874f881c2cb62d9587514beab7d7ab214c8ee67a47b7e7b95979cadc6d7aa1c6" gracePeriod=2 Sep 11 16:03:30.476356 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:03:30.476327 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-f7h2s" Sep 11 16:03:30.493948 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:03:30.493917 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-f7h2s"] Sep 11 16:03:30.501017 ip-10-0-136-96 kubenswrapper[2566]: E0911 16:03:30.500957 2566 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"b74f48bd-ec75-400e-a13c-7953339d6f6c\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"limitador-operator-controller-manager-74495c69f8-f7h2s\": pods \"limitador-operator-controller-manager-74495c69f8-f7h2s\" not found" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-f7h2s" Sep 11 16:03:30.503410 ip-10-0-136-96 kubenswrapper[2566]: E0911 16:03:30.503382 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-f7h2s\" is forbidden: User \"system:node:ip-10-0-136-96.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-96.ec2.internal' and this object" podUID="b74f48bd-ec75-400e-a13c-7953339d6f6c" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-f7h2s" Sep 11 16:03:30.515585 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:03:30.515559 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-p55g5"] Sep 11 16:03:30.515982 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:03:30.515952 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b74f48bd-ec75-400e-a13c-7953339d6f6c" containerName="manager" Sep 11 16:03:30.516070 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:03:30.515985 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b74f48bd-ec75-400e-a13c-7953339d6f6c" containerName="manager" Sep 11 16:03:30.516131 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:03:30.516119 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="b74f48bd-ec75-400e-a13c-7953339d6f6c" containerName="manager" Sep 11 16:03:30.519305 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:03:30.519284 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-p55g5" Sep 11 16:03:30.521137 ip-10-0-136-96 kubenswrapper[2566]: E0911 16:03:30.521075 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-f7h2s\" is forbidden: User \"system:node:ip-10-0-136-96.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-96.ec2.internal' and this object" podUID="b74f48bd-ec75-400e-a13c-7953339d6f6c" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-f7h2s" Sep 11 16:03:30.541362 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:03:30.541333 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-p55g5"] Sep 11 16:03:30.574654 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:03:30.574622 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tp5wp\" (UniqueName: \"kubernetes.io/projected/4246efe8-35df-4aaf-8bb9-f01789b50185-kube-api-access-tp5wp\") pod \"limitador-operator-controller-manager-74495c69f8-p55g5\" (UID: \"4246efe8-35df-4aaf-8bb9-f01789b50185\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-p55g5" Sep 11 16:03:30.675575 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:03:30.675541 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tp5wp\" (UniqueName: \"kubernetes.io/projected/4246efe8-35df-4aaf-8bb9-f01789b50185-kube-api-access-tp5wp\") pod \"limitador-operator-controller-manager-74495c69f8-p55g5\" (UID: \"4246efe8-35df-4aaf-8bb9-f01789b50185\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-p55g5" Sep 11 16:03:30.690220 ip-10-0-136-96 kubenswrapper[2566]: E0911 16:03:30.690182 2566 log.go:32] "RunPodSandbox from runtime service failed" err=< Sep 11 16:03:30.690220 ip-10-0-136-96 kubenswrapper[2566]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-7978cbdfbc-vm2dl_kuadrant-system_a4a932b0-63cc-44ed-9daf-9281962cec7e_0(0759e680055748e4e15df1997d32a7fabf6f6325dd8e4a0ec69353fdcaa0468f): error adding pod kuadrant-system_kuadrant-operator-controller-manager-7978cbdfbc-vm2dl to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"0759e680055748e4e15df1997d32a7fabf6f6325dd8e4a0ec69353fdcaa0468f" Netns:"/var/run/netns/ec34d65c-bdf0-4400-8669-1d17ed13fae8" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=kuadrant-system;K8S_POD_NAME=kuadrant-operator-controller-manager-7978cbdfbc-vm2dl;K8S_POD_INFRA_CONTAINER_ID=0759e680055748e4e15df1997d32a7fabf6f6325dd8e4a0ec69353fdcaa0468f;K8S_POD_UID=a4a932b0-63cc-44ed-9daf-9281962cec7e" Path:"" ERRORED: error configuring pod [kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-vm2dl] networking: [kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-vm2dl/a4a932b0-63cc-44ed-9daf-9281962cec7e:ovn-kubernetes]: error adding container to network "ovn-kubernetes": CNI request failed with status 400: '[kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-vm2dl 0759e680055748e4e15df1997d32a7fabf6f6325dd8e4a0ec69353fdcaa0468f network default NAD default NAD key default] failed to configure pod interface: pod deleted waiting for OVS port binding for 0a:58:0a:84:00:2f [10.132.0.47/23] Sep 11 16:03:30.690220 ip-10-0-136-96 kubenswrapper[2566]: ' Sep 11 16:03:30.690220 ip-10-0-136-96 kubenswrapper[2566]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Sep 11 16:03:30.690220 ip-10-0-136-96 kubenswrapper[2566]: > Sep 11 16:03:30.690438 ip-10-0-136-96 kubenswrapper[2566]: E0911 16:03:30.690256 2566 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err=< Sep 11 16:03:30.690438 ip-10-0-136-96 kubenswrapper[2566]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-7978cbdfbc-vm2dl_kuadrant-system_a4a932b0-63cc-44ed-9daf-9281962cec7e_0(0759e680055748e4e15df1997d32a7fabf6f6325dd8e4a0ec69353fdcaa0468f): error adding pod kuadrant-system_kuadrant-operator-controller-manager-7978cbdfbc-vm2dl to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"0759e680055748e4e15df1997d32a7fabf6f6325dd8e4a0ec69353fdcaa0468f" Netns:"/var/run/netns/ec34d65c-bdf0-4400-8669-1d17ed13fae8" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=kuadrant-system;K8S_POD_NAME=kuadrant-operator-controller-manager-7978cbdfbc-vm2dl;K8S_POD_INFRA_CONTAINER_ID=0759e680055748e4e15df1997d32a7fabf6f6325dd8e4a0ec69353fdcaa0468f;K8S_POD_UID=a4a932b0-63cc-44ed-9daf-9281962cec7e" Path:"" ERRORED: error configuring pod [kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-vm2dl] networking: [kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-vm2dl/a4a932b0-63cc-44ed-9daf-9281962cec7e:ovn-kubernetes]: error adding container to network "ovn-kubernetes": CNI request failed with status 400: '[kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-vm2dl 0759e680055748e4e15df1997d32a7fabf6f6325dd8e4a0ec69353fdcaa0468f network default NAD default NAD key default] failed to configure pod interface: pod deleted waiting for OVS port binding for 0a:58:0a:84:00:2f [10.132.0.47/23] Sep 11 16:03:30.690438 ip-10-0-136-96 kubenswrapper[2566]: ' Sep 11 16:03:30.690438 ip-10-0-136-96 kubenswrapper[2566]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Sep 11 16:03:30.690438 ip-10-0-136-96 kubenswrapper[2566]: > pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-vm2dl" Sep 11 16:03:30.698775 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:03:30.698743 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tp5wp\" (UniqueName: \"kubernetes.io/projected/4246efe8-35df-4aaf-8bb9-f01789b50185-kube-api-access-tp5wp\") pod \"limitador-operator-controller-manager-74495c69f8-p55g5\" (UID: \"4246efe8-35df-4aaf-8bb9-f01789b50185\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-p55g5" Sep 11 16:03:30.710766 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:03:30.710745 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-f7h2s" Sep 11 16:03:30.723754 ip-10-0-136-96 kubenswrapper[2566]: E0911 16:03:30.723717 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-f7h2s\" is forbidden: User \"system:node:ip-10-0-136-96.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-96.ec2.internal' and this object" podUID="b74f48bd-ec75-400e-a13c-7953339d6f6c" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-f7h2s" Sep 11 16:03:30.776311 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:03:30.776220 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k5xdn\" (UniqueName: \"kubernetes.io/projected/b74f48bd-ec75-400e-a13c-7953339d6f6c-kube-api-access-k5xdn\") pod \"b74f48bd-ec75-400e-a13c-7953339d6f6c\" (UID: \"b74f48bd-ec75-400e-a13c-7953339d6f6c\") " Sep 11 16:03:30.778387 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:03:30.778358 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b74f48bd-ec75-400e-a13c-7953339d6f6c-kube-api-access-k5xdn" (OuterVolumeSpecName: "kube-api-access-k5xdn") pod "b74f48bd-ec75-400e-a13c-7953339d6f6c" (UID: "b74f48bd-ec75-400e-a13c-7953339d6f6c"). InnerVolumeSpecName "kube-api-access-k5xdn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 16:03:30.849056 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:03:30.849016 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-p55g5" Sep 11 16:03:30.877462 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:03:30.877433 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5xdn\" (UniqueName: \"kubernetes.io/projected/b74f48bd-ec75-400e-a13c-7953339d6f6c-kube-api-access-k5xdn\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:03:30.907698 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:03:30.907657 2566 generic.go:356] "Generic (PLEG): container finished" podID="b74f48bd-ec75-400e-a13c-7953339d6f6c" containerID="874f881c2cb62d9587514beab7d7ab214c8ee67a47b7e7b95979cadc6d7aa1c6" exitCode=0 Sep 11 16:03:30.907925 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:03:30.907742 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-vm2dl" Sep 11 16:03:30.908247 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:03:30.908171 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-f7h2s" Sep 11 16:03:30.908341 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:03:30.908302 2566 scope.go:117] "RemoveContainer" containerID="874f881c2cb62d9587514beab7d7ab214c8ee67a47b7e7b95979cadc6d7aa1c6" Sep 11 16:03:30.911420 ip-10-0-136-96 kubenswrapper[2566]: E0911 16:03:30.911388 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-f7h2s\" is forbidden: User \"system:node:ip-10-0-136-96.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-96.ec2.internal' and this object" podUID="b74f48bd-ec75-400e-a13c-7953339d6f6c" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-f7h2s" Sep 11 16:03:30.913047 ip-10-0-136-96 kubenswrapper[2566]: E0911 16:03:30.913018 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-7978cbdfbc-vm2dl\" is forbidden: User \"system:node:ip-10-0-136-96.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-96.ec2.internal' and this object" podUID="a4a932b0-63cc-44ed-9daf-9281962cec7e" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-vm2dl" Sep 11 16:03:30.914756 ip-10-0-136-96 kubenswrapper[2566]: E0911 16:03:30.914720 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-f7h2s\" is forbidden: User \"system:node:ip-10-0-136-96.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-96.ec2.internal' and this object" podUID="b74f48bd-ec75-400e-a13c-7953339d6f6c" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-f7h2s" Sep 11 16:03:30.917756 ip-10-0-136-96 kubenswrapper[2566]: E0911 16:03:30.917727 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-7978cbdfbc-vm2dl\" is forbidden: User \"system:node:ip-10-0-136-96.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-96.ec2.internal' and this object" podUID="a4a932b0-63cc-44ed-9daf-9281962cec7e" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-vm2dl" Sep 11 16:03:30.918104 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:03:30.918084 2566 scope.go:117] "RemoveContainer" containerID="874f881c2cb62d9587514beab7d7ab214c8ee67a47b7e7b95979cadc6d7aa1c6" Sep 11 16:03:30.918400 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:03:30.918385 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-vm2dl" Sep 11 16:03:30.918480 ip-10-0-136-96 kubenswrapper[2566]: E0911 16:03:30.918429 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"874f881c2cb62d9587514beab7d7ab214c8ee67a47b7e7b95979cadc6d7aa1c6\": container with ID starting with 874f881c2cb62d9587514beab7d7ab214c8ee67a47b7e7b95979cadc6d7aa1c6 not found: ID does not exist" containerID="874f881c2cb62d9587514beab7d7ab214c8ee67a47b7e7b95979cadc6d7aa1c6" Sep 11 16:03:30.918480 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:03:30.918462 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"874f881c2cb62d9587514beab7d7ab214c8ee67a47b7e7b95979cadc6d7aa1c6"} err="failed to get container status \"874f881c2cb62d9587514beab7d7ab214c8ee67a47b7e7b95979cadc6d7aa1c6\": rpc error: code = NotFound desc = could not find container \"874f881c2cb62d9587514beab7d7ab214c8ee67a47b7e7b95979cadc6d7aa1c6\": container with ID starting with 874f881c2cb62d9587514beab7d7ab214c8ee67a47b7e7b95979cadc6d7aa1c6 not found: ID does not exist" Sep 11 16:03:30.920268 ip-10-0-136-96 kubenswrapper[2566]: E0911 16:03:30.920245 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-7978cbdfbc-vm2dl\" is forbidden: User \"system:node:ip-10-0-136-96.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-96.ec2.internal' and this object" podUID="a4a932b0-63cc-44ed-9daf-9281962cec7e" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-vm2dl" Sep 11 16:03:30.922169 ip-10-0-136-96 kubenswrapper[2566]: E0911 16:03:30.921864 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-f7h2s\" is forbidden: User \"system:node:ip-10-0-136-96.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-96.ec2.internal' and this object" podUID="b74f48bd-ec75-400e-a13c-7953339d6f6c" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-f7h2s" Sep 11 16:03:30.923847 ip-10-0-136-96 kubenswrapper[2566]: E0911 16:03:30.923821 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-f7h2s\" is forbidden: User \"system:node:ip-10-0-136-96.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-96.ec2.internal' and this object" podUID="b74f48bd-ec75-400e-a13c-7953339d6f6c" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-f7h2s" Sep 11 16:03:30.925375 ip-10-0-136-96 kubenswrapper[2566]: E0911 16:03:30.925350 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-7978cbdfbc-vm2dl\" is forbidden: User \"system:node:ip-10-0-136-96.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-96.ec2.internal' and this object" podUID="a4a932b0-63cc-44ed-9daf-9281962cec7e" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-vm2dl" Sep 11 16:03:30.978557 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:03:30.978526 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2ql8w\" (UniqueName: \"kubernetes.io/projected/a4a932b0-63cc-44ed-9daf-9281962cec7e-kube-api-access-2ql8w\") pod \"a4a932b0-63cc-44ed-9daf-9281962cec7e\" (UID: \"a4a932b0-63cc-44ed-9daf-9281962cec7e\") " Sep 11 16:03:30.979024 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:03:30.978631 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a4a932b0-63cc-44ed-9daf-9281962cec7e-extensions-socket-volume\") pod \"a4a932b0-63cc-44ed-9daf-9281962cec7e\" (UID: \"a4a932b0-63cc-44ed-9daf-9281962cec7e\") " Sep 11 16:03:30.979024 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:03:30.978892 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4a932b0-63cc-44ed-9daf-9281962cec7e-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "a4a932b0-63cc-44ed-9daf-9281962cec7e" (UID: "a4a932b0-63cc-44ed-9daf-9281962cec7e"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 16:03:30.980621 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:03:30.980600 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4a932b0-63cc-44ed-9daf-9281962cec7e-kube-api-access-2ql8w" (OuterVolumeSpecName: "kube-api-access-2ql8w") pod "a4a932b0-63cc-44ed-9daf-9281962cec7e" (UID: "a4a932b0-63cc-44ed-9daf-9281962cec7e"). InnerVolumeSpecName "kube-api-access-2ql8w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 16:03:31.080071 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:03:31.079993 2566 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a4a932b0-63cc-44ed-9daf-9281962cec7e-extensions-socket-volume\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:03:31.080071 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:03:31.080021 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ql8w\" (UniqueName: \"kubernetes.io/projected/a4a932b0-63cc-44ed-9daf-9281962cec7e-kube-api-access-2ql8w\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:03:31.123579 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:03:31.123552 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-p55g5"] Sep 11 16:03:31.126237 ip-10-0-136-96 kubenswrapper[2566]: W0911 16:03:31.126210 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4246efe8_35df_4aaf_8bb9_f01789b50185.slice/crio-8fd412c8ca4e4b811c6b9eaed824fdab9096fde1a8c5eeda5b3d371812eee0c2 WatchSource:0}: Error finding container 8fd412c8ca4e4b811c6b9eaed824fdab9096fde1a8c5eeda5b3d371812eee0c2: Status 404 returned error can't find the container with id 8fd412c8ca4e4b811c6b9eaed824fdab9096fde1a8c5eeda5b3d371812eee0c2 Sep 11 16:03:31.239055 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:03:31.239018 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a4a932b0-63cc-44ed-9daf-9281962cec7e" path="/var/lib/kubelet/pods/a4a932b0-63cc-44ed-9daf-9281962cec7e/volumes" Sep 11 16:03:31.239283 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:03:31.239269 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b74f48bd-ec75-400e-a13c-7953339d6f6c" path="/var/lib/kubelet/pods/b74f48bd-ec75-400e-a13c-7953339d6f6c/volumes" Sep 11 16:03:31.915259 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:03:31.915218 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-p55g5" event={"ID":"4246efe8-35df-4aaf-8bb9-f01789b50185","Type":"ContainerStarted","Data":"953c79b510da0df6cebde2f63c50c8f0e12439dd809b7e64f7679cecaf73d9fb"} Sep 11 16:03:31.915514 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:03:31.915274 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-p55g5" event={"ID":"4246efe8-35df-4aaf-8bb9-f01789b50185","Type":"ContainerStarted","Data":"8fd412c8ca4e4b811c6b9eaed824fdab9096fde1a8c5eeda5b3d371812eee0c2"} Sep 11 16:03:31.915514 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:03:31.915304 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-p55g5" Sep 11 16:03:31.916293 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:03:31.916276 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-vm2dl" Sep 11 16:03:31.949797 ip-10-0-136-96 kubenswrapper[2566]: E0911 16:03:31.949746 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-7978cbdfbc-vm2dl\" is forbidden: User \"system:node:ip-10-0-136-96.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-96.ec2.internal' and this object" podUID="a4a932b0-63cc-44ed-9daf-9281962cec7e" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-vm2dl" Sep 11 16:03:31.951236 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:03:31.951186 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-p55g5" podStartSLOduration=1.951169256 podStartE2EDuration="1.951169256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:03:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:03:31.947489047 +0000 UTC m=+519.162322955" watchObservedRunningTime="2026-09-11 16:03:31.951169256 +0000 UTC m=+519.166003196" Sep 11 16:03:33.237704 ip-10-0-136-96 kubenswrapper[2566]: E0911 16:03:33.237666 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-7978cbdfbc-vm2dl\" is forbidden: User \"system:node:ip-10-0-136-96.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-96.ec2.internal' and this object" podUID="a4a932b0-63cc-44ed-9daf-9281962cec7e" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-vm2dl" Sep 11 16:03:42.924404 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:03:42.924369 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-p55g5" Sep 11 16:04:02.235345 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:04:02.235312 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2gpqh"] Sep 11 16:04:02.240379 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:04:02.240358 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2gpqh" Sep 11 16:04:02.248346 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:04:02.248327 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-gg7k2\"" Sep 11 16:04:02.309003 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:04:02.308949 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2gpqh"] Sep 11 16:04:02.363717 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:04:02.363675 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4fb754a-0ba5-47b6-8fca-ec6a846499f5-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2gpqh\" (UID: \"b4fb754a-0ba5-47b6-8fca-ec6a846499f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2gpqh" Sep 11 16:04:02.363913 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:04:02.363732 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4fb754a-0ba5-47b6-8fca-ec6a846499f5-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2gpqh\" (UID: \"b4fb754a-0ba5-47b6-8fca-ec6a846499f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2gpqh" Sep 11 16:04:02.363913 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:04:02.363779 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4fb754a-0ba5-47b6-8fca-ec6a846499f5-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2gpqh\" (UID: \"b4fb754a-0ba5-47b6-8fca-ec6a846499f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2gpqh" Sep 11 16:04:02.363913 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:04:02.363815 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4fb754a-0ba5-47b6-8fca-ec6a846499f5-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2gpqh\" (UID: \"b4fb754a-0ba5-47b6-8fca-ec6a846499f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2gpqh" Sep 11 16:04:02.363913 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:04:02.363834 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4fb754a-0ba5-47b6-8fca-ec6a846499f5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2gpqh\" (UID: \"b4fb754a-0ba5-47b6-8fca-ec6a846499f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2gpqh" Sep 11 16:04:02.363913 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:04:02.363895 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4fb754a-0ba5-47b6-8fca-ec6a846499f5-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2gpqh\" (UID: \"b4fb754a-0ba5-47b6-8fca-ec6a846499f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2gpqh" Sep 11 16:04:02.364210 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:04:02.363945 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92g2x\" (UniqueName: \"kubernetes.io/projected/b4fb754a-0ba5-47b6-8fca-ec6a846499f5-kube-api-access-92g2x\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2gpqh\" (UID: \"b4fb754a-0ba5-47b6-8fca-ec6a846499f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2gpqh" Sep 11 16:04:02.364210 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:04:02.364023 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4fb754a-0ba5-47b6-8fca-ec6a846499f5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2gpqh\" (UID: \"b4fb754a-0ba5-47b6-8fca-ec6a846499f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2gpqh" Sep 11 16:04:02.364210 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:04:02.364085 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4fb754a-0ba5-47b6-8fca-ec6a846499f5-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2gpqh\" (UID: \"b4fb754a-0ba5-47b6-8fca-ec6a846499f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2gpqh" Sep 11 16:04:02.465227 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:04:02.465186 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4fb754a-0ba5-47b6-8fca-ec6a846499f5-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2gpqh\" (UID: \"b4fb754a-0ba5-47b6-8fca-ec6a846499f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2gpqh" Sep 11 16:04:02.465227 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:04:02.465238 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4fb754a-0ba5-47b6-8fca-ec6a846499f5-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2gpqh\" (UID: \"b4fb754a-0ba5-47b6-8fca-ec6a846499f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2gpqh" Sep 11 16:04:02.465516 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:04:02.465258 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4fb754a-0ba5-47b6-8fca-ec6a846499f5-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2gpqh\" (UID: \"b4fb754a-0ba5-47b6-8fca-ec6a846499f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2gpqh" Sep 11 16:04:02.465516 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:04:02.465291 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4fb754a-0ba5-47b6-8fca-ec6a846499f5-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2gpqh\" (UID: \"b4fb754a-0ba5-47b6-8fca-ec6a846499f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2gpqh" Sep 11 16:04:02.465516 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:04:02.465310 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4fb754a-0ba5-47b6-8fca-ec6a846499f5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2gpqh\" (UID: \"b4fb754a-0ba5-47b6-8fca-ec6a846499f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2gpqh" Sep 11 16:04:02.465516 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:04:02.465333 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4fb754a-0ba5-47b6-8fca-ec6a846499f5-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2gpqh\" (UID: \"b4fb754a-0ba5-47b6-8fca-ec6a846499f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2gpqh" Sep 11 16:04:02.465516 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:04:02.465373 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92g2x\" (UniqueName: \"kubernetes.io/projected/b4fb754a-0ba5-47b6-8fca-ec6a846499f5-kube-api-access-92g2x\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2gpqh\" (UID: \"b4fb754a-0ba5-47b6-8fca-ec6a846499f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2gpqh" Sep 11 16:04:02.465516 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:04:02.465410 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4fb754a-0ba5-47b6-8fca-ec6a846499f5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2gpqh\" (UID: \"b4fb754a-0ba5-47b6-8fca-ec6a846499f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2gpqh" Sep 11 16:04:02.465516 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:04:02.465469 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4fb754a-0ba5-47b6-8fca-ec6a846499f5-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2gpqh\" (UID: \"b4fb754a-0ba5-47b6-8fca-ec6a846499f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2gpqh" Sep 11 16:04:02.465909 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:04:02.465674 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4fb754a-0ba5-47b6-8fca-ec6a846499f5-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2gpqh\" (UID: \"b4fb754a-0ba5-47b6-8fca-ec6a846499f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2gpqh" Sep 11 16:04:02.465909 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:04:02.465705 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4fb754a-0ba5-47b6-8fca-ec6a846499f5-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2gpqh\" (UID: \"b4fb754a-0ba5-47b6-8fca-ec6a846499f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2gpqh" Sep 11 16:04:02.465909 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:04:02.465793 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4fb754a-0ba5-47b6-8fca-ec6a846499f5-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2gpqh\" (UID: \"b4fb754a-0ba5-47b6-8fca-ec6a846499f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2gpqh" Sep 11 16:04:02.466095 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:04:02.466003 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4fb754a-0ba5-47b6-8fca-ec6a846499f5-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2gpqh\" (UID: \"b4fb754a-0ba5-47b6-8fca-ec6a846499f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2gpqh" Sep 11 16:04:02.466330 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:04:02.466307 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4fb754a-0ba5-47b6-8fca-ec6a846499f5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2gpqh\" (UID: \"b4fb754a-0ba5-47b6-8fca-ec6a846499f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2gpqh" Sep 11 16:04:02.467932 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:04:02.467906 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4fb754a-0ba5-47b6-8fca-ec6a846499f5-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2gpqh\" (UID: \"b4fb754a-0ba5-47b6-8fca-ec6a846499f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2gpqh" Sep 11 16:04:02.468301 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:04:02.468282 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4fb754a-0ba5-47b6-8fca-ec6a846499f5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2gpqh\" (UID: \"b4fb754a-0ba5-47b6-8fca-ec6a846499f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2gpqh" Sep 11 16:04:02.474728 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:04:02.474705 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-92g2x\" (UniqueName: \"kubernetes.io/projected/b4fb754a-0ba5-47b6-8fca-ec6a846499f5-kube-api-access-92g2x\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2gpqh\" (UID: \"b4fb754a-0ba5-47b6-8fca-ec6a846499f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2gpqh" Sep 11 16:04:02.475411 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:04:02.475387 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4fb754a-0ba5-47b6-8fca-ec6a846499f5-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2gpqh\" (UID: \"b4fb754a-0ba5-47b6-8fca-ec6a846499f5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2gpqh" Sep 11 16:04:02.550206 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:04:02.550172 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2gpqh" Sep 11 16:04:02.836875 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:04:02.836847 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2gpqh"] Sep 11 16:04:02.837946 ip-10-0-136-96 kubenswrapper[2566]: W0911 16:04:02.837915 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4fb754a_0ba5_47b6_8fca_ec6a846499f5.slice/crio-e52022fcc11c3d9be1ecb7598d5075d0a0d3620958af0f1149a53a11a01611d7 WatchSource:0}: Error finding container e52022fcc11c3d9be1ecb7598d5075d0a0d3620958af0f1149a53a11a01611d7: Status 404 returned error can't find the container with id e52022fcc11c3d9be1ecb7598d5075d0a0d3620958af0f1149a53a11a01611d7 Sep 11 16:04:02.840137 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:04:02.840101 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 16:04:02.840200 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:04:02.840183 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 16:04:02.840245 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:04:02.840228 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 16:04:03.090297 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:04:03.090194 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2gpqh" event={"ID":"b4fb754a-0ba5-47b6-8fca-ec6a846499f5","Type":"ContainerStarted","Data":"0cbf63c74f3c313253ebd82657543297066f2db1c54912b0ebbce55fe0cae41b"} Sep 11 16:04:03.090297 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:04:03.090272 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2gpqh" event={"ID":"b4fb754a-0ba5-47b6-8fca-ec6a846499f5","Type":"ContainerStarted","Data":"e52022fcc11c3d9be1ecb7598d5075d0a0d3620958af0f1149a53a11a01611d7"} Sep 11 16:04:03.126801 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:04:03.126736 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2gpqh" podStartSLOduration=1.126714446 podStartE2EDuration="1.126714446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:04:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:04:03.123271562 +0000 UTC m=+550.338105467" watchObservedRunningTime="2026-09-11 16:04:03.126714446 +0000 UTC m=+550.341548352" Sep 11 16:04:03.550955 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:04:03.550911 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2gpqh" Sep 11 16:04:03.556246 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:04:03.556213 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2gpqh" Sep 11 16:04:04.095685 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:04:04.095652 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2gpqh" Sep 11 16:04:04.096581 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:04:04.096559 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2gpqh" Sep 11 16:04:17.754506 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:04:17.754467 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dx4xn"] Sep 11 16:04:17.757050 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:04:17.757027 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-dx4xn" Sep 11 16:04:17.759413 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:04:17.759388 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Sep 11 16:04:17.759503 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:04:17.759438 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-wrlk5\"" Sep 11 16:04:17.767072 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:04:17.767046 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dx4xn"] Sep 11 16:04:17.802134 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:04:17.802101 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmkg7\" (UniqueName: \"kubernetes.io/projected/1842dece-04ed-460c-879c-1e62e6548a67-kube-api-access-nmkg7\") pod \"limitador-limitador-84994f7ffc-dx4xn\" (UID: \"1842dece-04ed-460c-879c-1e62e6548a67\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-dx4xn" Sep 11 16:04:17.802134 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:04:17.802138 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1842dece-04ed-460c-879c-1e62e6548a67-config-file\") pod \"limitador-limitador-84994f7ffc-dx4xn\" (UID: \"1842dece-04ed-460c-879c-1e62e6548a67\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-dx4xn" Sep 11 16:04:17.807539 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:04:17.807511 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dx4xn"] Sep 11 16:04:17.902887 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:04:17.902847 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmkg7\" (UniqueName: \"kubernetes.io/projected/1842dece-04ed-460c-879c-1e62e6548a67-kube-api-access-nmkg7\") pod \"limitador-limitador-84994f7ffc-dx4xn\" (UID: \"1842dece-04ed-460c-879c-1e62e6548a67\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-dx4xn" Sep 11 16:04:17.903075 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:04:17.902894 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1842dece-04ed-460c-879c-1e62e6548a67-config-file\") pod \"limitador-limitador-84994f7ffc-dx4xn\" (UID: \"1842dece-04ed-460c-879c-1e62e6548a67\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-dx4xn" Sep 11 16:04:17.903526 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:04:17.903504 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1842dece-04ed-460c-879c-1e62e6548a67-config-file\") pod \"limitador-limitador-84994f7ffc-dx4xn\" (UID: \"1842dece-04ed-460c-879c-1e62e6548a67\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-dx4xn" Sep 11 16:04:17.911840 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:04:17.911814 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmkg7\" (UniqueName: \"kubernetes.io/projected/1842dece-04ed-460c-879c-1e62e6548a67-kube-api-access-nmkg7\") pod \"limitador-limitador-84994f7ffc-dx4xn\" (UID: \"1842dece-04ed-460c-879c-1e62e6548a67\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-dx4xn" Sep 11 16:04:18.068901 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:04:18.068858 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-dx4xn" Sep 11 16:04:18.340866 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:04:18.340788 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dx4xn"] Sep 11 16:04:18.344156 ip-10-0-136-96 kubenswrapper[2566]: W0911 16:04:18.344128 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1842dece_04ed_460c_879c_1e62e6548a67.slice/crio-c93b8aefcead2cda9da5622c4a5eca219deb411337623eea6ea686b0964e604b WatchSource:0}: Error finding container c93b8aefcead2cda9da5622c4a5eca219deb411337623eea6ea686b0964e604b: Status 404 returned error can't find the container with id c93b8aefcead2cda9da5622c4a5eca219deb411337623eea6ea686b0964e604b Sep 11 16:04:18.461513 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:04:18.461477 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-sqpkc"] Sep 11 16:04:18.464532 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:04:18.464515 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-sqpkc" Sep 11 16:04:18.466666 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:04:18.466650 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-4469b\"" Sep 11 16:04:18.474159 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:04:18.474136 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-sqpkc"] Sep 11 16:04:18.509293 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:04:18.509262 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b82hd\" (UniqueName: \"kubernetes.io/projected/b5c87369-3c99-4b44-9554-ab0430256596-kube-api-access-b82hd\") pod \"authorino-59c974bd48-sqpkc\" (UID: \"b5c87369-3c99-4b44-9554-ab0430256596\") " pod="kuadrant-system/authorino-59c974bd48-sqpkc" Sep 11 16:04:18.609859 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:04:18.609775 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b82hd\" (UniqueName: \"kubernetes.io/projected/b5c87369-3c99-4b44-9554-ab0430256596-kube-api-access-b82hd\") pod \"authorino-59c974bd48-sqpkc\" (UID: \"b5c87369-3c99-4b44-9554-ab0430256596\") " pod="kuadrant-system/authorino-59c974bd48-sqpkc" Sep 11 16:04:18.618930 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:04:18.618894 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b82hd\" (UniqueName: \"kubernetes.io/projected/b5c87369-3c99-4b44-9554-ab0430256596-kube-api-access-b82hd\") pod \"authorino-59c974bd48-sqpkc\" (UID: \"b5c87369-3c99-4b44-9554-ab0430256596\") " pod="kuadrant-system/authorino-59c974bd48-sqpkc" Sep 11 16:04:18.774857 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:04:18.774823 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-sqpkc" Sep 11 16:04:19.048422 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:04:19.048334 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-sqpkc"] Sep 11 16:04:19.050703 ip-10-0-136-96 kubenswrapper[2566]: W0911 16:04:19.050667 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb5c87369_3c99_4b44_9554_ab0430256596.slice/crio-ac95e09dce405e8a1e1869b32a62989e6927fca57ea2eb7f17318c1b6446b80c WatchSource:0}: Error finding container ac95e09dce405e8a1e1869b32a62989e6927fca57ea2eb7f17318c1b6446b80c: Status 404 returned error can't find the container with id ac95e09dce405e8a1e1869b32a62989e6927fca57ea2eb7f17318c1b6446b80c Sep 11 16:04:19.183526 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:04:19.183458 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-sqpkc" event={"ID":"b5c87369-3c99-4b44-9554-ab0430256596","Type":"ContainerStarted","Data":"ac95e09dce405e8a1e1869b32a62989e6927fca57ea2eb7f17318c1b6446b80c"} Sep 11 16:04:19.185057 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:04:19.185028 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-dx4xn" event={"ID":"1842dece-04ed-460c-879c-1e62e6548a67","Type":"ContainerStarted","Data":"c93b8aefcead2cda9da5622c4a5eca219deb411337623eea6ea686b0964e604b"} Sep 11 16:04:20.192299 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:04:20.192263 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-dx4xn" event={"ID":"1842dece-04ed-460c-879c-1e62e6548a67","Type":"ContainerStarted","Data":"2e16fbedb6ce0272c220b1a23cc7e87fafb470df3f20c6414c457f4bfb3fe77c"} Sep 11 16:04:20.192770 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:04:20.192330 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-dx4xn" Sep 11 16:04:20.216633 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:04:20.211880 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-dx4xn" podStartSLOduration=2.127390684 podStartE2EDuration="3.211861684s" totalImagesPullingTime="1.084471s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:04:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:04:20.208920188 +0000 UTC m=+567.423754106" watchObservedRunningTime="2026-09-11 16:04:20.211861684 +0000 UTC m=+567.426695592" Sep 11 16:04:25.222987 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:04:25.222947 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-sqpkc" event={"ID":"b5c87369-3c99-4b44-9554-ab0430256596","Type":"ContainerStarted","Data":"6b1d51af47c5f9cec6b39900f5d016562bcbf2c03b691f249d42869216be17e1"} Sep 11 16:04:25.249195 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:04:25.249145 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-sqpkc" podStartSLOduration=1.9469812050000002 podStartE2EDuration="7.24912778s" totalImagesPullingTime="5.302146575s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:04:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:04:25.247064949 +0000 UTC m=+572.461898853" watchObservedRunningTime="2026-09-11 16:04:25.24912778 +0000 UTC m=+572.463961706" Sep 11 16:04:31.198957 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:04:31.198878 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-dx4xn" Sep 11 16:04:51.858733 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:04:51.858698 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-sqpkc"] Sep 11 16:04:51.859266 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:04:51.858901 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-sqpkc" podUID="b5c87369-3c99-4b44-9554-ab0430256596" containerName="authorino" containerID="cri-o://6b1d51af47c5f9cec6b39900f5d016562bcbf2c03b691f249d42869216be17e1" gracePeriod=30 Sep 11 16:04:52.107260 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:04:52.107236 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-sqpkc" Sep 11 16:04:52.223829 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:04:52.223741 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b82hd\" (UniqueName: \"kubernetes.io/projected/b5c87369-3c99-4b44-9554-ab0430256596-kube-api-access-b82hd\") pod \"b5c87369-3c99-4b44-9554-ab0430256596\" (UID: \"b5c87369-3c99-4b44-9554-ab0430256596\") " Sep 11 16:04:52.225817 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:04:52.225788 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5c87369-3c99-4b44-9554-ab0430256596-kube-api-access-b82hd" (OuterVolumeSpecName: "kube-api-access-b82hd") pod "b5c87369-3c99-4b44-9554-ab0430256596" (UID: "b5c87369-3c99-4b44-9554-ab0430256596"). InnerVolumeSpecName "kube-api-access-b82hd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 16:04:52.324934 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:04:52.324895 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b82hd\" (UniqueName: \"kubernetes.io/projected/b5c87369-3c99-4b44-9554-ab0430256596-kube-api-access-b82hd\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:04:52.370772 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:04:52.370733 2566 generic.go:356] "Generic (PLEG): container finished" podID="b5c87369-3c99-4b44-9554-ab0430256596" containerID="6b1d51af47c5f9cec6b39900f5d016562bcbf2c03b691f249d42869216be17e1" exitCode=0 Sep 11 16:04:52.370772 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:04:52.370772 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-sqpkc" Sep 11 16:04:52.370993 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:04:52.370804 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-sqpkc" event={"ID":"b5c87369-3c99-4b44-9554-ab0430256596","Type":"ContainerDied","Data":"6b1d51af47c5f9cec6b39900f5d016562bcbf2c03b691f249d42869216be17e1"} Sep 11 16:04:52.370993 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:04:52.370844 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-sqpkc" event={"ID":"b5c87369-3c99-4b44-9554-ab0430256596","Type":"ContainerDied","Data":"ac95e09dce405e8a1e1869b32a62989e6927fca57ea2eb7f17318c1b6446b80c"} Sep 11 16:04:52.370993 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:04:52.370864 2566 scope.go:117] "RemoveContainer" containerID="6b1d51af47c5f9cec6b39900f5d016562bcbf2c03b691f249d42869216be17e1" Sep 11 16:04:52.380147 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:04:52.380128 2566 scope.go:117] "RemoveContainer" containerID="6b1d51af47c5f9cec6b39900f5d016562bcbf2c03b691f249d42869216be17e1" Sep 11 16:04:52.380469 ip-10-0-136-96 kubenswrapper[2566]: E0911 16:04:52.380448 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b1d51af47c5f9cec6b39900f5d016562bcbf2c03b691f249d42869216be17e1\": container with ID starting with 6b1d51af47c5f9cec6b39900f5d016562bcbf2c03b691f249d42869216be17e1 not found: ID does not exist" containerID="6b1d51af47c5f9cec6b39900f5d016562bcbf2c03b691f249d42869216be17e1" Sep 11 16:04:52.380538 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:04:52.380483 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b1d51af47c5f9cec6b39900f5d016562bcbf2c03b691f249d42869216be17e1"} err="failed to get container status \"6b1d51af47c5f9cec6b39900f5d016562bcbf2c03b691f249d42869216be17e1\": rpc error: code = NotFound desc = could not find container \"6b1d51af47c5f9cec6b39900f5d016562bcbf2c03b691f249d42869216be17e1\": container with ID starting with 6b1d51af47c5f9cec6b39900f5d016562bcbf2c03b691f249d42869216be17e1 not found: ID does not exist" Sep 11 16:04:52.397076 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:04:52.397044 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-sqpkc"] Sep 11 16:04:52.401467 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:04:52.401438 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-sqpkc"] Sep 11 16:04:53.230340 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:04:53.230311 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-xjvl5_7542ad77-c817-4370-91de-3b08db264753/console-operator/2.log" Sep 11 16:04:53.232623 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:04:53.232601 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-xjvl5_7542ad77-c817-4370-91de-3b08db264753/console-operator/2.log" Sep 11 16:04:53.238094 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:04:53.238069 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tgq5c_981d7d1e-f81f-48fc-8962-7831bc5c2b27/ovn-acl-logging/0.log" Sep 11 16:04:53.240451 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:04:53.240427 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tgq5c_981d7d1e-f81f-48fc-8962-7831bc5c2b27/ovn-acl-logging/0.log" Sep 11 16:04:53.240545 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:04:53.240461 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b5c87369-3c99-4b44-9554-ab0430256596" path="/var/lib/kubelet/pods/b5c87369-3c99-4b44-9554-ab0430256596/volumes" Sep 11 16:05:35.241010 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:05:35.240979 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dx4xn"] Sep 11 16:05:44.665745 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:05:44.665711 2566 generic.go:356] "Generic (PLEG): container finished" podID="b4fb754a-0ba5-47b6-8fca-ec6a846499f5" containerID="0cbf63c74f3c313253ebd82657543297066f2db1c54912b0ebbce55fe0cae41b" exitCode=0 Sep 11 16:05:44.666173 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:05:44.665790 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2gpqh" event={"ID":"b4fb754a-0ba5-47b6-8fca-ec6a846499f5","Type":"ContainerDied","Data":"0cbf63c74f3c313253ebd82657543297066f2db1c54912b0ebbce55fe0cae41b"} Sep 11 16:05:44.666223 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:05:44.666171 2566 scope.go:117] "RemoveContainer" containerID="0cbf63c74f3c313253ebd82657543297066f2db1c54912b0ebbce55fe0cae41b" Sep 11 16:05:44.667266 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:05:44.667236 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 16:05:44.667359 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:05:44.667345 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 16:05:44.667400 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:05:44.667382 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 16:05:45.673830 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:05:45.673794 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2gpqh" event={"ID":"b4fb754a-0ba5-47b6-8fca-ec6a846499f5","Type":"ContainerStarted","Data":"494bde730c11492dcb1129fbb240037648283bdf9c033a25de503da4f936f0e1"} Sep 11 16:05:46.550822 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:05:46.550765 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2gpqh" Sep 11 16:05:46.551766 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:05:46.551737 2566 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-2gpqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Sep 11 16:05:46.552047 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:05:46.552018 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2gpqh" podUID="b4fb754a-0ba5-47b6-8fca-ec6a846499f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Sep 11 16:05:47.550749 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:05:47.550700 2566 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-2gpqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Sep 11 16:05:47.551258 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:05:47.550779 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2gpqh" podUID="b4fb754a-0ba5-47b6-8fca-ec6a846499f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Sep 11 16:05:48.550746 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:05:48.550707 2566 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-2gpqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Sep 11 16:05:48.551109 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:05:48.551077 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2gpqh" podUID="b4fb754a-0ba5-47b6-8fca-ec6a846499f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Sep 11 16:05:49.096094 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:05:49.096057 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2gpqh" Sep 11 16:05:49.551496 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:05:49.551453 2566 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-2gpqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Sep 11 16:05:49.552108 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:05:49.552076 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2gpqh" podUID="b4fb754a-0ba5-47b6-8fca-ec6a846499f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Sep 11 16:05:50.551681 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:05:50.551633 2566 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-2gpqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Sep 11 16:05:50.552200 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:05:50.551705 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2gpqh" podUID="b4fb754a-0ba5-47b6-8fca-ec6a846499f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Sep 11 16:05:51.554662 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:05:51.554631 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2gpqh" Sep 11 16:05:51.555600 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:05:51.555582 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2gpqh" Sep 11 16:05:59.110384 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:05:59.110348 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dx4xn"] Sep 11 16:06:09.838436 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:06:09.838398 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dx4xn"] Sep 11 16:06:30.125356 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:06:30.125321 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dx4xn"] Sep 11 16:06:48.122619 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:06:48.122535 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dx4xn"] Sep 11 16:07:06.020158 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:07:06.020108 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dx4xn"] Sep 11 16:07:10.618786 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:07:10.618752 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dx4xn"] Sep 11 16:07:31.006188 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:07:31.006147 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-g54l9"] Sep 11 16:07:31.006740 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:07:31.006720 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5c87369-3c99-4b44-9554-ab0430256596" containerName="authorino" Sep 11 16:07:31.006838 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:07:31.006743 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5c87369-3c99-4b44-9554-ab0430256596" containerName="authorino" Sep 11 16:07:31.006896 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:07:31.006875 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5c87369-3c99-4b44-9554-ab0430256596" containerName="authorino" Sep 11 16:07:31.009177 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:07:31.009155 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g54l9" Sep 11 16:07:31.011600 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:07:31.011572 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 11 16:07:31.012273 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:07:31.012246 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 11 16:07:31.012393 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:07:31.012290 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-sxmds\"" Sep 11 16:07:31.012393 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:07:31.012258 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Sep 11 16:07:31.020780 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:07:31.020755 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-g54l9"] Sep 11 16:07:31.093781 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:07:31.093742 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a53fb426-3ed8-49f0-9c0e-041a23cd5e5a-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-g54l9\" (UID: \"a53fb426-3ed8-49f0-9c0e-041a23cd5e5a\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g54l9" Sep 11 16:07:31.093781 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:07:31.093781 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7q5mm\" (UniqueName: \"kubernetes.io/projected/a53fb426-3ed8-49f0-9c0e-041a23cd5e5a-kube-api-access-7q5mm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-g54l9\" (UID: \"a53fb426-3ed8-49f0-9c0e-041a23cd5e5a\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g54l9" Sep 11 16:07:31.094029 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:07:31.093806 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a53fb426-3ed8-49f0-9c0e-041a23cd5e5a-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-g54l9\" (UID: \"a53fb426-3ed8-49f0-9c0e-041a23cd5e5a\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g54l9" Sep 11 16:07:31.094029 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:07:31.093950 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a53fb426-3ed8-49f0-9c0e-041a23cd5e5a-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-g54l9\" (UID: \"a53fb426-3ed8-49f0-9c0e-041a23cd5e5a\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g54l9" Sep 11 16:07:31.094029 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:07:31.094021 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a53fb426-3ed8-49f0-9c0e-041a23cd5e5a-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-g54l9\" (UID: \"a53fb426-3ed8-49f0-9c0e-041a23cd5e5a\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g54l9" Sep 11 16:07:31.094132 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:07:31.094043 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a53fb426-3ed8-49f0-9c0e-041a23cd5e5a-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-g54l9\" (UID: \"a53fb426-3ed8-49f0-9c0e-041a23cd5e5a\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g54l9" Sep 11 16:07:31.195277 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:07:31.195162 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a53fb426-3ed8-49f0-9c0e-041a23cd5e5a-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-g54l9\" (UID: \"a53fb426-3ed8-49f0-9c0e-041a23cd5e5a\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g54l9" Sep 11 16:07:31.195277 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:07:31.195228 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7q5mm\" (UniqueName: \"kubernetes.io/projected/a53fb426-3ed8-49f0-9c0e-041a23cd5e5a-kube-api-access-7q5mm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-g54l9\" (UID: \"a53fb426-3ed8-49f0-9c0e-041a23cd5e5a\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g54l9" Sep 11 16:07:31.195277 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:07:31.195251 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a53fb426-3ed8-49f0-9c0e-041a23cd5e5a-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-g54l9\" (UID: \"a53fb426-3ed8-49f0-9c0e-041a23cd5e5a\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g54l9" Sep 11 16:07:31.195583 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:07:31.195306 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a53fb426-3ed8-49f0-9c0e-041a23cd5e5a-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-g54l9\" (UID: \"a53fb426-3ed8-49f0-9c0e-041a23cd5e5a\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g54l9" Sep 11 16:07:31.195583 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:07:31.195330 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a53fb426-3ed8-49f0-9c0e-041a23cd5e5a-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-g54l9\" (UID: \"a53fb426-3ed8-49f0-9c0e-041a23cd5e5a\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g54l9" Sep 11 16:07:31.195583 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:07:31.195355 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a53fb426-3ed8-49f0-9c0e-041a23cd5e5a-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-g54l9\" (UID: \"a53fb426-3ed8-49f0-9c0e-041a23cd5e5a\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g54l9" Sep 11 16:07:31.195767 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:07:31.195659 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a53fb426-3ed8-49f0-9c0e-041a23cd5e5a-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-g54l9\" (UID: \"a53fb426-3ed8-49f0-9c0e-041a23cd5e5a\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g54l9" Sep 11 16:07:31.195828 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:07:31.195805 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a53fb426-3ed8-49f0-9c0e-041a23cd5e5a-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-g54l9\" (UID: \"a53fb426-3ed8-49f0-9c0e-041a23cd5e5a\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g54l9" Sep 11 16:07:31.195828 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:07:31.195815 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a53fb426-3ed8-49f0-9c0e-041a23cd5e5a-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-g54l9\" (UID: \"a53fb426-3ed8-49f0-9c0e-041a23cd5e5a\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g54l9" Sep 11 16:07:31.197539 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:07:31.197512 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a53fb426-3ed8-49f0-9c0e-041a23cd5e5a-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-g54l9\" (UID: \"a53fb426-3ed8-49f0-9c0e-041a23cd5e5a\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g54l9" Sep 11 16:07:31.197819 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:07:31.197801 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a53fb426-3ed8-49f0-9c0e-041a23cd5e5a-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-g54l9\" (UID: \"a53fb426-3ed8-49f0-9c0e-041a23cd5e5a\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g54l9" Sep 11 16:07:31.204243 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:07:31.204222 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7q5mm\" (UniqueName: \"kubernetes.io/projected/a53fb426-3ed8-49f0-9c0e-041a23cd5e5a-kube-api-access-7q5mm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-g54l9\" (UID: \"a53fb426-3ed8-49f0-9c0e-041a23cd5e5a\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g54l9" Sep 11 16:07:31.319184 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:07:31.319136 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g54l9" Sep 11 16:07:31.601631 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:07:31.601603 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-g54l9"] Sep 11 16:07:31.603669 ip-10-0-136-96 kubenswrapper[2566]: W0911 16:07:31.603640 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda53fb426_3ed8_49f0_9c0e_041a23cd5e5a.slice/crio-fa4be31234674ae8260864eb3b1a0adc0b7b2cf0ea16332317143ef588f86804 WatchSource:0}: Error finding container fa4be31234674ae8260864eb3b1a0adc0b7b2cf0ea16332317143ef588f86804: Status 404 returned error can't find the container with id fa4be31234674ae8260864eb3b1a0adc0b7b2cf0ea16332317143ef588f86804 Sep 11 16:07:31.605108 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:07:31.605087 2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 16:07:32.271050 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:07:32.271008 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g54l9" event={"ID":"a53fb426-3ed8-49f0-9c0e-041a23cd5e5a","Type":"ContainerStarted","Data":"fa4be31234674ae8260864eb3b1a0adc0b7b2cf0ea16332317143ef588f86804"} Sep 11 16:07:33.034925 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:07:33.034889 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dx4xn"] Sep 11 16:07:33.278119 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:07:33.278083 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g54l9" event={"ID":"a53fb426-3ed8-49f0-9c0e-041a23cd5e5a","Type":"ContainerStarted","Data":"9949708e9aeb787c9f761abd04ebeca4e1e8b334de097e94b368346f9cd2a837"} Sep 11 16:07:33.351256 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:07:33.351149 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g54l9" podStartSLOduration=2.428831119 podStartE2EDuration="3.351132073s" totalImagesPullingTime="922.300954ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:07:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:07:33.347735085 +0000 UTC m=+760.562568989" watchObservedRunningTime="2026-09-11 16:07:33.351132073 +0000 UTC m=+760.565965971" Sep 11 16:07:41.320056 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:07:41.320022 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g54l9" Sep 11 16:07:41.320585 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:07:41.320071 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g54l9" Sep 11 16:07:41.332800 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:07:41.332771 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g54l9" Sep 11 16:07:42.345585 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:07:42.345555 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g54l9" Sep 11 16:09:14.373214 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:14.373176 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-65696b975d-t9k9v"] Sep 11 16:09:14.375737 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:14.375716 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-65696b975d-t9k9v" Sep 11 16:09:14.378677 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:14.378657 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Sep 11 16:09:14.378762 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:14.378668 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-4469b\"" Sep 11 16:09:14.402713 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:14.402683 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-65696b975d-t9k9v"] Sep 11 16:09:14.464272 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:14.464233 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9face0b1-6f4b-4bff-a174-b928e8fe3c42-tls-cert\") pod \"authorino-65696b975d-t9k9v\" (UID: \"9face0b1-6f4b-4bff-a174-b928e8fe3c42\") " pod="kuadrant-system/authorino-65696b975d-t9k9v" Sep 11 16:09:14.464450 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:14.464298 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nc8m\" (UniqueName: \"kubernetes.io/projected/9face0b1-6f4b-4bff-a174-b928e8fe3c42-kube-api-access-6nc8m\") pod \"authorino-65696b975d-t9k9v\" (UID: \"9face0b1-6f4b-4bff-a174-b928e8fe3c42\") " pod="kuadrant-system/authorino-65696b975d-t9k9v" Sep 11 16:09:14.565221 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:14.565187 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9face0b1-6f4b-4bff-a174-b928e8fe3c42-tls-cert\") pod \"authorino-65696b975d-t9k9v\" (UID: \"9face0b1-6f4b-4bff-a174-b928e8fe3c42\") " pod="kuadrant-system/authorino-65696b975d-t9k9v" Sep 11 16:09:14.565387 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:14.565251 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nc8m\" (UniqueName: \"kubernetes.io/projected/9face0b1-6f4b-4bff-a174-b928e8fe3c42-kube-api-access-6nc8m\") pod \"authorino-65696b975d-t9k9v\" (UID: \"9face0b1-6f4b-4bff-a174-b928e8fe3c42\") " pod="kuadrant-system/authorino-65696b975d-t9k9v" Sep 11 16:09:14.567768 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:14.567739 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9face0b1-6f4b-4bff-a174-b928e8fe3c42-tls-cert\") pod \"authorino-65696b975d-t9k9v\" (UID: \"9face0b1-6f4b-4bff-a174-b928e8fe3c42\") " pod="kuadrant-system/authorino-65696b975d-t9k9v" Sep 11 16:09:14.573031 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:14.572999 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nc8m\" (UniqueName: \"kubernetes.io/projected/9face0b1-6f4b-4bff-a174-b928e8fe3c42-kube-api-access-6nc8m\") pod \"authorino-65696b975d-t9k9v\" (UID: \"9face0b1-6f4b-4bff-a174-b928e8fe3c42\") " pod="kuadrant-system/authorino-65696b975d-t9k9v" Sep 11 16:09:14.685383 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:14.685285 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-65696b975d-t9k9v" Sep 11 16:09:14.958760 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:14.958728 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-65696b975d-t9k9v"] Sep 11 16:09:14.960650 ip-10-0-136-96 kubenswrapper[2566]: W0911 16:09:14.960617 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9face0b1_6f4b_4bff_a174_b928e8fe3c42.slice/crio-344e092d261d75bdd65ac349b9d728ba4b89e5495a1eb485e8a7c198074f57d2 WatchSource:0}: Error finding container 344e092d261d75bdd65ac349b9d728ba4b89e5495a1eb485e8a7c198074f57d2: Status 404 returned error can't find the container with id 344e092d261d75bdd65ac349b9d728ba4b89e5495a1eb485e8a7c198074f57d2 Sep 11 16:09:15.866817 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:15.866778 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-65696b975d-t9k9v" event={"ID":"9face0b1-6f4b-4bff-a174-b928e8fe3c42","Type":"ContainerStarted","Data":"90e02189b8b2fc8d971b679a6bc36b8ad398f300e1d57f913c24d5717b1ad265"} Sep 11 16:09:15.867220 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:15.866831 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-65696b975d-t9k9v" event={"ID":"9face0b1-6f4b-4bff-a174-b928e8fe3c42","Type":"ContainerStarted","Data":"344e092d261d75bdd65ac349b9d728ba4b89e5495a1eb485e8a7c198074f57d2"} Sep 11 16:09:15.883032 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:15.882964 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-65696b975d-t9k9v" podStartSLOduration=1.151310042 podStartE2EDuration="1.882948837s" totalImagesPullingTime="731.638795ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:09:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:09:15.881858664 +0000 UTC m=+863.096692569" watchObservedRunningTime="2026-09-11 16:09:15.882948837 +0000 UTC m=+863.097782800" Sep 11 16:09:33.728011 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:33.727961 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-8e3121-openshift-default-7f6ffd865d-vbjmp"] Sep 11 16:09:33.731848 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:33.731824 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8e3121-openshift-default-7f6ffd865d-vbjmp" Sep 11 16:09:33.734199 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:33.734168 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-8e3121-openshift-default-dockercfg-64zpn\"" Sep 11 16:09:33.748006 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:33.747957 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-8e3121-openshift-default-7f6ffd865d-vbjmp"] Sep 11 16:09:33.843809 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:33.843765 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96800483-9c43-4f65-b7ae-a46b5a7d35b7-istio-podinfo\") pod \"e2e-inf-a-8e3121-openshift-default-7f6ffd865d-vbjmp\" (UID: \"96800483-9c43-4f65-b7ae-a46b5a7d35b7\") " pod="openshift-ingress/e2e-inf-a-8e3121-openshift-default-7f6ffd865d-vbjmp" Sep 11 16:09:33.843987 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:33.843826 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96800483-9c43-4f65-b7ae-a46b5a7d35b7-workload-socket\") pod \"e2e-inf-a-8e3121-openshift-default-7f6ffd865d-vbjmp\" (UID: \"96800483-9c43-4f65-b7ae-a46b5a7d35b7\") " pod="openshift-ingress/e2e-inf-a-8e3121-openshift-default-7f6ffd865d-vbjmp" Sep 11 16:09:33.843987 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:33.843854 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96800483-9c43-4f65-b7ae-a46b5a7d35b7-credential-socket\") pod \"e2e-inf-a-8e3121-openshift-default-7f6ffd865d-vbjmp\" (UID: \"96800483-9c43-4f65-b7ae-a46b5a7d35b7\") " pod="openshift-ingress/e2e-inf-a-8e3121-openshift-default-7f6ffd865d-vbjmp" Sep 11 16:09:33.843987 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:33.843950 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96800483-9c43-4f65-b7ae-a46b5a7d35b7-istio-envoy\") pod \"e2e-inf-a-8e3121-openshift-default-7f6ffd865d-vbjmp\" (UID: \"96800483-9c43-4f65-b7ae-a46b5a7d35b7\") " pod="openshift-ingress/e2e-inf-a-8e3121-openshift-default-7f6ffd865d-vbjmp" Sep 11 16:09:33.844101 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:33.843996 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5ffk\" (UniqueName: \"kubernetes.io/projected/96800483-9c43-4f65-b7ae-a46b5a7d35b7-kube-api-access-s5ffk\") pod \"e2e-inf-a-8e3121-openshift-default-7f6ffd865d-vbjmp\" (UID: \"96800483-9c43-4f65-b7ae-a46b5a7d35b7\") " pod="openshift-ingress/e2e-inf-a-8e3121-openshift-default-7f6ffd865d-vbjmp" Sep 11 16:09:33.844101 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:33.844029 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96800483-9c43-4f65-b7ae-a46b5a7d35b7-istio-data\") pod \"e2e-inf-a-8e3121-openshift-default-7f6ffd865d-vbjmp\" (UID: \"96800483-9c43-4f65-b7ae-a46b5a7d35b7\") " pod="openshift-ingress/e2e-inf-a-8e3121-openshift-default-7f6ffd865d-vbjmp" Sep 11 16:09:33.844167 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:33.844117 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96800483-9c43-4f65-b7ae-a46b5a7d35b7-workload-certs\") pod \"e2e-inf-a-8e3121-openshift-default-7f6ffd865d-vbjmp\" (UID: \"96800483-9c43-4f65-b7ae-a46b5a7d35b7\") " pod="openshift-ingress/e2e-inf-a-8e3121-openshift-default-7f6ffd865d-vbjmp" Sep 11 16:09:33.844167 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:33.844159 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96800483-9c43-4f65-b7ae-a46b5a7d35b7-istio-token\") pod \"e2e-inf-a-8e3121-openshift-default-7f6ffd865d-vbjmp\" (UID: \"96800483-9c43-4f65-b7ae-a46b5a7d35b7\") " pod="openshift-ingress/e2e-inf-a-8e3121-openshift-default-7f6ffd865d-vbjmp" Sep 11 16:09:33.844231 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:33.844176 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96800483-9c43-4f65-b7ae-a46b5a7d35b7-istiod-ca-cert\") pod \"e2e-inf-a-8e3121-openshift-default-7f6ffd865d-vbjmp\" (UID: \"96800483-9c43-4f65-b7ae-a46b5a7d35b7\") " pod="openshift-ingress/e2e-inf-a-8e3121-openshift-default-7f6ffd865d-vbjmp" Sep 11 16:09:33.945181 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:33.945144 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96800483-9c43-4f65-b7ae-a46b5a7d35b7-workload-certs\") pod \"e2e-inf-a-8e3121-openshift-default-7f6ffd865d-vbjmp\" (UID: \"96800483-9c43-4f65-b7ae-a46b5a7d35b7\") " pod="openshift-ingress/e2e-inf-a-8e3121-openshift-default-7f6ffd865d-vbjmp" Sep 11 16:09:33.945181 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:33.945185 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96800483-9c43-4f65-b7ae-a46b5a7d35b7-istio-token\") pod \"e2e-inf-a-8e3121-openshift-default-7f6ffd865d-vbjmp\" (UID: \"96800483-9c43-4f65-b7ae-a46b5a7d35b7\") " pod="openshift-ingress/e2e-inf-a-8e3121-openshift-default-7f6ffd865d-vbjmp" Sep 11 16:09:33.945400 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:33.945206 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96800483-9c43-4f65-b7ae-a46b5a7d35b7-istiod-ca-cert\") pod \"e2e-inf-a-8e3121-openshift-default-7f6ffd865d-vbjmp\" (UID: \"96800483-9c43-4f65-b7ae-a46b5a7d35b7\") " pod="openshift-ingress/e2e-inf-a-8e3121-openshift-default-7f6ffd865d-vbjmp" Sep 11 16:09:33.945400 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:33.945263 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96800483-9c43-4f65-b7ae-a46b5a7d35b7-istio-podinfo\") pod \"e2e-inf-a-8e3121-openshift-default-7f6ffd865d-vbjmp\" (UID: \"96800483-9c43-4f65-b7ae-a46b5a7d35b7\") " pod="openshift-ingress/e2e-inf-a-8e3121-openshift-default-7f6ffd865d-vbjmp" Sep 11 16:09:33.945400 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:33.945298 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96800483-9c43-4f65-b7ae-a46b5a7d35b7-workload-socket\") pod \"e2e-inf-a-8e3121-openshift-default-7f6ffd865d-vbjmp\" (UID: \"96800483-9c43-4f65-b7ae-a46b5a7d35b7\") " pod="openshift-ingress/e2e-inf-a-8e3121-openshift-default-7f6ffd865d-vbjmp" Sep 11 16:09:33.945528 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:33.945420 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96800483-9c43-4f65-b7ae-a46b5a7d35b7-credential-socket\") pod \"e2e-inf-a-8e3121-openshift-default-7f6ffd865d-vbjmp\" (UID: \"96800483-9c43-4f65-b7ae-a46b5a7d35b7\") " pod="openshift-ingress/e2e-inf-a-8e3121-openshift-default-7f6ffd865d-vbjmp" Sep 11 16:09:33.945574 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:33.945551 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96800483-9c43-4f65-b7ae-a46b5a7d35b7-istio-envoy\") pod \"e2e-inf-a-8e3121-openshift-default-7f6ffd865d-vbjmp\" (UID: \"96800483-9c43-4f65-b7ae-a46b5a7d35b7\") " pod="openshift-ingress/e2e-inf-a-8e3121-openshift-default-7f6ffd865d-vbjmp" Sep 11 16:09:33.945618 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:33.945577 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5ffk\" (UniqueName: \"kubernetes.io/projected/96800483-9c43-4f65-b7ae-a46b5a7d35b7-kube-api-access-s5ffk\") pod \"e2e-inf-a-8e3121-openshift-default-7f6ffd865d-vbjmp\" (UID: \"96800483-9c43-4f65-b7ae-a46b5a7d35b7\") " pod="openshift-ingress/e2e-inf-a-8e3121-openshift-default-7f6ffd865d-vbjmp" Sep 11 16:09:33.945618 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:33.945602 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96800483-9c43-4f65-b7ae-a46b5a7d35b7-istio-data\") pod \"e2e-inf-a-8e3121-openshift-default-7f6ffd865d-vbjmp\" (UID: \"96800483-9c43-4f65-b7ae-a46b5a7d35b7\") " pod="openshift-ingress/e2e-inf-a-8e3121-openshift-default-7f6ffd865d-vbjmp" Sep 11 16:09:33.945727 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:33.945625 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96800483-9c43-4f65-b7ae-a46b5a7d35b7-workload-socket\") pod \"e2e-inf-a-8e3121-openshift-default-7f6ffd865d-vbjmp\" (UID: \"96800483-9c43-4f65-b7ae-a46b5a7d35b7\") " pod="openshift-ingress/e2e-inf-a-8e3121-openshift-default-7f6ffd865d-vbjmp" Sep 11 16:09:33.945727 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:33.945672 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96800483-9c43-4f65-b7ae-a46b5a7d35b7-workload-certs\") pod \"e2e-inf-a-8e3121-openshift-default-7f6ffd865d-vbjmp\" (UID: \"96800483-9c43-4f65-b7ae-a46b5a7d35b7\") " pod="openshift-ingress/e2e-inf-a-8e3121-openshift-default-7f6ffd865d-vbjmp" Sep 11 16:09:33.945831 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:33.945755 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96800483-9c43-4f65-b7ae-a46b5a7d35b7-credential-socket\") pod \"e2e-inf-a-8e3121-openshift-default-7f6ffd865d-vbjmp\" (UID: \"96800483-9c43-4f65-b7ae-a46b5a7d35b7\") " pod="openshift-ingress/e2e-inf-a-8e3121-openshift-default-7f6ffd865d-vbjmp" Sep 11 16:09:33.945962 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:33.945940 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96800483-9c43-4f65-b7ae-a46b5a7d35b7-istio-data\") pod \"e2e-inf-a-8e3121-openshift-default-7f6ffd865d-vbjmp\" (UID: \"96800483-9c43-4f65-b7ae-a46b5a7d35b7\") " pod="openshift-ingress/e2e-inf-a-8e3121-openshift-default-7f6ffd865d-vbjmp" Sep 11 16:09:33.946070 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:33.946053 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96800483-9c43-4f65-b7ae-a46b5a7d35b7-istiod-ca-cert\") pod \"e2e-inf-a-8e3121-openshift-default-7f6ffd865d-vbjmp\" (UID: \"96800483-9c43-4f65-b7ae-a46b5a7d35b7\") " pod="openshift-ingress/e2e-inf-a-8e3121-openshift-default-7f6ffd865d-vbjmp" Sep 11 16:09:33.947790 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:33.947769 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96800483-9c43-4f65-b7ae-a46b5a7d35b7-istio-envoy\") pod \"e2e-inf-a-8e3121-openshift-default-7f6ffd865d-vbjmp\" (UID: \"96800483-9c43-4f65-b7ae-a46b5a7d35b7\") " pod="openshift-ingress/e2e-inf-a-8e3121-openshift-default-7f6ffd865d-vbjmp" Sep 11 16:09:33.948050 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:33.948032 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96800483-9c43-4f65-b7ae-a46b5a7d35b7-istio-podinfo\") pod \"e2e-inf-a-8e3121-openshift-default-7f6ffd865d-vbjmp\" (UID: \"96800483-9c43-4f65-b7ae-a46b5a7d35b7\") " pod="openshift-ingress/e2e-inf-a-8e3121-openshift-default-7f6ffd865d-vbjmp" Sep 11 16:09:33.953827 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:33.953797 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96800483-9c43-4f65-b7ae-a46b5a7d35b7-istio-token\") pod \"e2e-inf-a-8e3121-openshift-default-7f6ffd865d-vbjmp\" (UID: \"96800483-9c43-4f65-b7ae-a46b5a7d35b7\") " pod="openshift-ingress/e2e-inf-a-8e3121-openshift-default-7f6ffd865d-vbjmp" Sep 11 16:09:33.953945 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:33.953930 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5ffk\" (UniqueName: \"kubernetes.io/projected/96800483-9c43-4f65-b7ae-a46b5a7d35b7-kube-api-access-s5ffk\") pod \"e2e-inf-a-8e3121-openshift-default-7f6ffd865d-vbjmp\" (UID: \"96800483-9c43-4f65-b7ae-a46b5a7d35b7\") " pod="openshift-ingress/e2e-inf-a-8e3121-openshift-default-7f6ffd865d-vbjmp" Sep 11 16:09:34.045926 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:34.045888 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8e3121-openshift-default-7f6ffd865d-vbjmp" Sep 11 16:09:34.328041 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:34.328006 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-8e3121-openshift-default-7f6ffd865d-vbjmp"] Sep 11 16:09:34.330162 ip-10-0-136-96 kubenswrapper[2566]: W0911 16:09:34.330132 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96800483_9c43_4f65_b7ae_a46b5a7d35b7.slice/crio-05b6ce85583b9d60218f4e73111033c98d9e72f69ea085eb86077c647ca8c8aa WatchSource:0}: Error finding container 05b6ce85583b9d60218f4e73111033c98d9e72f69ea085eb86077c647ca8c8aa: Status 404 returned error can't find the container with id 05b6ce85583b9d60218f4e73111033c98d9e72f69ea085eb86077c647ca8c8aa Sep 11 16:09:34.332699 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:34.332664 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 16:09:34.332791 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:34.332742 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 16:09:34.332791 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:34.332783 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 16:09:34.977546 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:34.977497 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8e3121-openshift-default-7f6ffd865d-vbjmp" event={"ID":"96800483-9c43-4f65-b7ae-a46b5a7d35b7","Type":"ContainerStarted","Data":"7e91796dff1eebcc4d261b64ebc8b043bec53348f548846eddf543ffa770ce4e"} Sep 11 16:09:34.977988 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:34.977609 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8e3121-openshift-default-7f6ffd865d-vbjmp" event={"ID":"96800483-9c43-4f65-b7ae-a46b5a7d35b7","Type":"ContainerStarted","Data":"05b6ce85583b9d60218f4e73111033c98d9e72f69ea085eb86077c647ca8c8aa"} Sep 11 16:09:35.004615 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:35.004565 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-8e3121-openshift-default-7f6ffd865d-vbjmp" podStartSLOduration=2.004549445 podStartE2EDuration="2.004549445s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:09:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:09:35.00247653 +0000 UTC m=+882.217310435" watchObservedRunningTime="2026-09-11 16:09:35.004549445 +0000 UTC m=+882.219383350" Sep 11 16:09:35.046303 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:35.046271 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-8e3121-openshift-default-7f6ffd865d-vbjmp" Sep 11 16:09:35.051652 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:35.051613 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-8e3121-openshift-default-7f6ffd865d-vbjmp" Sep 11 16:09:35.392047 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:35.392014 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7104-f4ba5c"] Sep 11 16:09:35.395676 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:35.395656 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7104-f4ba5c" Sep 11 16:09:35.397959 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:35.397936 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-vcw5k\"" Sep 11 16:09:35.402222 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:35.402198 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7104-f4ba5c"] Sep 11 16:09:35.460257 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:35.460223 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mffvd\" (UniqueName: \"kubernetes.io/projected/8e6e1a0f-175e-4f54-b11a-9626dcb880d8-kube-api-access-mffvd\") pod \"test-curl-7104-f4ba5c\" (UID: \"8e6e1a0f-175e-4f54-b11a-9626dcb880d8\") " pod="opendatahub/test-curl-7104-f4ba5c" Sep 11 16:09:35.561788 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:35.561756 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mffvd\" (UniqueName: \"kubernetes.io/projected/8e6e1a0f-175e-4f54-b11a-9626dcb880d8-kube-api-access-mffvd\") pod \"test-curl-7104-f4ba5c\" (UID: \"8e6e1a0f-175e-4f54-b11a-9626dcb880d8\") " pod="opendatahub/test-curl-7104-f4ba5c" Sep 11 16:09:35.571245 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:35.571213 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mffvd\" (UniqueName: \"kubernetes.io/projected/8e6e1a0f-175e-4f54-b11a-9626dcb880d8-kube-api-access-mffvd\") pod \"test-curl-7104-f4ba5c\" (UID: \"8e6e1a0f-175e-4f54-b11a-9626dcb880d8\") " pod="opendatahub/test-curl-7104-f4ba5c" Sep 11 16:09:35.707496 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:35.707404 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7104-f4ba5c" Sep 11 16:09:35.982714 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:35.982684 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7104-f4ba5c"] Sep 11 16:09:35.983668 ip-10-0-136-96 kubenswrapper[2566]: W0911 16:09:35.983643 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8e6e1a0f_175e_4f54_b11a_9626dcb880d8.slice/crio-3c33b0e2c965fa33f351c93138e6f69745317691787c276c0567db35cc57ba66 WatchSource:0}: Error finding container 3c33b0e2c965fa33f351c93138e6f69745317691787c276c0567db35cc57ba66: Status 404 returned error can't find the container with id 3c33b0e2c965fa33f351c93138e6f69745317691787c276c0567db35cc57ba66 Sep 11 16:09:35.985154 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:35.985120 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-8e3121-openshift-default-7f6ffd865d-vbjmp" Sep 11 16:09:35.986088 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:35.986068 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-8e3121-openshift-default-7f6ffd865d-vbjmp" Sep 11 16:09:36.942296 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:36.942258 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dx4xn"] Sep 11 16:09:36.995382 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:36.995340 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7104-f4ba5c" event={"ID":"8e6e1a0f-175e-4f54-b11a-9626dcb880d8","Type":"ContainerStarted","Data":"3c33b0e2c965fa33f351c93138e6f69745317691787c276c0567db35cc57ba66"} Sep 11 16:09:39.019700 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:39.019657 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7104-f4ba5c" event={"ID":"8e6e1a0f-175e-4f54-b11a-9626dcb880d8","Type":"ContainerStarted","Data":"8d8f7280ccc7ad57ca10ea7056eff7484903bf27278ee6fff66387fee581f37a"} Sep 11 16:09:39.036789 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:39.036740 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7104-f4ba5c" podStartSLOduration=1.732377899 podStartE2EDuration="4.036724703s" totalImagesPullingTime="2.304346804s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:09:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:09:39.035738626 +0000 UTC m=+886.250572532" watchObservedRunningTime="2026-09-11 16:09:39.036724703 +0000 UTC m=+886.251558607" Sep 11 16:09:39.643802 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:39.643764 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7104-f4ba5c"] Sep 11 16:09:39.653064 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:39.653032 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7104-f4ba5c"] Sep 11 16:09:39.784598 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:39.784565 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7104-05747d"] Sep 11 16:09:39.784992 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:39.784961 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e6e1a0f-175e-4f54-b11a-9626dcb880d8" containerName="test-curl-7104-f4ba5c" Sep 11 16:09:39.785061 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:39.784995 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e6e1a0f-175e-4f54-b11a-9626dcb880d8" containerName="test-curl-7104-f4ba5c" Sep 11 16:09:39.785099 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:39.785082 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e6e1a0f-175e-4f54-b11a-9626dcb880d8" containerName="test-curl-7104-f4ba5c" Sep 11 16:09:39.788348 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:39.788327 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7104-05747d" Sep 11 16:09:39.795542 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:39.795515 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7104-05747d"] Sep 11 16:09:39.906339 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:39.906296 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvzvd\" (UniqueName: \"kubernetes.io/projected/ff6de5f5-1c65-4a98-9d12-c34b8f4db5dc-kube-api-access-zvzvd\") pod \"test-curl-7104-05747d\" (UID: \"ff6de5f5-1c65-4a98-9d12-c34b8f4db5dc\") " pod="opendatahub/test-curl-7104-05747d" Sep 11 16:09:40.007453 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:40.007413 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvzvd\" (UniqueName: \"kubernetes.io/projected/ff6de5f5-1c65-4a98-9d12-c34b8f4db5dc-kube-api-access-zvzvd\") pod \"test-curl-7104-05747d\" (UID: \"ff6de5f5-1c65-4a98-9d12-c34b8f4db5dc\") " pod="opendatahub/test-curl-7104-05747d" Sep 11 16:09:40.023613 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:40.023578 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvzvd\" (UniqueName: \"kubernetes.io/projected/ff6de5f5-1c65-4a98-9d12-c34b8f4db5dc-kube-api-access-zvzvd\") pod \"test-curl-7104-05747d\" (UID: \"ff6de5f5-1c65-4a98-9d12-c34b8f4db5dc\") " pod="opendatahub/test-curl-7104-05747d" Sep 11 16:09:40.026305 ip-10-0-136-96 kubenswrapper[2566]: E0911 16:09:40.026280 2566 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7104-f4ba5c" container="test-curl-7104-f4ba5c" Sep 11 16:09:40.119744 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:40.119660 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7104-05747d" Sep 11 16:09:40.397521 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:40.397491 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7104-05747d"] Sep 11 16:09:40.398568 ip-10-0-136-96 kubenswrapper[2566]: W0911 16:09:40.398544 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podff6de5f5_1c65_4a98_9d12_c34b8f4db5dc.slice/crio-0f04ada50cc33bf3a2c07ce901ee825e1ee5ef5d5e28f38aaa9fd2e41e9acf07 WatchSource:0}: Error finding container 0f04ada50cc33bf3a2c07ce901ee825e1ee5ef5d5e28f38aaa9fd2e41e9acf07: Status 404 returned error can't find the container with id 0f04ada50cc33bf3a2c07ce901ee825e1ee5ef5d5e28f38aaa9fd2e41e9acf07 Sep 11 16:09:41.033735 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:41.033698 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7104-05747d" event={"ID":"ff6de5f5-1c65-4a98-9d12-c34b8f4db5dc","Type":"ContainerStarted","Data":"aed79947e998592ff7d8052e491493891c2c3d2e7340bbe46680cc0c9408be12"} Sep 11 16:09:41.033735 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:41.033743 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7104-05747d" event={"ID":"ff6de5f5-1c65-4a98-9d12-c34b8f4db5dc","Type":"ContainerStarted","Data":"0f04ada50cc33bf3a2c07ce901ee825e1ee5ef5d5e28f38aaa9fd2e41e9acf07"} Sep 11 16:09:41.034192 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:41.033862 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7104-f4ba5c" podUID="8e6e1a0f-175e-4f54-b11a-9626dcb880d8" containerName="test-curl-7104-f4ba5c" containerID="cri-o://8d8f7280ccc7ad57ca10ea7056eff7484903bf27278ee6fff66387fee581f37a" gracePeriod=2 Sep 11 16:09:41.060437 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:41.060372 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7104-05747d" podStartSLOduration=2.060356387 podStartE2EDuration="2.060356387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:09:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:09:41.05866103 +0000 UTC m=+888.273494938" watchObservedRunningTime="2026-09-11 16:09:41.060356387 +0000 UTC m=+888.275190292" Sep 11 16:09:41.060984 ip-10-0-136-96 kubenswrapper[2566]: E0911 16:09:41.060942 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7104-f4ba5c\" is forbidden: User \"system:node:ip-10-0-136-96.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-96.ec2.internal' and this object" podUID="8e6e1a0f-175e-4f54-b11a-9626dcb880d8" pod="opendatahub/test-curl-7104-f4ba5c" Sep 11 16:09:41.686952 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:41.686911 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7104-05747d"] Sep 11 16:09:41.694451 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:41.694416 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7104-05747d"] Sep 11 16:09:41.952469 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:41.952439 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-a6560b69-openshift-default-5c8f947cc4-qcw28"] Sep 11 16:09:41.952886 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:41.952871 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff6de5f5-1c65-4a98-9d12-c34b8f4db5dc" containerName="test-curl-7104-05747d" Sep 11 16:09:41.952940 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:41.952888 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff6de5f5-1c65-4a98-9d12-c34b8f4db5dc" containerName="test-curl-7104-05747d" Sep 11 16:09:41.952994 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:41.952986 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff6de5f5-1c65-4a98-9d12-c34b8f4db5dc" containerName="test-curl-7104-05747d" Sep 11 16:09:41.956378 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:41.956357 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a6560b69-openshift-default-5c8f947cc4-qcw28" Sep 11 16:09:41.959162 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:41.959135 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-a6560b69-openshift-default-dockercfg-zntqd\"" Sep 11 16:09:41.977083 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:41.977035 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-a6560b69-openshift-default-5c8f947cc4-qcw28"] Sep 11 16:09:42.028695 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:42.028659 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f755540-c766-4b46-a53f-dce9a5c44ce8-istio-podinfo\") pod \"e2e-ait-a6560b69-openshift-default-5c8f947cc4-qcw28\" (UID: \"4f755540-c766-4b46-a53f-dce9a5c44ce8\") " pod="openshift-ingress/e2e-ait-a6560b69-openshift-default-5c8f947cc4-qcw28" Sep 11 16:09:42.028882 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:42.028703 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xg5w\" (UniqueName: \"kubernetes.io/projected/4f755540-c766-4b46-a53f-dce9a5c44ce8-kube-api-access-2xg5w\") pod \"e2e-ait-a6560b69-openshift-default-5c8f947cc4-qcw28\" (UID: \"4f755540-c766-4b46-a53f-dce9a5c44ce8\") " pod="openshift-ingress/e2e-ait-a6560b69-openshift-default-5c8f947cc4-qcw28" Sep 11 16:09:42.028882 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:42.028779 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f755540-c766-4b46-a53f-dce9a5c44ce8-workload-socket\") pod \"e2e-ait-a6560b69-openshift-default-5c8f947cc4-qcw28\" (UID: \"4f755540-c766-4b46-a53f-dce9a5c44ce8\") " pod="openshift-ingress/e2e-ait-a6560b69-openshift-default-5c8f947cc4-qcw28" Sep 11 16:09:42.028882 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:42.028833 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f755540-c766-4b46-a53f-dce9a5c44ce8-credential-socket\") pod \"e2e-ait-a6560b69-openshift-default-5c8f947cc4-qcw28\" (UID: \"4f755540-c766-4b46-a53f-dce9a5c44ce8\") " pod="openshift-ingress/e2e-ait-a6560b69-openshift-default-5c8f947cc4-qcw28" Sep 11 16:09:42.029047 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:42.028909 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f755540-c766-4b46-a53f-dce9a5c44ce8-istio-data\") pod \"e2e-ait-a6560b69-openshift-default-5c8f947cc4-qcw28\" (UID: \"4f755540-c766-4b46-a53f-dce9a5c44ce8\") " pod="openshift-ingress/e2e-ait-a6560b69-openshift-default-5c8f947cc4-qcw28" Sep 11 16:09:42.029047 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:42.028940 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f755540-c766-4b46-a53f-dce9a5c44ce8-workload-certs\") pod \"e2e-ait-a6560b69-openshift-default-5c8f947cc4-qcw28\" (UID: \"4f755540-c766-4b46-a53f-dce9a5c44ce8\") " pod="openshift-ingress/e2e-ait-a6560b69-openshift-default-5c8f947cc4-qcw28" Sep 11 16:09:42.029155 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:42.029058 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f755540-c766-4b46-a53f-dce9a5c44ce8-istiod-ca-cert\") pod \"e2e-ait-a6560b69-openshift-default-5c8f947cc4-qcw28\" (UID: \"4f755540-c766-4b46-a53f-dce9a5c44ce8\") " pod="openshift-ingress/e2e-ait-a6560b69-openshift-default-5c8f947cc4-qcw28" Sep 11 16:09:42.029155 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:42.029094 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f755540-c766-4b46-a53f-dce9a5c44ce8-istio-envoy\") pod \"e2e-ait-a6560b69-openshift-default-5c8f947cc4-qcw28\" (UID: \"4f755540-c766-4b46-a53f-dce9a5c44ce8\") " pod="openshift-ingress/e2e-ait-a6560b69-openshift-default-5c8f947cc4-qcw28" Sep 11 16:09:42.029155 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:42.029123 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f755540-c766-4b46-a53f-dce9a5c44ce8-istio-token\") pod \"e2e-ait-a6560b69-openshift-default-5c8f947cc4-qcw28\" (UID: \"4f755540-c766-4b46-a53f-dce9a5c44ce8\") " pod="openshift-ingress/e2e-ait-a6560b69-openshift-default-5c8f947cc4-qcw28" Sep 11 16:09:42.040909 ip-10-0-136-96 kubenswrapper[2566]: E0911 16:09:42.040878 2566 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7104-05747d" container="test-curl-7104-05747d" Sep 11 16:09:42.130676 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:42.130640 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f755540-c766-4b46-a53f-dce9a5c44ce8-istio-podinfo\") pod \"e2e-ait-a6560b69-openshift-default-5c8f947cc4-qcw28\" (UID: \"4f755540-c766-4b46-a53f-dce9a5c44ce8\") " pod="openshift-ingress/e2e-ait-a6560b69-openshift-default-5c8f947cc4-qcw28" Sep 11 16:09:42.130676 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:42.130689 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xg5w\" (UniqueName: \"kubernetes.io/projected/4f755540-c766-4b46-a53f-dce9a5c44ce8-kube-api-access-2xg5w\") pod \"e2e-ait-a6560b69-openshift-default-5c8f947cc4-qcw28\" (UID: \"4f755540-c766-4b46-a53f-dce9a5c44ce8\") " pod="openshift-ingress/e2e-ait-a6560b69-openshift-default-5c8f947cc4-qcw28" Sep 11 16:09:42.130933 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:42.130727 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f755540-c766-4b46-a53f-dce9a5c44ce8-workload-socket\") pod \"e2e-ait-a6560b69-openshift-default-5c8f947cc4-qcw28\" (UID: \"4f755540-c766-4b46-a53f-dce9a5c44ce8\") " pod="openshift-ingress/e2e-ait-a6560b69-openshift-default-5c8f947cc4-qcw28" Sep 11 16:09:42.130933 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:42.130756 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f755540-c766-4b46-a53f-dce9a5c44ce8-credential-socket\") pod \"e2e-ait-a6560b69-openshift-default-5c8f947cc4-qcw28\" (UID: \"4f755540-c766-4b46-a53f-dce9a5c44ce8\") " pod="openshift-ingress/e2e-ait-a6560b69-openshift-default-5c8f947cc4-qcw28" Sep 11 16:09:42.130933 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:42.130807 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f755540-c766-4b46-a53f-dce9a5c44ce8-istio-data\") pod \"e2e-ait-a6560b69-openshift-default-5c8f947cc4-qcw28\" (UID: \"4f755540-c766-4b46-a53f-dce9a5c44ce8\") " pod="openshift-ingress/e2e-ait-a6560b69-openshift-default-5c8f947cc4-qcw28" Sep 11 16:09:42.130933 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:42.130837 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f755540-c766-4b46-a53f-dce9a5c44ce8-workload-certs\") pod \"e2e-ait-a6560b69-openshift-default-5c8f947cc4-qcw28\" (UID: \"4f755540-c766-4b46-a53f-dce9a5c44ce8\") " pod="openshift-ingress/e2e-ait-a6560b69-openshift-default-5c8f947cc4-qcw28" Sep 11 16:09:42.130933 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:42.130904 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f755540-c766-4b46-a53f-dce9a5c44ce8-istiod-ca-cert\") pod \"e2e-ait-a6560b69-openshift-default-5c8f947cc4-qcw28\" (UID: \"4f755540-c766-4b46-a53f-dce9a5c44ce8\") " pod="openshift-ingress/e2e-ait-a6560b69-openshift-default-5c8f947cc4-qcw28" Sep 11 16:09:42.131210 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:42.130937 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f755540-c766-4b46-a53f-dce9a5c44ce8-istio-envoy\") pod \"e2e-ait-a6560b69-openshift-default-5c8f947cc4-qcw28\" (UID: \"4f755540-c766-4b46-a53f-dce9a5c44ce8\") " pod="openshift-ingress/e2e-ait-a6560b69-openshift-default-5c8f947cc4-qcw28" Sep 11 16:09:42.131210 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:42.130965 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f755540-c766-4b46-a53f-dce9a5c44ce8-istio-token\") pod \"e2e-ait-a6560b69-openshift-default-5c8f947cc4-qcw28\" (UID: \"4f755540-c766-4b46-a53f-dce9a5c44ce8\") " pod="openshift-ingress/e2e-ait-a6560b69-openshift-default-5c8f947cc4-qcw28" Sep 11 16:09:42.131318 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:42.131295 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f755540-c766-4b46-a53f-dce9a5c44ce8-workload-socket\") pod \"e2e-ait-a6560b69-openshift-default-5c8f947cc4-qcw28\" (UID: \"4f755540-c766-4b46-a53f-dce9a5c44ce8\") " pod="openshift-ingress/e2e-ait-a6560b69-openshift-default-5c8f947cc4-qcw28" Sep 11 16:09:42.131489 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:42.131441 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f755540-c766-4b46-a53f-dce9a5c44ce8-workload-certs\") pod \"e2e-ait-a6560b69-openshift-default-5c8f947cc4-qcw28\" (UID: \"4f755540-c766-4b46-a53f-dce9a5c44ce8\") " pod="openshift-ingress/e2e-ait-a6560b69-openshift-default-5c8f947cc4-qcw28" Sep 11 16:09:42.131620 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:42.131490 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f755540-c766-4b46-a53f-dce9a5c44ce8-credential-socket\") pod \"e2e-ait-a6560b69-openshift-default-5c8f947cc4-qcw28\" (UID: \"4f755540-c766-4b46-a53f-dce9a5c44ce8\") " pod="openshift-ingress/e2e-ait-a6560b69-openshift-default-5c8f947cc4-qcw28" Sep 11 16:09:42.131731 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:42.131711 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f755540-c766-4b46-a53f-dce9a5c44ce8-istio-data\") pod \"e2e-ait-a6560b69-openshift-default-5c8f947cc4-qcw28\" (UID: \"4f755540-c766-4b46-a53f-dce9a5c44ce8\") " pod="openshift-ingress/e2e-ait-a6560b69-openshift-default-5c8f947cc4-qcw28" Sep 11 16:09:42.131792 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:42.131716 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f755540-c766-4b46-a53f-dce9a5c44ce8-istiod-ca-cert\") pod \"e2e-ait-a6560b69-openshift-default-5c8f947cc4-qcw28\" (UID: \"4f755540-c766-4b46-a53f-dce9a5c44ce8\") " pod="openshift-ingress/e2e-ait-a6560b69-openshift-default-5c8f947cc4-qcw28" Sep 11 16:09:42.133413 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:42.133381 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f755540-c766-4b46-a53f-dce9a5c44ce8-istio-podinfo\") pod \"e2e-ait-a6560b69-openshift-default-5c8f947cc4-qcw28\" (UID: \"4f755540-c766-4b46-a53f-dce9a5c44ce8\") " pod="openshift-ingress/e2e-ait-a6560b69-openshift-default-5c8f947cc4-qcw28" Sep 11 16:09:42.133510 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:42.133429 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f755540-c766-4b46-a53f-dce9a5c44ce8-istio-envoy\") pod \"e2e-ait-a6560b69-openshift-default-5c8f947cc4-qcw28\" (UID: \"4f755540-c766-4b46-a53f-dce9a5c44ce8\") " pod="openshift-ingress/e2e-ait-a6560b69-openshift-default-5c8f947cc4-qcw28" Sep 11 16:09:42.143830 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:42.143801 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f755540-c766-4b46-a53f-dce9a5c44ce8-istio-token\") pod \"e2e-ait-a6560b69-openshift-default-5c8f947cc4-qcw28\" (UID: \"4f755540-c766-4b46-a53f-dce9a5c44ce8\") " pod="openshift-ingress/e2e-ait-a6560b69-openshift-default-5c8f947cc4-qcw28" Sep 11 16:09:42.143933 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:42.143877 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xg5w\" (UniqueName: \"kubernetes.io/projected/4f755540-c766-4b46-a53f-dce9a5c44ce8-kube-api-access-2xg5w\") pod \"e2e-ait-a6560b69-openshift-default-5c8f947cc4-qcw28\" (UID: \"4f755540-c766-4b46-a53f-dce9a5c44ce8\") " pod="openshift-ingress/e2e-ait-a6560b69-openshift-default-5c8f947cc4-qcw28" Sep 11 16:09:42.268288 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:42.268246 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a6560b69-openshift-default-5c8f947cc4-qcw28" Sep 11 16:09:42.560043 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:42.559611 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-9c5b09-openshift-default-66c68b44fd-7dtg6"] Sep 11 16:09:42.573996 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:42.573295 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-9c5b09-openshift-default-66c68b44fd-7dtg6" Sep 11 16:09:42.577955 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:42.577842 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-a6560b69-openshift-default-5c8f947cc4-qcw28"] Sep 11 16:09:42.579640 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:42.579281 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-9c5b09-openshift-default-dockercfg-qnw24\"" Sep 11 16:09:42.602406 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:42.602334 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 16:09:42.602544 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:42.602443 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 16:09:42.602544 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:42.602492 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 16:09:42.608295 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:42.607442 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-9c5b09-openshift-default-66c68b44fd-7dtg6"] Sep 11 16:09:42.637535 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:42.636643 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c028f61-e84c-4c34-9a94-7c7e0c9a2212-istio-envoy\") pod \"e2e-autores-9c5b09-openshift-default-66c68b44fd-7dtg6\" (UID: \"3c028f61-e84c-4c34-9a94-7c7e0c9a2212\") " pod="openshift-ingress/e2e-autores-9c5b09-openshift-default-66c68b44fd-7dtg6" Sep 11 16:09:42.637535 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:42.636790 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c028f61-e84c-4c34-9a94-7c7e0c9a2212-istio-token\") pod \"e2e-autores-9c5b09-openshift-default-66c68b44fd-7dtg6\" (UID: \"3c028f61-e84c-4c34-9a94-7c7e0c9a2212\") " pod="openshift-ingress/e2e-autores-9c5b09-openshift-default-66c68b44fd-7dtg6" Sep 11 16:09:42.637535 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:42.636838 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c028f61-e84c-4c34-9a94-7c7e0c9a2212-credential-socket\") pod \"e2e-autores-9c5b09-openshift-default-66c68b44fd-7dtg6\" (UID: \"3c028f61-e84c-4c34-9a94-7c7e0c9a2212\") " pod="openshift-ingress/e2e-autores-9c5b09-openshift-default-66c68b44fd-7dtg6" Sep 11 16:09:42.637535 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:42.636879 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xz76r\" (UniqueName: \"kubernetes.io/projected/3c028f61-e84c-4c34-9a94-7c7e0c9a2212-kube-api-access-xz76r\") pod \"e2e-autores-9c5b09-openshift-default-66c68b44fd-7dtg6\" (UID: \"3c028f61-e84c-4c34-9a94-7c7e0c9a2212\") " pod="openshift-ingress/e2e-autores-9c5b09-openshift-default-66c68b44fd-7dtg6" Sep 11 16:09:42.637535 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:42.636950 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c028f61-e84c-4c34-9a94-7c7e0c9a2212-istio-data\") pod \"e2e-autores-9c5b09-openshift-default-66c68b44fd-7dtg6\" (UID: \"3c028f61-e84c-4c34-9a94-7c7e0c9a2212\") " pod="openshift-ingress/e2e-autores-9c5b09-openshift-default-66c68b44fd-7dtg6" Sep 11 16:09:42.637535 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:42.637025 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c028f61-e84c-4c34-9a94-7c7e0c9a2212-istio-podinfo\") pod \"e2e-autores-9c5b09-openshift-default-66c68b44fd-7dtg6\" (UID: \"3c028f61-e84c-4c34-9a94-7c7e0c9a2212\") " pod="openshift-ingress/e2e-autores-9c5b09-openshift-default-66c68b44fd-7dtg6" Sep 11 16:09:42.637535 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:42.637063 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c028f61-e84c-4c34-9a94-7c7e0c9a2212-istiod-ca-cert\") pod \"e2e-autores-9c5b09-openshift-default-66c68b44fd-7dtg6\" (UID: \"3c028f61-e84c-4c34-9a94-7c7e0c9a2212\") " pod="openshift-ingress/e2e-autores-9c5b09-openshift-default-66c68b44fd-7dtg6" Sep 11 16:09:42.637535 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:42.637109 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c028f61-e84c-4c34-9a94-7c7e0c9a2212-workload-certs\") pod \"e2e-autores-9c5b09-openshift-default-66c68b44fd-7dtg6\" (UID: \"3c028f61-e84c-4c34-9a94-7c7e0c9a2212\") " pod="openshift-ingress/e2e-autores-9c5b09-openshift-default-66c68b44fd-7dtg6" Sep 11 16:09:42.637535 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:42.637142 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c028f61-e84c-4c34-9a94-7c7e0c9a2212-workload-socket\") pod \"e2e-autores-9c5b09-openshift-default-66c68b44fd-7dtg6\" (UID: \"3c028f61-e84c-4c34-9a94-7c7e0c9a2212\") " pod="openshift-ingress/e2e-autores-9c5b09-openshift-default-66c68b44fd-7dtg6" Sep 11 16:09:42.739676 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:42.738124 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c028f61-e84c-4c34-9a94-7c7e0c9a2212-istio-data\") pod \"e2e-autores-9c5b09-openshift-default-66c68b44fd-7dtg6\" (UID: \"3c028f61-e84c-4c34-9a94-7c7e0c9a2212\") " pod="openshift-ingress/e2e-autores-9c5b09-openshift-default-66c68b44fd-7dtg6" Sep 11 16:09:42.739676 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:42.738186 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c028f61-e84c-4c34-9a94-7c7e0c9a2212-istio-podinfo\") pod \"e2e-autores-9c5b09-openshift-default-66c68b44fd-7dtg6\" (UID: \"3c028f61-e84c-4c34-9a94-7c7e0c9a2212\") " pod="openshift-ingress/e2e-autores-9c5b09-openshift-default-66c68b44fd-7dtg6" Sep 11 16:09:42.739676 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:42.738218 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c028f61-e84c-4c34-9a94-7c7e0c9a2212-istiod-ca-cert\") pod \"e2e-autores-9c5b09-openshift-default-66c68b44fd-7dtg6\" (UID: \"3c028f61-e84c-4c34-9a94-7c7e0c9a2212\") " pod="openshift-ingress/e2e-autores-9c5b09-openshift-default-66c68b44fd-7dtg6" Sep 11 16:09:42.739676 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:42.738265 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c028f61-e84c-4c34-9a94-7c7e0c9a2212-workload-certs\") pod \"e2e-autores-9c5b09-openshift-default-66c68b44fd-7dtg6\" (UID: \"3c028f61-e84c-4c34-9a94-7c7e0c9a2212\") " pod="openshift-ingress/e2e-autores-9c5b09-openshift-default-66c68b44fd-7dtg6" Sep 11 16:09:42.739676 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:42.738317 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c028f61-e84c-4c34-9a94-7c7e0c9a2212-workload-socket\") pod \"e2e-autores-9c5b09-openshift-default-66c68b44fd-7dtg6\" (UID: \"3c028f61-e84c-4c34-9a94-7c7e0c9a2212\") " pod="openshift-ingress/e2e-autores-9c5b09-openshift-default-66c68b44fd-7dtg6" Sep 11 16:09:42.739676 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:42.738355 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c028f61-e84c-4c34-9a94-7c7e0c9a2212-istio-envoy\") pod \"e2e-autores-9c5b09-openshift-default-66c68b44fd-7dtg6\" (UID: \"3c028f61-e84c-4c34-9a94-7c7e0c9a2212\") " pod="openshift-ingress/e2e-autores-9c5b09-openshift-default-66c68b44fd-7dtg6" Sep 11 16:09:42.739676 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:42.738419 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c028f61-e84c-4c34-9a94-7c7e0c9a2212-istio-token\") pod \"e2e-autores-9c5b09-openshift-default-66c68b44fd-7dtg6\" (UID: \"3c028f61-e84c-4c34-9a94-7c7e0c9a2212\") " pod="openshift-ingress/e2e-autores-9c5b09-openshift-default-66c68b44fd-7dtg6" Sep 11 16:09:42.739676 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:42.738456 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c028f61-e84c-4c34-9a94-7c7e0c9a2212-credential-socket\") pod \"e2e-autores-9c5b09-openshift-default-66c68b44fd-7dtg6\" (UID: \"3c028f61-e84c-4c34-9a94-7c7e0c9a2212\") " pod="openshift-ingress/e2e-autores-9c5b09-openshift-default-66c68b44fd-7dtg6" Sep 11 16:09:42.739676 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:42.738497 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xz76r\" (UniqueName: \"kubernetes.io/projected/3c028f61-e84c-4c34-9a94-7c7e0c9a2212-kube-api-access-xz76r\") pod \"e2e-autores-9c5b09-openshift-default-66c68b44fd-7dtg6\" (UID: \"3c028f61-e84c-4c34-9a94-7c7e0c9a2212\") " pod="openshift-ingress/e2e-autores-9c5b09-openshift-default-66c68b44fd-7dtg6" Sep 11 16:09:42.739676 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:42.739268 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c028f61-e84c-4c34-9a94-7c7e0c9a2212-istio-data\") pod \"e2e-autores-9c5b09-openshift-default-66c68b44fd-7dtg6\" (UID: \"3c028f61-e84c-4c34-9a94-7c7e0c9a2212\") " pod="openshift-ingress/e2e-autores-9c5b09-openshift-default-66c68b44fd-7dtg6" Sep 11 16:09:42.740377 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:42.740154 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c028f61-e84c-4c34-9a94-7c7e0c9a2212-workload-socket\") pod \"e2e-autores-9c5b09-openshift-default-66c68b44fd-7dtg6\" (UID: \"3c028f61-e84c-4c34-9a94-7c7e0c9a2212\") " pod="openshift-ingress/e2e-autores-9c5b09-openshift-default-66c68b44fd-7dtg6" Sep 11 16:09:42.741167 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:42.740823 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c028f61-e84c-4c34-9a94-7c7e0c9a2212-istiod-ca-cert\") pod \"e2e-autores-9c5b09-openshift-default-66c68b44fd-7dtg6\" (UID: \"3c028f61-e84c-4c34-9a94-7c7e0c9a2212\") " pod="openshift-ingress/e2e-autores-9c5b09-openshift-default-66c68b44fd-7dtg6" Sep 11 16:09:42.741167 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:42.741128 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c028f61-e84c-4c34-9a94-7c7e0c9a2212-credential-socket\") pod \"e2e-autores-9c5b09-openshift-default-66c68b44fd-7dtg6\" (UID: \"3c028f61-e84c-4c34-9a94-7c7e0c9a2212\") " pod="openshift-ingress/e2e-autores-9c5b09-openshift-default-66c68b44fd-7dtg6" Sep 11 16:09:42.741996 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:42.741386 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c028f61-e84c-4c34-9a94-7c7e0c9a2212-workload-certs\") pod \"e2e-autores-9c5b09-openshift-default-66c68b44fd-7dtg6\" (UID: \"3c028f61-e84c-4c34-9a94-7c7e0c9a2212\") " pod="openshift-ingress/e2e-autores-9c5b09-openshift-default-66c68b44fd-7dtg6" Sep 11 16:09:42.747411 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:42.742672 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c028f61-e84c-4c34-9a94-7c7e0c9a2212-istio-podinfo\") pod \"e2e-autores-9c5b09-openshift-default-66c68b44fd-7dtg6\" (UID: \"3c028f61-e84c-4c34-9a94-7c7e0c9a2212\") " pod="openshift-ingress/e2e-autores-9c5b09-openshift-default-66c68b44fd-7dtg6" Sep 11 16:09:42.747411 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:42.746462 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c028f61-e84c-4c34-9a94-7c7e0c9a2212-istio-envoy\") pod \"e2e-autores-9c5b09-openshift-default-66c68b44fd-7dtg6\" (UID: \"3c028f61-e84c-4c34-9a94-7c7e0c9a2212\") " pod="openshift-ingress/e2e-autores-9c5b09-openshift-default-66c68b44fd-7dtg6" Sep 11 16:09:42.750460 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:42.750428 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c028f61-e84c-4c34-9a94-7c7e0c9a2212-istio-token\") pod \"e2e-autores-9c5b09-openshift-default-66c68b44fd-7dtg6\" (UID: \"3c028f61-e84c-4c34-9a94-7c7e0c9a2212\") " pod="openshift-ingress/e2e-autores-9c5b09-openshift-default-66c68b44fd-7dtg6" Sep 11 16:09:42.751016 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:42.750962 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xz76r\" (UniqueName: \"kubernetes.io/projected/3c028f61-e84c-4c34-9a94-7c7e0c9a2212-kube-api-access-xz76r\") pod \"e2e-autores-9c5b09-openshift-default-66c68b44fd-7dtg6\" (UID: \"3c028f61-e84c-4c34-9a94-7c7e0c9a2212\") " pod="openshift-ingress/e2e-autores-9c5b09-openshift-default-66c68b44fd-7dtg6" Sep 11 16:09:42.904274 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:42.904173 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-9c5b09-openshift-default-66c68b44fd-7dtg6" Sep 11 16:09:43.064030 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:43.062249 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7104-05747d" podUID="ff6de5f5-1c65-4a98-9d12-c34b8f4db5dc" containerName="test-curl-7104-05747d" containerID="cri-o://aed79947e998592ff7d8052e491493891c2c3d2e7340bbe46680cc0c9408be12" gracePeriod=2 Sep 11 16:09:43.064030 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:43.063126 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a6560b69-openshift-default-5c8f947cc4-qcw28" event={"ID":"4f755540-c766-4b46-a53f-dce9a5c44ce8","Type":"ContainerStarted","Data":"ce243feda875ee9607bcad9a840670b0bfe697c8944f5618604ecdc029bb4da1"} Sep 11 16:09:43.064030 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:43.063180 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a6560b69-openshift-default-5c8f947cc4-qcw28" event={"ID":"4f755540-c766-4b46-a53f-dce9a5c44ce8","Type":"ContainerStarted","Data":"c4609b4611b5ff465493134edf9251906ff2af7b19056a441c9c9f3c2c27d77d"} Sep 11 16:09:43.108569 ip-10-0-136-96 kubenswrapper[2566]: E0911 16:09:43.108525 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7104-05747d\" is forbidden: User \"system:node:ip-10-0-136-96.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-96.ec2.internal' and this object" podUID="ff6de5f5-1c65-4a98-9d12-c34b8f4db5dc" pod="opendatahub/test-curl-7104-05747d" Sep 11 16:09:43.110155 ip-10-0-136-96 kubenswrapper[2566]: E0911 16:09:43.110119 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7104-05747d\" is forbidden: User \"system:node:ip-10-0-136-96.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-96.ec2.internal' and this object" podUID="ff6de5f5-1c65-4a98-9d12-c34b8f4db5dc" pod="opendatahub/test-curl-7104-05747d" Sep 11 16:09:43.259562 ip-10-0-136-96 kubenswrapper[2566]: E0911 16:09:43.259472 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7104-05747d\" is forbidden: User \"system:node:ip-10-0-136-96.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-96.ec2.internal' and this object" podUID="ff6de5f5-1c65-4a98-9d12-c34b8f4db5dc" pod="opendatahub/test-curl-7104-05747d" Sep 11 16:09:43.268994 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:43.262827 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-a6560b69-openshift-default-5c8f947cc4-qcw28" podStartSLOduration=2.262798295 podStartE2EDuration="2.262798295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:09:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:09:43.106824159 +0000 UTC m=+890.321658064" watchObservedRunningTime="2026-09-11 16:09:43.262798295 +0000 UTC m=+890.477632200" Sep 11 16:09:43.288274 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:43.269705 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-a6560b69-openshift-default-5c8f947cc4-qcw28" Sep 11 16:09:43.288612 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:43.285577 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 16:09:43.288925 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:43.288597 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-a6560b69-openshift-default-5c8f947cc4-qcw28" Sep 11 16:09:43.289109 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:43.288906 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-9c5b09-openshift-default-66c68b44fd-7dtg6"] Sep 11 16:09:43.289271 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:43.289239 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 16:09:43.289429 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:43.289403 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 16:09:43.399442 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:43.399412 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7104-f4ba5c" Sep 11 16:09:43.401473 ip-10-0-136-96 kubenswrapper[2566]: E0911 16:09:43.401439 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7104-f4ba5c\" is forbidden: User \"system:node:ip-10-0-136-96.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-96.ec2.internal' and this object" podUID="8e6e1a0f-175e-4f54-b11a-9626dcb880d8" pod="opendatahub/test-curl-7104-f4ba5c" Sep 11 16:09:43.554720 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:43.554657 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mffvd\" (UniqueName: \"kubernetes.io/projected/8e6e1a0f-175e-4f54-b11a-9626dcb880d8-kube-api-access-mffvd\") pod \"8e6e1a0f-175e-4f54-b11a-9626dcb880d8\" (UID: \"8e6e1a0f-175e-4f54-b11a-9626dcb880d8\") " Sep 11 16:09:43.568667 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:43.560147 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e6e1a0f-175e-4f54-b11a-9626dcb880d8-kube-api-access-mffvd" (OuterVolumeSpecName: "kube-api-access-mffvd") pod "8e6e1a0f-175e-4f54-b11a-9626dcb880d8" (UID: "8e6e1a0f-175e-4f54-b11a-9626dcb880d8"). InnerVolumeSpecName "kube-api-access-mffvd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 16:09:43.659993 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:43.659651 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mffvd\" (UniqueName: \"kubernetes.io/projected/8e6e1a0f-175e-4f54-b11a-9626dcb880d8-kube-api-access-mffvd\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:09:44.069304 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:44.069257 2566 generic.go:356] "Generic (PLEG): container finished" podID="8e6e1a0f-175e-4f54-b11a-9626dcb880d8" containerID="8d8f7280ccc7ad57ca10ea7056eff7484903bf27278ee6fff66387fee581f37a" exitCode=137 Sep 11 16:09:44.069304 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:44.069302 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7104-f4ba5c" Sep 11 16:09:44.069835 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:44.069353 2566 scope.go:117] "RemoveContainer" containerID="8d8f7280ccc7ad57ca10ea7056eff7484903bf27278ee6fff66387fee581f37a" Sep 11 16:09:44.071398 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:44.071353 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-9c5b09-openshift-default-66c68b44fd-7dtg6" event={"ID":"3c028f61-e84c-4c34-9a94-7c7e0c9a2212","Type":"ContainerStarted","Data":"6a1a9469ace076fbd71923f1224d7f272ac7b9e76b82d85c0bddecf0a72d9ce1"} Sep 11 16:09:44.071537 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:44.071434 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-9c5b09-openshift-default-66c68b44fd-7dtg6" event={"ID":"3c028f61-e84c-4c34-9a94-7c7e0c9a2212","Type":"ContainerStarted","Data":"1a28367d0e1cb4cd3ef14af4108af09bfc39e55000fc7ab1f3aa1e0f48699cf8"} Sep 11 16:09:44.071694 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:44.071631 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-a6560b69-openshift-default-5c8f947cc4-qcw28" Sep 11 16:09:44.072038 ip-10-0-136-96 kubenswrapper[2566]: E0911 16:09:44.071898 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7104-f4ba5c\" is forbidden: User \"system:node:ip-10-0-136-96.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-96.ec2.internal' and this object" podUID="8e6e1a0f-175e-4f54-b11a-9626dcb880d8" pod="opendatahub/test-curl-7104-f4ba5c" Sep 11 16:09:44.073021 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:44.072964 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-a6560b69-openshift-default-5c8f947cc4-qcw28" Sep 11 16:09:44.079891 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:44.079875 2566 scope.go:117] "RemoveContainer" containerID="8d8f7280ccc7ad57ca10ea7056eff7484903bf27278ee6fff66387fee581f37a" Sep 11 16:09:44.080210 ip-10-0-136-96 kubenswrapper[2566]: E0911 16:09:44.080185 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d8f7280ccc7ad57ca10ea7056eff7484903bf27278ee6fff66387fee581f37a\": container with ID starting with 8d8f7280ccc7ad57ca10ea7056eff7484903bf27278ee6fff66387fee581f37a not found: ID does not exist" containerID="8d8f7280ccc7ad57ca10ea7056eff7484903bf27278ee6fff66387fee581f37a" Sep 11 16:09:44.080271 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:44.080218 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d8f7280ccc7ad57ca10ea7056eff7484903bf27278ee6fff66387fee581f37a"} err="failed to get container status \"8d8f7280ccc7ad57ca10ea7056eff7484903bf27278ee6fff66387fee581f37a\": rpc error: code = NotFound desc = could not find container \"8d8f7280ccc7ad57ca10ea7056eff7484903bf27278ee6fff66387fee581f37a\": container with ID starting with 8d8f7280ccc7ad57ca10ea7056eff7484903bf27278ee6fff66387fee581f37a not found: ID does not exist" Sep 11 16:09:44.093875 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:44.093829 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-9c5b09-openshift-default-66c68b44fd-7dtg6" podStartSLOduration=2.09381477 podStartE2EDuration="2.09381477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:09:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:09:44.092046407 +0000 UTC m=+891.306880313" watchObservedRunningTime="2026-09-11 16:09:44.09381477 +0000 UTC m=+891.308648735" Sep 11 16:09:44.094257 ip-10-0-136-96 kubenswrapper[2566]: E0911 16:09:44.094224 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7104-f4ba5c\" is forbidden: User \"system:node:ip-10-0-136-96.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-96.ec2.internal' and this object" podUID="8e6e1a0f-175e-4f54-b11a-9626dcb880d8" pod="opendatahub/test-curl-7104-f4ba5c" Sep 11 16:09:44.101036 ip-10-0-136-96 kubenswrapper[2566]: E0911 16:09:44.099178 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7104-f4ba5c\" is forbidden: User \"system:node:ip-10-0-136-96.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-96.ec2.internal' and this object" podUID="8e6e1a0f-175e-4f54-b11a-9626dcb880d8" pod="opendatahub/test-curl-7104-f4ba5c" Sep 11 16:09:44.904936 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:44.904892 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-9c5b09-openshift-default-66c68b44fd-7dtg6" Sep 11 16:09:44.910127 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:44.910100 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-9c5b09-openshift-default-66c68b44fd-7dtg6" Sep 11 16:09:44.945320 ip-10-0-136-96 kubenswrapper[2566]: E0911 16:09:44.945269 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7104-f4ba5c\" is forbidden: User \"system:node:ip-10-0-136-96.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-96.ec2.internal' and this object" podUID="8e6e1a0f-175e-4f54-b11a-9626dcb880d8" pod="opendatahub/test-curl-7104-f4ba5c" Sep 11 16:09:45.080219 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:45.080191 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-9c5b09-openshift-default-66c68b44fd-7dtg6" Sep 11 16:09:45.081095 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:45.081075 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-9c5b09-openshift-default-66c68b44fd-7dtg6" Sep 11 16:09:45.103622 ip-10-0-136-96 kubenswrapper[2566]: E0911 16:09:45.103576 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7104-f4ba5c\" is forbidden: User \"system:node:ip-10-0-136-96.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-96.ec2.internal' and this object" podUID="8e6e1a0f-175e-4f54-b11a-9626dcb880d8" pod="opendatahub/test-curl-7104-f4ba5c" Sep 11 16:09:45.255674 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:45.255630 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e6e1a0f-175e-4f54-b11a-9626dcb880d8" path="/var/lib/kubelet/pods/8e6e1a0f-175e-4f54-b11a-9626dcb880d8/volumes" Sep 11 16:09:45.321465 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:45.321439 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7104-05747d" Sep 11 16:09:45.488999 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:45.488884 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zvzvd\" (UniqueName: \"kubernetes.io/projected/ff6de5f5-1c65-4a98-9d12-c34b8f4db5dc-kube-api-access-zvzvd\") pod \"ff6de5f5-1c65-4a98-9d12-c34b8f4db5dc\" (UID: \"ff6de5f5-1c65-4a98-9d12-c34b8f4db5dc\") " Sep 11 16:09:45.491111 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:45.491080 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff6de5f5-1c65-4a98-9d12-c34b8f4db5dc-kube-api-access-zvzvd" (OuterVolumeSpecName: "kube-api-access-zvzvd") pod "ff6de5f5-1c65-4a98-9d12-c34b8f4db5dc" (UID: "ff6de5f5-1c65-4a98-9d12-c34b8f4db5dc"). InnerVolumeSpecName "kube-api-access-zvzvd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 16:09:45.590534 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:45.590499 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvzvd\" (UniqueName: \"kubernetes.io/projected/ff6de5f5-1c65-4a98-9d12-c34b8f4db5dc-kube-api-access-zvzvd\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:09:46.087517 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:46.087476 2566 generic.go:356] "Generic (PLEG): container finished" podID="ff6de5f5-1c65-4a98-9d12-c34b8f4db5dc" containerID="aed79947e998592ff7d8052e491493891c2c3d2e7340bbe46680cc0c9408be12" exitCode=137 Sep 11 16:09:46.088037 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:46.087578 2566 scope.go:117] "RemoveContainer" containerID="aed79947e998592ff7d8052e491493891c2c3d2e7340bbe46680cc0c9408be12" Sep 11 16:09:46.088124 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:46.088081 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7104-05747d" Sep 11 16:09:46.100245 ip-10-0-136-96 kubenswrapper[2566]: E0911 16:09:46.100214 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7104-05747d\" is forbidden: User \"system:node:ip-10-0-136-96.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-96.ec2.internal' and this object" podUID="ff6de5f5-1c65-4a98-9d12-c34b8f4db5dc" pod="opendatahub/test-curl-7104-05747d" Sep 11 16:09:46.101930 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:46.101915 2566 scope.go:117] "RemoveContainer" containerID="aed79947e998592ff7d8052e491493891c2c3d2e7340bbe46680cc0c9408be12" Sep 11 16:09:46.102290 ip-10-0-136-96 kubenswrapper[2566]: E0911 16:09:46.102269 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aed79947e998592ff7d8052e491493891c2c3d2e7340bbe46680cc0c9408be12\": container with ID starting with aed79947e998592ff7d8052e491493891c2c3d2e7340bbe46680cc0c9408be12 not found: ID does not exist" containerID="aed79947e998592ff7d8052e491493891c2c3d2e7340bbe46680cc0c9408be12" Sep 11 16:09:46.102354 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:46.102300 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aed79947e998592ff7d8052e491493891c2c3d2e7340bbe46680cc0c9408be12"} err="failed to get container status \"aed79947e998592ff7d8052e491493891c2c3d2e7340bbe46680cc0c9408be12\": rpc error: code = NotFound desc = could not find container \"aed79947e998592ff7d8052e491493891c2c3d2e7340bbe46680cc0c9408be12\": container with ID starting with aed79947e998592ff7d8052e491493891c2c3d2e7340bbe46680cc0c9408be12 not found: ID does not exist" Sep 11 16:09:47.242232 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:47.242191 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff6de5f5-1c65-4a98-9d12-c34b8f4db5dc" path="/var/lib/kubelet/pods/ff6de5f5-1c65-4a98-9d12-c34b8f4db5dc/volumes" Sep 11 16:09:47.441717 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:47.441660 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dx4xn"] Sep 11 16:09:51.637468 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:51.637423 2566 patch_prober.go:28] interesting pod/e2e-inf-a-8e3121-openshift-default-7f6ffd865d-vbjmp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": read tcp 10.132.0.2:34192->10.132.0.54:15021: read: connection reset by peer" start-of-body= Sep 11 16:09:51.637827 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:51.637504 2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-8e3121-openshift-default-7f6ffd865d-vbjmp" podUID="96800483-9c43-4f65-b7ae-a46b5a7d35b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": read tcp 10.132.0.2:34192->10.132.0.54:15021: read: connection reset by peer" Sep 11 16:09:52.130993 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:52.130938 2566 generic.go:356] "Generic (PLEG): container finished" podID="96800483-9c43-4f65-b7ae-a46b5a7d35b7" containerID="7e91796dff1eebcc4d261b64ebc8b043bec53348f548846eddf543ffa770ce4e" exitCode=0 Sep 11 16:09:52.131165 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:52.131016 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8e3121-openshift-default-7f6ffd865d-vbjmp" event={"ID":"96800483-9c43-4f65-b7ae-a46b5a7d35b7","Type":"ContainerDied","Data":"7e91796dff1eebcc4d261b64ebc8b043bec53348f548846eddf543ffa770ce4e"} Sep 11 16:09:52.131465 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:52.131452 2566 scope.go:117] "RemoveContainer" containerID="7e91796dff1eebcc4d261b64ebc8b043bec53348f548846eddf543ffa770ce4e" Sep 11 16:09:52.132559 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:52.132534 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 16:09:52.132617 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:52.132606 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 16:09:52.132660 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:52.132645 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 16:09:52.555896 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:52.555860 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-2b1bd5-openshift-default-687f66dffb-jdh2v"] Sep 11 16:09:52.559277 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:52.559257 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2b1bd5-openshift-default-687f66dffb-jdh2v" Sep 11 16:09:52.562106 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:52.562081 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-2b1bd5-openshift-default-dockercfg-b5dlw\"" Sep 11 16:09:52.575313 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:52.575285 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-2b1bd5-openshift-default-687f66dffb-jdh2v"] Sep 11 16:09:52.663388 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:52.663353 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c311820f-99df-4371-8967-d5518a1e299b-istio-envoy\") pod \"e2e-inf-b-2b1bd5-openshift-default-687f66dffb-jdh2v\" (UID: \"c311820f-99df-4371-8967-d5518a1e299b\") " pod="openshift-ingress/e2e-inf-b-2b1bd5-openshift-default-687f66dffb-jdh2v" Sep 11 16:09:52.664012 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:52.663937 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c311820f-99df-4371-8967-d5518a1e299b-workload-socket\") pod \"e2e-inf-b-2b1bd5-openshift-default-687f66dffb-jdh2v\" (UID: \"c311820f-99df-4371-8967-d5518a1e299b\") " pod="openshift-ingress/e2e-inf-b-2b1bd5-openshift-default-687f66dffb-jdh2v" Sep 11 16:09:52.664210 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:52.664194 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c311820f-99df-4371-8967-d5518a1e299b-istio-data\") pod \"e2e-inf-b-2b1bd5-openshift-default-687f66dffb-jdh2v\" (UID: \"c311820f-99df-4371-8967-d5518a1e299b\") " pod="openshift-ingress/e2e-inf-b-2b1bd5-openshift-default-687f66dffb-jdh2v" Sep 11 16:09:52.664382 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:52.664365 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c311820f-99df-4371-8967-d5518a1e299b-workload-certs\") pod \"e2e-inf-b-2b1bd5-openshift-default-687f66dffb-jdh2v\" (UID: \"c311820f-99df-4371-8967-d5518a1e299b\") " pod="openshift-ingress/e2e-inf-b-2b1bd5-openshift-default-687f66dffb-jdh2v" Sep 11 16:09:52.664557 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:52.664542 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c311820f-99df-4371-8967-d5518a1e299b-istio-token\") pod \"e2e-inf-b-2b1bd5-openshift-default-687f66dffb-jdh2v\" (UID: \"c311820f-99df-4371-8967-d5518a1e299b\") " pod="openshift-ingress/e2e-inf-b-2b1bd5-openshift-default-687f66dffb-jdh2v" Sep 11 16:09:52.664829 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:52.664757 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c311820f-99df-4371-8967-d5518a1e299b-istio-podinfo\") pod \"e2e-inf-b-2b1bd5-openshift-default-687f66dffb-jdh2v\" (UID: \"c311820f-99df-4371-8967-d5518a1e299b\") " pod="openshift-ingress/e2e-inf-b-2b1bd5-openshift-default-687f66dffb-jdh2v" Sep 11 16:09:52.664829 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:52.664820 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c311820f-99df-4371-8967-d5518a1e299b-istiod-ca-cert\") pod \"e2e-inf-b-2b1bd5-openshift-default-687f66dffb-jdh2v\" (UID: \"c311820f-99df-4371-8967-d5518a1e299b\") " pod="openshift-ingress/e2e-inf-b-2b1bd5-openshift-default-687f66dffb-jdh2v" Sep 11 16:09:52.665003 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:52.664895 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c311820f-99df-4371-8967-d5518a1e299b-credential-socket\") pod \"e2e-inf-b-2b1bd5-openshift-default-687f66dffb-jdh2v\" (UID: \"c311820f-99df-4371-8967-d5518a1e299b\") " pod="openshift-ingress/e2e-inf-b-2b1bd5-openshift-default-687f66dffb-jdh2v" Sep 11 16:09:52.665003 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:52.664940 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqpwt\" (UniqueName: \"kubernetes.io/projected/c311820f-99df-4371-8967-d5518a1e299b-kube-api-access-sqpwt\") pod \"e2e-inf-b-2b1bd5-openshift-default-687f66dffb-jdh2v\" (UID: \"c311820f-99df-4371-8967-d5518a1e299b\") " pod="openshift-ingress/e2e-inf-b-2b1bd5-openshift-default-687f66dffb-jdh2v" Sep 11 16:09:52.766410 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:52.766380 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c311820f-99df-4371-8967-d5518a1e299b-credential-socket\") pod \"e2e-inf-b-2b1bd5-openshift-default-687f66dffb-jdh2v\" (UID: \"c311820f-99df-4371-8967-d5518a1e299b\") " pod="openshift-ingress/e2e-inf-b-2b1bd5-openshift-default-687f66dffb-jdh2v" Sep 11 16:09:52.766568 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:52.766423 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqpwt\" (UniqueName: \"kubernetes.io/projected/c311820f-99df-4371-8967-d5518a1e299b-kube-api-access-sqpwt\") pod \"e2e-inf-b-2b1bd5-openshift-default-687f66dffb-jdh2v\" (UID: \"c311820f-99df-4371-8967-d5518a1e299b\") " pod="openshift-ingress/e2e-inf-b-2b1bd5-openshift-default-687f66dffb-jdh2v" Sep 11 16:09:52.766568 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:52.766468 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c311820f-99df-4371-8967-d5518a1e299b-istio-envoy\") pod \"e2e-inf-b-2b1bd5-openshift-default-687f66dffb-jdh2v\" (UID: \"c311820f-99df-4371-8967-d5518a1e299b\") " pod="openshift-ingress/e2e-inf-b-2b1bd5-openshift-default-687f66dffb-jdh2v" Sep 11 16:09:52.766568 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:52.766499 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c311820f-99df-4371-8967-d5518a1e299b-workload-socket\") pod \"e2e-inf-b-2b1bd5-openshift-default-687f66dffb-jdh2v\" (UID: \"c311820f-99df-4371-8967-d5518a1e299b\") " pod="openshift-ingress/e2e-inf-b-2b1bd5-openshift-default-687f66dffb-jdh2v" Sep 11 16:09:52.766681 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:52.766613 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c311820f-99df-4371-8967-d5518a1e299b-istio-data\") pod \"e2e-inf-b-2b1bd5-openshift-default-687f66dffb-jdh2v\" (UID: \"c311820f-99df-4371-8967-d5518a1e299b\") " pod="openshift-ingress/e2e-inf-b-2b1bd5-openshift-default-687f66dffb-jdh2v" Sep 11 16:09:52.766724 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:52.766680 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c311820f-99df-4371-8967-d5518a1e299b-workload-certs\") pod \"e2e-inf-b-2b1bd5-openshift-default-687f66dffb-jdh2v\" (UID: \"c311820f-99df-4371-8967-d5518a1e299b\") " pod="openshift-ingress/e2e-inf-b-2b1bd5-openshift-default-687f66dffb-jdh2v" Sep 11 16:09:52.766770 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:52.766733 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c311820f-99df-4371-8967-d5518a1e299b-istio-token\") pod \"e2e-inf-b-2b1bd5-openshift-default-687f66dffb-jdh2v\" (UID: \"c311820f-99df-4371-8967-d5518a1e299b\") " pod="openshift-ingress/e2e-inf-b-2b1bd5-openshift-default-687f66dffb-jdh2v" Sep 11 16:09:52.766823 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:52.766783 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c311820f-99df-4371-8967-d5518a1e299b-istio-podinfo\") pod \"e2e-inf-b-2b1bd5-openshift-default-687f66dffb-jdh2v\" (UID: \"c311820f-99df-4371-8967-d5518a1e299b\") " pod="openshift-ingress/e2e-inf-b-2b1bd5-openshift-default-687f66dffb-jdh2v" Sep 11 16:09:52.766823 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:52.766799 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c311820f-99df-4371-8967-d5518a1e299b-credential-socket\") pod \"e2e-inf-b-2b1bd5-openshift-default-687f66dffb-jdh2v\" (UID: \"c311820f-99df-4371-8967-d5518a1e299b\") " pod="openshift-ingress/e2e-inf-b-2b1bd5-openshift-default-687f66dffb-jdh2v" Sep 11 16:09:52.766823 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:52.766814 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c311820f-99df-4371-8967-d5518a1e299b-istiod-ca-cert\") pod \"e2e-inf-b-2b1bd5-openshift-default-687f66dffb-jdh2v\" (UID: \"c311820f-99df-4371-8967-d5518a1e299b\") " pod="openshift-ingress/e2e-inf-b-2b1bd5-openshift-default-687f66dffb-jdh2v" Sep 11 16:09:52.767014 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:52.766890 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c311820f-99df-4371-8967-d5518a1e299b-workload-socket\") pod \"e2e-inf-b-2b1bd5-openshift-default-687f66dffb-jdh2v\" (UID: \"c311820f-99df-4371-8967-d5518a1e299b\") " pod="openshift-ingress/e2e-inf-b-2b1bd5-openshift-default-687f66dffb-jdh2v" Sep 11 16:09:52.767014 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:52.767002 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c311820f-99df-4371-8967-d5518a1e299b-istio-data\") pod \"e2e-inf-b-2b1bd5-openshift-default-687f66dffb-jdh2v\" (UID: \"c311820f-99df-4371-8967-d5518a1e299b\") " pod="openshift-ingress/e2e-inf-b-2b1bd5-openshift-default-687f66dffb-jdh2v" Sep 11 16:09:52.767167 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:52.767143 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c311820f-99df-4371-8967-d5518a1e299b-workload-certs\") pod \"e2e-inf-b-2b1bd5-openshift-default-687f66dffb-jdh2v\" (UID: \"c311820f-99df-4371-8967-d5518a1e299b\") " pod="openshift-ingress/e2e-inf-b-2b1bd5-openshift-default-687f66dffb-jdh2v" Sep 11 16:09:52.767632 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:52.767607 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c311820f-99df-4371-8967-d5518a1e299b-istiod-ca-cert\") pod \"e2e-inf-b-2b1bd5-openshift-default-687f66dffb-jdh2v\" (UID: \"c311820f-99df-4371-8967-d5518a1e299b\") " pod="openshift-ingress/e2e-inf-b-2b1bd5-openshift-default-687f66dffb-jdh2v" Sep 11 16:09:52.768830 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:52.768797 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c311820f-99df-4371-8967-d5518a1e299b-istio-envoy\") pod \"e2e-inf-b-2b1bd5-openshift-default-687f66dffb-jdh2v\" (UID: \"c311820f-99df-4371-8967-d5518a1e299b\") " pod="openshift-ingress/e2e-inf-b-2b1bd5-openshift-default-687f66dffb-jdh2v" Sep 11 16:09:52.769204 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:52.769182 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c311820f-99df-4371-8967-d5518a1e299b-istio-podinfo\") pod \"e2e-inf-b-2b1bd5-openshift-default-687f66dffb-jdh2v\" (UID: \"c311820f-99df-4371-8967-d5518a1e299b\") " pod="openshift-ingress/e2e-inf-b-2b1bd5-openshift-default-687f66dffb-jdh2v" Sep 11 16:09:52.775111 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:52.775084 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c311820f-99df-4371-8967-d5518a1e299b-istio-token\") pod \"e2e-inf-b-2b1bd5-openshift-default-687f66dffb-jdh2v\" (UID: \"c311820f-99df-4371-8967-d5518a1e299b\") " pod="openshift-ingress/e2e-inf-b-2b1bd5-openshift-default-687f66dffb-jdh2v" Sep 11 16:09:52.775180 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:52.775110 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqpwt\" (UniqueName: \"kubernetes.io/projected/c311820f-99df-4371-8967-d5518a1e299b-kube-api-access-sqpwt\") pod \"e2e-inf-b-2b1bd5-openshift-default-687f66dffb-jdh2v\" (UID: \"c311820f-99df-4371-8967-d5518a1e299b\") " pod="openshift-ingress/e2e-inf-b-2b1bd5-openshift-default-687f66dffb-jdh2v" Sep 11 16:09:52.871351 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:52.871263 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2b1bd5-openshift-default-687f66dffb-jdh2v" Sep 11 16:09:53.146689 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:53.146575 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8e3121-openshift-default-7f6ffd865d-vbjmp" event={"ID":"96800483-9c43-4f65-b7ae-a46b5a7d35b7","Type":"ContainerStarted","Data":"227e4736d0c7482ecc2421407ce9fabf242e373c74b2ef273586d5e88cfd6b49"} Sep 11 16:09:53.170166 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:53.170053 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-2b1bd5-openshift-default-687f66dffb-jdh2v"] Sep 11 16:09:53.173337 ip-10-0-136-96 kubenswrapper[2566]: W0911 16:09:53.173309 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc311820f_99df_4371_8967_d5518a1e299b.slice/crio-bf320695f877cccad43b6a537af43358f43781849675d8e860daf3b42a781ff5 WatchSource:0}: Error finding container bf320695f877cccad43b6a537af43358f43781849675d8e860daf3b42a781ff5: Status 404 returned error can't find the container with id bf320695f877cccad43b6a537af43358f43781849675d8e860daf3b42a781ff5 Sep 11 16:09:53.175769 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:53.175738 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 16:09:53.175852 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:53.175796 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 16:09:53.175852 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:53.175825 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 16:09:53.317814 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:53.317784 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-xjvl5_7542ad77-c817-4370-91de-3b08db264753/console-operator/2.log" Sep 11 16:09:53.320132 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:53.320109 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-xjvl5_7542ad77-c817-4370-91de-3b08db264753/console-operator/2.log" Sep 11 16:09:53.328239 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:53.328103 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tgq5c_981d7d1e-f81f-48fc-8962-7831bc5c2b27/ovn-acl-logging/0.log" Sep 11 16:09:53.330349 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:53.330329 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tgq5c_981d7d1e-f81f-48fc-8962-7831bc5c2b27/ovn-acl-logging/0.log" Sep 11 16:09:54.047079 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:54.047036 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-8e3121-openshift-default-7f6ffd865d-vbjmp" Sep 11 16:09:54.048000 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:54.047387 2566 patch_prober.go:28] interesting pod/e2e-inf-a-8e3121-openshift-default-7f6ffd865d-vbjmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Sep 11 16:09:54.048000 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:54.047711 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8e3121-openshift-default-7f6ffd865d-vbjmp" podUID="96800483-9c43-4f65-b7ae-a46b5a7d35b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Sep 11 16:09:54.159617 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:54.159582 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2b1bd5-openshift-default-687f66dffb-jdh2v" event={"ID":"c311820f-99df-4371-8967-d5518a1e299b","Type":"ContainerStarted","Data":"0b224b3169b2365e5a3ef2f92a07c4f8b9a14f8b46456b2a50013044873aa8a4"} Sep 11 16:09:54.159922 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:54.159865 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2b1bd5-openshift-default-687f66dffb-jdh2v" event={"ID":"c311820f-99df-4371-8967-d5518a1e299b","Type":"ContainerStarted","Data":"bf320695f877cccad43b6a537af43358f43781849675d8e860daf3b42a781ff5"} Sep 11 16:09:54.183503 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:54.183451 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-2b1bd5-openshift-default-687f66dffb-jdh2v" podStartSLOduration=2.183435801 podStartE2EDuration="2.183435801s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:09:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:09:54.180813865 +0000 UTC m=+901.395647770" watchObservedRunningTime="2026-09-11 16:09:54.183435801 +0000 UTC m=+901.398269706" Sep 11 16:09:54.872138 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:54.872097 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-2b1bd5-openshift-default-687f66dffb-jdh2v" Sep 11 16:09:54.878082 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:54.878056 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-2b1bd5-openshift-default-687f66dffb-jdh2v" Sep 11 16:09:55.046631 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:55.046593 2566 patch_prober.go:28] interesting pod/e2e-inf-a-8e3121-openshift-default-7f6ffd865d-vbjmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Sep 11 16:09:55.046909 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:55.046877 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8e3121-openshift-default-7f6ffd865d-vbjmp" podUID="96800483-9c43-4f65-b7ae-a46b5a7d35b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Sep 11 16:09:55.169617 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:55.169489 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-2b1bd5-openshift-default-687f66dffb-jdh2v" Sep 11 16:09:55.172216 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:55.171318 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-2b1bd5-openshift-default-687f66dffb-jdh2v" Sep 11 16:09:56.047222 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:56.047178 2566 patch_prober.go:28] interesting pod/e2e-inf-a-8e3121-openshift-default-7f6ffd865d-vbjmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Sep 11 16:09:56.047516 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:56.047487 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8e3121-openshift-default-7f6ffd865d-vbjmp" podUID="96800483-9c43-4f65-b7ae-a46b5a7d35b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Sep 11 16:09:57.047186 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:57.047134 2566 patch_prober.go:28] interesting pod/e2e-inf-a-8e3121-openshift-default-7f6ffd865d-vbjmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Sep 11 16:09:57.047693 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:57.047207 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8e3121-openshift-default-7f6ffd865d-vbjmp" podUID="96800483-9c43-4f65-b7ae-a46b5a7d35b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Sep 11 16:09:58.047399 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:58.047352 2566 patch_prober.go:28] interesting pod/e2e-inf-a-8e3121-openshift-default-7f6ffd865d-vbjmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Sep 11 16:09:58.047882 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:58.047424 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8e3121-openshift-default-7f6ffd865d-vbjmp" podUID="96800483-9c43-4f65-b7ae-a46b5a7d35b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Sep 11 16:09:59.067671 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:59.067619 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-8e3121-openshift-default-7f6ffd865d-vbjmp" Sep 11 16:09:59.070148 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:59.070128 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-8e3121-openshift-default-7f6ffd865d-vbjmp" Sep 11 16:09:59.071104 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:59.071084 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-8e3121-openshift-default-7f6ffd865d-vbjmp" Sep 11 16:09:59.823606 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:59.823563 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a6560b69-rqd4v"] Sep 11 16:09:59.830654 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:59.830625 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a6560b69-rqd4v" Sep 11 16:09:59.832951 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:59.832928 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 16:09:59.833193 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:59.833139 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qbzdr\"" Sep 11 16:09:59.834704 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:59.834680 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a6560b69-rqd4v"] Sep 11 16:09:59.865658 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:59.865619 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59d68d7b-64e1-42d4-a5df-8378d1484f5c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a6560b69-rqd4v\" (UID: \"59d68d7b-64e1-42d4-a5df-8378d1484f5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a6560b69-rqd4v" Sep 11 16:09:59.970581 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:59.970540 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59d68d7b-64e1-42d4-a5df-8378d1484f5c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a6560b69-rqd4v\" (UID: \"59d68d7b-64e1-42d4-a5df-8378d1484f5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a6560b69-rqd4v" Sep 11 16:09:59.971912 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:09:59.971885 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59d68d7b-64e1-42d4-a5df-8378d1484f5c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a6560b69-rqd4v\" (UID: \"59d68d7b-64e1-42d4-a5df-8378d1484f5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a6560b69-rqd4v" Sep 11 16:10:00.164761 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:00.164656 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a6560b69-rqd4v" Sep 11 16:10:00.462051 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:00.461546 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a6560b69-rqd4v"] Sep 11 16:10:01.224434 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:01.224399 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a6560b69-rqd4v" event={"ID":"59d68d7b-64e1-42d4-a5df-8378d1484f5c","Type":"ContainerStarted","Data":"2ab5b89552fe6fedb4eebfd8fa22a42fbfa385c36bde2c7bcbd80a5c34455cc2"} Sep 11 16:10:04.164279 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:04.164230 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dx4xn"] Sep 11 16:10:04.252508 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:04.252462 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a6560b69-rqd4v" event={"ID":"59d68d7b-64e1-42d4-a5df-8378d1484f5c","Type":"ContainerStarted","Data":"ec9248bc34432f1a2b283acba1cece731b054aead8be06bedc56e92c59284dea"} Sep 11 16:10:04.274962 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:04.274886 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a6560b69-rqd4v" podStartSLOduration=2.524900235 podStartE2EDuration="5.274863386s" totalImagesPullingTime="2.749963151s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:09:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:10:04.27424016 +0000 UTC m=+911.489074066" watchObservedRunningTime="2026-09-11 16:10:04.274863386 +0000 UTC m=+911.489697293" Sep 11 16:10:05.259793 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:05.259756 2566 generic.go:356] "Generic (PLEG): container finished" podID="59d68d7b-64e1-42d4-a5df-8378d1484f5c" containerID="ec9248bc34432f1a2b283acba1cece731b054aead8be06bedc56e92c59284dea" exitCode=0 Sep 11 16:10:05.260231 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:05.259854 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a6560b69-rqd4v" event={"ID":"59d68d7b-64e1-42d4-a5df-8378d1484f5c","Type":"ContainerDied","Data":"ec9248bc34432f1a2b283acba1cece731b054aead8be06bedc56e92c59284dea"} Sep 11 16:10:06.271332 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:06.271304 2566 generic.go:356] "Generic (PLEG): container finished" podID="3c028f61-e84c-4c34-9a94-7c7e0c9a2212" containerID="6a1a9469ace076fbd71923f1224d7f272ac7b9e76b82d85c0bddecf0a72d9ce1" exitCode=0 Sep 11 16:10:06.271736 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:06.271365 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-9c5b09-openshift-default-66c68b44fd-7dtg6" event={"ID":"3c028f61-e84c-4c34-9a94-7c7e0c9a2212","Type":"ContainerDied","Data":"6a1a9469ace076fbd71923f1224d7f272ac7b9e76b82d85c0bddecf0a72d9ce1"} Sep 11 16:10:06.271781 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:06.271734 2566 scope.go:117] "RemoveContainer" containerID="6a1a9469ace076fbd71923f1224d7f272ac7b9e76b82d85c0bddecf0a72d9ce1" Sep 11 16:10:06.272882 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:06.272852 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 16:10:06.273002 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:06.272918 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 16:10:06.273002 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:06.272959 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 16:10:06.274436 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:06.274255 2566 generic.go:356] "Generic (PLEG): container finished" podID="4f755540-c766-4b46-a53f-dce9a5c44ce8" containerID="ce243feda875ee9607bcad9a840670b0bfe697c8944f5618604ecdc029bb4da1" exitCode=0 Sep 11 16:10:06.274436 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:06.274309 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a6560b69-openshift-default-5c8f947cc4-qcw28" event={"ID":"4f755540-c766-4b46-a53f-dce9a5c44ce8","Type":"ContainerDied","Data":"ce243feda875ee9607bcad9a840670b0bfe697c8944f5618604ecdc029bb4da1"} Sep 11 16:10:06.274734 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:06.274704 2566 scope.go:117] "RemoveContainer" containerID="ce243feda875ee9607bcad9a840670b0bfe697c8944f5618604ecdc029bb4da1" Sep 11 16:10:06.276692 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:06.276560 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 16:10:06.276692 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:06.276648 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 16:10:06.276842 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:06.276693 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 16:10:06.541993 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:06.541090 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a6560b69-rqd4v" Sep 11 16:10:06.572603 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:06.570282 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59d68d7b-64e1-42d4-a5df-8378d1484f5c-maas-api-service-ca\") pod \"59d68d7b-64e1-42d4-a5df-8378d1484f5c\" (UID: \"59d68d7b-64e1-42d4-a5df-8378d1484f5c\") " Sep 11 16:10:06.572603 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:06.571044 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59d68d7b-64e1-42d4-a5df-8378d1484f5c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "59d68d7b-64e1-42d4-a5df-8378d1484f5c" (UID: "59d68d7b-64e1-42d4-a5df-8378d1484f5c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 16:10:06.674709 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:06.674607 2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59d68d7b-64e1-42d4-a5df-8378d1484f5c-maas-api-service-ca\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:10:07.286676 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:07.286627 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a6560b69-rqd4v" event={"ID":"59d68d7b-64e1-42d4-a5df-8378d1484f5c","Type":"ContainerDied","Data":"2ab5b89552fe6fedb4eebfd8fa22a42fbfa385c36bde2c7bcbd80a5c34455cc2"} Sep 11 16:10:07.287173 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:07.286701 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2ab5b89552fe6fedb4eebfd8fa22a42fbfa385c36bde2c7bcbd80a5c34455cc2" Sep 11 16:10:07.287173 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:07.286769 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a6560b69-rqd4v" Sep 11 16:10:07.296474 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:07.296433 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-9c5b09-openshift-default-66c68b44fd-7dtg6" event={"ID":"3c028f61-e84c-4c34-9a94-7c7e0c9a2212","Type":"ContainerStarted","Data":"abde8929462db8fc40c28736b0901ceb355eaf2b60f51c106c8b916fb2a63b75"} Sep 11 16:10:07.301249 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:07.301213 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a6560b69-openshift-default-5c8f947cc4-qcw28" event={"ID":"4f755540-c766-4b46-a53f-dce9a5c44ce8","Type":"ContainerStarted","Data":"541ab1bbc4f0da1c5c924332bbc2f7857320e9d473b10178a703f6de0fac857a"} Sep 11 16:10:07.904943 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:07.904896 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-9c5b09-openshift-default-66c68b44fd-7dtg6" Sep 11 16:10:07.905184 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:07.905066 2566 patch_prober.go:28] interesting pod/e2e-autores-9c5b09-openshift-default-66c68b44fd-7dtg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Sep 11 16:10:07.905184 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:07.905123 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-9c5b09-openshift-default-66c68b44fd-7dtg6" podUID="3c028f61-e84c-4c34-9a94-7c7e0c9a2212" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Sep 11 16:10:08.269344 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:08.269310 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-a6560b69-openshift-default-5c8f947cc4-qcw28" Sep 11 16:10:08.269705 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:08.269682 2566 patch_prober.go:28] interesting pod/e2e-ait-a6560b69-openshift-default-5c8f947cc4-qcw28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Sep 11 16:10:08.269858 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:08.269835 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a6560b69-openshift-default-5c8f947cc4-qcw28" podUID="4f755540-c766-4b46-a53f-dce9a5c44ce8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Sep 11 16:10:08.905309 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:08.905270 2566 patch_prober.go:28] interesting pod/e2e-autores-9c5b09-openshift-default-66c68b44fd-7dtg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Sep 11 16:10:08.905889 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:08.905857 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-9c5b09-openshift-default-66c68b44fd-7dtg6" podUID="3c028f61-e84c-4c34-9a94-7c7e0c9a2212" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Sep 11 16:10:09.269745 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:09.269687 2566 patch_prober.go:28] interesting pod/e2e-ait-a6560b69-openshift-default-5c8f947cc4-qcw28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Sep 11 16:10:09.269990 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:09.269772 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a6560b69-openshift-default-5c8f947cc4-qcw28" podUID="4f755540-c766-4b46-a53f-dce9a5c44ce8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Sep 11 16:10:09.905282 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:09.905237 2566 patch_prober.go:28] interesting pod/e2e-autores-9c5b09-openshift-default-66c68b44fd-7dtg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Sep 11 16:10:09.905501 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:09.905309 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-9c5b09-openshift-default-66c68b44fd-7dtg6" podUID="3c028f61-e84c-4c34-9a94-7c7e0c9a2212" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Sep 11 16:10:10.269644 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:10.269601 2566 patch_prober.go:28] interesting pod/e2e-ait-a6560b69-openshift-default-5c8f947cc4-qcw28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Sep 11 16:10:10.269940 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:10.269907 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a6560b69-openshift-default-5c8f947cc4-qcw28" podUID="4f755540-c766-4b46-a53f-dce9a5c44ce8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Sep 11 16:10:10.904824 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:10.904784 2566 patch_prober.go:28] interesting pod/e2e-autores-9c5b09-openshift-default-66c68b44fd-7dtg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Sep 11 16:10:10.905184 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:10.905144 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-9c5b09-openshift-default-66c68b44fd-7dtg6" podUID="3c028f61-e84c-4c34-9a94-7c7e0c9a2212" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Sep 11 16:10:11.269686 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:11.269650 2566 patch_prober.go:28] interesting pod/e2e-ait-a6560b69-openshift-default-5c8f947cc4-qcw28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Sep 11 16:10:11.270330 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:11.270276 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a6560b69-openshift-default-5c8f947cc4-qcw28" podUID="4f755540-c766-4b46-a53f-dce9a5c44ce8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Sep 11 16:10:11.493586 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:11.493548 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a6560b69-rqd4v"] Sep 11 16:10:11.500103 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:11.500071 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a6560b69-rqd4v"] Sep 11 16:10:11.623608 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:11.623517 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-a6560b69-openshift-default-5c8f947cc4-qcw28"] Sep 11 16:10:11.624304 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:11.624038 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-a6560b69-openshift-default-5c8f947cc4-qcw28" podUID="4f755540-c766-4b46-a53f-dce9a5c44ce8" containerName="istio-proxy" containerID="cri-o://541ab1bbc4f0da1c5c924332bbc2f7857320e9d473b10178a703f6de0fac857a" gracePeriod=30 Sep 11 16:10:11.905370 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:11.905282 2566 patch_prober.go:28] interesting pod/e2e-autores-9c5b09-openshift-default-66c68b44fd-7dtg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Sep 11 16:10:11.905370 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:11.905346 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-9c5b09-openshift-default-66c68b44fd-7dtg6" podUID="3c028f61-e84c-4c34-9a94-7c7e0c9a2212" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Sep 11 16:10:12.699158 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:12.699114 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-f8b85f0c-gw-openshift-default-bc5d5c4d-27r4v"] Sep 11 16:10:12.700452 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:12.700424 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59d68d7b-64e1-42d4-a5df-8378d1484f5c" containerName="revoke-keys" Sep 11 16:10:12.700650 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:12.700626 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="59d68d7b-64e1-42d4-a5df-8378d1484f5c" containerName="revoke-keys" Sep 11 16:10:12.700909 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:12.700893 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="59d68d7b-64e1-42d4-a5df-8378d1484f5c" containerName="revoke-keys" Sep 11 16:10:12.704428 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:12.704371 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-f8b85f0c-gw-openshift-default-bc5d5c4d-27r4v" Sep 11 16:10:12.706887 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:12.706870 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-f8b85f0c-gw-openshift-default-dockercfg-jlmlz\"" Sep 11 16:10:12.707102 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:12.707064 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-f8b85f0c-gw-openshift-default-bc5d5c4d-27r4v"] Sep 11 16:10:12.782732 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:12.782674 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/186654d0-9025-46d3-880c-25da1cd4df1c-workload-certs\") pod \"e2e-migrate-f8b85f0c-gw-openshift-default-bc5d5c4d-27r4v\" (UID: \"186654d0-9025-46d3-880c-25da1cd4df1c\") " pod="openshift-ingress/e2e-migrate-f8b85f0c-gw-openshift-default-bc5d5c4d-27r4v" Sep 11 16:10:12.782732 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:12.782742 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/186654d0-9025-46d3-880c-25da1cd4df1c-istio-envoy\") pod \"e2e-migrate-f8b85f0c-gw-openshift-default-bc5d5c4d-27r4v\" (UID: \"186654d0-9025-46d3-880c-25da1cd4df1c\") " pod="openshift-ingress/e2e-migrate-f8b85f0c-gw-openshift-default-bc5d5c4d-27r4v" Sep 11 16:10:12.783044 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:12.782769 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/186654d0-9025-46d3-880c-25da1cd4df1c-istio-podinfo\") pod \"e2e-migrate-f8b85f0c-gw-openshift-default-bc5d5c4d-27r4v\" (UID: \"186654d0-9025-46d3-880c-25da1cd4df1c\") " pod="openshift-ingress/e2e-migrate-f8b85f0c-gw-openshift-default-bc5d5c4d-27r4v" Sep 11 16:10:12.783044 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:12.782802 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/186654d0-9025-46d3-880c-25da1cd4df1c-istio-data\") pod \"e2e-migrate-f8b85f0c-gw-openshift-default-bc5d5c4d-27r4v\" (UID: \"186654d0-9025-46d3-880c-25da1cd4df1c\") " pod="openshift-ingress/e2e-migrate-f8b85f0c-gw-openshift-default-bc5d5c4d-27r4v" Sep 11 16:10:12.783044 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:12.782859 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/186654d0-9025-46d3-880c-25da1cd4df1c-istiod-ca-cert\") pod \"e2e-migrate-f8b85f0c-gw-openshift-default-bc5d5c4d-27r4v\" (UID: \"186654d0-9025-46d3-880c-25da1cd4df1c\") " pod="openshift-ingress/e2e-migrate-f8b85f0c-gw-openshift-default-bc5d5c4d-27r4v" Sep 11 16:10:12.783044 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:12.782887 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/186654d0-9025-46d3-880c-25da1cd4df1c-istio-token\") pod \"e2e-migrate-f8b85f0c-gw-openshift-default-bc5d5c4d-27r4v\" (UID: \"186654d0-9025-46d3-880c-25da1cd4df1c\") " pod="openshift-ingress/e2e-migrate-f8b85f0c-gw-openshift-default-bc5d5c4d-27r4v" Sep 11 16:10:12.783044 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:12.782953 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8ncs\" (UniqueName: \"kubernetes.io/projected/186654d0-9025-46d3-880c-25da1cd4df1c-kube-api-access-r8ncs\") pod \"e2e-migrate-f8b85f0c-gw-openshift-default-bc5d5c4d-27r4v\" (UID: \"186654d0-9025-46d3-880c-25da1cd4df1c\") " pod="openshift-ingress/e2e-migrate-f8b85f0c-gw-openshift-default-bc5d5c4d-27r4v" Sep 11 16:10:12.783044 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:12.783042 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/186654d0-9025-46d3-880c-25da1cd4df1c-workload-socket\") pod \"e2e-migrate-f8b85f0c-gw-openshift-default-bc5d5c4d-27r4v\" (UID: \"186654d0-9025-46d3-880c-25da1cd4df1c\") " pod="openshift-ingress/e2e-migrate-f8b85f0c-gw-openshift-default-bc5d5c4d-27r4v" Sep 11 16:10:12.783368 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:12.783103 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/186654d0-9025-46d3-880c-25da1cd4df1c-credential-socket\") pod \"e2e-migrate-f8b85f0c-gw-openshift-default-bc5d5c4d-27r4v\" (UID: \"186654d0-9025-46d3-880c-25da1cd4df1c\") " pod="openshift-ingress/e2e-migrate-f8b85f0c-gw-openshift-default-bc5d5c4d-27r4v" Sep 11 16:10:12.883843 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:12.883792 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/186654d0-9025-46d3-880c-25da1cd4df1c-workload-certs\") pod \"e2e-migrate-f8b85f0c-gw-openshift-default-bc5d5c4d-27r4v\" (UID: \"186654d0-9025-46d3-880c-25da1cd4df1c\") " pod="openshift-ingress/e2e-migrate-f8b85f0c-gw-openshift-default-bc5d5c4d-27r4v" Sep 11 16:10:12.884062 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:12.883850 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/186654d0-9025-46d3-880c-25da1cd4df1c-istio-envoy\") pod \"e2e-migrate-f8b85f0c-gw-openshift-default-bc5d5c4d-27r4v\" (UID: \"186654d0-9025-46d3-880c-25da1cd4df1c\") " pod="openshift-ingress/e2e-migrate-f8b85f0c-gw-openshift-default-bc5d5c4d-27r4v" Sep 11 16:10:12.884062 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:12.883876 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/186654d0-9025-46d3-880c-25da1cd4df1c-istio-podinfo\") pod \"e2e-migrate-f8b85f0c-gw-openshift-default-bc5d5c4d-27r4v\" (UID: \"186654d0-9025-46d3-880c-25da1cd4df1c\") " pod="openshift-ingress/e2e-migrate-f8b85f0c-gw-openshift-default-bc5d5c4d-27r4v" Sep 11 16:10:12.884062 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:12.883901 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/186654d0-9025-46d3-880c-25da1cd4df1c-istio-data\") pod \"e2e-migrate-f8b85f0c-gw-openshift-default-bc5d5c4d-27r4v\" (UID: \"186654d0-9025-46d3-880c-25da1cd4df1c\") " pod="openshift-ingress/e2e-migrate-f8b85f0c-gw-openshift-default-bc5d5c4d-27r4v" Sep 11 16:10:12.884062 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:12.883954 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/186654d0-9025-46d3-880c-25da1cd4df1c-istiod-ca-cert\") pod \"e2e-migrate-f8b85f0c-gw-openshift-default-bc5d5c4d-27r4v\" (UID: \"186654d0-9025-46d3-880c-25da1cd4df1c\") " pod="openshift-ingress/e2e-migrate-f8b85f0c-gw-openshift-default-bc5d5c4d-27r4v" Sep 11 16:10:12.884062 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:12.884009 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/186654d0-9025-46d3-880c-25da1cd4df1c-istio-token\") pod \"e2e-migrate-f8b85f0c-gw-openshift-default-bc5d5c4d-27r4v\" (UID: \"186654d0-9025-46d3-880c-25da1cd4df1c\") " pod="openshift-ingress/e2e-migrate-f8b85f0c-gw-openshift-default-bc5d5c4d-27r4v" Sep 11 16:10:12.884338 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:12.884069 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8ncs\" (UniqueName: \"kubernetes.io/projected/186654d0-9025-46d3-880c-25da1cd4df1c-kube-api-access-r8ncs\") pod \"e2e-migrate-f8b85f0c-gw-openshift-default-bc5d5c4d-27r4v\" (UID: \"186654d0-9025-46d3-880c-25da1cd4df1c\") " pod="openshift-ingress/e2e-migrate-f8b85f0c-gw-openshift-default-bc5d5c4d-27r4v" Sep 11 16:10:12.884338 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:12.884142 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/186654d0-9025-46d3-880c-25da1cd4df1c-workload-socket\") pod \"e2e-migrate-f8b85f0c-gw-openshift-default-bc5d5c4d-27r4v\" (UID: \"186654d0-9025-46d3-880c-25da1cd4df1c\") " pod="openshift-ingress/e2e-migrate-f8b85f0c-gw-openshift-default-bc5d5c4d-27r4v" Sep 11 16:10:12.884338 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:12.884201 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/186654d0-9025-46d3-880c-25da1cd4df1c-credential-socket\") pod \"e2e-migrate-f8b85f0c-gw-openshift-default-bc5d5c4d-27r4v\" (UID: \"186654d0-9025-46d3-880c-25da1cd4df1c\") " pod="openshift-ingress/e2e-migrate-f8b85f0c-gw-openshift-default-bc5d5c4d-27r4v" Sep 11 16:10:12.884338 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:12.884302 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/186654d0-9025-46d3-880c-25da1cd4df1c-workload-certs\") pod \"e2e-migrate-f8b85f0c-gw-openshift-default-bc5d5c4d-27r4v\" (UID: \"186654d0-9025-46d3-880c-25da1cd4df1c\") " pod="openshift-ingress/e2e-migrate-f8b85f0c-gw-openshift-default-bc5d5c4d-27r4v" Sep 11 16:10:12.884544 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:12.884438 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/186654d0-9025-46d3-880c-25da1cd4df1c-istio-data\") pod \"e2e-migrate-f8b85f0c-gw-openshift-default-bc5d5c4d-27r4v\" (UID: \"186654d0-9025-46d3-880c-25da1cd4df1c\") " pod="openshift-ingress/e2e-migrate-f8b85f0c-gw-openshift-default-bc5d5c4d-27r4v" Sep 11 16:10:12.884544 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:12.884506 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/186654d0-9025-46d3-880c-25da1cd4df1c-credential-socket\") pod \"e2e-migrate-f8b85f0c-gw-openshift-default-bc5d5c4d-27r4v\" (UID: \"186654d0-9025-46d3-880c-25da1cd4df1c\") " pod="openshift-ingress/e2e-migrate-f8b85f0c-gw-openshift-default-bc5d5c4d-27r4v" Sep 11 16:10:12.884782 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:12.884756 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/186654d0-9025-46d3-880c-25da1cd4df1c-workload-socket\") pod \"e2e-migrate-f8b85f0c-gw-openshift-default-bc5d5c4d-27r4v\" (UID: \"186654d0-9025-46d3-880c-25da1cd4df1c\") " pod="openshift-ingress/e2e-migrate-f8b85f0c-gw-openshift-default-bc5d5c4d-27r4v" Sep 11 16:10:12.885216 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:12.885192 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/186654d0-9025-46d3-880c-25da1cd4df1c-istiod-ca-cert\") pod \"e2e-migrate-f8b85f0c-gw-openshift-default-bc5d5c4d-27r4v\" (UID: \"186654d0-9025-46d3-880c-25da1cd4df1c\") " pod="openshift-ingress/e2e-migrate-f8b85f0c-gw-openshift-default-bc5d5c4d-27r4v" Sep 11 16:10:12.887230 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:12.887201 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/186654d0-9025-46d3-880c-25da1cd4df1c-istio-envoy\") pod \"e2e-migrate-f8b85f0c-gw-openshift-default-bc5d5c4d-27r4v\" (UID: \"186654d0-9025-46d3-880c-25da1cd4df1c\") " pod="openshift-ingress/e2e-migrate-f8b85f0c-gw-openshift-default-bc5d5c4d-27r4v" Sep 11 16:10:12.887666 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:12.887645 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/186654d0-9025-46d3-880c-25da1cd4df1c-istio-podinfo\") pod \"e2e-migrate-f8b85f0c-gw-openshift-default-bc5d5c4d-27r4v\" (UID: \"186654d0-9025-46d3-880c-25da1cd4df1c\") " pod="openshift-ingress/e2e-migrate-f8b85f0c-gw-openshift-default-bc5d5c4d-27r4v" Sep 11 16:10:12.897726 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:12.897665 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/186654d0-9025-46d3-880c-25da1cd4df1c-istio-token\") pod \"e2e-migrate-f8b85f0c-gw-openshift-default-bc5d5c4d-27r4v\" (UID: \"186654d0-9025-46d3-880c-25da1cd4df1c\") " pod="openshift-ingress/e2e-migrate-f8b85f0c-gw-openshift-default-bc5d5c4d-27r4v" Sep 11 16:10:12.898720 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:12.898686 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8ncs\" (UniqueName: \"kubernetes.io/projected/186654d0-9025-46d3-880c-25da1cd4df1c-kube-api-access-r8ncs\") pod \"e2e-migrate-f8b85f0c-gw-openshift-default-bc5d5c4d-27r4v\" (UID: \"186654d0-9025-46d3-880c-25da1cd4df1c\") " pod="openshift-ingress/e2e-migrate-f8b85f0c-gw-openshift-default-bc5d5c4d-27r4v" Sep 11 16:10:12.905508 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:12.905379 2566 patch_prober.go:28] interesting pod/e2e-autores-9c5b09-openshift-default-66c68b44fd-7dtg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Sep 11 16:10:12.905508 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:12.905442 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-9c5b09-openshift-default-66c68b44fd-7dtg6" podUID="3c028f61-e84c-4c34-9a94-7c7e0c9a2212" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Sep 11 16:10:13.018310 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:13.018204 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-f8b85f0c-gw-openshift-default-bc5d5c4d-27r4v" Sep 11 16:10:13.242249 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:13.242212 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="59d68d7b-64e1-42d4-a5df-8378d1484f5c" path="/var/lib/kubelet/pods/59d68d7b-64e1-42d4-a5df-8378d1484f5c/volumes" Sep 11 16:10:13.350018 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:13.349985 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-f8b85f0c-gw-openshift-default-bc5d5c4d-27r4v"] Sep 11 16:10:13.350400 ip-10-0-136-96 kubenswrapper[2566]: W0911 16:10:13.350379 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod186654d0_9025_46d3_880c_25da1cd4df1c.slice/crio-1c1c0514baac1f7bd46e195a4de86202a891803d4ba79bff39fb95e049485e4d WatchSource:0}: Error finding container 1c1c0514baac1f7bd46e195a4de86202a891803d4ba79bff39fb95e049485e4d: Status 404 returned error can't find the container with id 1c1c0514baac1f7bd46e195a4de86202a891803d4ba79bff39fb95e049485e4d Sep 11 16:10:13.353042 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:13.353005 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 16:10:13.353143 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:13.353072 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 16:10:13.353143 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:13.353100 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 16:10:13.909206 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:13.909177 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-9c5b09-openshift-default-66c68b44fd-7dtg6" Sep 11 16:10:13.909631 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:13.909544 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-9c5b09-openshift-default-66c68b44fd-7dtg6" Sep 11 16:10:13.910382 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:13.910364 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-9c5b09-openshift-default-66c68b44fd-7dtg6" Sep 11 16:10:14.072028 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:14.071989 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-a6560b69-openshift-default-5c8f947cc4-qcw28" Sep 11 16:10:14.358920 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:14.358886 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-f8b85f0c-gw-openshift-default-bc5d5c4d-27r4v" event={"ID":"186654d0-9025-46d3-880c-25da1cd4df1c","Type":"ContainerStarted","Data":"fcffbf65068e934ddcf030d7116a5256ba132f8f752787324d9adbaf3c8fd850"} Sep 11 16:10:14.359155 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:14.358939 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-f8b85f0c-gw-openshift-default-bc5d5c4d-27r4v" event={"ID":"186654d0-9025-46d3-880c-25da1cd4df1c","Type":"ContainerStarted","Data":"1c1c0514baac1f7bd46e195a4de86202a891803d4ba79bff39fb95e049485e4d"} Sep 11 16:10:14.398336 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:14.398275 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-f8b85f0c-gw-openshift-default-bc5d5c4d-27r4v" podStartSLOduration=2.398259244 podStartE2EDuration="2.398259244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:10:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:10:14.397705785 +0000 UTC m=+921.612539690" watchObservedRunningTime="2026-09-11 16:10:14.398259244 +0000 UTC m=+921.613093149" Sep 11 16:10:15.018951 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:15.018905 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-f8b85f0c-gw-openshift-default-bc5d5c4d-27r4v" Sep 11 16:10:15.025347 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:15.025315 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-f8b85f0c-gw-openshift-default-bc5d5c4d-27r4v" Sep 11 16:10:15.365399 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:15.365360 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-f8b85f0c-gw-openshift-default-bc5d5c4d-27r4v" Sep 11 16:10:15.366563 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:15.366436 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-f8b85f0c-gw-openshift-default-bc5d5c4d-27r4v" Sep 11 16:10:16.372190 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:16.372155 2566 generic.go:356] "Generic (PLEG): container finished" podID="c311820f-99df-4371-8967-d5518a1e299b" containerID="0b224b3169b2365e5a3ef2f92a07c4f8b9a14f8b46456b2a50013044873aa8a4" exitCode=0 Sep 11 16:10:16.372586 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:16.372226 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2b1bd5-openshift-default-687f66dffb-jdh2v" event={"ID":"c311820f-99df-4371-8967-d5518a1e299b","Type":"ContainerDied","Data":"0b224b3169b2365e5a3ef2f92a07c4f8b9a14f8b46456b2a50013044873aa8a4"} Sep 11 16:10:16.372771 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:16.372751 2566 scope.go:117] "RemoveContainer" containerID="0b224b3169b2365e5a3ef2f92a07c4f8b9a14f8b46456b2a50013044873aa8a4" Sep 11 16:10:16.373929 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:16.373902 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 16:10:16.374039 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:16.373984 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 16:10:16.374039 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:16.374018 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 16:10:17.380253 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:17.380213 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2b1bd5-openshift-default-687f66dffb-jdh2v" event={"ID":"c311820f-99df-4371-8967-d5518a1e299b","Type":"ContainerStarted","Data":"5cb2622d6ae86291b94297e89ff86797e65bb74ca9d7c7b38e99a4f375dbbd24"} Sep 11 16:10:17.871894 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:17.871852 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-2b1bd5-openshift-default-687f66dffb-jdh2v" Sep 11 16:10:17.872120 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:17.872101 2566 patch_prober.go:28] interesting pod/e2e-inf-b-2b1bd5-openshift-default-687f66dffb-jdh2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Sep 11 16:10:17.872180 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:17.872158 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2b1bd5-openshift-default-687f66dffb-jdh2v" podUID="c311820f-99df-4371-8967-d5518a1e299b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Sep 11 16:10:18.333670 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:18.333642 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a6560b69-openshift-default-5c8f947cc4-qcw28" Sep 11 16:10:18.389750 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:18.389664 2566 generic.go:356] "Generic (PLEG): container finished" podID="4f755540-c766-4b46-a53f-dce9a5c44ce8" containerID="541ab1bbc4f0da1c5c924332bbc2f7857320e9d473b10178a703f6de0fac857a" exitCode=0 Sep 11 16:10:18.390243 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:18.390043 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a6560b69-openshift-default-5c8f947cc4-qcw28" event={"ID":"4f755540-c766-4b46-a53f-dce9a5c44ce8","Type":"ContainerDied","Data":"541ab1bbc4f0da1c5c924332bbc2f7857320e9d473b10178a703f6de0fac857a"} Sep 11 16:10:18.390243 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:18.390087 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a6560b69-openshift-default-5c8f947cc4-qcw28" Sep 11 16:10:18.390243 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:18.390095 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a6560b69-openshift-default-5c8f947cc4-qcw28" event={"ID":"4f755540-c766-4b46-a53f-dce9a5c44ce8","Type":"ContainerDied","Data":"c4609b4611b5ff465493134edf9251906ff2af7b19056a441c9c9f3c2c27d77d"} Sep 11 16:10:18.390243 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:18.390128 2566 scope.go:117] "RemoveContainer" containerID="541ab1bbc4f0da1c5c924332bbc2f7857320e9d473b10178a703f6de0fac857a" Sep 11 16:10:18.402074 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:18.402047 2566 scope.go:117] "RemoveContainer" containerID="ce243feda875ee9607bcad9a840670b0bfe697c8944f5618604ecdc029bb4da1" Sep 11 16:10:18.424326 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:18.424204 2566 scope.go:117] "RemoveContainer" containerID="541ab1bbc4f0da1c5c924332bbc2f7857320e9d473b10178a703f6de0fac857a" Sep 11 16:10:18.430514 ip-10-0-136-96 kubenswrapper[2566]: E0911 16:10:18.429353 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"541ab1bbc4f0da1c5c924332bbc2f7857320e9d473b10178a703f6de0fac857a\": container with ID starting with 541ab1bbc4f0da1c5c924332bbc2f7857320e9d473b10178a703f6de0fac857a not found: ID does not exist" containerID="541ab1bbc4f0da1c5c924332bbc2f7857320e9d473b10178a703f6de0fac857a" Sep 11 16:10:18.430514 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:18.429406 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"541ab1bbc4f0da1c5c924332bbc2f7857320e9d473b10178a703f6de0fac857a"} err="failed to get container status \"541ab1bbc4f0da1c5c924332bbc2f7857320e9d473b10178a703f6de0fac857a\": rpc error: code = NotFound desc = could not find container \"541ab1bbc4f0da1c5c924332bbc2f7857320e9d473b10178a703f6de0fac857a\": container with ID starting with 541ab1bbc4f0da1c5c924332bbc2f7857320e9d473b10178a703f6de0fac857a not found: ID does not exist" Sep 11 16:10:18.430514 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:18.429435 2566 scope.go:117] "RemoveContainer" containerID="ce243feda875ee9607bcad9a840670b0bfe697c8944f5618604ecdc029bb4da1" Sep 11 16:10:18.431183 ip-10-0-136-96 kubenswrapper[2566]: E0911 16:10:18.431090 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce243feda875ee9607bcad9a840670b0bfe697c8944f5618604ecdc029bb4da1\": container with ID starting with ce243feda875ee9607bcad9a840670b0bfe697c8944f5618604ecdc029bb4da1 not found: ID does not exist" containerID="ce243feda875ee9607bcad9a840670b0bfe697c8944f5618604ecdc029bb4da1" Sep 11 16:10:18.431183 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:18.431128 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce243feda875ee9607bcad9a840670b0bfe697c8944f5618604ecdc029bb4da1"} err="failed to get container status \"ce243feda875ee9607bcad9a840670b0bfe697c8944f5618604ecdc029bb4da1\": rpc error: code = NotFound desc = could not find container \"ce243feda875ee9607bcad9a840670b0bfe697c8944f5618604ecdc029bb4da1\": container with ID starting with ce243feda875ee9607bcad9a840670b0bfe697c8944f5618604ecdc029bb4da1 not found: ID does not exist" Sep 11 16:10:18.463996 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:18.460332 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f755540-c766-4b46-a53f-dce9a5c44ce8-credential-socket\") pod \"4f755540-c766-4b46-a53f-dce9a5c44ce8\" (UID: \"4f755540-c766-4b46-a53f-dce9a5c44ce8\") " Sep 11 16:10:18.463996 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:18.460407 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f755540-c766-4b46-a53f-dce9a5c44ce8-istio-data\") pod \"4f755540-c766-4b46-a53f-dce9a5c44ce8\" (UID: \"4f755540-c766-4b46-a53f-dce9a5c44ce8\") " Sep 11 16:10:18.463996 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:18.460461 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f755540-c766-4b46-a53f-dce9a5c44ce8-istiod-ca-cert\") pod \"4f755540-c766-4b46-a53f-dce9a5c44ce8\" (UID: \"4f755540-c766-4b46-a53f-dce9a5c44ce8\") " Sep 11 16:10:18.463996 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:18.460488 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2xg5w\" (UniqueName: \"kubernetes.io/projected/4f755540-c766-4b46-a53f-dce9a5c44ce8-kube-api-access-2xg5w\") pod \"4f755540-c766-4b46-a53f-dce9a5c44ce8\" (UID: \"4f755540-c766-4b46-a53f-dce9a5c44ce8\") " Sep 11 16:10:18.463996 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:18.460532 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f755540-c766-4b46-a53f-dce9a5c44ce8-istio-envoy\") pod \"4f755540-c766-4b46-a53f-dce9a5c44ce8\" (UID: \"4f755540-c766-4b46-a53f-dce9a5c44ce8\") " Sep 11 16:10:18.463996 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:18.460567 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f755540-c766-4b46-a53f-dce9a5c44ce8-istio-podinfo\") pod \"4f755540-c766-4b46-a53f-dce9a5c44ce8\" (UID: \"4f755540-c766-4b46-a53f-dce9a5c44ce8\") " Sep 11 16:10:18.463996 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:18.460618 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f755540-c766-4b46-a53f-dce9a5c44ce8-workload-certs\") pod \"4f755540-c766-4b46-a53f-dce9a5c44ce8\" (UID: \"4f755540-c766-4b46-a53f-dce9a5c44ce8\") " Sep 11 16:10:18.463996 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:18.460660 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f755540-c766-4b46-a53f-dce9a5c44ce8-istio-token\") pod \"4f755540-c766-4b46-a53f-dce9a5c44ce8\" (UID: \"4f755540-c766-4b46-a53f-dce9a5c44ce8\") " Sep 11 16:10:18.463996 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:18.460727 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f755540-c766-4b46-a53f-dce9a5c44ce8-workload-socket\") pod \"4f755540-c766-4b46-a53f-dce9a5c44ce8\" (UID: \"4f755540-c766-4b46-a53f-dce9a5c44ce8\") " Sep 11 16:10:18.463996 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:18.461300 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f755540-c766-4b46-a53f-dce9a5c44ce8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4f755540-c766-4b46-a53f-dce9a5c44ce8" (UID: "4f755540-c766-4b46-a53f-dce9a5c44ce8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 16:10:18.463996 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:18.461472 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f755540-c766-4b46-a53f-dce9a5c44ce8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4f755540-c766-4b46-a53f-dce9a5c44ce8" (UID: "4f755540-c766-4b46-a53f-dce9a5c44ce8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 16:10:18.463996 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:18.461596 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f755540-c766-4b46-a53f-dce9a5c44ce8-istio-data" (OuterVolumeSpecName: "istio-data") pod "4f755540-c766-4b46-a53f-dce9a5c44ce8" (UID: "4f755540-c766-4b46-a53f-dce9a5c44ce8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 16:10:18.463996 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:18.461824 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f755540-c766-4b46-a53f-dce9a5c44ce8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4f755540-c766-4b46-a53f-dce9a5c44ce8" (UID: "4f755540-c766-4b46-a53f-dce9a5c44ce8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 16:10:18.463996 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:18.462732 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f755540-c766-4b46-a53f-dce9a5c44ce8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4f755540-c766-4b46-a53f-dce9a5c44ce8" (UID: "4f755540-c766-4b46-a53f-dce9a5c44ce8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 16:10:18.470932 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:18.470883 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f755540-c766-4b46-a53f-dce9a5c44ce8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4f755540-c766-4b46-a53f-dce9a5c44ce8" (UID: "4f755540-c766-4b46-a53f-dce9a5c44ce8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 16:10:18.471273 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:18.471253 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4f755540-c766-4b46-a53f-dce9a5c44ce8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4f755540-c766-4b46-a53f-dce9a5c44ce8" (UID: "4f755540-c766-4b46-a53f-dce9a5c44ce8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 16:10:18.471624 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:18.471595 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f755540-c766-4b46-a53f-dce9a5c44ce8-kube-api-access-2xg5w" (OuterVolumeSpecName: "kube-api-access-2xg5w") pod "4f755540-c766-4b46-a53f-dce9a5c44ce8" (UID: "4f755540-c766-4b46-a53f-dce9a5c44ce8"). InnerVolumeSpecName "kube-api-access-2xg5w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 16:10:18.471831 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:18.471810 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f755540-c766-4b46-a53f-dce9a5c44ce8-istio-token" (OuterVolumeSpecName: "istio-token") pod "4f755540-c766-4b46-a53f-dce9a5c44ce8" (UID: "4f755540-c766-4b46-a53f-dce9a5c44ce8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 16:10:18.562059 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:18.561899 2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f755540-c766-4b46-a53f-dce9a5c44ce8-credential-socket\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:10:18.562059 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:18.561942 2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f755540-c766-4b46-a53f-dce9a5c44ce8-istio-data\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:10:18.562059 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:18.561957 2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f755540-c766-4b46-a53f-dce9a5c44ce8-istiod-ca-cert\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:10:18.562059 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:18.562000 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xg5w\" (UniqueName: \"kubernetes.io/projected/4f755540-c766-4b46-a53f-dce9a5c44ce8-kube-api-access-2xg5w\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:10:18.562059 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:18.562019 2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f755540-c766-4b46-a53f-dce9a5c44ce8-istio-envoy\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:10:18.562059 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:18.562031 2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f755540-c766-4b46-a53f-dce9a5c44ce8-istio-podinfo\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:10:18.562059 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:18.562042 2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f755540-c766-4b46-a53f-dce9a5c44ce8-workload-certs\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:10:18.562059 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:18.562054 2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f755540-c766-4b46-a53f-dce9a5c44ce8-istio-token\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:10:18.562059 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:18.562066 2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f755540-c766-4b46-a53f-dce9a5c44ce8-workload-socket\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:10:18.729789 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:18.729672 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-a6560b69-openshift-default-5c8f947cc4-qcw28"] Sep 11 16:10:18.757224 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:18.757173 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-a6560b69-openshift-default-5c8f947cc4-qcw28"] Sep 11 16:10:18.872120 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:18.872082 2566 patch_prober.go:28] interesting pod/e2e-inf-b-2b1bd5-openshift-default-687f66dffb-jdh2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Sep 11 16:10:18.872308 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:18.872141 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2b1bd5-openshift-default-687f66dffb-jdh2v" podUID="c311820f-99df-4371-8967-d5518a1e299b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Sep 11 16:10:19.242869 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:19.242823 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f755540-c766-4b46-a53f-dce9a5c44ce8" path="/var/lib/kubelet/pods/4f755540-c766-4b46-a53f-dce9a5c44ce8/volumes" Sep 11 16:10:19.854783 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:19.854722 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dx4xn"] Sep 11 16:10:19.872012 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:19.871932 2566 patch_prober.go:28] interesting pod/e2e-inf-b-2b1bd5-openshift-default-687f66dffb-jdh2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Sep 11 16:10:19.872012 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:19.872018 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2b1bd5-openshift-default-687f66dffb-jdh2v" podUID="c311820f-99df-4371-8967-d5518a1e299b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Sep 11 16:10:20.872472 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:20.872432 2566 patch_prober.go:28] interesting pod/e2e-inf-b-2b1bd5-openshift-default-687f66dffb-jdh2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Sep 11 16:10:20.872859 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:20.872502 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2b1bd5-openshift-default-687f66dffb-jdh2v" podUID="c311820f-99df-4371-8967-d5518a1e299b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Sep 11 16:10:21.871751 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:21.871714 2566 patch_prober.go:28] interesting pod/e2e-inf-b-2b1bd5-openshift-default-687f66dffb-jdh2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Sep 11 16:10:21.871928 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:21.871774 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2b1bd5-openshift-default-687f66dffb-jdh2v" podUID="c311820f-99df-4371-8967-d5518a1e299b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Sep 11 16:10:22.875801 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:22.875768 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-2b1bd5-openshift-default-687f66dffb-jdh2v" Sep 11 16:10:22.876193 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:22.876025 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-2b1bd5-openshift-default-687f66dffb-jdh2v" Sep 11 16:10:22.876652 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:22.876629 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-2b1bd5-openshift-default-687f66dffb-jdh2v" Sep 11 16:10:33.139047 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:33.136910 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dx4xn"] Sep 11 16:10:36.518637 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:36.518605 2566 generic.go:356] "Generic (PLEG): container finished" podID="186654d0-9025-46d3-880c-25da1cd4df1c" containerID="fcffbf65068e934ddcf030d7116a5256ba132f8f752787324d9adbaf3c8fd850" exitCode=0 Sep 11 16:10:36.519086 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:36.518681 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-f8b85f0c-gw-openshift-default-bc5d5c4d-27r4v" event={"ID":"186654d0-9025-46d3-880c-25da1cd4df1c","Type":"ContainerDied","Data":"fcffbf65068e934ddcf030d7116a5256ba132f8f752787324d9adbaf3c8fd850"} Sep 11 16:10:36.519191 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:36.519174 2566 scope.go:117] "RemoveContainer" containerID="fcffbf65068e934ddcf030d7116a5256ba132f8f752787324d9adbaf3c8fd850" Sep 11 16:10:36.520340 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:36.520313 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 16:10:36.520399 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:36.520380 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 16:10:36.520440 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:36.520426 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 16:10:37.527710 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:37.527676 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-f8b85f0c-gw-openshift-default-bc5d5c4d-27r4v" event={"ID":"186654d0-9025-46d3-880c-25da1cd4df1c","Type":"ContainerStarted","Data":"7376178b93d1edbefe43dcedfe1f536cf16e3f1493f934131313cfd931a0552f"} Sep 11 16:10:38.018579 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:38.018539 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-f8b85f0c-gw-openshift-default-bc5d5c4d-27r4v" Sep 11 16:10:38.018783 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:38.018758 2566 patch_prober.go:28] interesting pod/e2e-migrate-f8b85f0c-gw-openshift-default-bc5d5c4d-27r4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Sep 11 16:10:38.018875 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:38.018820 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-f8b85f0c-gw-openshift-default-bc5d5c4d-27r4v" podUID="186654d0-9025-46d3-880c-25da1cd4df1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Sep 11 16:10:39.019087 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:39.019052 2566 patch_prober.go:28] interesting pod/e2e-migrate-f8b85f0c-gw-openshift-default-bc5d5c4d-27r4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Sep 11 16:10:39.019465 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:39.019112 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-f8b85f0c-gw-openshift-default-bc5d5c4d-27r4v" podUID="186654d0-9025-46d3-880c-25da1cd4df1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Sep 11 16:10:40.018835 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:40.018799 2566 patch_prober.go:28] interesting pod/e2e-migrate-f8b85f0c-gw-openshift-default-bc5d5c4d-27r4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Sep 11 16:10:40.019039 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:40.018871 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-f8b85f0c-gw-openshift-default-bc5d5c4d-27r4v" podUID="186654d0-9025-46d3-880c-25da1cd4df1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Sep 11 16:10:40.869425 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:40.869383 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f8b85f0c-wf9gc"] Sep 11 16:10:40.869843 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:40.869828 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f755540-c766-4b46-a53f-dce9a5c44ce8" containerName="istio-proxy" Sep 11 16:10:40.869843 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:40.869844 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f755540-c766-4b46-a53f-dce9a5c44ce8" containerName="istio-proxy" Sep 11 16:10:40.869940 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:40.869929 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f755540-c766-4b46-a53f-dce9a5c44ce8" containerName="istio-proxy" Sep 11 16:10:40.872253 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:40.872233 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f8b85f0c-wf9gc" Sep 11 16:10:40.874386 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:40.874364 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 16:10:40.874723 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:40.874703 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qbzdr\"" Sep 11 16:10:40.882656 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:40.882626 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f8b85f0c-wf9gc"] Sep 11 16:10:40.986833 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:40.986796 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f256ba0b-f8b3-49a8-901f-90112d7c98bf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f8b85f0c-wf9gc\" (UID: \"f256ba0b-f8b3-49a8-901f-90112d7c98bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f8b85f0c-wf9gc" Sep 11 16:10:41.019174 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:41.019134 2566 patch_prober.go:28] interesting pod/e2e-migrate-f8b85f0c-gw-openshift-default-bc5d5c4d-27r4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Sep 11 16:10:41.019378 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:41.019215 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-f8b85f0c-gw-openshift-default-bc5d5c4d-27r4v" podUID="186654d0-9025-46d3-880c-25da1cd4df1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Sep 11 16:10:41.088511 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:41.088472 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f256ba0b-f8b3-49a8-901f-90112d7c98bf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f8b85f0c-wf9gc\" (UID: \"f256ba0b-f8b3-49a8-901f-90112d7c98bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f8b85f0c-wf9gc" Sep 11 16:10:41.089200 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:41.089173 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f256ba0b-f8b3-49a8-901f-90112d7c98bf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f8b85f0c-wf9gc\" (UID: \"f256ba0b-f8b3-49a8-901f-90112d7c98bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f8b85f0c-wf9gc" Sep 11 16:10:41.184237 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:41.184152 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f8b85f0c-wf9gc" Sep 11 16:10:41.501171 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:41.501142 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f8b85f0c-wf9gc"] Sep 11 16:10:41.503201 ip-10-0-136-96 kubenswrapper[2566]: W0911 16:10:41.503170 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf256ba0b_f8b3_49a8_901f_90112d7c98bf.slice/crio-927941b2c34670a3203a64db4a2f63ea215d89ff732e80c17f6b28ef9938e9e4 WatchSource:0}: Error finding container 927941b2c34670a3203a64db4a2f63ea215d89ff732e80c17f6b28ef9938e9e4: Status 404 returned error can't find the container with id 927941b2c34670a3203a64db4a2f63ea215d89ff732e80c17f6b28ef9938e9e4 Sep 11 16:10:41.569706 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:41.569664 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f8b85f0c-wf9gc" event={"ID":"f256ba0b-f8b3-49a8-901f-90112d7c98bf","Type":"ContainerStarted","Data":"927941b2c34670a3203a64db4a2f63ea215d89ff732e80c17f6b28ef9938e9e4"} Sep 11 16:10:42.018834 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:42.018802 2566 patch_prober.go:28] interesting pod/e2e-migrate-f8b85f0c-gw-openshift-default-bc5d5c4d-27r4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Sep 11 16:10:42.019245 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:42.018855 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-f8b85f0c-gw-openshift-default-bc5d5c4d-27r4v" podUID="186654d0-9025-46d3-880c-25da1cd4df1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Sep 11 16:10:42.577225 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:42.577193 2566 generic.go:356] "Generic (PLEG): container finished" podID="f256ba0b-f8b3-49a8-901f-90112d7c98bf" containerID="cec3bd2c980a438e8c1168eea8b5560a310203ffaaf7efb273f5d3a626ecdfc1" exitCode=0 Sep 11 16:10:42.577395 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:42.577237 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f8b85f0c-wf9gc" event={"ID":"f256ba0b-f8b3-49a8-901f-90112d7c98bf","Type":"ContainerDied","Data":"cec3bd2c980a438e8c1168eea8b5560a310203ffaaf7efb273f5d3a626ecdfc1"} Sep 11 16:10:43.019049 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:43.019011 2566 patch_prober.go:28] interesting pod/e2e-migrate-f8b85f0c-gw-openshift-default-bc5d5c4d-27r4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Sep 11 16:10:43.019430 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:43.019071 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-f8b85f0c-gw-openshift-default-bc5d5c4d-27r4v" podUID="186654d0-9025-46d3-880c-25da1cd4df1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Sep 11 16:10:43.731863 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:43.731840 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f8b85f0c-wf9gc" Sep 11 16:10:43.823667 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:43.823629 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f256ba0b-f8b3-49a8-901f-90112d7c98bf-maas-api-service-ca\") pod \"f256ba0b-f8b3-49a8-901f-90112d7c98bf\" (UID: \"f256ba0b-f8b3-49a8-901f-90112d7c98bf\") " Sep 11 16:10:43.824067 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:43.824046 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f256ba0b-f8b3-49a8-901f-90112d7c98bf-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f256ba0b-f8b3-49a8-901f-90112d7c98bf" (UID: "f256ba0b-f8b3-49a8-901f-90112d7c98bf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 16:10:43.924707 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:43.924619 2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f256ba0b-f8b3-49a8-901f-90112d7c98bf-maas-api-service-ca\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:10:44.023291 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:44.023254 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-f8b85f0c-gw-openshift-default-bc5d5c4d-27r4v" Sep 11 16:10:44.023743 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:44.023592 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-f8b85f0c-gw-openshift-default-bc5d5c4d-27r4v" Sep 11 16:10:44.024148 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:44.024125 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-f8b85f0c-gw-openshift-default-bc5d5c4d-27r4v" Sep 11 16:10:44.590604 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:44.590571 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f8b85f0c-wf9gc" Sep 11 16:10:44.590604 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:44.590598 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f8b85f0c-wf9gc" event={"ID":"f256ba0b-f8b3-49a8-901f-90112d7c98bf","Type":"ContainerDied","Data":"927941b2c34670a3203a64db4a2f63ea215d89ff732e80c17f6b28ef9938e9e4"} Sep 11 16:10:44.590847 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:44.590654 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="927941b2c34670a3203a64db4a2f63ea215d89ff732e80c17f6b28ef9938e9e4" Sep 11 16:10:46.736293 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:46.736252 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dx4xn"] Sep 11 16:10:47.840285 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:47.840243 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f8b85f0c-wf9gc"] Sep 11 16:10:47.843133 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:47.843105 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f8b85f0c-wf9gc"] Sep 11 16:10:47.962353 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:47.962319 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-f8b85f0c-gw-openshift-default-bc5d5c4d-27r4v"] Sep 11 16:10:47.962536 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:47.962502 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-f8b85f0c-gw-openshift-default-bc5d5c4d-27r4v" podUID="186654d0-9025-46d3-880c-25da1cd4df1c" containerName="istio-proxy" containerID="cri-o://7376178b93d1edbefe43dcedfe1f536cf16e3f1493f934131313cfd931a0552f" gracePeriod=30 Sep 11 16:10:48.691761 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:48.691729 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-20aaef10-openshift-default-7d59d8dc9b-bm8sx"] Sep 11 16:10:48.692181 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:48.692168 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f755540-c766-4b46-a53f-dce9a5c44ce8" containerName="istio-proxy" Sep 11 16:10:48.692181 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:48.692182 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f755540-c766-4b46-a53f-dce9a5c44ce8" containerName="istio-proxy" Sep 11 16:10:48.692276 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:48.692192 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f256ba0b-f8b3-49a8-901f-90112d7c98bf" containerName="revoke-keys" Sep 11 16:10:48.692276 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:48.692198 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="f256ba0b-f8b3-49a8-901f-90112d7c98bf" containerName="revoke-keys" Sep 11 16:10:48.692276 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:48.692268 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f755540-c766-4b46-a53f-dce9a5c44ce8" containerName="istio-proxy" Sep 11 16:10:48.692276 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:48.692276 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="f256ba0b-f8b3-49a8-901f-90112d7c98bf" containerName="revoke-keys" Sep 11 16:10:48.694432 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:48.694413 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-20aaef10-openshift-default-7d59d8dc9b-bm8sx" Sep 11 16:10:48.696531 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:48.696513 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-20aaef10-openshift-default-dockercfg-d9d92\"" Sep 11 16:10:48.711061 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:48.711039 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-20aaef10-openshift-default-7d59d8dc9b-bm8sx"] Sep 11 16:10:48.775107 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:48.775063 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c21c3cd-eed2-4036-90af-a330ede0d337-istiod-ca-cert\") pod \"e2e-ait-20aaef10-openshift-default-7d59d8dc9b-bm8sx\" (UID: \"8c21c3cd-eed2-4036-90af-a330ede0d337\") " pod="openshift-ingress/e2e-ait-20aaef10-openshift-default-7d59d8dc9b-bm8sx" Sep 11 16:10:48.775303 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:48.775125 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c21c3cd-eed2-4036-90af-a330ede0d337-credential-socket\") pod \"e2e-ait-20aaef10-openshift-default-7d59d8dc9b-bm8sx\" (UID: \"8c21c3cd-eed2-4036-90af-a330ede0d337\") " pod="openshift-ingress/e2e-ait-20aaef10-openshift-default-7d59d8dc9b-bm8sx" Sep 11 16:10:48.775303 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:48.775194 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c21c3cd-eed2-4036-90af-a330ede0d337-istio-podinfo\") pod \"e2e-ait-20aaef10-openshift-default-7d59d8dc9b-bm8sx\" (UID: \"8c21c3cd-eed2-4036-90af-a330ede0d337\") " pod="openshift-ingress/e2e-ait-20aaef10-openshift-default-7d59d8dc9b-bm8sx" Sep 11 16:10:48.775303 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:48.775227 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnkng\" (UniqueName: \"kubernetes.io/projected/8c21c3cd-eed2-4036-90af-a330ede0d337-kube-api-access-dnkng\") pod \"e2e-ait-20aaef10-openshift-default-7d59d8dc9b-bm8sx\" (UID: \"8c21c3cd-eed2-4036-90af-a330ede0d337\") " pod="openshift-ingress/e2e-ait-20aaef10-openshift-default-7d59d8dc9b-bm8sx" Sep 11 16:10:48.775303 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:48.775260 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c21c3cd-eed2-4036-90af-a330ede0d337-istio-data\") pod \"e2e-ait-20aaef10-openshift-default-7d59d8dc9b-bm8sx\" (UID: \"8c21c3cd-eed2-4036-90af-a330ede0d337\") " pod="openshift-ingress/e2e-ait-20aaef10-openshift-default-7d59d8dc9b-bm8sx" Sep 11 16:10:48.775303 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:48.775291 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c21c3cd-eed2-4036-90af-a330ede0d337-istio-envoy\") pod \"e2e-ait-20aaef10-openshift-default-7d59d8dc9b-bm8sx\" (UID: \"8c21c3cd-eed2-4036-90af-a330ede0d337\") " pod="openshift-ingress/e2e-ait-20aaef10-openshift-default-7d59d8dc9b-bm8sx" Sep 11 16:10:48.775598 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:48.775324 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c21c3cd-eed2-4036-90af-a330ede0d337-workload-socket\") pod \"e2e-ait-20aaef10-openshift-default-7d59d8dc9b-bm8sx\" (UID: \"8c21c3cd-eed2-4036-90af-a330ede0d337\") " pod="openshift-ingress/e2e-ait-20aaef10-openshift-default-7d59d8dc9b-bm8sx" Sep 11 16:10:48.775598 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:48.775368 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c21c3cd-eed2-4036-90af-a330ede0d337-workload-certs\") pod \"e2e-ait-20aaef10-openshift-default-7d59d8dc9b-bm8sx\" (UID: \"8c21c3cd-eed2-4036-90af-a330ede0d337\") " pod="openshift-ingress/e2e-ait-20aaef10-openshift-default-7d59d8dc9b-bm8sx" Sep 11 16:10:48.775598 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:48.775404 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c21c3cd-eed2-4036-90af-a330ede0d337-istio-token\") pod \"e2e-ait-20aaef10-openshift-default-7d59d8dc9b-bm8sx\" (UID: \"8c21c3cd-eed2-4036-90af-a330ede0d337\") " pod="openshift-ingress/e2e-ait-20aaef10-openshift-default-7d59d8dc9b-bm8sx" Sep 11 16:10:48.876381 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:48.876345 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c21c3cd-eed2-4036-90af-a330ede0d337-istio-podinfo\") pod \"e2e-ait-20aaef10-openshift-default-7d59d8dc9b-bm8sx\" (UID: \"8c21c3cd-eed2-4036-90af-a330ede0d337\") " pod="openshift-ingress/e2e-ait-20aaef10-openshift-default-7d59d8dc9b-bm8sx" Sep 11 16:10:48.876381 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:48.876385 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnkng\" (UniqueName: \"kubernetes.io/projected/8c21c3cd-eed2-4036-90af-a330ede0d337-kube-api-access-dnkng\") pod \"e2e-ait-20aaef10-openshift-default-7d59d8dc9b-bm8sx\" (UID: \"8c21c3cd-eed2-4036-90af-a330ede0d337\") " pod="openshift-ingress/e2e-ait-20aaef10-openshift-default-7d59d8dc9b-bm8sx" Sep 11 16:10:48.876866 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:48.876409 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c21c3cd-eed2-4036-90af-a330ede0d337-istio-data\") pod \"e2e-ait-20aaef10-openshift-default-7d59d8dc9b-bm8sx\" (UID: \"8c21c3cd-eed2-4036-90af-a330ede0d337\") " pod="openshift-ingress/e2e-ait-20aaef10-openshift-default-7d59d8dc9b-bm8sx" Sep 11 16:10:48.876866 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:48.876438 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c21c3cd-eed2-4036-90af-a330ede0d337-istio-envoy\") pod \"e2e-ait-20aaef10-openshift-default-7d59d8dc9b-bm8sx\" (UID: \"8c21c3cd-eed2-4036-90af-a330ede0d337\") " pod="openshift-ingress/e2e-ait-20aaef10-openshift-default-7d59d8dc9b-bm8sx" Sep 11 16:10:48.876866 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:48.876466 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c21c3cd-eed2-4036-90af-a330ede0d337-workload-socket\") pod \"e2e-ait-20aaef10-openshift-default-7d59d8dc9b-bm8sx\" (UID: \"8c21c3cd-eed2-4036-90af-a330ede0d337\") " pod="openshift-ingress/e2e-ait-20aaef10-openshift-default-7d59d8dc9b-bm8sx" Sep 11 16:10:48.876866 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:48.876494 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c21c3cd-eed2-4036-90af-a330ede0d337-workload-certs\") pod \"e2e-ait-20aaef10-openshift-default-7d59d8dc9b-bm8sx\" (UID: \"8c21c3cd-eed2-4036-90af-a330ede0d337\") " pod="openshift-ingress/e2e-ait-20aaef10-openshift-default-7d59d8dc9b-bm8sx" Sep 11 16:10:48.876866 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:48.876523 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c21c3cd-eed2-4036-90af-a330ede0d337-istio-token\") pod \"e2e-ait-20aaef10-openshift-default-7d59d8dc9b-bm8sx\" (UID: \"8c21c3cd-eed2-4036-90af-a330ede0d337\") " pod="openshift-ingress/e2e-ait-20aaef10-openshift-default-7d59d8dc9b-bm8sx" Sep 11 16:10:48.876866 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:48.876619 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c21c3cd-eed2-4036-90af-a330ede0d337-istiod-ca-cert\") pod \"e2e-ait-20aaef10-openshift-default-7d59d8dc9b-bm8sx\" (UID: \"8c21c3cd-eed2-4036-90af-a330ede0d337\") " pod="openshift-ingress/e2e-ait-20aaef10-openshift-default-7d59d8dc9b-bm8sx" Sep 11 16:10:48.876866 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:48.876657 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c21c3cd-eed2-4036-90af-a330ede0d337-credential-socket\") pod \"e2e-ait-20aaef10-openshift-default-7d59d8dc9b-bm8sx\" (UID: \"8c21c3cd-eed2-4036-90af-a330ede0d337\") " pod="openshift-ingress/e2e-ait-20aaef10-openshift-default-7d59d8dc9b-bm8sx" Sep 11 16:10:48.877204 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:48.876865 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c21c3cd-eed2-4036-90af-a330ede0d337-istio-data\") pod \"e2e-ait-20aaef10-openshift-default-7d59d8dc9b-bm8sx\" (UID: \"8c21c3cd-eed2-4036-90af-a330ede0d337\") " pod="openshift-ingress/e2e-ait-20aaef10-openshift-default-7d59d8dc9b-bm8sx" Sep 11 16:10:48.877204 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:48.876900 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c21c3cd-eed2-4036-90af-a330ede0d337-workload-socket\") pod \"e2e-ait-20aaef10-openshift-default-7d59d8dc9b-bm8sx\" (UID: \"8c21c3cd-eed2-4036-90af-a330ede0d337\") " pod="openshift-ingress/e2e-ait-20aaef10-openshift-default-7d59d8dc9b-bm8sx" Sep 11 16:10:48.877204 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:48.876964 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c21c3cd-eed2-4036-90af-a330ede0d337-workload-certs\") pod \"e2e-ait-20aaef10-openshift-default-7d59d8dc9b-bm8sx\" (UID: \"8c21c3cd-eed2-4036-90af-a330ede0d337\") " pod="openshift-ingress/e2e-ait-20aaef10-openshift-default-7d59d8dc9b-bm8sx" Sep 11 16:10:48.877204 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:48.877165 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c21c3cd-eed2-4036-90af-a330ede0d337-credential-socket\") pod \"e2e-ait-20aaef10-openshift-default-7d59d8dc9b-bm8sx\" (UID: \"8c21c3cd-eed2-4036-90af-a330ede0d337\") " pod="openshift-ingress/e2e-ait-20aaef10-openshift-default-7d59d8dc9b-bm8sx" Sep 11 16:10:48.877334 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:48.877276 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c21c3cd-eed2-4036-90af-a330ede0d337-istiod-ca-cert\") pod \"e2e-ait-20aaef10-openshift-default-7d59d8dc9b-bm8sx\" (UID: \"8c21c3cd-eed2-4036-90af-a330ede0d337\") " pod="openshift-ingress/e2e-ait-20aaef10-openshift-default-7d59d8dc9b-bm8sx" Sep 11 16:10:48.878960 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:48.878930 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c21c3cd-eed2-4036-90af-a330ede0d337-istio-envoy\") pod \"e2e-ait-20aaef10-openshift-default-7d59d8dc9b-bm8sx\" (UID: \"8c21c3cd-eed2-4036-90af-a330ede0d337\") " pod="openshift-ingress/e2e-ait-20aaef10-openshift-default-7d59d8dc9b-bm8sx" Sep 11 16:10:48.879094 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:48.879037 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c21c3cd-eed2-4036-90af-a330ede0d337-istio-podinfo\") pod \"e2e-ait-20aaef10-openshift-default-7d59d8dc9b-bm8sx\" (UID: \"8c21c3cd-eed2-4036-90af-a330ede0d337\") " pod="openshift-ingress/e2e-ait-20aaef10-openshift-default-7d59d8dc9b-bm8sx" Sep 11 16:10:48.886485 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:48.886466 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c21c3cd-eed2-4036-90af-a330ede0d337-istio-token\") pod \"e2e-ait-20aaef10-openshift-default-7d59d8dc9b-bm8sx\" (UID: \"8c21c3cd-eed2-4036-90af-a330ede0d337\") " pod="openshift-ingress/e2e-ait-20aaef10-openshift-default-7d59d8dc9b-bm8sx" Sep 11 16:10:48.888849 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:48.888828 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnkng\" (UniqueName: \"kubernetes.io/projected/8c21c3cd-eed2-4036-90af-a330ede0d337-kube-api-access-dnkng\") pod \"e2e-ait-20aaef10-openshift-default-7d59d8dc9b-bm8sx\" (UID: \"8c21c3cd-eed2-4036-90af-a330ede0d337\") " pod="openshift-ingress/e2e-ait-20aaef10-openshift-default-7d59d8dc9b-bm8sx" Sep 11 16:10:49.004669 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:49.004574 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-20aaef10-openshift-default-7d59d8dc9b-bm8sx" Sep 11 16:10:49.240127 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:49.240095 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f256ba0b-f8b3-49a8-901f-90112d7c98bf" path="/var/lib/kubelet/pods/f256ba0b-f8b3-49a8-901f-90112d7c98bf/volumes" Sep 11 16:10:49.311674 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:49.311647 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-20aaef10-openshift-default-7d59d8dc9b-bm8sx"] Sep 11 16:10:49.313684 ip-10-0-136-96 kubenswrapper[2566]: W0911 16:10:49.313654 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c21c3cd_eed2_4036_90af_a330ede0d337.slice/crio-27390d48104cf872ac10470b175e600d0ba5cb7bcb1bc4b9a3d10c982ee9e99a WatchSource:0}: Error finding container 27390d48104cf872ac10470b175e600d0ba5cb7bcb1bc4b9a3d10c982ee9e99a: Status 404 returned error can't find the container with id 27390d48104cf872ac10470b175e600d0ba5cb7bcb1bc4b9a3d10c982ee9e99a Sep 11 16:10:49.315964 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:49.315930 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 16:10:49.316051 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:49.316038 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 16:10:49.316096 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:49.316089 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 16:10:49.625467 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:49.625378 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-20aaef10-openshift-default-7d59d8dc9b-bm8sx" event={"ID":"8c21c3cd-eed2-4036-90af-a330ede0d337","Type":"ContainerStarted","Data":"eeb8b78f52e1edc177abcecb65d55ac7ec886e00631d150f5beb453211d4c7ba"} Sep 11 16:10:49.625608 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:49.625469 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-20aaef10-openshift-default-7d59d8dc9b-bm8sx" event={"ID":"8c21c3cd-eed2-4036-90af-a330ede0d337","Type":"ContainerStarted","Data":"27390d48104cf872ac10470b175e600d0ba5cb7bcb1bc4b9a3d10c982ee9e99a"} Sep 11 16:10:49.671938 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:49.671883 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-20aaef10-openshift-default-7d59d8dc9b-bm8sx" podStartSLOduration=1.67186572 podStartE2EDuration="1.67186572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:10:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:10:49.666573739 +0000 UTC m=+956.881407645" watchObservedRunningTime="2026-09-11 16:10:49.67186572 +0000 UTC m=+956.886699625" Sep 11 16:10:50.005129 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:50.005038 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-20aaef10-openshift-default-7d59d8dc9b-bm8sx" Sep 11 16:10:50.010340 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:50.010315 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-20aaef10-openshift-default-7d59d8dc9b-bm8sx" Sep 11 16:10:50.631610 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:50.631577 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-20aaef10-openshift-default-7d59d8dc9b-bm8sx" Sep 11 16:10:50.632544 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:50.632522 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-20aaef10-openshift-default-7d59d8dc9b-bm8sx" Sep 11 16:10:53.229388 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:53.229314 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-f8b85f0c-gw-openshift-default-bc5d5c4d-27r4v" Sep 11 16:10:53.324338 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:53.324304 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/186654d0-9025-46d3-880c-25da1cd4df1c-credential-socket\") pod \"186654d0-9025-46d3-880c-25da1cd4df1c\" (UID: \"186654d0-9025-46d3-880c-25da1cd4df1c\") " Sep 11 16:10:53.324595 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:53.324359 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/186654d0-9025-46d3-880c-25da1cd4df1c-istiod-ca-cert\") pod \"186654d0-9025-46d3-880c-25da1cd4df1c\" (UID: \"186654d0-9025-46d3-880c-25da1cd4df1c\") " Sep 11 16:10:53.324595 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:53.324413 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/186654d0-9025-46d3-880c-25da1cd4df1c-istio-token\") pod \"186654d0-9025-46d3-880c-25da1cd4df1c\" (UID: \"186654d0-9025-46d3-880c-25da1cd4df1c\") " Sep 11 16:10:53.324595 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:53.324439 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/186654d0-9025-46d3-880c-25da1cd4df1c-istio-envoy\") pod \"186654d0-9025-46d3-880c-25da1cd4df1c\" (UID: \"186654d0-9025-46d3-880c-25da1cd4df1c\") " Sep 11 16:10:53.324595 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:53.324460 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r8ncs\" (UniqueName: \"kubernetes.io/projected/186654d0-9025-46d3-880c-25da1cd4df1c-kube-api-access-r8ncs\") pod \"186654d0-9025-46d3-880c-25da1cd4df1c\" (UID: \"186654d0-9025-46d3-880c-25da1cd4df1c\") " Sep 11 16:10:53.324832 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:53.324669 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/186654d0-9025-46d3-880c-25da1cd4df1c-istio-podinfo\") pod \"186654d0-9025-46d3-880c-25da1cd4df1c\" (UID: \"186654d0-9025-46d3-880c-25da1cd4df1c\") " Sep 11 16:10:53.324832 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:53.324664 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/186654d0-9025-46d3-880c-25da1cd4df1c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "186654d0-9025-46d3-880c-25da1cd4df1c" (UID: "186654d0-9025-46d3-880c-25da1cd4df1c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 16:10:53.324832 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:53.324722 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/186654d0-9025-46d3-880c-25da1cd4df1c-istio-data\") pod \"186654d0-9025-46d3-880c-25da1cd4df1c\" (UID: \"186654d0-9025-46d3-880c-25da1cd4df1c\") " Sep 11 16:10:53.324832 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:53.324752 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/186654d0-9025-46d3-880c-25da1cd4df1c-workload-socket\") pod \"186654d0-9025-46d3-880c-25da1cd4df1c\" (UID: \"186654d0-9025-46d3-880c-25da1cd4df1c\") " Sep 11 16:10:53.324832 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:53.324816 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/186654d0-9025-46d3-880c-25da1cd4df1c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "186654d0-9025-46d3-880c-25da1cd4df1c" (UID: "186654d0-9025-46d3-880c-25da1cd4df1c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 16:10:53.325129 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:53.324849 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/186654d0-9025-46d3-880c-25da1cd4df1c-workload-certs\") pod \"186654d0-9025-46d3-880c-25da1cd4df1c\" (UID: \"186654d0-9025-46d3-880c-25da1cd4df1c\") " Sep 11 16:10:53.325129 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:53.324930 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/186654d0-9025-46d3-880c-25da1cd4df1c-istio-data" (OuterVolumeSpecName: "istio-data") pod "186654d0-9025-46d3-880c-25da1cd4df1c" (UID: "186654d0-9025-46d3-880c-25da1cd4df1c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 16:10:53.325269 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:53.325230 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/186654d0-9025-46d3-880c-25da1cd4df1c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "186654d0-9025-46d3-880c-25da1cd4df1c" (UID: "186654d0-9025-46d3-880c-25da1cd4df1c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 16:10:53.325340 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:53.325251 2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/186654d0-9025-46d3-880c-25da1cd4df1c-credential-socket\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:10:53.325340 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:53.325293 2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/186654d0-9025-46d3-880c-25da1cd4df1c-istiod-ca-cert\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:10:53.325340 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:53.325302 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/186654d0-9025-46d3-880c-25da1cd4df1c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "186654d0-9025-46d3-880c-25da1cd4df1c" (UID: "186654d0-9025-46d3-880c-25da1cd4df1c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 16:10:53.325340 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:53.325310 2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/186654d0-9025-46d3-880c-25da1cd4df1c-istio-data\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:10:53.327180 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:53.327134 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/186654d0-9025-46d3-880c-25da1cd4df1c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "186654d0-9025-46d3-880c-25da1cd4df1c" (UID: "186654d0-9025-46d3-880c-25da1cd4df1c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 16:10:53.327426 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:53.327394 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/186654d0-9025-46d3-880c-25da1cd4df1c-istio-token" (OuterVolumeSpecName: "istio-token") pod "186654d0-9025-46d3-880c-25da1cd4df1c" (UID: "186654d0-9025-46d3-880c-25da1cd4df1c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 16:10:53.327638 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:53.327616 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/186654d0-9025-46d3-880c-25da1cd4df1c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "186654d0-9025-46d3-880c-25da1cd4df1c" (UID: "186654d0-9025-46d3-880c-25da1cd4df1c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 16:10:53.327750 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:53.327698 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/186654d0-9025-46d3-880c-25da1cd4df1c-kube-api-access-r8ncs" (OuterVolumeSpecName: "kube-api-access-r8ncs") pod "186654d0-9025-46d3-880c-25da1cd4df1c" (UID: "186654d0-9025-46d3-880c-25da1cd4df1c"). InnerVolumeSpecName "kube-api-access-r8ncs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 16:10:53.426770 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:53.426729 2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/186654d0-9025-46d3-880c-25da1cd4df1c-istio-token\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:10:53.426770 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:53.426764 2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/186654d0-9025-46d3-880c-25da1cd4df1c-istio-envoy\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:10:53.426770 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:53.426774 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8ncs\" (UniqueName: \"kubernetes.io/projected/186654d0-9025-46d3-880c-25da1cd4df1c-kube-api-access-r8ncs\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:10:53.427038 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:53.426783 2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/186654d0-9025-46d3-880c-25da1cd4df1c-istio-podinfo\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:10:53.427038 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:53.426792 2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/186654d0-9025-46d3-880c-25da1cd4df1c-workload-socket\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:10:53.427038 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:53.426800 2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/186654d0-9025-46d3-880c-25da1cd4df1c-workload-certs\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:10:53.651924 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:53.651883 2566 generic.go:356] "Generic (PLEG): container finished" podID="186654d0-9025-46d3-880c-25da1cd4df1c" containerID="7376178b93d1edbefe43dcedfe1f536cf16e3f1493f934131313cfd931a0552f" exitCode=0 Sep 11 16:10:53.652112 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:53.651952 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-f8b85f0c-gw-openshift-default-bc5d5c4d-27r4v" Sep 11 16:10:53.652112 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:53.651987 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-f8b85f0c-gw-openshift-default-bc5d5c4d-27r4v" event={"ID":"186654d0-9025-46d3-880c-25da1cd4df1c","Type":"ContainerDied","Data":"7376178b93d1edbefe43dcedfe1f536cf16e3f1493f934131313cfd931a0552f"} Sep 11 16:10:53.652112 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:53.652074 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-f8b85f0c-gw-openshift-default-bc5d5c4d-27r4v" event={"ID":"186654d0-9025-46d3-880c-25da1cd4df1c","Type":"ContainerDied","Data":"1c1c0514baac1f7bd46e195a4de86202a891803d4ba79bff39fb95e049485e4d"} Sep 11 16:10:53.652112 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:53.652088 2566 scope.go:117] "RemoveContainer" containerID="7376178b93d1edbefe43dcedfe1f536cf16e3f1493f934131313cfd931a0552f" Sep 11 16:10:53.662203 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:53.662184 2566 scope.go:117] "RemoveContainer" containerID="fcffbf65068e934ddcf030d7116a5256ba132f8f752787324d9adbaf3c8fd850" Sep 11 16:10:53.670850 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:53.670834 2566 scope.go:117] "RemoveContainer" containerID="7376178b93d1edbefe43dcedfe1f536cf16e3f1493f934131313cfd931a0552f" Sep 11 16:10:53.671132 ip-10-0-136-96 kubenswrapper[2566]: E0911 16:10:53.671115 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7376178b93d1edbefe43dcedfe1f536cf16e3f1493f934131313cfd931a0552f\": container with ID starting with 7376178b93d1edbefe43dcedfe1f536cf16e3f1493f934131313cfd931a0552f not found: ID does not exist" containerID="7376178b93d1edbefe43dcedfe1f536cf16e3f1493f934131313cfd931a0552f" Sep 11 16:10:53.671217 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:53.671145 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7376178b93d1edbefe43dcedfe1f536cf16e3f1493f934131313cfd931a0552f"} err="failed to get container status \"7376178b93d1edbefe43dcedfe1f536cf16e3f1493f934131313cfd931a0552f\": rpc error: code = NotFound desc = could not find container \"7376178b93d1edbefe43dcedfe1f536cf16e3f1493f934131313cfd931a0552f\": container with ID starting with 7376178b93d1edbefe43dcedfe1f536cf16e3f1493f934131313cfd931a0552f not found: ID does not exist" Sep 11 16:10:53.671217 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:53.671168 2566 scope.go:117] "RemoveContainer" containerID="fcffbf65068e934ddcf030d7116a5256ba132f8f752787324d9adbaf3c8fd850" Sep 11 16:10:53.671432 ip-10-0-136-96 kubenswrapper[2566]: E0911 16:10:53.671412 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcffbf65068e934ddcf030d7116a5256ba132f8f752787324d9adbaf3c8fd850\": container with ID starting with fcffbf65068e934ddcf030d7116a5256ba132f8f752787324d9adbaf3c8fd850 not found: ID does not exist" containerID="fcffbf65068e934ddcf030d7116a5256ba132f8f752787324d9adbaf3c8fd850" Sep 11 16:10:53.671484 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:53.671438 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcffbf65068e934ddcf030d7116a5256ba132f8f752787324d9adbaf3c8fd850"} err="failed to get container status \"fcffbf65068e934ddcf030d7116a5256ba132f8f752787324d9adbaf3c8fd850\": rpc error: code = NotFound desc = could not find container \"fcffbf65068e934ddcf030d7116a5256ba132f8f752787324d9adbaf3c8fd850\": container with ID starting with fcffbf65068e934ddcf030d7116a5256ba132f8f752787324d9adbaf3c8fd850 not found: ID does not exist" Sep 11 16:10:54.032523 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:54.032487 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-f8b85f0c-gw-openshift-default-bc5d5c4d-27r4v"] Sep 11 16:10:54.045101 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:54.045068 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-f8b85f0c-gw-openshift-default-bc5d5c4d-27r4v"] Sep 11 16:10:55.240271 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:10:55.240226 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="186654d0-9025-46d3-880c-25da1cd4df1c" path="/var/lib/kubelet/pods/186654d0-9025-46d3-880c-25da1cd4df1c/volumes" Sep 11 16:11:00.323360 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:00.323315 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dx4xn"] Sep 11 16:11:10.762163 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:10.762130 2566 generic.go:356] "Generic (PLEG): container finished" podID="8c21c3cd-eed2-4036-90af-a330ede0d337" containerID="eeb8b78f52e1edc177abcecb65d55ac7ec886e00631d150f5beb453211d4c7ba" exitCode=0 Sep 11 16:11:10.762769 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:10.762200 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-20aaef10-openshift-default-7d59d8dc9b-bm8sx" event={"ID":"8c21c3cd-eed2-4036-90af-a330ede0d337","Type":"ContainerDied","Data":"eeb8b78f52e1edc177abcecb65d55ac7ec886e00631d150f5beb453211d4c7ba"} Sep 11 16:11:10.762769 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:10.762602 2566 scope.go:117] "RemoveContainer" containerID="eeb8b78f52e1edc177abcecb65d55ac7ec886e00631d150f5beb453211d4c7ba" Sep 11 16:11:10.766874 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:10.766839 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 16:11:10.767015 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:10.766913 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 16:11:10.767015 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:10.766947 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 16:11:11.219520 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:11.219422 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dx4xn"] Sep 11 16:11:11.774938 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:11.774899 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-20aaef10-openshift-default-7d59d8dc9b-bm8sx" event={"ID":"8c21c3cd-eed2-4036-90af-a330ede0d337","Type":"ContainerStarted","Data":"4fa37d24c46d01b9e595e982f557bc19420f38c57959e6f4665d50f96bc72044"} Sep 11 16:11:12.005523 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:12.005484 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-20aaef10-openshift-default-7d59d8dc9b-bm8sx" Sep 11 16:11:12.005739 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:12.005716 2566 patch_prober.go:28] interesting pod/e2e-ait-20aaef10-openshift-default-7d59d8dc9b-bm8sx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Sep 11 16:11:12.005792 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:12.005776 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-20aaef10-openshift-default-7d59d8dc9b-bm8sx" podUID="8c21c3cd-eed2-4036-90af-a330ede0d337" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Sep 11 16:11:13.005050 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:13.005009 2566 patch_prober.go:28] interesting pod/e2e-ait-20aaef10-openshift-default-7d59d8dc9b-bm8sx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Sep 11 16:11:13.005463 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:13.005085 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-20aaef10-openshift-default-7d59d8dc9b-bm8sx" podUID="8c21c3cd-eed2-4036-90af-a330ede0d337" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Sep 11 16:11:14.005260 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:14.005218 2566 patch_prober.go:28] interesting pod/e2e-ait-20aaef10-openshift-default-7d59d8dc9b-bm8sx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Sep 11 16:11:14.005634 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:14.005283 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-20aaef10-openshift-default-7d59d8dc9b-bm8sx" podUID="8c21c3cd-eed2-4036-90af-a330ede0d337" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Sep 11 16:11:15.005449 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:15.005412 2566 patch_prober.go:28] interesting pod/e2e-ait-20aaef10-openshift-default-7d59d8dc9b-bm8sx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Sep 11 16:11:15.005835 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:15.005502 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-20aaef10-openshift-default-7d59d8dc9b-bm8sx" podUID="8c21c3cd-eed2-4036-90af-a330ede0d337" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Sep 11 16:11:16.005490 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:16.005409 2566 patch_prober.go:28] interesting pod/e2e-ait-20aaef10-openshift-default-7d59d8dc9b-bm8sx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Sep 11 16:11:16.005490 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:16.005472 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-20aaef10-openshift-default-7d59d8dc9b-bm8sx" podUID="8c21c3cd-eed2-4036-90af-a330ede0d337" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Sep 11 16:11:17.009864 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:17.009829 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-20aaef10-openshift-default-7d59d8dc9b-bm8sx" Sep 11 16:11:17.010318 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:17.010222 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-20aaef10-openshift-default-7d59d8dc9b-bm8sx" Sep 11 16:11:17.010965 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:17.010947 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-20aaef10-openshift-default-7d59d8dc9b-bm8sx" Sep 11 16:11:22.055188 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:22.055150 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-20aaef10-fsqpp"] Sep 11 16:11:22.055589 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:22.055576 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="186654d0-9025-46d3-880c-25da1cd4df1c" containerName="istio-proxy" Sep 11 16:11:22.055634 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:22.055590 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="186654d0-9025-46d3-880c-25da1cd4df1c" containerName="istio-proxy" Sep 11 16:11:22.055634 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:22.055598 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="186654d0-9025-46d3-880c-25da1cd4df1c" containerName="istio-proxy" Sep 11 16:11:22.055634 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:22.055604 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="186654d0-9025-46d3-880c-25da1cd4df1c" containerName="istio-proxy" Sep 11 16:11:22.055731 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:22.055676 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="186654d0-9025-46d3-880c-25da1cd4df1c" containerName="istio-proxy" Sep 11 16:11:22.055731 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:22.055687 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="186654d0-9025-46d3-880c-25da1cd4df1c" containerName="istio-proxy" Sep 11 16:11:22.057704 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:22.057688 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-20aaef10-fsqpp" Sep 11 16:11:22.061167 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:22.061138 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qbzdr\"" Sep 11 16:11:22.062311 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:22.062294 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 16:11:22.078258 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:22.078225 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-20aaef10-fsqpp"] Sep 11 16:11:22.142595 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:22.142548 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94004803-2dd9-4574-bdad-a12d20881514-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-20aaef10-fsqpp\" (UID: \"94004803-2dd9-4574-bdad-a12d20881514\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-20aaef10-fsqpp" Sep 11 16:11:22.243690 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:22.243651 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94004803-2dd9-4574-bdad-a12d20881514-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-20aaef10-fsqpp\" (UID: \"94004803-2dd9-4574-bdad-a12d20881514\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-20aaef10-fsqpp" Sep 11 16:11:22.244338 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:22.244315 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94004803-2dd9-4574-bdad-a12d20881514-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-20aaef10-fsqpp\" (UID: \"94004803-2dd9-4574-bdad-a12d20881514\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-20aaef10-fsqpp" Sep 11 16:11:22.368918 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:22.368831 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-20aaef10-fsqpp" Sep 11 16:11:22.644770 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:22.644735 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-20aaef10-fsqpp"] Sep 11 16:11:22.646055 ip-10-0-136-96 kubenswrapper[2566]: W0911 16:11:22.646023 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94004803_2dd9_4574_bdad_a12d20881514.slice/crio-e74a9ed74efdaf258c5db32e62de0114011783a46a3b85bfc95c112044706df4 WatchSource:0}: Error finding container e74a9ed74efdaf258c5db32e62de0114011783a46a3b85bfc95c112044706df4: Status 404 returned error can't find the container with id e74a9ed74efdaf258c5db32e62de0114011783a46a3b85bfc95c112044706df4 Sep 11 16:11:22.845037 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:22.844999 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-20aaef10-fsqpp" event={"ID":"94004803-2dd9-4574-bdad-a12d20881514","Type":"ContainerStarted","Data":"e74a9ed74efdaf258c5db32e62de0114011783a46a3b85bfc95c112044706df4"} Sep 11 16:11:23.852611 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:23.852574 2566 generic.go:356] "Generic (PLEG): container finished" podID="94004803-2dd9-4574-bdad-a12d20881514" containerID="5e2fc177a46b80d225fc692497577b5e9df4d44333c5b04d881972b697449a59" exitCode=0 Sep 11 16:11:23.853083 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:23.852669 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-20aaef10-fsqpp" event={"ID":"94004803-2dd9-4574-bdad-a12d20881514","Type":"ContainerDied","Data":"5e2fc177a46b80d225fc692497577b5e9df4d44333c5b04d881972b697449a59"} Sep 11 16:11:23.942105 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:23.942071 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dx4xn"] Sep 11 16:11:25.020675 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:25.020641 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-20aaef10-fsqpp" Sep 11 16:11:25.173671 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:25.173558 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94004803-2dd9-4574-bdad-a12d20881514-maas-api-service-ca\") pod \"94004803-2dd9-4574-bdad-a12d20881514\" (UID: \"94004803-2dd9-4574-bdad-a12d20881514\") " Sep 11 16:11:25.174052 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:25.174022 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94004803-2dd9-4574-bdad-a12d20881514-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "94004803-2dd9-4574-bdad-a12d20881514" (UID: "94004803-2dd9-4574-bdad-a12d20881514"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 16:11:25.274935 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:25.274897 2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94004803-2dd9-4574-bdad-a12d20881514-maas-api-service-ca\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:11:25.868251 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:25.868221 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-20aaef10-fsqpp" Sep 11 16:11:25.868419 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:25.868260 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-20aaef10-fsqpp" event={"ID":"94004803-2dd9-4574-bdad-a12d20881514","Type":"ContainerDied","Data":"e74a9ed74efdaf258c5db32e62de0114011783a46a3b85bfc95c112044706df4"} Sep 11 16:11:25.868419 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:25.868307 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e74a9ed74efdaf258c5db32e62de0114011783a46a3b85bfc95c112044706df4" Sep 11 16:11:33.787184 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:33.787143 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-20aaef10-fsqpp"] Sep 11 16:11:33.789680 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:33.789658 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-20aaef10-fsqpp"] Sep 11 16:11:34.943631 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:34.943587 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-20aaef10-openshift-default-7d59d8dc9b-bm8sx"] Sep 11 16:11:34.944051 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:34.943806 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-20aaef10-openshift-default-7d59d8dc9b-bm8sx" podUID="8c21c3cd-eed2-4036-90af-a330ede0d337" containerName="istio-proxy" containerID="cri-o://4fa37d24c46d01b9e595e982f557bc19420f38c57959e6f4665d50f96bc72044" gracePeriod=30 Sep 11 16:11:35.240515 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:35.240417 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94004803-2dd9-4574-bdad-a12d20881514" path="/var/lib/kubelet/pods/94004803-2dd9-4574-bdad-a12d20881514/volumes" Sep 11 16:11:35.714369 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:35.714335 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-455ccc2b-openshift-default-65c5c8ff6-cj554"] Sep 11 16:11:35.714794 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:35.714779 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94004803-2dd9-4574-bdad-a12d20881514" containerName="revoke-keys" Sep 11 16:11:35.714794 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:35.714795 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="94004803-2dd9-4574-bdad-a12d20881514" containerName="revoke-keys" Sep 11 16:11:35.714911 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:35.714889 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="94004803-2dd9-4574-bdad-a12d20881514" containerName="revoke-keys" Sep 11 16:11:35.717173 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:35.717156 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-455ccc2b-openshift-default-65c5c8ff6-cj554" Sep 11 16:11:35.721466 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:35.721447 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-455ccc2b-openshift-default-dockercfg-zpfq7\"" Sep 11 16:11:35.735560 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:35.735531 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-455ccc2b-openshift-default-65c5c8ff6-cj554"] Sep 11 16:11:35.787527 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:35.787487 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2z6s\" (UniqueName: \"kubernetes.io/projected/f72e085d-36b9-4aeb-a3f1-59209208092a-kube-api-access-c2z6s\") pod \"e2e-derive-455ccc2b-openshift-default-65c5c8ff6-cj554\" (UID: \"f72e085d-36b9-4aeb-a3f1-59209208092a\") " pod="openshift-ingress/e2e-derive-455ccc2b-openshift-default-65c5c8ff6-cj554" Sep 11 16:11:35.787708 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:35.787540 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f72e085d-36b9-4aeb-a3f1-59209208092a-workload-socket\") pod \"e2e-derive-455ccc2b-openshift-default-65c5c8ff6-cj554\" (UID: \"f72e085d-36b9-4aeb-a3f1-59209208092a\") " pod="openshift-ingress/e2e-derive-455ccc2b-openshift-default-65c5c8ff6-cj554" Sep 11 16:11:35.787708 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:35.787614 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f72e085d-36b9-4aeb-a3f1-59209208092a-istio-data\") pod \"e2e-derive-455ccc2b-openshift-default-65c5c8ff6-cj554\" (UID: \"f72e085d-36b9-4aeb-a3f1-59209208092a\") " pod="openshift-ingress/e2e-derive-455ccc2b-openshift-default-65c5c8ff6-cj554" Sep 11 16:11:35.787708 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:35.787654 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f72e085d-36b9-4aeb-a3f1-59209208092a-workload-certs\") pod \"e2e-derive-455ccc2b-openshift-default-65c5c8ff6-cj554\" (UID: \"f72e085d-36b9-4aeb-a3f1-59209208092a\") " pod="openshift-ingress/e2e-derive-455ccc2b-openshift-default-65c5c8ff6-cj554" Sep 11 16:11:35.787708 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:35.787690 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f72e085d-36b9-4aeb-a3f1-59209208092a-istio-podinfo\") pod \"e2e-derive-455ccc2b-openshift-default-65c5c8ff6-cj554\" (UID: \"f72e085d-36b9-4aeb-a3f1-59209208092a\") " pod="openshift-ingress/e2e-derive-455ccc2b-openshift-default-65c5c8ff6-cj554" Sep 11 16:11:35.787873 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:35.787713 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f72e085d-36b9-4aeb-a3f1-59209208092a-istio-envoy\") pod \"e2e-derive-455ccc2b-openshift-default-65c5c8ff6-cj554\" (UID: \"f72e085d-36b9-4aeb-a3f1-59209208092a\") " pod="openshift-ingress/e2e-derive-455ccc2b-openshift-default-65c5c8ff6-cj554" Sep 11 16:11:35.787873 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:35.787777 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f72e085d-36b9-4aeb-a3f1-59209208092a-istio-token\") pod \"e2e-derive-455ccc2b-openshift-default-65c5c8ff6-cj554\" (UID: \"f72e085d-36b9-4aeb-a3f1-59209208092a\") " pod="openshift-ingress/e2e-derive-455ccc2b-openshift-default-65c5c8ff6-cj554" Sep 11 16:11:35.787873 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:35.787824 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f72e085d-36b9-4aeb-a3f1-59209208092a-istiod-ca-cert\") pod \"e2e-derive-455ccc2b-openshift-default-65c5c8ff6-cj554\" (UID: \"f72e085d-36b9-4aeb-a3f1-59209208092a\") " pod="openshift-ingress/e2e-derive-455ccc2b-openshift-default-65c5c8ff6-cj554" Sep 11 16:11:35.787873 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:35.787864 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f72e085d-36b9-4aeb-a3f1-59209208092a-credential-socket\") pod \"e2e-derive-455ccc2b-openshift-default-65c5c8ff6-cj554\" (UID: \"f72e085d-36b9-4aeb-a3f1-59209208092a\") " pod="openshift-ingress/e2e-derive-455ccc2b-openshift-default-65c5c8ff6-cj554" Sep 11 16:11:35.888826 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:35.888792 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f72e085d-36b9-4aeb-a3f1-59209208092a-istio-data\") pod \"e2e-derive-455ccc2b-openshift-default-65c5c8ff6-cj554\" (UID: \"f72e085d-36b9-4aeb-a3f1-59209208092a\") " pod="openshift-ingress/e2e-derive-455ccc2b-openshift-default-65c5c8ff6-cj554" Sep 11 16:11:35.889016 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:35.888841 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f72e085d-36b9-4aeb-a3f1-59209208092a-workload-certs\") pod \"e2e-derive-455ccc2b-openshift-default-65c5c8ff6-cj554\" (UID: \"f72e085d-36b9-4aeb-a3f1-59209208092a\") " pod="openshift-ingress/e2e-derive-455ccc2b-openshift-default-65c5c8ff6-cj554" Sep 11 16:11:35.889016 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:35.888886 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f72e085d-36b9-4aeb-a3f1-59209208092a-istio-podinfo\") pod \"e2e-derive-455ccc2b-openshift-default-65c5c8ff6-cj554\" (UID: \"f72e085d-36b9-4aeb-a3f1-59209208092a\") " pod="openshift-ingress/e2e-derive-455ccc2b-openshift-default-65c5c8ff6-cj554" Sep 11 16:11:35.889016 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:35.888903 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f72e085d-36b9-4aeb-a3f1-59209208092a-istio-envoy\") pod \"e2e-derive-455ccc2b-openshift-default-65c5c8ff6-cj554\" (UID: \"f72e085d-36b9-4aeb-a3f1-59209208092a\") " pod="openshift-ingress/e2e-derive-455ccc2b-openshift-default-65c5c8ff6-cj554" Sep 11 16:11:35.889016 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:35.888931 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f72e085d-36b9-4aeb-a3f1-59209208092a-istio-token\") pod \"e2e-derive-455ccc2b-openshift-default-65c5c8ff6-cj554\" (UID: \"f72e085d-36b9-4aeb-a3f1-59209208092a\") " pod="openshift-ingress/e2e-derive-455ccc2b-openshift-default-65c5c8ff6-cj554" Sep 11 16:11:35.889016 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:35.888964 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f72e085d-36b9-4aeb-a3f1-59209208092a-istiod-ca-cert\") pod \"e2e-derive-455ccc2b-openshift-default-65c5c8ff6-cj554\" (UID: \"f72e085d-36b9-4aeb-a3f1-59209208092a\") " pod="openshift-ingress/e2e-derive-455ccc2b-openshift-default-65c5c8ff6-cj554" Sep 11 16:11:35.889279 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:35.889033 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f72e085d-36b9-4aeb-a3f1-59209208092a-credential-socket\") pod \"e2e-derive-455ccc2b-openshift-default-65c5c8ff6-cj554\" (UID: \"f72e085d-36b9-4aeb-a3f1-59209208092a\") " pod="openshift-ingress/e2e-derive-455ccc2b-openshift-default-65c5c8ff6-cj554" Sep 11 16:11:35.889279 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:35.889078 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2z6s\" (UniqueName: \"kubernetes.io/projected/f72e085d-36b9-4aeb-a3f1-59209208092a-kube-api-access-c2z6s\") pod \"e2e-derive-455ccc2b-openshift-default-65c5c8ff6-cj554\" (UID: \"f72e085d-36b9-4aeb-a3f1-59209208092a\") " pod="openshift-ingress/e2e-derive-455ccc2b-openshift-default-65c5c8ff6-cj554" Sep 11 16:11:35.889411 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:35.889275 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f72e085d-36b9-4aeb-a3f1-59209208092a-workload-socket\") pod \"e2e-derive-455ccc2b-openshift-default-65c5c8ff6-cj554\" (UID: \"f72e085d-36b9-4aeb-a3f1-59209208092a\") " pod="openshift-ingress/e2e-derive-455ccc2b-openshift-default-65c5c8ff6-cj554" Sep 11 16:11:35.889411 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:35.889337 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f72e085d-36b9-4aeb-a3f1-59209208092a-istio-data\") pod \"e2e-derive-455ccc2b-openshift-default-65c5c8ff6-cj554\" (UID: \"f72e085d-36b9-4aeb-a3f1-59209208092a\") " pod="openshift-ingress/e2e-derive-455ccc2b-openshift-default-65c5c8ff6-cj554" Sep 11 16:11:35.889508 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:35.889486 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f72e085d-36b9-4aeb-a3f1-59209208092a-credential-socket\") pod \"e2e-derive-455ccc2b-openshift-default-65c5c8ff6-cj554\" (UID: \"f72e085d-36b9-4aeb-a3f1-59209208092a\") " pod="openshift-ingress/e2e-derive-455ccc2b-openshift-default-65c5c8ff6-cj554" Sep 11 16:11:35.889631 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:35.889610 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f72e085d-36b9-4aeb-a3f1-59209208092a-workload-socket\") pod \"e2e-derive-455ccc2b-openshift-default-65c5c8ff6-cj554\" (UID: \"f72e085d-36b9-4aeb-a3f1-59209208092a\") " pod="openshift-ingress/e2e-derive-455ccc2b-openshift-default-65c5c8ff6-cj554" Sep 11 16:11:35.889888 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:35.889860 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f72e085d-36b9-4aeb-a3f1-59209208092a-workload-certs\") pod \"e2e-derive-455ccc2b-openshift-default-65c5c8ff6-cj554\" (UID: \"f72e085d-36b9-4aeb-a3f1-59209208092a\") " pod="openshift-ingress/e2e-derive-455ccc2b-openshift-default-65c5c8ff6-cj554" Sep 11 16:11:35.890051 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:35.890031 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f72e085d-36b9-4aeb-a3f1-59209208092a-istiod-ca-cert\") pod \"e2e-derive-455ccc2b-openshift-default-65c5c8ff6-cj554\" (UID: \"f72e085d-36b9-4aeb-a3f1-59209208092a\") " pod="openshift-ingress/e2e-derive-455ccc2b-openshift-default-65c5c8ff6-cj554" Sep 11 16:11:35.891725 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:35.891701 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f72e085d-36b9-4aeb-a3f1-59209208092a-istio-podinfo\") pod \"e2e-derive-455ccc2b-openshift-default-65c5c8ff6-cj554\" (UID: \"f72e085d-36b9-4aeb-a3f1-59209208092a\") " pod="openshift-ingress/e2e-derive-455ccc2b-openshift-default-65c5c8ff6-cj554" Sep 11 16:11:35.891808 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:35.891728 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f72e085d-36b9-4aeb-a3f1-59209208092a-istio-envoy\") pod \"e2e-derive-455ccc2b-openshift-default-65c5c8ff6-cj554\" (UID: \"f72e085d-36b9-4aeb-a3f1-59209208092a\") " pod="openshift-ingress/e2e-derive-455ccc2b-openshift-default-65c5c8ff6-cj554" Sep 11 16:11:35.900112 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:35.900066 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2z6s\" (UniqueName: \"kubernetes.io/projected/f72e085d-36b9-4aeb-a3f1-59209208092a-kube-api-access-c2z6s\") pod \"e2e-derive-455ccc2b-openshift-default-65c5c8ff6-cj554\" (UID: \"f72e085d-36b9-4aeb-a3f1-59209208092a\") " pod="openshift-ingress/e2e-derive-455ccc2b-openshift-default-65c5c8ff6-cj554" Sep 11 16:11:35.901220 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:35.901190 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f72e085d-36b9-4aeb-a3f1-59209208092a-istio-token\") pod \"e2e-derive-455ccc2b-openshift-default-65c5c8ff6-cj554\" (UID: \"f72e085d-36b9-4aeb-a3f1-59209208092a\") " pod="openshift-ingress/e2e-derive-455ccc2b-openshift-default-65c5c8ff6-cj554" Sep 11 16:11:36.027557 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:36.027520 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-455ccc2b-openshift-default-65c5c8ff6-cj554" Sep 11 16:11:36.312980 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:36.312941 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-455ccc2b-openshift-default-65c5c8ff6-cj554"] Sep 11 16:11:36.314091 ip-10-0-136-96 kubenswrapper[2566]: W0911 16:11:36.314047 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf72e085d_36b9_4aeb_a3f1_59209208092a.slice/crio-0eed13b6e38fa2677b2bfd59f4e30c30b9dff41434baffc1a080c91ba0f06577 WatchSource:0}: Error finding container 0eed13b6e38fa2677b2bfd59f4e30c30b9dff41434baffc1a080c91ba0f06577: Status 404 returned error can't find the container with id 0eed13b6e38fa2677b2bfd59f4e30c30b9dff41434baffc1a080c91ba0f06577 Sep 11 16:11:36.316706 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:36.316669 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 16:11:36.316799 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:36.316762 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 16:11:36.316840 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:36.316812 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 16:11:36.943998 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:36.943938 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-455ccc2b-openshift-default-65c5c8ff6-cj554" event={"ID":"f72e085d-36b9-4aeb-a3f1-59209208092a","Type":"ContainerStarted","Data":"b459a0d7f2f85849af002b5f60d0c15f55215ac6e41ccf5e875b8bc5562e76a9"} Sep 11 16:11:36.944173 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:36.944024 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-455ccc2b-openshift-default-65c5c8ff6-cj554" event={"ID":"f72e085d-36b9-4aeb-a3f1-59209208092a","Type":"ContainerStarted","Data":"0eed13b6e38fa2677b2bfd59f4e30c30b9dff41434baffc1a080c91ba0f06577"} Sep 11 16:11:36.971495 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:36.971448 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-455ccc2b-openshift-default-65c5c8ff6-cj554" podStartSLOduration=1.971432573 podStartE2EDuration="1.971432573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:11:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:11:36.968885538 +0000 UTC m=+1004.183719442" watchObservedRunningTime="2026-09-11 16:11:36.971432573 +0000 UTC m=+1004.186266478" Sep 11 16:11:37.027835 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:37.027804 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-455ccc2b-openshift-default-65c5c8ff6-cj554" Sep 11 16:11:37.033319 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:37.033294 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-455ccc2b-openshift-default-65c5c8ff6-cj554" Sep 11 16:11:37.472879 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:37.472844 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dx4xn"] Sep 11 16:11:37.951965 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:37.951934 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-455ccc2b-openshift-default-65c5c8ff6-cj554" Sep 11 16:11:37.952875 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:37.952854 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-455ccc2b-openshift-default-65c5c8ff6-cj554" Sep 11 16:11:40.209693 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:40.209632 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-20aaef10-openshift-default-7d59d8dc9b-bm8sx" Sep 11 16:11:40.234335 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:40.234307 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c21c3cd-eed2-4036-90af-a330ede0d337-istiod-ca-cert\") pod \"8c21c3cd-eed2-4036-90af-a330ede0d337\" (UID: \"8c21c3cd-eed2-4036-90af-a330ede0d337\") " Sep 11 16:11:40.234335 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:40.234342 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c21c3cd-eed2-4036-90af-a330ede0d337-credential-socket\") pod \"8c21c3cd-eed2-4036-90af-a330ede0d337\" (UID: \"8c21c3cd-eed2-4036-90af-a330ede0d337\") " Sep 11 16:11:40.234553 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:40.234363 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c21c3cd-eed2-4036-90af-a330ede0d337-istio-data\") pod \"8c21c3cd-eed2-4036-90af-a330ede0d337\" (UID: \"8c21c3cd-eed2-4036-90af-a330ede0d337\") " Sep 11 16:11:40.234553 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:40.234402 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c21c3cd-eed2-4036-90af-a330ede0d337-istio-token\") pod \"8c21c3cd-eed2-4036-90af-a330ede0d337\" (UID: \"8c21c3cd-eed2-4036-90af-a330ede0d337\") " Sep 11 16:11:40.234553 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:40.234447 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c21c3cd-eed2-4036-90af-a330ede0d337-workload-certs\") pod \"8c21c3cd-eed2-4036-90af-a330ede0d337\" (UID: \"8c21c3cd-eed2-4036-90af-a330ede0d337\") " Sep 11 16:11:40.234553 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:40.234526 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c21c3cd-eed2-4036-90af-a330ede0d337-istio-envoy\") pod \"8c21c3cd-eed2-4036-90af-a330ede0d337\" (UID: \"8c21c3cd-eed2-4036-90af-a330ede0d337\") " Sep 11 16:11:40.234761 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:40.234562 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c21c3cd-eed2-4036-90af-a330ede0d337-istio-podinfo\") pod \"8c21c3cd-eed2-4036-90af-a330ede0d337\" (UID: \"8c21c3cd-eed2-4036-90af-a330ede0d337\") " Sep 11 16:11:40.234761 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:40.234595 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dnkng\" (UniqueName: \"kubernetes.io/projected/8c21c3cd-eed2-4036-90af-a330ede0d337-kube-api-access-dnkng\") pod \"8c21c3cd-eed2-4036-90af-a330ede0d337\" (UID: \"8c21c3cd-eed2-4036-90af-a330ede0d337\") " Sep 11 16:11:40.234761 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:40.234637 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c21c3cd-eed2-4036-90af-a330ede0d337-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8c21c3cd-eed2-4036-90af-a330ede0d337" (UID: "8c21c3cd-eed2-4036-90af-a330ede0d337"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 16:11:40.234761 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:40.234649 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c21c3cd-eed2-4036-90af-a330ede0d337-workload-socket\") pod \"8c21c3cd-eed2-4036-90af-a330ede0d337\" (UID: \"8c21c3cd-eed2-4036-90af-a330ede0d337\") " Sep 11 16:11:40.235004 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:40.234775 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c21c3cd-eed2-4036-90af-a330ede0d337-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8c21c3cd-eed2-4036-90af-a330ede0d337" (UID: "8c21c3cd-eed2-4036-90af-a330ede0d337"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 16:11:40.235004 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:40.234781 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c21c3cd-eed2-4036-90af-a330ede0d337-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8c21c3cd-eed2-4036-90af-a330ede0d337" (UID: "8c21c3cd-eed2-4036-90af-a330ede0d337"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 16:11:40.235004 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:40.234948 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c21c3cd-eed2-4036-90af-a330ede0d337-istio-data" (OuterVolumeSpecName: "istio-data") pod "8c21c3cd-eed2-4036-90af-a330ede0d337" (UID: "8c21c3cd-eed2-4036-90af-a330ede0d337"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 16:11:40.235160 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:40.235100 2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c21c3cd-eed2-4036-90af-a330ede0d337-istiod-ca-cert\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:11:40.235160 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:40.235123 2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c21c3cd-eed2-4036-90af-a330ede0d337-credential-socket\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:11:40.235160 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:40.235138 2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c21c3cd-eed2-4036-90af-a330ede0d337-istio-data\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:11:40.235160 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:40.235152 2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c21c3cd-eed2-4036-90af-a330ede0d337-workload-certs\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:11:40.235359 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:40.235177 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c21c3cd-eed2-4036-90af-a330ede0d337-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8c21c3cd-eed2-4036-90af-a330ede0d337" (UID: "8c21c3cd-eed2-4036-90af-a330ede0d337"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 16:11:40.237436 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:40.237397 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c21c3cd-eed2-4036-90af-a330ede0d337-kube-api-access-dnkng" (OuterVolumeSpecName: "kube-api-access-dnkng") pod "8c21c3cd-eed2-4036-90af-a330ede0d337" (UID: "8c21c3cd-eed2-4036-90af-a330ede0d337"). InnerVolumeSpecName "kube-api-access-dnkng". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 16:11:40.237563 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:40.237474 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c21c3cd-eed2-4036-90af-a330ede0d337-istio-token" (OuterVolumeSpecName: "istio-token") pod "8c21c3cd-eed2-4036-90af-a330ede0d337" (UID: "8c21c3cd-eed2-4036-90af-a330ede0d337"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 16:11:40.237563 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:40.237496 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8c21c3cd-eed2-4036-90af-a330ede0d337-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8c21c3cd-eed2-4036-90af-a330ede0d337" (UID: "8c21c3cd-eed2-4036-90af-a330ede0d337"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 16:11:40.237563 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:40.237543 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c21c3cd-eed2-4036-90af-a330ede0d337-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8c21c3cd-eed2-4036-90af-a330ede0d337" (UID: "8c21c3cd-eed2-4036-90af-a330ede0d337"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 16:11:40.336549 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:40.336510 2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c21c3cd-eed2-4036-90af-a330ede0d337-istio-token\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:11:40.336549 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:40.336538 2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c21c3cd-eed2-4036-90af-a330ede0d337-istio-envoy\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:11:40.336549 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:40.336548 2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c21c3cd-eed2-4036-90af-a330ede0d337-istio-podinfo\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:11:40.336549 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:40.336557 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnkng\" (UniqueName: \"kubernetes.io/projected/8c21c3cd-eed2-4036-90af-a330ede0d337-kube-api-access-dnkng\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:11:40.336819 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:40.336568 2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c21c3cd-eed2-4036-90af-a330ede0d337-workload-socket\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:11:40.972334 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:40.972302 2566 generic.go:356] "Generic (PLEG): container finished" podID="8c21c3cd-eed2-4036-90af-a330ede0d337" containerID="4fa37d24c46d01b9e595e982f557bc19420f38c57959e6f4665d50f96bc72044" exitCode=0 Sep 11 16:11:40.972507 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:40.972371 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-20aaef10-openshift-default-7d59d8dc9b-bm8sx" Sep 11 16:11:40.972507 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:40.972388 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-20aaef10-openshift-default-7d59d8dc9b-bm8sx" event={"ID":"8c21c3cd-eed2-4036-90af-a330ede0d337","Type":"ContainerDied","Data":"4fa37d24c46d01b9e595e982f557bc19420f38c57959e6f4665d50f96bc72044"} Sep 11 16:11:40.972507 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:40.972468 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-20aaef10-openshift-default-7d59d8dc9b-bm8sx" event={"ID":"8c21c3cd-eed2-4036-90af-a330ede0d337","Type":"ContainerDied","Data":"27390d48104cf872ac10470b175e600d0ba5cb7bcb1bc4b9a3d10c982ee9e99a"} Sep 11 16:11:40.972507 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:40.972490 2566 scope.go:117] "RemoveContainer" containerID="4fa37d24c46d01b9e595e982f557bc19420f38c57959e6f4665d50f96bc72044" Sep 11 16:11:40.982591 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:40.982571 2566 scope.go:117] "RemoveContainer" containerID="eeb8b78f52e1edc177abcecb65d55ac7ec886e00631d150f5beb453211d4c7ba" Sep 11 16:11:40.992028 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:40.992003 2566 scope.go:117] "RemoveContainer" containerID="4fa37d24c46d01b9e595e982f557bc19420f38c57959e6f4665d50f96bc72044" Sep 11 16:11:40.992358 ip-10-0-136-96 kubenswrapper[2566]: E0911 16:11:40.992337 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fa37d24c46d01b9e595e982f557bc19420f38c57959e6f4665d50f96bc72044\": container with ID starting with 4fa37d24c46d01b9e595e982f557bc19420f38c57959e6f4665d50f96bc72044 not found: ID does not exist" containerID="4fa37d24c46d01b9e595e982f557bc19420f38c57959e6f4665d50f96bc72044" Sep 11 16:11:40.992415 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:40.992368 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fa37d24c46d01b9e595e982f557bc19420f38c57959e6f4665d50f96bc72044"} err="failed to get container status \"4fa37d24c46d01b9e595e982f557bc19420f38c57959e6f4665d50f96bc72044\": rpc error: code = NotFound desc = could not find container \"4fa37d24c46d01b9e595e982f557bc19420f38c57959e6f4665d50f96bc72044\": container with ID starting with 4fa37d24c46d01b9e595e982f557bc19420f38c57959e6f4665d50f96bc72044 not found: ID does not exist" Sep 11 16:11:40.992415 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:40.992386 2566 scope.go:117] "RemoveContainer" containerID="eeb8b78f52e1edc177abcecb65d55ac7ec886e00631d150f5beb453211d4c7ba" Sep 11 16:11:40.992733 ip-10-0-136-96 kubenswrapper[2566]: E0911 16:11:40.992663 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eeb8b78f52e1edc177abcecb65d55ac7ec886e00631d150f5beb453211d4c7ba\": container with ID starting with eeb8b78f52e1edc177abcecb65d55ac7ec886e00631d150f5beb453211d4c7ba not found: ID does not exist" containerID="eeb8b78f52e1edc177abcecb65d55ac7ec886e00631d150f5beb453211d4c7ba" Sep 11 16:11:40.992733 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:40.992695 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eeb8b78f52e1edc177abcecb65d55ac7ec886e00631d150f5beb453211d4c7ba"} err="failed to get container status \"eeb8b78f52e1edc177abcecb65d55ac7ec886e00631d150f5beb453211d4c7ba\": rpc error: code = NotFound desc = could not find container \"eeb8b78f52e1edc177abcecb65d55ac7ec886e00631d150f5beb453211d4c7ba\": container with ID starting with eeb8b78f52e1edc177abcecb65d55ac7ec886e00631d150f5beb453211d4c7ba not found: ID does not exist" Sep 11 16:11:40.996177 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:40.996141 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-20aaef10-openshift-default-7d59d8dc9b-bm8sx"] Sep 11 16:11:41.000832 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:41.000809 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-20aaef10-openshift-default-7d59d8dc9b-bm8sx"] Sep 11 16:11:41.239434 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:41.239358 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c21c3cd-eed2-4036-90af-a330ede0d337" path="/var/lib/kubelet/pods/8c21c3cd-eed2-4036-90af-a330ede0d337/volumes" Sep 11 16:11:53.844619 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:53.844574 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dx4xn"] Sep 11 16:11:58.100737 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:58.100703 2566 generic.go:356] "Generic (PLEG): container finished" podID="f72e085d-36b9-4aeb-a3f1-59209208092a" containerID="b459a0d7f2f85849af002b5f60d0c15f55215ac6e41ccf5e875b8bc5562e76a9" exitCode=0 Sep 11 16:11:58.101180 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:58.100778 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-455ccc2b-openshift-default-65c5c8ff6-cj554" event={"ID":"f72e085d-36b9-4aeb-a3f1-59209208092a","Type":"ContainerDied","Data":"b459a0d7f2f85849af002b5f60d0c15f55215ac6e41ccf5e875b8bc5562e76a9"} Sep 11 16:11:58.101273 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:58.101248 2566 scope.go:117] "RemoveContainer" containerID="b459a0d7f2f85849af002b5f60d0c15f55215ac6e41ccf5e875b8bc5562e76a9" Sep 11 16:11:58.102595 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:58.102570 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 16:11:58.102676 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:58.102641 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 16:11:58.102721 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:58.102692 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 16:11:59.109980 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:11:59.109946 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-455ccc2b-openshift-default-65c5c8ff6-cj554" event={"ID":"f72e085d-36b9-4aeb-a3f1-59209208092a","Type":"ContainerStarted","Data":"4edb74f8fd87e39287c77c46903f9e0f1ef01d3464cc66709f22ca6b91ad02e2"} Sep 11 16:12:00.028411 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:00.028378 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-455ccc2b-openshift-default-65c5c8ff6-cj554" Sep 11 16:12:00.028665 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:00.028635 2566 patch_prober.go:28] interesting pod/e2e-derive-455ccc2b-openshift-default-65c5c8ff6-cj554 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Sep 11 16:12:00.028723 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:00.028703 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-455ccc2b-openshift-default-65c5c8ff6-cj554" podUID="f72e085d-36b9-4aeb-a3f1-59209208092a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Sep 11 16:12:01.027918 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:01.027878 2566 patch_prober.go:28] interesting pod/e2e-derive-455ccc2b-openshift-default-65c5c8ff6-cj554 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Sep 11 16:12:01.028376 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:01.027934 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-455ccc2b-openshift-default-65c5c8ff6-cj554" podUID="f72e085d-36b9-4aeb-a3f1-59209208092a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Sep 11 16:12:02.027989 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:02.027933 2566 patch_prober.go:28] interesting pod/e2e-derive-455ccc2b-openshift-default-65c5c8ff6-cj554 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Sep 11 16:12:02.028377 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:02.028022 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-455ccc2b-openshift-default-65c5c8ff6-cj554" podUID="f72e085d-36b9-4aeb-a3f1-59209208092a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Sep 11 16:12:03.028169 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:03.028126 2566 patch_prober.go:28] interesting pod/e2e-derive-455ccc2b-openshift-default-65c5c8ff6-cj554 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Sep 11 16:12:03.028666 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:03.028200 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-455ccc2b-openshift-default-65c5c8ff6-cj554" podUID="f72e085d-36b9-4aeb-a3f1-59209208092a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Sep 11 16:12:04.028395 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:04.028359 2566 patch_prober.go:28] interesting pod/e2e-derive-455ccc2b-openshift-default-65c5c8ff6-cj554 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Sep 11 16:12:04.028822 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:04.028424 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-455ccc2b-openshift-default-65c5c8ff6-cj554" podUID="f72e085d-36b9-4aeb-a3f1-59209208092a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Sep 11 16:12:05.031953 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:05.031919 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-455ccc2b-openshift-default-65c5c8ff6-cj554" Sep 11 16:12:05.032343 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:05.032221 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-455ccc2b-openshift-default-65c5c8ff6-cj554" Sep 11 16:12:05.033055 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:05.033036 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-455ccc2b-openshift-default-65c5c8ff6-cj554" Sep 11 16:12:05.164373 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:05.164343 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dx4xn"] Sep 11 16:12:08.310649 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:08.310612 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-455ccc2b-t8rvp"] Sep 11 16:12:08.311103 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:08.311087 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c21c3cd-eed2-4036-90af-a330ede0d337" containerName="istio-proxy" Sep 11 16:12:08.311165 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:08.311107 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c21c3cd-eed2-4036-90af-a330ede0d337" containerName="istio-proxy" Sep 11 16:12:08.311165 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:08.311120 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c21c3cd-eed2-4036-90af-a330ede0d337" containerName="istio-proxy" Sep 11 16:12:08.311165 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:08.311125 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c21c3cd-eed2-4036-90af-a330ede0d337" containerName="istio-proxy" Sep 11 16:12:08.311276 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:08.311193 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c21c3cd-eed2-4036-90af-a330ede0d337" containerName="istio-proxy" Sep 11 16:12:08.314355 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:08.314338 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-455ccc2b-t8rvp" Sep 11 16:12:08.318230 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:08.318199 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 16:12:08.318397 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:08.318382 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qbzdr\"" Sep 11 16:12:08.324836 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:08.324808 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-455ccc2b-t8rvp"] Sep 11 16:12:08.331276 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:08.331251 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b92f917-0c3c-40eb-8b59-2a1a40c9a8e0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-455ccc2b-t8rvp\" (UID: \"2b92f917-0c3c-40eb-8b59-2a1a40c9a8e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-455ccc2b-t8rvp" Sep 11 16:12:08.432722 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:08.432687 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b92f917-0c3c-40eb-8b59-2a1a40c9a8e0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-455ccc2b-t8rvp\" (UID: \"2b92f917-0c3c-40eb-8b59-2a1a40c9a8e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-455ccc2b-t8rvp" Sep 11 16:12:08.433350 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:08.433329 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b92f917-0c3c-40eb-8b59-2a1a40c9a8e0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-455ccc2b-t8rvp\" (UID: \"2b92f917-0c3c-40eb-8b59-2a1a40c9a8e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-455ccc2b-t8rvp" Sep 11 16:12:08.626069 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:08.625966 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-455ccc2b-t8rvp" Sep 11 16:12:08.909297 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:08.909269 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-455ccc2b-t8rvp"] Sep 11 16:12:08.910729 ip-10-0-136-96 kubenswrapper[2566]: W0911 16:12:08.910701 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b92f917_0c3c_40eb_8b59_2a1a40c9a8e0.slice/crio-c710fbc0946533dc313e1afbfdd916f6cc1efa919e06bf288232dde805546dbf WatchSource:0}: Error finding container c710fbc0946533dc313e1afbfdd916f6cc1efa919e06bf288232dde805546dbf: Status 404 returned error can't find the container with id c710fbc0946533dc313e1afbfdd916f6cc1efa919e06bf288232dde805546dbf Sep 11 16:12:09.186700 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:09.186667 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-455ccc2b-t8rvp" event={"ID":"2b92f917-0c3c-40eb-8b59-2a1a40c9a8e0","Type":"ContainerStarted","Data":"9d5c419a420bd1a2f356ba495dbbf7db4a7d0de1ea013c82ac81ea0c50223511"} Sep 11 16:12:09.186858 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:09.186718 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-455ccc2b-t8rvp" event={"ID":"2b92f917-0c3c-40eb-8b59-2a1a40c9a8e0","Type":"ContainerStarted","Data":"c710fbc0946533dc313e1afbfdd916f6cc1efa919e06bf288232dde805546dbf"} Sep 11 16:12:09.211240 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:09.211180 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-455ccc2b-t8rvp" podStartSLOduration=1.211161378 podStartE2EDuration="1.211161378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:12:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:12:09.208421351 +0000 UTC m=+1036.423255257" watchObservedRunningTime="2026-09-11 16:12:09.211161378 +0000 UTC m=+1036.425995283" Sep 11 16:12:10.194053 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:10.194019 2566 generic.go:356] "Generic (PLEG): container finished" podID="2b92f917-0c3c-40eb-8b59-2a1a40c9a8e0" containerID="9d5c419a420bd1a2f356ba495dbbf7db4a7d0de1ea013c82ac81ea0c50223511" exitCode=0 Sep 11 16:12:10.194436 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:10.194075 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-455ccc2b-t8rvp" event={"ID":"2b92f917-0c3c-40eb-8b59-2a1a40c9a8e0","Type":"ContainerDied","Data":"9d5c419a420bd1a2f356ba495dbbf7db4a7d0de1ea013c82ac81ea0c50223511"} Sep 11 16:12:11.351622 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:11.351596 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-455ccc2b-t8rvp" Sep 11 16:12:11.464189 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:11.464108 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b92f917-0c3c-40eb-8b59-2a1a40c9a8e0-maas-api-service-ca\") pod \"2b92f917-0c3c-40eb-8b59-2a1a40c9a8e0\" (UID: \"2b92f917-0c3c-40eb-8b59-2a1a40c9a8e0\") " Sep 11 16:12:11.464484 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:11.464463 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b92f917-0c3c-40eb-8b59-2a1a40c9a8e0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2b92f917-0c3c-40eb-8b59-2a1a40c9a8e0" (UID: "2b92f917-0c3c-40eb-8b59-2a1a40c9a8e0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 16:12:11.565155 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:11.565119 2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b92f917-0c3c-40eb-8b59-2a1a40c9a8e0-maas-api-service-ca\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:12:12.208459 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:12.208417 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-455ccc2b-t8rvp" event={"ID":"2b92f917-0c3c-40eb-8b59-2a1a40c9a8e0","Type":"ContainerDied","Data":"c710fbc0946533dc313e1afbfdd916f6cc1efa919e06bf288232dde805546dbf"} Sep 11 16:12:12.208617 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:12.208481 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c710fbc0946533dc313e1afbfdd916f6cc1efa919e06bf288232dde805546dbf" Sep 11 16:12:12.208617 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:12.208438 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-455ccc2b-t8rvp" Sep 11 16:12:21.453900 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:21.453859 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-455ccc2b-t8rvp"] Sep 11 16:12:21.457578 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:21.457552 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-455ccc2b-t8rvp"] Sep 11 16:12:21.601156 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:21.601122 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-455ccc2b-openshift-default-65c5c8ff6-cj554"] Sep 11 16:12:21.601376 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:21.601337 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-455ccc2b-openshift-default-65c5c8ff6-cj554" podUID="f72e085d-36b9-4aeb-a3f1-59209208092a" containerName="istio-proxy" containerID="cri-o://4edb74f8fd87e39287c77c46903f9e0f1ef01d3464cc66709f22ca6b91ad02e2" gracePeriod=30 Sep 11 16:12:22.943399 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:22.943363 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dx4xn"] Sep 11 16:12:23.241424 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:23.241339 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b92f917-0c3c-40eb-8b59-2a1a40c9a8e0" path="/var/lib/kubelet/pods/2b92f917-0c3c-40eb-8b59-2a1a40c9a8e0/volumes" Sep 11 16:12:25.014111 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:25.014077 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-9c5b09-jjz6d"] Sep 11 16:12:25.014629 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:25.014610 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b92f917-0c3c-40eb-8b59-2a1a40c9a8e0" containerName="revoke-keys" Sep 11 16:12:25.014677 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:25.014631 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b92f917-0c3c-40eb-8b59-2a1a40c9a8e0" containerName="revoke-keys" Sep 11 16:12:25.014713 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:25.014708 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b92f917-0c3c-40eb-8b59-2a1a40c9a8e0" containerName="revoke-keys" Sep 11 16:12:25.014752 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:25.014717 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c21c3cd-eed2-4036-90af-a330ede0d337" containerName="istio-proxy" Sep 11 16:12:25.017782 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:25.017767 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-9c5b09-jjz6d" Sep 11 16:12:25.020101 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:25.020081 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qbzdr\"" Sep 11 16:12:25.020759 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:25.020742 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 16:12:25.028874 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:25.028848 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-9c5b09-jjz6d"] Sep 11 16:12:25.104322 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:25.104284 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2153a5bc-fbcf-4461-bc04-ef0125c78e46-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-9c5b09-jjz6d\" (UID: \"2153a5bc-fbcf-4461-bc04-ef0125c78e46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-9c5b09-jjz6d" Sep 11 16:12:25.205547 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:25.205507 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2153a5bc-fbcf-4461-bc04-ef0125c78e46-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-9c5b09-jjz6d\" (UID: \"2153a5bc-fbcf-4461-bc04-ef0125c78e46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-9c5b09-jjz6d" Sep 11 16:12:25.206277 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:25.206254 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2153a5bc-fbcf-4461-bc04-ef0125c78e46-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-9c5b09-jjz6d\" (UID: \"2153a5bc-fbcf-4461-bc04-ef0125c78e46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-9c5b09-jjz6d" Sep 11 16:12:25.329996 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:25.329897 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-9c5b09-jjz6d" Sep 11 16:12:25.617325 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:25.617234 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-9c5b09-jjz6d"] Sep 11 16:12:25.623413 ip-10-0-136-96 kubenswrapper[2566]: W0911 16:12:25.623374 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2153a5bc_fbcf_4461_bc04_ef0125c78e46.slice/crio-f9ca376d3a879b382e9201c4dcca0bea6fddef599c2817d98b80431787bd768f WatchSource:0}: Error finding container f9ca376d3a879b382e9201c4dcca0bea6fddef599c2817d98b80431787bd768f: Status 404 returned error can't find the container with id f9ca376d3a879b382e9201c4dcca0bea6fddef599c2817d98b80431787bd768f Sep 11 16:12:26.296296 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:26.296258 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-9c5b09-jjz6d" event={"ID":"2153a5bc-fbcf-4461-bc04-ef0125c78e46","Type":"ContainerStarted","Data":"84791abd3445f9ddf2ddb9fb510dc853e7d22e89dba0e9af34797562b85c3512"} Sep 11 16:12:26.296701 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:26.296322 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-9c5b09-jjz6d" event={"ID":"2153a5bc-fbcf-4461-bc04-ef0125c78e46","Type":"ContainerStarted","Data":"f9ca376d3a879b382e9201c4dcca0bea6fddef599c2817d98b80431787bd768f"} Sep 11 16:12:26.854850 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:26.854823 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-455ccc2b-openshift-default-65c5c8ff6-cj554" Sep 11 16:12:26.891176 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:26.891028 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-9c5b09-jjz6d" podStartSLOduration=2.891006548 podStartE2EDuration="2.891006548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:12:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:12:26.320739434 +0000 UTC m=+1053.535573338" watchObservedRunningTime="2026-09-11 16:12:26.891006548 +0000 UTC m=+1054.105840453" Sep 11 16:12:26.922582 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:26.922551 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f72e085d-36b9-4aeb-a3f1-59209208092a-istio-token\") pod \"f72e085d-36b9-4aeb-a3f1-59209208092a\" (UID: \"f72e085d-36b9-4aeb-a3f1-59209208092a\") " Sep 11 16:12:26.922745 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:26.922606 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f72e085d-36b9-4aeb-a3f1-59209208092a-istio-data\") pod \"f72e085d-36b9-4aeb-a3f1-59209208092a\" (UID: \"f72e085d-36b9-4aeb-a3f1-59209208092a\") " Sep 11 16:12:26.922745 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:26.922648 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f72e085d-36b9-4aeb-a3f1-59209208092a-istiod-ca-cert\") pod \"f72e085d-36b9-4aeb-a3f1-59209208092a\" (UID: \"f72e085d-36b9-4aeb-a3f1-59209208092a\") " Sep 11 16:12:26.922745 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:26.922674 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f72e085d-36b9-4aeb-a3f1-59209208092a-credential-socket\") pod \"f72e085d-36b9-4aeb-a3f1-59209208092a\" (UID: \"f72e085d-36b9-4aeb-a3f1-59209208092a\") " Sep 11 16:12:26.922745 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:26.922739 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c2z6s\" (UniqueName: \"kubernetes.io/projected/f72e085d-36b9-4aeb-a3f1-59209208092a-kube-api-access-c2z6s\") pod \"f72e085d-36b9-4aeb-a3f1-59209208092a\" (UID: \"f72e085d-36b9-4aeb-a3f1-59209208092a\") " Sep 11 16:12:26.922928 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:26.922763 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f72e085d-36b9-4aeb-a3f1-59209208092a-workload-certs\") pod \"f72e085d-36b9-4aeb-a3f1-59209208092a\" (UID: \"f72e085d-36b9-4aeb-a3f1-59209208092a\") " Sep 11 16:12:26.922928 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:26.922791 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f72e085d-36b9-4aeb-a3f1-59209208092a-workload-socket\") pod \"f72e085d-36b9-4aeb-a3f1-59209208092a\" (UID: \"f72e085d-36b9-4aeb-a3f1-59209208092a\") " Sep 11 16:12:26.922928 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:26.922862 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f72e085d-36b9-4aeb-a3f1-59209208092a-istio-envoy\") pod \"f72e085d-36b9-4aeb-a3f1-59209208092a\" (UID: \"f72e085d-36b9-4aeb-a3f1-59209208092a\") " Sep 11 16:12:26.922928 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:26.922908 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f72e085d-36b9-4aeb-a3f1-59209208092a-istio-data" (OuterVolumeSpecName: "istio-data") pod "f72e085d-36b9-4aeb-a3f1-59209208092a" (UID: "f72e085d-36b9-4aeb-a3f1-59209208092a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 16:12:26.923175 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:26.923087 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f72e085d-36b9-4aeb-a3f1-59209208092a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f72e085d-36b9-4aeb-a3f1-59209208092a" (UID: "f72e085d-36b9-4aeb-a3f1-59209208092a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 16:12:26.923175 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:26.923105 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f72e085d-36b9-4aeb-a3f1-59209208092a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f72e085d-36b9-4aeb-a3f1-59209208092a" (UID: "f72e085d-36b9-4aeb-a3f1-59209208092a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 16:12:26.923175 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:26.923122 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f72e085d-36b9-4aeb-a3f1-59209208092a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f72e085d-36b9-4aeb-a3f1-59209208092a" (UID: "f72e085d-36b9-4aeb-a3f1-59209208092a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 16:12:26.923175 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:26.923161 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f72e085d-36b9-4aeb-a3f1-59209208092a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f72e085d-36b9-4aeb-a3f1-59209208092a" (UID: "f72e085d-36b9-4aeb-a3f1-59209208092a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 16:12:26.923359 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:26.923223 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f72e085d-36b9-4aeb-a3f1-59209208092a-istio-podinfo\") pod \"f72e085d-36b9-4aeb-a3f1-59209208092a\" (UID: \"f72e085d-36b9-4aeb-a3f1-59209208092a\") " Sep 11 16:12:26.923650 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:26.923623 2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f72e085d-36b9-4aeb-a3f1-59209208092a-istio-data\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:12:26.923765 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:26.923656 2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f72e085d-36b9-4aeb-a3f1-59209208092a-istiod-ca-cert\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:12:26.923765 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:26.923671 2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f72e085d-36b9-4aeb-a3f1-59209208092a-credential-socket\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:12:26.923765 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:26.923685 2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f72e085d-36b9-4aeb-a3f1-59209208092a-workload-certs\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:12:26.923765 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:26.923699 2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f72e085d-36b9-4aeb-a3f1-59209208092a-workload-socket\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:12:26.925107 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:26.925081 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f72e085d-36b9-4aeb-a3f1-59209208092a-istio-token" (OuterVolumeSpecName: "istio-token") pod "f72e085d-36b9-4aeb-a3f1-59209208092a" (UID: "f72e085d-36b9-4aeb-a3f1-59209208092a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 16:12:26.925242 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:26.925106 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f72e085d-36b9-4aeb-a3f1-59209208092a-kube-api-access-c2z6s" (OuterVolumeSpecName: "kube-api-access-c2z6s") pod "f72e085d-36b9-4aeb-a3f1-59209208092a" (UID: "f72e085d-36b9-4aeb-a3f1-59209208092a"). InnerVolumeSpecName "kube-api-access-c2z6s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 16:12:26.925242 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:26.925119 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f72e085d-36b9-4aeb-a3f1-59209208092a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f72e085d-36b9-4aeb-a3f1-59209208092a" (UID: "f72e085d-36b9-4aeb-a3f1-59209208092a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 16:12:26.925242 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:26.925216 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f72e085d-36b9-4aeb-a3f1-59209208092a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f72e085d-36b9-4aeb-a3f1-59209208092a" (UID: "f72e085d-36b9-4aeb-a3f1-59209208092a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 16:12:27.024439 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:27.024405 2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f72e085d-36b9-4aeb-a3f1-59209208092a-istio-envoy\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:12:27.024439 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:27.024435 2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f72e085d-36b9-4aeb-a3f1-59209208092a-istio-podinfo\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:12:27.024439 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:27.024445 2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f72e085d-36b9-4aeb-a3f1-59209208092a-istio-token\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:12:27.024676 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:27.024453 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2z6s\" (UniqueName: \"kubernetes.io/projected/f72e085d-36b9-4aeb-a3f1-59209208092a-kube-api-access-c2z6s\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:12:27.303879 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:27.303836 2566 generic.go:356] "Generic (PLEG): container finished" podID="f72e085d-36b9-4aeb-a3f1-59209208092a" containerID="4edb74f8fd87e39287c77c46903f9e0f1ef01d3464cc66709f22ca6b91ad02e2" exitCode=0 Sep 11 16:12:27.304345 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:27.303885 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-455ccc2b-openshift-default-65c5c8ff6-cj554" Sep 11 16:12:27.304345 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:27.303955 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-455ccc2b-openshift-default-65c5c8ff6-cj554" event={"ID":"f72e085d-36b9-4aeb-a3f1-59209208092a","Type":"ContainerDied","Data":"4edb74f8fd87e39287c77c46903f9e0f1ef01d3464cc66709f22ca6b91ad02e2"} Sep 11 16:12:27.304345 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:27.304039 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-455ccc2b-openshift-default-65c5c8ff6-cj554" event={"ID":"f72e085d-36b9-4aeb-a3f1-59209208092a","Type":"ContainerDied","Data":"0eed13b6e38fa2677b2bfd59f4e30c30b9dff41434baffc1a080c91ba0f06577"} Sep 11 16:12:27.304345 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:27.304065 2566 scope.go:117] "RemoveContainer" containerID="4edb74f8fd87e39287c77c46903f9e0f1ef01d3464cc66709f22ca6b91ad02e2" Sep 11 16:12:27.306059 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:27.306035 2566 generic.go:356] "Generic (PLEG): container finished" podID="2153a5bc-fbcf-4461-bc04-ef0125c78e46" containerID="84791abd3445f9ddf2ddb9fb510dc853e7d22e89dba0e9af34797562b85c3512" exitCode=0 Sep 11 16:12:27.306203 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:27.306095 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-9c5b09-jjz6d" event={"ID":"2153a5bc-fbcf-4461-bc04-ef0125c78e46","Type":"ContainerDied","Data":"84791abd3445f9ddf2ddb9fb510dc853e7d22e89dba0e9af34797562b85c3512"} Sep 11 16:12:27.314081 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:27.314064 2566 scope.go:117] "RemoveContainer" containerID="b459a0d7f2f85849af002b5f60d0c15f55215ac6e41ccf5e875b8bc5562e76a9" Sep 11 16:12:27.323847 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:27.323823 2566 scope.go:117] "RemoveContainer" containerID="4edb74f8fd87e39287c77c46903f9e0f1ef01d3464cc66709f22ca6b91ad02e2" Sep 11 16:12:27.324082 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:27.324055 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-455ccc2b-openshift-default-65c5c8ff6-cj554"] Sep 11 16:12:27.324204 ip-10-0-136-96 kubenswrapper[2566]: E0911 16:12:27.324166 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4edb74f8fd87e39287c77c46903f9e0f1ef01d3464cc66709f22ca6b91ad02e2\": container with ID starting with 4edb74f8fd87e39287c77c46903f9e0f1ef01d3464cc66709f22ca6b91ad02e2 not found: ID does not exist" containerID="4edb74f8fd87e39287c77c46903f9e0f1ef01d3464cc66709f22ca6b91ad02e2" Sep 11 16:12:27.324255 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:27.324198 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4edb74f8fd87e39287c77c46903f9e0f1ef01d3464cc66709f22ca6b91ad02e2"} err="failed to get container status \"4edb74f8fd87e39287c77c46903f9e0f1ef01d3464cc66709f22ca6b91ad02e2\": rpc error: code = NotFound desc = could not find container \"4edb74f8fd87e39287c77c46903f9e0f1ef01d3464cc66709f22ca6b91ad02e2\": container with ID starting with 4edb74f8fd87e39287c77c46903f9e0f1ef01d3464cc66709f22ca6b91ad02e2 not found: ID does not exist" Sep 11 16:12:27.324255 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:27.324222 2566 scope.go:117] "RemoveContainer" containerID="b459a0d7f2f85849af002b5f60d0c15f55215ac6e41ccf5e875b8bc5562e76a9" Sep 11 16:12:27.324506 ip-10-0-136-96 kubenswrapper[2566]: E0911 16:12:27.324488 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b459a0d7f2f85849af002b5f60d0c15f55215ac6e41ccf5e875b8bc5562e76a9\": container with ID starting with b459a0d7f2f85849af002b5f60d0c15f55215ac6e41ccf5e875b8bc5562e76a9 not found: ID does not exist" containerID="b459a0d7f2f85849af002b5f60d0c15f55215ac6e41ccf5e875b8bc5562e76a9" Sep 11 16:12:27.324553 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:27.324512 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b459a0d7f2f85849af002b5f60d0c15f55215ac6e41ccf5e875b8bc5562e76a9"} err="failed to get container status \"b459a0d7f2f85849af002b5f60d0c15f55215ac6e41ccf5e875b8bc5562e76a9\": rpc error: code = NotFound desc = could not find container \"b459a0d7f2f85849af002b5f60d0c15f55215ac6e41ccf5e875b8bc5562e76a9\": container with ID starting with b459a0d7f2f85849af002b5f60d0c15f55215ac6e41ccf5e875b8bc5562e76a9 not found: ID does not exist" Sep 11 16:12:27.330357 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:27.330330 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-455ccc2b-openshift-default-65c5c8ff6-cj554"] Sep 11 16:12:28.443502 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:28.443471 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-9c5b09-jjz6d" Sep 11 16:12:28.540551 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:28.540520 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2153a5bc-fbcf-4461-bc04-ef0125c78e46-maas-api-service-ca\") pod \"2153a5bc-fbcf-4461-bc04-ef0125c78e46\" (UID: \"2153a5bc-fbcf-4461-bc04-ef0125c78e46\") " Sep 11 16:12:28.540901 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:28.540874 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2153a5bc-fbcf-4461-bc04-ef0125c78e46-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2153a5bc-fbcf-4461-bc04-ef0125c78e46" (UID: "2153a5bc-fbcf-4461-bc04-ef0125c78e46"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 16:12:28.642236 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:28.642162 2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2153a5bc-fbcf-4461-bc04-ef0125c78e46-maas-api-service-ca\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:12:29.240191 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:29.240155 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f72e085d-36b9-4aeb-a3f1-59209208092a" path="/var/lib/kubelet/pods/f72e085d-36b9-4aeb-a3f1-59209208092a/volumes" Sep 11 16:12:29.321699 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:29.321670 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-9c5b09-jjz6d" Sep 11 16:12:29.321870 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:29.321709 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-9c5b09-jjz6d" event={"ID":"2153a5bc-fbcf-4461-bc04-ef0125c78e46","Type":"ContainerDied","Data":"f9ca376d3a879b382e9201c4dcca0bea6fddef599c2817d98b80431787bd768f"} Sep 11 16:12:29.321870 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:29.321758 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f9ca376d3a879b382e9201c4dcca0bea6fddef599c2817d98b80431787bd768f" Sep 11 16:12:35.826576 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:35.826538 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-9c5b09-jjz6d"] Sep 11 16:12:35.829676 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:35.829648 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-9c5b09-jjz6d"] Sep 11 16:12:37.240161 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:37.240124 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2153a5bc-fbcf-4461-bc04-ef0125c78e46" path="/var/lib/kubelet/pods/2153a5bc-fbcf-4461-bc04-ef0125c78e46/volumes" Sep 11 16:12:37.240533 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:37.240503 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dx4xn"] Sep 11 16:12:42.917826 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:42.917778 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-9c5b09-openshift-default-66c68b44fd-7dtg6"] Sep 11 16:12:42.918337 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:42.918111 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-9c5b09-openshift-default-66c68b44fd-7dtg6" podUID="3c028f61-e84c-4c34-9a94-7c7e0c9a2212" containerName="istio-proxy" containerID="cri-o://abde8929462db8fc40c28736b0901ceb355eaf2b60f51c106c8b916fb2a63b75" gracePeriod=30 Sep 11 16:12:43.822726 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:43.822689 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-dacdbe-openshift-default-885dc786d-cjdg8"] Sep 11 16:12:43.823209 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:43.823191 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f72e085d-36b9-4aeb-a3f1-59209208092a" containerName="istio-proxy" Sep 11 16:12:43.823209 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:43.823210 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="f72e085d-36b9-4aeb-a3f1-59209208092a" containerName="istio-proxy" Sep 11 16:12:43.823374 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:43.823221 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2153a5bc-fbcf-4461-bc04-ef0125c78e46" containerName="revoke-keys" Sep 11 16:12:43.823374 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:43.823229 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="2153a5bc-fbcf-4461-bc04-ef0125c78e46" containerName="revoke-keys" Sep 11 16:12:43.823480 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:43.823377 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="f72e085d-36b9-4aeb-a3f1-59209208092a" containerName="istio-proxy" Sep 11 16:12:43.823480 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:43.823390 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="2153a5bc-fbcf-4461-bc04-ef0125c78e46" containerName="revoke-keys" Sep 11 16:12:43.823480 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:43.823400 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="f72e085d-36b9-4aeb-a3f1-59209208092a" containerName="istio-proxy" Sep 11 16:12:43.826955 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:43.826935 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-dacdbe-openshift-default-885dc786d-cjdg8" Sep 11 16:12:43.831100 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:43.831081 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-dacdbe-openshift-default-dockercfg-xj6wm\"" Sep 11 16:12:43.849913 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:43.849886 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-dacdbe-openshift-default-885dc786d-cjdg8"] Sep 11 16:12:43.891881 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:43.891842 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7ee410e-6e10-49e4-af3a-c826b06ec137-istio-podinfo\") pod \"e2e-explicit-dacdbe-openshift-default-885dc786d-cjdg8\" (UID: \"a7ee410e-6e10-49e4-af3a-c826b06ec137\") " pod="openshift-ingress/e2e-explicit-dacdbe-openshift-default-885dc786d-cjdg8" Sep 11 16:12:43.892225 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:43.892197 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7ee410e-6e10-49e4-af3a-c826b06ec137-istiod-ca-cert\") pod \"e2e-explicit-dacdbe-openshift-default-885dc786d-cjdg8\" (UID: \"a7ee410e-6e10-49e4-af3a-c826b06ec137\") " pod="openshift-ingress/e2e-explicit-dacdbe-openshift-default-885dc786d-cjdg8" Sep 11 16:12:43.892388 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:43.892369 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7ee410e-6e10-49e4-af3a-c826b06ec137-istio-envoy\") pod \"e2e-explicit-dacdbe-openshift-default-885dc786d-cjdg8\" (UID: \"a7ee410e-6e10-49e4-af3a-c826b06ec137\") " pod="openshift-ingress/e2e-explicit-dacdbe-openshift-default-885dc786d-cjdg8" Sep 11 16:12:43.892514 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:43.892499 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7ee410e-6e10-49e4-af3a-c826b06ec137-istio-data\") pod \"e2e-explicit-dacdbe-openshift-default-885dc786d-cjdg8\" (UID: \"a7ee410e-6e10-49e4-af3a-c826b06ec137\") " pod="openshift-ingress/e2e-explicit-dacdbe-openshift-default-885dc786d-cjdg8" Sep 11 16:12:43.892633 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:43.892615 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7ee410e-6e10-49e4-af3a-c826b06ec137-credential-socket\") pod \"e2e-explicit-dacdbe-openshift-default-885dc786d-cjdg8\" (UID: \"a7ee410e-6e10-49e4-af3a-c826b06ec137\") " pod="openshift-ingress/e2e-explicit-dacdbe-openshift-default-885dc786d-cjdg8" Sep 11 16:12:43.892733 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:43.892712 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7ee410e-6e10-49e4-af3a-c826b06ec137-workload-certs\") pod \"e2e-explicit-dacdbe-openshift-default-885dc786d-cjdg8\" (UID: \"a7ee410e-6e10-49e4-af3a-c826b06ec137\") " pod="openshift-ingress/e2e-explicit-dacdbe-openshift-default-885dc786d-cjdg8" Sep 11 16:12:43.892793 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:43.892752 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8dgm\" (UniqueName: \"kubernetes.io/projected/a7ee410e-6e10-49e4-af3a-c826b06ec137-kube-api-access-w8dgm\") pod \"e2e-explicit-dacdbe-openshift-default-885dc786d-cjdg8\" (UID: \"a7ee410e-6e10-49e4-af3a-c826b06ec137\") " pod="openshift-ingress/e2e-explicit-dacdbe-openshift-default-885dc786d-cjdg8" Sep 11 16:12:43.893086 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:43.893055 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7ee410e-6e10-49e4-af3a-c826b06ec137-workload-socket\") pod \"e2e-explicit-dacdbe-openshift-default-885dc786d-cjdg8\" (UID: \"a7ee410e-6e10-49e4-af3a-c826b06ec137\") " pod="openshift-ingress/e2e-explicit-dacdbe-openshift-default-885dc786d-cjdg8" Sep 11 16:12:43.893331 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:43.893305 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7ee410e-6e10-49e4-af3a-c826b06ec137-istio-token\") pod \"e2e-explicit-dacdbe-openshift-default-885dc786d-cjdg8\" (UID: \"a7ee410e-6e10-49e4-af3a-c826b06ec137\") " pod="openshift-ingress/e2e-explicit-dacdbe-openshift-default-885dc786d-cjdg8" Sep 11 16:12:43.912882 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:43.912847 2566 patch_prober.go:28] interesting pod/e2e-autores-9c5b09-openshift-default-66c68b44fd-7dtg6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 11 16:12:43.913240 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:43.913193 2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-9c5b09-openshift-default-66c68b44fd-7dtg6" podUID="3c028f61-e84c-4c34-9a94-7c7e0c9a2212" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 11 16:12:43.994944 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:43.994907 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7ee410e-6e10-49e4-af3a-c826b06ec137-istiod-ca-cert\") pod \"e2e-explicit-dacdbe-openshift-default-885dc786d-cjdg8\" (UID: \"a7ee410e-6e10-49e4-af3a-c826b06ec137\") " pod="openshift-ingress/e2e-explicit-dacdbe-openshift-default-885dc786d-cjdg8" Sep 11 16:12:43.994944 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:43.994951 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7ee410e-6e10-49e4-af3a-c826b06ec137-istio-envoy\") pod \"e2e-explicit-dacdbe-openshift-default-885dc786d-cjdg8\" (UID: \"a7ee410e-6e10-49e4-af3a-c826b06ec137\") " pod="openshift-ingress/e2e-explicit-dacdbe-openshift-default-885dc786d-cjdg8" Sep 11 16:12:43.995450 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:43.995001 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7ee410e-6e10-49e4-af3a-c826b06ec137-istio-data\") pod \"e2e-explicit-dacdbe-openshift-default-885dc786d-cjdg8\" (UID: \"a7ee410e-6e10-49e4-af3a-c826b06ec137\") " pod="openshift-ingress/e2e-explicit-dacdbe-openshift-default-885dc786d-cjdg8" Sep 11 16:12:43.995450 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:43.995026 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7ee410e-6e10-49e4-af3a-c826b06ec137-credential-socket\") pod \"e2e-explicit-dacdbe-openshift-default-885dc786d-cjdg8\" (UID: \"a7ee410e-6e10-49e4-af3a-c826b06ec137\") " pod="openshift-ingress/e2e-explicit-dacdbe-openshift-default-885dc786d-cjdg8" Sep 11 16:12:43.995450 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:43.995079 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7ee410e-6e10-49e4-af3a-c826b06ec137-workload-certs\") pod \"e2e-explicit-dacdbe-openshift-default-885dc786d-cjdg8\" (UID: \"a7ee410e-6e10-49e4-af3a-c826b06ec137\") " pod="openshift-ingress/e2e-explicit-dacdbe-openshift-default-885dc786d-cjdg8" Sep 11 16:12:43.995450 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:43.995112 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8dgm\" (UniqueName: \"kubernetes.io/projected/a7ee410e-6e10-49e4-af3a-c826b06ec137-kube-api-access-w8dgm\") pod \"e2e-explicit-dacdbe-openshift-default-885dc786d-cjdg8\" (UID: \"a7ee410e-6e10-49e4-af3a-c826b06ec137\") " pod="openshift-ingress/e2e-explicit-dacdbe-openshift-default-885dc786d-cjdg8" Sep 11 16:12:43.995450 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:43.995141 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7ee410e-6e10-49e4-af3a-c826b06ec137-workload-socket\") pod \"e2e-explicit-dacdbe-openshift-default-885dc786d-cjdg8\" (UID: \"a7ee410e-6e10-49e4-af3a-c826b06ec137\") " pod="openshift-ingress/e2e-explicit-dacdbe-openshift-default-885dc786d-cjdg8" Sep 11 16:12:43.995450 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:43.995181 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7ee410e-6e10-49e4-af3a-c826b06ec137-istio-token\") pod \"e2e-explicit-dacdbe-openshift-default-885dc786d-cjdg8\" (UID: \"a7ee410e-6e10-49e4-af3a-c826b06ec137\") " pod="openshift-ingress/e2e-explicit-dacdbe-openshift-default-885dc786d-cjdg8" Sep 11 16:12:43.995450 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:43.995275 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7ee410e-6e10-49e4-af3a-c826b06ec137-istio-podinfo\") pod \"e2e-explicit-dacdbe-openshift-default-885dc786d-cjdg8\" (UID: \"a7ee410e-6e10-49e4-af3a-c826b06ec137\") " pod="openshift-ingress/e2e-explicit-dacdbe-openshift-default-885dc786d-cjdg8" Sep 11 16:12:43.995804 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:43.995486 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7ee410e-6e10-49e4-af3a-c826b06ec137-credential-socket\") pod \"e2e-explicit-dacdbe-openshift-default-885dc786d-cjdg8\" (UID: \"a7ee410e-6e10-49e4-af3a-c826b06ec137\") " pod="openshift-ingress/e2e-explicit-dacdbe-openshift-default-885dc786d-cjdg8" Sep 11 16:12:43.995804 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:43.995655 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7ee410e-6e10-49e4-af3a-c826b06ec137-workload-certs\") pod \"e2e-explicit-dacdbe-openshift-default-885dc786d-cjdg8\" (UID: \"a7ee410e-6e10-49e4-af3a-c826b06ec137\") " pod="openshift-ingress/e2e-explicit-dacdbe-openshift-default-885dc786d-cjdg8" Sep 11 16:12:43.995804 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:43.995685 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7ee410e-6e10-49e4-af3a-c826b06ec137-istio-data\") pod \"e2e-explicit-dacdbe-openshift-default-885dc786d-cjdg8\" (UID: \"a7ee410e-6e10-49e4-af3a-c826b06ec137\") " pod="openshift-ingress/e2e-explicit-dacdbe-openshift-default-885dc786d-cjdg8" Sep 11 16:12:43.995908 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:43.995837 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7ee410e-6e10-49e4-af3a-c826b06ec137-istiod-ca-cert\") pod \"e2e-explicit-dacdbe-openshift-default-885dc786d-cjdg8\" (UID: \"a7ee410e-6e10-49e4-af3a-c826b06ec137\") " pod="openshift-ingress/e2e-explicit-dacdbe-openshift-default-885dc786d-cjdg8" Sep 11 16:12:43.995908 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:43.995899 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7ee410e-6e10-49e4-af3a-c826b06ec137-workload-socket\") pod \"e2e-explicit-dacdbe-openshift-default-885dc786d-cjdg8\" (UID: \"a7ee410e-6e10-49e4-af3a-c826b06ec137\") " pod="openshift-ingress/e2e-explicit-dacdbe-openshift-default-885dc786d-cjdg8" Sep 11 16:12:43.997348 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:43.997325 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7ee410e-6e10-49e4-af3a-c826b06ec137-istio-envoy\") pod \"e2e-explicit-dacdbe-openshift-default-885dc786d-cjdg8\" (UID: \"a7ee410e-6e10-49e4-af3a-c826b06ec137\") " pod="openshift-ingress/e2e-explicit-dacdbe-openshift-default-885dc786d-cjdg8" Sep 11 16:12:43.997616 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:43.997596 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7ee410e-6e10-49e4-af3a-c826b06ec137-istio-podinfo\") pod \"e2e-explicit-dacdbe-openshift-default-885dc786d-cjdg8\" (UID: \"a7ee410e-6e10-49e4-af3a-c826b06ec137\") " pod="openshift-ingress/e2e-explicit-dacdbe-openshift-default-885dc786d-cjdg8" Sep 11 16:12:44.007821 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:44.007794 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8dgm\" (UniqueName: \"kubernetes.io/projected/a7ee410e-6e10-49e4-af3a-c826b06ec137-kube-api-access-w8dgm\") pod \"e2e-explicit-dacdbe-openshift-default-885dc786d-cjdg8\" (UID: \"a7ee410e-6e10-49e4-af3a-c826b06ec137\") " pod="openshift-ingress/e2e-explicit-dacdbe-openshift-default-885dc786d-cjdg8" Sep 11 16:12:44.007998 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:44.007963 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7ee410e-6e10-49e4-af3a-c826b06ec137-istio-token\") pod \"e2e-explicit-dacdbe-openshift-default-885dc786d-cjdg8\" (UID: \"a7ee410e-6e10-49e4-af3a-c826b06ec137\") " pod="openshift-ingress/e2e-explicit-dacdbe-openshift-default-885dc786d-cjdg8" Sep 11 16:12:44.138727 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:44.138632 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-dacdbe-openshift-default-885dc786d-cjdg8" Sep 11 16:12:44.420794 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:44.420765 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-dacdbe-openshift-default-885dc786d-cjdg8"] Sep 11 16:12:44.422603 ip-10-0-136-96 kubenswrapper[2566]: W0911 16:12:44.422577 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7ee410e_6e10_49e4_af3a_c826b06ec137.slice/crio-484b10d01418bb68bb152dfbc0b58e02d6d3bfb2f032bcb13c5221cc36d66618 WatchSource:0}: Error finding container 484b10d01418bb68bb152dfbc0b58e02d6d3bfb2f032bcb13c5221cc36d66618: Status 404 returned error can't find the container with id 484b10d01418bb68bb152dfbc0b58e02d6d3bfb2f032bcb13c5221cc36d66618 Sep 11 16:12:44.424600 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:44.424571 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 16:12:44.424682 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:44.424631 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 16:12:44.424682 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:44.424671 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 16:12:45.429625 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:45.429592 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-dacdbe-openshift-default-885dc786d-cjdg8" event={"ID":"a7ee410e-6e10-49e4-af3a-c826b06ec137","Type":"ContainerStarted","Data":"79fbb574951f008dd2ac020baeba850a78e99d758ee86d7f5e9ce8fced6f52a8"} Sep 11 16:12:45.430020 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:45.429641 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-dacdbe-openshift-default-885dc786d-cjdg8" event={"ID":"a7ee410e-6e10-49e4-af3a-c826b06ec137","Type":"ContainerStarted","Data":"484b10d01418bb68bb152dfbc0b58e02d6d3bfb2f032bcb13c5221cc36d66618"} Sep 11 16:12:45.455256 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:45.455200 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-dacdbe-openshift-default-885dc786d-cjdg8" podStartSLOduration=2.455181662 podStartE2EDuration="2.455181662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:12:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:12:45.45354244 +0000 UTC m=+1072.668376345" watchObservedRunningTime="2026-09-11 16:12:45.455181662 +0000 UTC m=+1072.670015626" Sep 11 16:12:46.139808 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:46.139768 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-dacdbe-openshift-default-885dc786d-cjdg8" Sep 11 16:12:46.144564 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:46.144537 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-dacdbe-openshift-default-885dc786d-cjdg8" Sep 11 16:12:46.436990 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:46.436878 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-dacdbe-openshift-default-885dc786d-cjdg8" Sep 11 16:12:46.444250 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:46.444215 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-dacdbe-openshift-default-885dc786d-cjdg8" Sep 11 16:12:48.180920 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:48.180853 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-9c5b09-openshift-default-66c68b44fd-7dtg6" Sep 11 16:12:48.238941 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:48.238900 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c028f61-e84c-4c34-9a94-7c7e0c9a2212-workload-socket\") pod \"3c028f61-e84c-4c34-9a94-7c7e0c9a2212\" (UID: \"3c028f61-e84c-4c34-9a94-7c7e0c9a2212\") " Sep 11 16:12:48.239134 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:48.238999 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c028f61-e84c-4c34-9a94-7c7e0c9a2212-istio-data\") pod \"3c028f61-e84c-4c34-9a94-7c7e0c9a2212\" (UID: \"3c028f61-e84c-4c34-9a94-7c7e0c9a2212\") " Sep 11 16:12:48.239134 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:48.239071 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xz76r\" (UniqueName: \"kubernetes.io/projected/3c028f61-e84c-4c34-9a94-7c7e0c9a2212-kube-api-access-xz76r\") pod \"3c028f61-e84c-4c34-9a94-7c7e0c9a2212\" (UID: \"3c028f61-e84c-4c34-9a94-7c7e0c9a2212\") " Sep 11 16:12:48.239134 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:48.239120 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c028f61-e84c-4c34-9a94-7c7e0c9a2212-credential-socket\") pod \"3c028f61-e84c-4c34-9a94-7c7e0c9a2212\" (UID: \"3c028f61-e84c-4c34-9a94-7c7e0c9a2212\") " Sep 11 16:12:48.239336 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:48.239154 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c028f61-e84c-4c34-9a94-7c7e0c9a2212-istio-envoy\") pod \"3c028f61-e84c-4c34-9a94-7c7e0c9a2212\" (UID: \"3c028f61-e84c-4c34-9a94-7c7e0c9a2212\") " Sep 11 16:12:48.239336 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:48.239198 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c028f61-e84c-4c34-9a94-7c7e0c9a2212-istiod-ca-cert\") pod \"3c028f61-e84c-4c34-9a94-7c7e0c9a2212\" (UID: \"3c028f61-e84c-4c34-9a94-7c7e0c9a2212\") " Sep 11 16:12:48.239336 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:48.239244 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c028f61-e84c-4c34-9a94-7c7e0c9a2212-istio-podinfo\") pod \"3c028f61-e84c-4c34-9a94-7c7e0c9a2212\" (UID: \"3c028f61-e84c-4c34-9a94-7c7e0c9a2212\") " Sep 11 16:12:48.239336 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:48.239273 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c028f61-e84c-4c34-9a94-7c7e0c9a2212-istio-token\") pod \"3c028f61-e84c-4c34-9a94-7c7e0c9a2212\" (UID: \"3c028f61-e84c-4c34-9a94-7c7e0c9a2212\") " Sep 11 16:12:48.239336 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:48.239258 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c028f61-e84c-4c34-9a94-7c7e0c9a2212-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3c028f61-e84c-4c34-9a94-7c7e0c9a2212" (UID: "3c028f61-e84c-4c34-9a94-7c7e0c9a2212"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 16:12:48.239336 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:48.239314 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c028f61-e84c-4c34-9a94-7c7e0c9a2212-workload-certs\") pod \"3c028f61-e84c-4c34-9a94-7c7e0c9a2212\" (UID: \"3c028f61-e84c-4c34-9a94-7c7e0c9a2212\") " Sep 11 16:12:48.239640 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:48.239490 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c028f61-e84c-4c34-9a94-7c7e0c9a2212-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3c028f61-e84c-4c34-9a94-7c7e0c9a2212" (UID: "3c028f61-e84c-4c34-9a94-7c7e0c9a2212"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 16:12:48.239715 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:48.239695 2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c028f61-e84c-4c34-9a94-7c7e0c9a2212-credential-socket\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:12:48.239773 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:48.239722 2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c028f61-e84c-4c34-9a94-7c7e0c9a2212-workload-socket\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:12:48.239999 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:48.239948 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c028f61-e84c-4c34-9a94-7c7e0c9a2212-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3c028f61-e84c-4c34-9a94-7c7e0c9a2212" (UID: "3c028f61-e84c-4c34-9a94-7c7e0c9a2212"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 16:12:48.240101 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:48.240043 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c028f61-e84c-4c34-9a94-7c7e0c9a2212-istio-data" (OuterVolumeSpecName: "istio-data") pod "3c028f61-e84c-4c34-9a94-7c7e0c9a2212" (UID: "3c028f61-e84c-4c34-9a94-7c7e0c9a2212"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 16:12:48.240189 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:48.240162 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c028f61-e84c-4c34-9a94-7c7e0c9a2212-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3c028f61-e84c-4c34-9a94-7c7e0c9a2212" (UID: "3c028f61-e84c-4c34-9a94-7c7e0c9a2212"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 16:12:48.241754 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:48.241725 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c028f61-e84c-4c34-9a94-7c7e0c9a2212-istio-token" (OuterVolumeSpecName: "istio-token") pod "3c028f61-e84c-4c34-9a94-7c7e0c9a2212" (UID: "3c028f61-e84c-4c34-9a94-7c7e0c9a2212"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 16:12:48.241893 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:48.241879 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c028f61-e84c-4c34-9a94-7c7e0c9a2212-kube-api-access-xz76r" (OuterVolumeSpecName: "kube-api-access-xz76r") pod "3c028f61-e84c-4c34-9a94-7c7e0c9a2212" (UID: "3c028f61-e84c-4c34-9a94-7c7e0c9a2212"). InnerVolumeSpecName "kube-api-access-xz76r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 16:12:48.242093 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:48.241944 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c028f61-e84c-4c34-9a94-7c7e0c9a2212-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3c028f61-e84c-4c34-9a94-7c7e0c9a2212" (UID: "3c028f61-e84c-4c34-9a94-7c7e0c9a2212"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 16:12:48.242340 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:48.242317 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3c028f61-e84c-4c34-9a94-7c7e0c9a2212-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3c028f61-e84c-4c34-9a94-7c7e0c9a2212" (UID: "3c028f61-e84c-4c34-9a94-7c7e0c9a2212"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 16:12:48.340706 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:48.340669 2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c028f61-e84c-4c34-9a94-7c7e0c9a2212-istio-podinfo\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:12:48.340706 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:48.340699 2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c028f61-e84c-4c34-9a94-7c7e0c9a2212-istio-token\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:12:48.340706 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:48.340708 2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c028f61-e84c-4c34-9a94-7c7e0c9a2212-workload-certs\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:12:48.340706 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:48.340716 2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c028f61-e84c-4c34-9a94-7c7e0c9a2212-istio-data\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:12:48.340706 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:48.340727 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xz76r\" (UniqueName: \"kubernetes.io/projected/3c028f61-e84c-4c34-9a94-7c7e0c9a2212-kube-api-access-xz76r\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:12:48.341086 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:48.340741 2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c028f61-e84c-4c34-9a94-7c7e0c9a2212-istio-envoy\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:12:48.341086 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:48.340754 2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c028f61-e84c-4c34-9a94-7c7e0c9a2212-istiod-ca-cert\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:12:48.453052 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:48.452945 2566 generic.go:356] "Generic (PLEG): container finished" podID="3c028f61-e84c-4c34-9a94-7c7e0c9a2212" containerID="abde8929462db8fc40c28736b0901ceb355eaf2b60f51c106c8b916fb2a63b75" exitCode=0 Sep 11 16:12:48.453052 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:48.453005 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-9c5b09-openshift-default-66c68b44fd-7dtg6" Sep 11 16:12:48.453052 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:48.453036 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-9c5b09-openshift-default-66c68b44fd-7dtg6" event={"ID":"3c028f61-e84c-4c34-9a94-7c7e0c9a2212","Type":"ContainerDied","Data":"abde8929462db8fc40c28736b0901ceb355eaf2b60f51c106c8b916fb2a63b75"} Sep 11 16:12:48.453322 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:48.453090 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-9c5b09-openshift-default-66c68b44fd-7dtg6" event={"ID":"3c028f61-e84c-4c34-9a94-7c7e0c9a2212","Type":"ContainerDied","Data":"1a28367d0e1cb4cd3ef14af4108af09bfc39e55000fc7ab1f3aa1e0f48699cf8"} Sep 11 16:12:48.453322 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:48.453113 2566 scope.go:117] "RemoveContainer" containerID="abde8929462db8fc40c28736b0901ceb355eaf2b60f51c106c8b916fb2a63b75" Sep 11 16:12:48.463233 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:48.463210 2566 scope.go:117] "RemoveContainer" containerID="6a1a9469ace076fbd71923f1224d7f272ac7b9e76b82d85c0bddecf0a72d9ce1" Sep 11 16:12:48.472205 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:48.472188 2566 scope.go:117] "RemoveContainer" containerID="abde8929462db8fc40c28736b0901ceb355eaf2b60f51c106c8b916fb2a63b75" Sep 11 16:12:48.472462 ip-10-0-136-96 kubenswrapper[2566]: E0911 16:12:48.472441 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abde8929462db8fc40c28736b0901ceb355eaf2b60f51c106c8b916fb2a63b75\": container with ID starting with abde8929462db8fc40c28736b0901ceb355eaf2b60f51c106c8b916fb2a63b75 not found: ID does not exist" containerID="abde8929462db8fc40c28736b0901ceb355eaf2b60f51c106c8b916fb2a63b75" Sep 11 16:12:48.472521 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:48.472471 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abde8929462db8fc40c28736b0901ceb355eaf2b60f51c106c8b916fb2a63b75"} err="failed to get container status \"abde8929462db8fc40c28736b0901ceb355eaf2b60f51c106c8b916fb2a63b75\": rpc error: code = NotFound desc = could not find container \"abde8929462db8fc40c28736b0901ceb355eaf2b60f51c106c8b916fb2a63b75\": container with ID starting with abde8929462db8fc40c28736b0901ceb355eaf2b60f51c106c8b916fb2a63b75 not found: ID does not exist" Sep 11 16:12:48.472521 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:48.472488 2566 scope.go:117] "RemoveContainer" containerID="6a1a9469ace076fbd71923f1224d7f272ac7b9e76b82d85c0bddecf0a72d9ce1" Sep 11 16:12:48.472776 ip-10-0-136-96 kubenswrapper[2566]: E0911 16:12:48.472757 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a1a9469ace076fbd71923f1224d7f272ac7b9e76b82d85c0bddecf0a72d9ce1\": container with ID starting with 6a1a9469ace076fbd71923f1224d7f272ac7b9e76b82d85c0bddecf0a72d9ce1 not found: ID does not exist" containerID="6a1a9469ace076fbd71923f1224d7f272ac7b9e76b82d85c0bddecf0a72d9ce1" Sep 11 16:12:48.472823 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:48.472783 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a1a9469ace076fbd71923f1224d7f272ac7b9e76b82d85c0bddecf0a72d9ce1"} err="failed to get container status \"6a1a9469ace076fbd71923f1224d7f272ac7b9e76b82d85c0bddecf0a72d9ce1\": rpc error: code = NotFound desc = could not find container \"6a1a9469ace076fbd71923f1224d7f272ac7b9e76b82d85c0bddecf0a72d9ce1\": container with ID starting with 6a1a9469ace076fbd71923f1224d7f272ac7b9e76b82d85c0bddecf0a72d9ce1 not found: ID does not exist" Sep 11 16:12:48.503646 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:48.503572 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-9c5b09-openshift-default-66c68b44fd-7dtg6"] Sep 11 16:12:48.506234 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:48.506211 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-9c5b09-openshift-default-66c68b44fd-7dtg6"] Sep 11 16:12:49.239963 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:49.239916 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c028f61-e84c-4c34-9a94-7c7e0c9a2212" path="/var/lib/kubelet/pods/3c028f61-e84c-4c34-9a94-7c7e0c9a2212/volumes" Sep 11 16:12:49.453648 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:12:49.453612 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dx4xn"] Sep 11 16:13:01.926919 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:13:01.926882 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dx4xn"] Sep 11 16:13:05.565662 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:13:05.565575 2566 generic.go:356] "Generic (PLEG): container finished" podID="a7ee410e-6e10-49e4-af3a-c826b06ec137" containerID="79fbb574951f008dd2ac020baeba850a78e99d758ee86d7f5e9ce8fced6f52a8" exitCode=0 Sep 11 16:13:05.566060 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:13:05.565652 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-dacdbe-openshift-default-885dc786d-cjdg8" event={"ID":"a7ee410e-6e10-49e4-af3a-c826b06ec137","Type":"ContainerDied","Data":"79fbb574951f008dd2ac020baeba850a78e99d758ee86d7f5e9ce8fced6f52a8"} Sep 11 16:13:05.566128 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:13:05.566113 2566 scope.go:117] "RemoveContainer" containerID="79fbb574951f008dd2ac020baeba850a78e99d758ee86d7f5e9ce8fced6f52a8" Sep 11 16:13:05.567393 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:13:05.567329 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 16:13:05.567531 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:13:05.567418 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 16:13:05.567531 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:13:05.567462 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 16:13:06.574052 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:13:06.574007 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-dacdbe-openshift-default-885dc786d-cjdg8" event={"ID":"a7ee410e-6e10-49e4-af3a-c826b06ec137","Type":"ContainerStarted","Data":"d41fa0f75ced236c2e3d9612dfb2ecb17ee34a0553543127a05b5965610aaf22"} Sep 11 16:13:07.139498 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:13:07.139358 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-dacdbe-openshift-default-885dc786d-cjdg8" Sep 11 16:13:07.139721 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:13:07.139680 2566 patch_prober.go:28] interesting pod/e2e-explicit-dacdbe-openshift-default-885dc786d-cjdg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Sep 11 16:13:07.139797 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:13:07.139750 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-dacdbe-openshift-default-885dc786d-cjdg8" podUID="a7ee410e-6e10-49e4-af3a-c826b06ec137" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Sep 11 16:13:08.139990 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:13:08.139931 2566 patch_prober.go:28] interesting pod/e2e-explicit-dacdbe-openshift-default-885dc786d-cjdg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Sep 11 16:13:08.140466 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:13:08.140031 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-dacdbe-openshift-default-885dc786d-cjdg8" podUID="a7ee410e-6e10-49e4-af3a-c826b06ec137" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Sep 11 16:13:09.139295 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:13:09.139252 2566 patch_prober.go:28] interesting pod/e2e-explicit-dacdbe-openshift-default-885dc786d-cjdg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Sep 11 16:13:09.139501 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:13:09.139335 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-dacdbe-openshift-default-885dc786d-cjdg8" podUID="a7ee410e-6e10-49e4-af3a-c826b06ec137" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Sep 11 16:13:10.140110 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:13:10.140070 2566 patch_prober.go:28] interesting pod/e2e-explicit-dacdbe-openshift-default-885dc786d-cjdg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Sep 11 16:13:10.140706 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:13:10.140670 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-dacdbe-openshift-default-885dc786d-cjdg8" podUID="a7ee410e-6e10-49e4-af3a-c826b06ec137" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Sep 11 16:13:11.140107 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:13:11.140065 2566 patch_prober.go:28] interesting pod/e2e-explicit-dacdbe-openshift-default-885dc786d-cjdg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Sep 11 16:13:11.140385 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:13:11.140354 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-dacdbe-openshift-default-885dc786d-cjdg8" podUID="a7ee410e-6e10-49e4-af3a-c826b06ec137" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Sep 11 16:13:12.143297 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:13:12.143268 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-dacdbe-openshift-default-885dc786d-cjdg8" Sep 11 16:13:12.143690 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:13:12.143658 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-dacdbe-openshift-default-885dc786d-cjdg8" Sep 11 16:13:12.144489 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:13:12.144468 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-dacdbe-openshift-default-885dc786d-cjdg8" Sep 11 16:13:19.556402 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:13:19.556366 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dx4xn"] Sep 11 16:13:30.216562 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:13:30.216526 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dx4xn"] Sep 11 16:13:40.927135 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:13:40.927091 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dx4xn"] Sep 11 16:13:45.056791 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:13:45.056750 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-dacdbe-crh6r"] Sep 11 16:13:45.057234 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:13:45.057215 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f72e085d-36b9-4aeb-a3f1-59209208092a" containerName="istio-proxy" Sep 11 16:13:45.057234 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:13:45.057227 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="f72e085d-36b9-4aeb-a3f1-59209208092a" containerName="istio-proxy" Sep 11 16:13:45.057315 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:13:45.057242 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c028f61-e84c-4c34-9a94-7c7e0c9a2212" containerName="istio-proxy" Sep 11 16:13:45.057315 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:13:45.057248 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c028f61-e84c-4c34-9a94-7c7e0c9a2212" containerName="istio-proxy" Sep 11 16:13:45.057315 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:13:45.057256 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c028f61-e84c-4c34-9a94-7c7e0c9a2212" containerName="istio-proxy" Sep 11 16:13:45.057315 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:13:45.057262 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c028f61-e84c-4c34-9a94-7c7e0c9a2212" containerName="istio-proxy" Sep 11 16:13:45.057442 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:13:45.057322 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c028f61-e84c-4c34-9a94-7c7e0c9a2212" containerName="istio-proxy" Sep 11 16:13:45.059484 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:13:45.059462 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-dacdbe-crh6r" Sep 11 16:13:45.061879 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:13:45.061860 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qbzdr\"" Sep 11 16:13:45.062298 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:13:45.062280 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 16:13:45.082479 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:13:45.082437 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-dacdbe-crh6r"] Sep 11 16:13:45.153208 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:13:45.153176 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7418b53a-8c59-4dbd-aa76-8c9479fa9451-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-dacdbe-crh6r\" (UID: \"7418b53a-8c59-4dbd-aa76-8c9479fa9451\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-dacdbe-crh6r" Sep 11 16:13:45.254626 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:13:45.254592 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7418b53a-8c59-4dbd-aa76-8c9479fa9451-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-dacdbe-crh6r\" (UID: \"7418b53a-8c59-4dbd-aa76-8c9479fa9451\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-dacdbe-crh6r" Sep 11 16:13:45.255255 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:13:45.255231 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7418b53a-8c59-4dbd-aa76-8c9479fa9451-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-dacdbe-crh6r\" (UID: \"7418b53a-8c59-4dbd-aa76-8c9479fa9451\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-dacdbe-crh6r" Sep 11 16:13:45.369828 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:13:45.369744 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-dacdbe-crh6r" Sep 11 16:13:45.666559 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:13:45.666529 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-dacdbe-crh6r"] Sep 11 16:13:45.669254 ip-10-0-136-96 kubenswrapper[2566]: W0911 16:13:45.669223 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7418b53a_8c59_4dbd_aa76_8c9479fa9451.slice/crio-937cc4f3553f9933d2511406afbb1a0e3d4ecfd9c67d511e2adc094a8214ebc6 WatchSource:0}: Error finding container 937cc4f3553f9933d2511406afbb1a0e3d4ecfd9c67d511e2adc094a8214ebc6: Status 404 returned error can't find the container with id 937cc4f3553f9933d2511406afbb1a0e3d4ecfd9c67d511e2adc094a8214ebc6 Sep 11 16:13:45.828656 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:13:45.828607 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-dacdbe-crh6r" event={"ID":"7418b53a-8c59-4dbd-aa76-8c9479fa9451","Type":"ContainerStarted","Data":"937cc4f3553f9933d2511406afbb1a0e3d4ecfd9c67d511e2adc094a8214ebc6"} Sep 11 16:13:46.835930 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:13:46.835892 2566 generic.go:356] "Generic (PLEG): container finished" podID="7418b53a-8c59-4dbd-aa76-8c9479fa9451" containerID="1484f8ab357b22383e38dca1786d2f8739188d9e78729cc93c6baafe7ac482c2" exitCode=0 Sep 11 16:13:46.836363 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:13:46.836002 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-dacdbe-crh6r" event={"ID":"7418b53a-8c59-4dbd-aa76-8c9479fa9451","Type":"ContainerDied","Data":"1484f8ab357b22383e38dca1786d2f8739188d9e78729cc93c6baafe7ac482c2"} Sep 11 16:13:47.975066 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:13:47.975037 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-dacdbe-crh6r" Sep 11 16:13:48.083669 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:13:48.083621 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7418b53a-8c59-4dbd-aa76-8c9479fa9451-maas-api-service-ca\") pod \"7418b53a-8c59-4dbd-aa76-8c9479fa9451\" (UID: \"7418b53a-8c59-4dbd-aa76-8c9479fa9451\") " Sep 11 16:13:48.083993 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:13:48.083951 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7418b53a-8c59-4dbd-aa76-8c9479fa9451-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7418b53a-8c59-4dbd-aa76-8c9479fa9451" (UID: "7418b53a-8c59-4dbd-aa76-8c9479fa9451"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 16:13:48.184706 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:13:48.184622 2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7418b53a-8c59-4dbd-aa76-8c9479fa9451-maas-api-service-ca\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:13:48.849416 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:13:48.849382 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-dacdbe-crh6r" Sep 11 16:13:48.849416 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:13:48.849408 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-dacdbe-crh6r" event={"ID":"7418b53a-8c59-4dbd-aa76-8c9479fa9451","Type":"ContainerDied","Data":"937cc4f3553f9933d2511406afbb1a0e3d4ecfd9c67d511e2adc094a8214ebc6"} Sep 11 16:13:48.849645 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:13:48.849489 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="937cc4f3553f9933d2511406afbb1a0e3d4ecfd9c67d511e2adc094a8214ebc6" Sep 11 16:13:56.094212 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:13:56.094174 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-dacdbe-crh6r"] Sep 11 16:13:56.100961 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:13:56.100927 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-dacdbe-crh6r"] Sep 11 16:13:57.239799 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:13:57.239758 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7418b53a-8c59-4dbd-aa76-8c9479fa9451" path="/var/lib/kubelet/pods/7418b53a-8c59-4dbd-aa76-8c9479fa9451/volumes" Sep 11 16:13:59.072438 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:13:59.072400 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dx4xn"] Sep 11 16:14:02.684157 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:02.684119 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-faa154ea-openshift-default-6db9786fc6-48297"] Sep 11 16:14:02.684710 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:02.684540 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7418b53a-8c59-4dbd-aa76-8c9479fa9451" containerName="revoke-keys" Sep 11 16:14:02.684710 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:02.684554 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="7418b53a-8c59-4dbd-aa76-8c9479fa9451" containerName="revoke-keys" Sep 11 16:14:02.684710 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:02.684680 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="7418b53a-8c59-4dbd-aa76-8c9479fa9451" containerName="revoke-keys" Sep 11 16:14:02.684710 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:02.684698 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c028f61-e84c-4c34-9a94-7c7e0c9a2212" containerName="istio-proxy" Sep 11 16:14:02.688073 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:02.688054 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-faa154ea-openshift-default-6db9786fc6-48297" Sep 11 16:14:02.692336 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:02.692307 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-faa154ea-openshift-default-dockercfg-g7wfz\"" Sep 11 16:14:02.711230 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:02.711198 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-faa154ea-openshift-default-6db9786fc6-48297"] Sep 11 16:14:02.832796 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:02.832752 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c8db87c-c55f-4c6a-91ba-e45267d81a93-istiod-ca-cert\") pod \"e2e-mt-faa154ea-openshift-default-6db9786fc6-48297\" (UID: \"0c8db87c-c55f-4c6a-91ba-e45267d81a93\") " pod="openshift-ingress/e2e-mt-faa154ea-openshift-default-6db9786fc6-48297" Sep 11 16:14:02.833014 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:02.832823 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c8db87c-c55f-4c6a-91ba-e45267d81a93-workload-socket\") pod \"e2e-mt-faa154ea-openshift-default-6db9786fc6-48297\" (UID: \"0c8db87c-c55f-4c6a-91ba-e45267d81a93\") " pod="openshift-ingress/e2e-mt-faa154ea-openshift-default-6db9786fc6-48297" Sep 11 16:14:02.833014 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:02.832853 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c8db87c-c55f-4c6a-91ba-e45267d81a93-istio-envoy\") pod \"e2e-mt-faa154ea-openshift-default-6db9786fc6-48297\" (UID: \"0c8db87c-c55f-4c6a-91ba-e45267d81a93\") " pod="openshift-ingress/e2e-mt-faa154ea-openshift-default-6db9786fc6-48297" Sep 11 16:14:02.833014 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:02.832876 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcz82\" (UniqueName: \"kubernetes.io/projected/0c8db87c-c55f-4c6a-91ba-e45267d81a93-kube-api-access-rcz82\") pod \"e2e-mt-faa154ea-openshift-default-6db9786fc6-48297\" (UID: \"0c8db87c-c55f-4c6a-91ba-e45267d81a93\") " pod="openshift-ingress/e2e-mt-faa154ea-openshift-default-6db9786fc6-48297" Sep 11 16:14:02.833014 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:02.832912 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c8db87c-c55f-4c6a-91ba-e45267d81a93-credential-socket\") pod \"e2e-mt-faa154ea-openshift-default-6db9786fc6-48297\" (UID: \"0c8db87c-c55f-4c6a-91ba-e45267d81a93\") " pod="openshift-ingress/e2e-mt-faa154ea-openshift-default-6db9786fc6-48297" Sep 11 16:14:02.833014 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:02.832938 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c8db87c-c55f-4c6a-91ba-e45267d81a93-istio-token\") pod \"e2e-mt-faa154ea-openshift-default-6db9786fc6-48297\" (UID: \"0c8db87c-c55f-4c6a-91ba-e45267d81a93\") " pod="openshift-ingress/e2e-mt-faa154ea-openshift-default-6db9786fc6-48297" Sep 11 16:14:02.833014 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:02.832965 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c8db87c-c55f-4c6a-91ba-e45267d81a93-istio-data\") pod \"e2e-mt-faa154ea-openshift-default-6db9786fc6-48297\" (UID: \"0c8db87c-c55f-4c6a-91ba-e45267d81a93\") " pod="openshift-ingress/e2e-mt-faa154ea-openshift-default-6db9786fc6-48297" Sep 11 16:14:02.833259 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:02.833097 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c8db87c-c55f-4c6a-91ba-e45267d81a93-istio-podinfo\") pod \"e2e-mt-faa154ea-openshift-default-6db9786fc6-48297\" (UID: \"0c8db87c-c55f-4c6a-91ba-e45267d81a93\") " pod="openshift-ingress/e2e-mt-faa154ea-openshift-default-6db9786fc6-48297" Sep 11 16:14:02.833259 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:02.833130 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c8db87c-c55f-4c6a-91ba-e45267d81a93-workload-certs\") pod \"e2e-mt-faa154ea-openshift-default-6db9786fc6-48297\" (UID: \"0c8db87c-c55f-4c6a-91ba-e45267d81a93\") " pod="openshift-ingress/e2e-mt-faa154ea-openshift-default-6db9786fc6-48297" Sep 11 16:14:02.933891 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:02.933860 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c8db87c-c55f-4c6a-91ba-e45267d81a93-istio-data\") pod \"e2e-mt-faa154ea-openshift-default-6db9786fc6-48297\" (UID: \"0c8db87c-c55f-4c6a-91ba-e45267d81a93\") " pod="openshift-ingress/e2e-mt-faa154ea-openshift-default-6db9786fc6-48297" Sep 11 16:14:02.934116 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:02.933922 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c8db87c-c55f-4c6a-91ba-e45267d81a93-istio-podinfo\") pod \"e2e-mt-faa154ea-openshift-default-6db9786fc6-48297\" (UID: \"0c8db87c-c55f-4c6a-91ba-e45267d81a93\") " pod="openshift-ingress/e2e-mt-faa154ea-openshift-default-6db9786fc6-48297" Sep 11 16:14:02.934116 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:02.933940 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c8db87c-c55f-4c6a-91ba-e45267d81a93-workload-certs\") pod \"e2e-mt-faa154ea-openshift-default-6db9786fc6-48297\" (UID: \"0c8db87c-c55f-4c6a-91ba-e45267d81a93\") " pod="openshift-ingress/e2e-mt-faa154ea-openshift-default-6db9786fc6-48297" Sep 11 16:14:02.934116 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:02.933994 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c8db87c-c55f-4c6a-91ba-e45267d81a93-istiod-ca-cert\") pod \"e2e-mt-faa154ea-openshift-default-6db9786fc6-48297\" (UID: \"0c8db87c-c55f-4c6a-91ba-e45267d81a93\") " pod="openshift-ingress/e2e-mt-faa154ea-openshift-default-6db9786fc6-48297" Sep 11 16:14:02.934116 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:02.934039 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c8db87c-c55f-4c6a-91ba-e45267d81a93-workload-socket\") pod \"e2e-mt-faa154ea-openshift-default-6db9786fc6-48297\" (UID: \"0c8db87c-c55f-4c6a-91ba-e45267d81a93\") " pod="openshift-ingress/e2e-mt-faa154ea-openshift-default-6db9786fc6-48297" Sep 11 16:14:02.934116 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:02.934055 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c8db87c-c55f-4c6a-91ba-e45267d81a93-istio-envoy\") pod \"e2e-mt-faa154ea-openshift-default-6db9786fc6-48297\" (UID: \"0c8db87c-c55f-4c6a-91ba-e45267d81a93\") " pod="openshift-ingress/e2e-mt-faa154ea-openshift-default-6db9786fc6-48297" Sep 11 16:14:02.934116 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:02.934071 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcz82\" (UniqueName: \"kubernetes.io/projected/0c8db87c-c55f-4c6a-91ba-e45267d81a93-kube-api-access-rcz82\") pod \"e2e-mt-faa154ea-openshift-default-6db9786fc6-48297\" (UID: \"0c8db87c-c55f-4c6a-91ba-e45267d81a93\") " pod="openshift-ingress/e2e-mt-faa154ea-openshift-default-6db9786fc6-48297" Sep 11 16:14:02.934116 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:02.934096 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c8db87c-c55f-4c6a-91ba-e45267d81a93-credential-socket\") pod \"e2e-mt-faa154ea-openshift-default-6db9786fc6-48297\" (UID: \"0c8db87c-c55f-4c6a-91ba-e45267d81a93\") " pod="openshift-ingress/e2e-mt-faa154ea-openshift-default-6db9786fc6-48297" Sep 11 16:14:02.934568 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:02.934294 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c8db87c-c55f-4c6a-91ba-e45267d81a93-istio-data\") pod \"e2e-mt-faa154ea-openshift-default-6db9786fc6-48297\" (UID: \"0c8db87c-c55f-4c6a-91ba-e45267d81a93\") " pod="openshift-ingress/e2e-mt-faa154ea-openshift-default-6db9786fc6-48297" Sep 11 16:14:02.934568 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:02.934354 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c8db87c-c55f-4c6a-91ba-e45267d81a93-workload-certs\") pod \"e2e-mt-faa154ea-openshift-default-6db9786fc6-48297\" (UID: \"0c8db87c-c55f-4c6a-91ba-e45267d81a93\") " pod="openshift-ingress/e2e-mt-faa154ea-openshift-default-6db9786fc6-48297" Sep 11 16:14:02.934568 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:02.934394 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c8db87c-c55f-4c6a-91ba-e45267d81a93-workload-socket\") pod \"e2e-mt-faa154ea-openshift-default-6db9786fc6-48297\" (UID: \"0c8db87c-c55f-4c6a-91ba-e45267d81a93\") " pod="openshift-ingress/e2e-mt-faa154ea-openshift-default-6db9786fc6-48297" Sep 11 16:14:02.934568 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:02.934413 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c8db87c-c55f-4c6a-91ba-e45267d81a93-istio-token\") pod \"e2e-mt-faa154ea-openshift-default-6db9786fc6-48297\" (UID: \"0c8db87c-c55f-4c6a-91ba-e45267d81a93\") " pod="openshift-ingress/e2e-mt-faa154ea-openshift-default-6db9786fc6-48297" Sep 11 16:14:02.934804 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:02.934679 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c8db87c-c55f-4c6a-91ba-e45267d81a93-credential-socket\") pod \"e2e-mt-faa154ea-openshift-default-6db9786fc6-48297\" (UID: \"0c8db87c-c55f-4c6a-91ba-e45267d81a93\") " pod="openshift-ingress/e2e-mt-faa154ea-openshift-default-6db9786fc6-48297" Sep 11 16:14:02.934804 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:02.934728 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c8db87c-c55f-4c6a-91ba-e45267d81a93-istiod-ca-cert\") pod \"e2e-mt-faa154ea-openshift-default-6db9786fc6-48297\" (UID: \"0c8db87c-c55f-4c6a-91ba-e45267d81a93\") " pod="openshift-ingress/e2e-mt-faa154ea-openshift-default-6db9786fc6-48297" Sep 11 16:14:02.936287 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:02.936256 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c8db87c-c55f-4c6a-91ba-e45267d81a93-istio-envoy\") pod \"e2e-mt-faa154ea-openshift-default-6db9786fc6-48297\" (UID: \"0c8db87c-c55f-4c6a-91ba-e45267d81a93\") " pod="openshift-ingress/e2e-mt-faa154ea-openshift-default-6db9786fc6-48297" Sep 11 16:14:02.936536 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:02.936518 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c8db87c-c55f-4c6a-91ba-e45267d81a93-istio-podinfo\") pod \"e2e-mt-faa154ea-openshift-default-6db9786fc6-48297\" (UID: \"0c8db87c-c55f-4c6a-91ba-e45267d81a93\") " pod="openshift-ingress/e2e-mt-faa154ea-openshift-default-6db9786fc6-48297" Sep 11 16:14:02.943778 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:02.943748 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c8db87c-c55f-4c6a-91ba-e45267d81a93-istio-token\") pod \"e2e-mt-faa154ea-openshift-default-6db9786fc6-48297\" (UID: \"0c8db87c-c55f-4c6a-91ba-e45267d81a93\") " pod="openshift-ingress/e2e-mt-faa154ea-openshift-default-6db9786fc6-48297" Sep 11 16:14:02.944042 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:02.944020 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcz82\" (UniqueName: \"kubernetes.io/projected/0c8db87c-c55f-4c6a-91ba-e45267d81a93-kube-api-access-rcz82\") pod \"e2e-mt-faa154ea-openshift-default-6db9786fc6-48297\" (UID: \"0c8db87c-c55f-4c6a-91ba-e45267d81a93\") " pod="openshift-ingress/e2e-mt-faa154ea-openshift-default-6db9786fc6-48297" Sep 11 16:14:03.000125 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:03.000087 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-faa154ea-openshift-default-6db9786fc6-48297" Sep 11 16:14:03.286384 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:03.286358 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-faa154ea-openshift-default-6db9786fc6-48297"] Sep 11 16:14:03.288803 ip-10-0-136-96 kubenswrapper[2566]: W0911 16:14:03.288773 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c8db87c_c55f_4c6a_91ba_e45267d81a93.slice/crio-0a19bf01098097ef5ab859441bb62379eb8aad487edccbc42c3e461b9c6c27ed WatchSource:0}: Error finding container 0a19bf01098097ef5ab859441bb62379eb8aad487edccbc42c3e461b9c6c27ed: Status 404 returned error can't find the container with id 0a19bf01098097ef5ab859441bb62379eb8aad487edccbc42c3e461b9c6c27ed Sep 11 16:14:03.291293 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:03.291262 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 16:14:03.291404 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:03.291325 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 16:14:03.291404 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:03.291352 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 16:14:03.952813 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:03.952761 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-faa154ea-openshift-default-6db9786fc6-48297" event={"ID":"0c8db87c-c55f-4c6a-91ba-e45267d81a93","Type":"ContainerStarted","Data":"e7617fb86e9d6ad73b57c0475e60bb41173dc5dd7cb86d1a72695ce92cd84d12"} Sep 11 16:14:03.953267 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:03.952845 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-faa154ea-openshift-default-6db9786fc6-48297" event={"ID":"0c8db87c-c55f-4c6a-91ba-e45267d81a93","Type":"ContainerStarted","Data":"0a19bf01098097ef5ab859441bb62379eb8aad487edccbc42c3e461b9c6c27ed"} Sep 11 16:14:03.982156 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:03.982095 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-faa154ea-openshift-default-6db9786fc6-48297" podStartSLOduration=1.982074049 podStartE2EDuration="1.982074049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:14:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:14:03.980288194 +0000 UTC m=+1151.195122100" watchObservedRunningTime="2026-09-11 16:14:03.982074049 +0000 UTC m=+1151.196907955" Sep 11 16:14:04.000432 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:04.000400 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-faa154ea-openshift-default-6db9786fc6-48297" Sep 11 16:14:04.005453 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:04.005430 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-faa154ea-openshift-default-6db9786fc6-48297" Sep 11 16:14:04.263199 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:04.263170 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-dacdbe-openshift-default-885dc786d-cjdg8"] Sep 11 16:14:04.263495 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:04.263461 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-dacdbe-openshift-default-885dc786d-cjdg8" podUID="a7ee410e-6e10-49e4-af3a-c826b06ec137" containerName="istio-proxy" containerID="cri-o://d41fa0f75ced236c2e3d9612dfb2ecb17ee34a0553543127a05b5965610aaf22" gracePeriod=30 Sep 11 16:14:04.959857 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:04.959819 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-faa154ea-openshift-default-6db9786fc6-48297" Sep 11 16:14:04.960842 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:04.960818 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-faa154ea-openshift-default-6db9786fc6-48297" Sep 11 16:14:05.057345 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:05.057302 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-3c2e6c-openshift-default-849cf5c868-txrmb"] Sep 11 16:14:05.061149 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:05.061131 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-3c2e6c-openshift-default-849cf5c868-txrmb" Sep 11 16:14:05.064097 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:05.064067 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-3c2e6c-openshift-default-dockercfg-sqbwp\"" Sep 11 16:14:05.077818 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:05.077789 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-3c2e6c-openshift-default-849cf5c868-txrmb"] Sep 11 16:14:05.161619 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:05.161571 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e28ddf9-c591-47d2-964b-8fde681c414d-workload-socket\") pod \"e2e-noten-3c2e6c-openshift-default-849cf5c868-txrmb\" (UID: \"9e28ddf9-c591-47d2-964b-8fde681c414d\") " pod="openshift-ingress/e2e-noten-3c2e6c-openshift-default-849cf5c868-txrmb" Sep 11 16:14:05.161891 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:05.161869 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e28ddf9-c591-47d2-964b-8fde681c414d-istio-token\") pod \"e2e-noten-3c2e6c-openshift-default-849cf5c868-txrmb\" (UID: \"9e28ddf9-c591-47d2-964b-8fde681c414d\") " pod="openshift-ingress/e2e-noten-3c2e6c-openshift-default-849cf5c868-txrmb" Sep 11 16:14:05.162060 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:05.162044 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e28ddf9-c591-47d2-964b-8fde681c414d-istio-data\") pod \"e2e-noten-3c2e6c-openshift-default-849cf5c868-txrmb\" (UID: \"9e28ddf9-c591-47d2-964b-8fde681c414d\") " pod="openshift-ingress/e2e-noten-3c2e6c-openshift-default-849cf5c868-txrmb" Sep 11 16:14:05.162185 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:05.162168 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dxkb\" (UniqueName: \"kubernetes.io/projected/9e28ddf9-c591-47d2-964b-8fde681c414d-kube-api-access-4dxkb\") pod \"e2e-noten-3c2e6c-openshift-default-849cf5c868-txrmb\" (UID: \"9e28ddf9-c591-47d2-964b-8fde681c414d\") " pod="openshift-ingress/e2e-noten-3c2e6c-openshift-default-849cf5c868-txrmb" Sep 11 16:14:05.162323 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:05.162300 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e28ddf9-c591-47d2-964b-8fde681c414d-credential-socket\") pod \"e2e-noten-3c2e6c-openshift-default-849cf5c868-txrmb\" (UID: \"9e28ddf9-c591-47d2-964b-8fde681c414d\") " pod="openshift-ingress/e2e-noten-3c2e6c-openshift-default-849cf5c868-txrmb" Sep 11 16:14:05.162456 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:05.162340 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e28ddf9-c591-47d2-964b-8fde681c414d-istio-podinfo\") pod \"e2e-noten-3c2e6c-openshift-default-849cf5c868-txrmb\" (UID: \"9e28ddf9-c591-47d2-964b-8fde681c414d\") " pod="openshift-ingress/e2e-noten-3c2e6c-openshift-default-849cf5c868-txrmb" Sep 11 16:14:05.162456 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:05.162383 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e28ddf9-c591-47d2-964b-8fde681c414d-istio-envoy\") pod \"e2e-noten-3c2e6c-openshift-default-849cf5c868-txrmb\" (UID: \"9e28ddf9-c591-47d2-964b-8fde681c414d\") " pod="openshift-ingress/e2e-noten-3c2e6c-openshift-default-849cf5c868-txrmb" Sep 11 16:14:05.162456 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:05.162428 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e28ddf9-c591-47d2-964b-8fde681c414d-workload-certs\") pod \"e2e-noten-3c2e6c-openshift-default-849cf5c868-txrmb\" (UID: \"9e28ddf9-c591-47d2-964b-8fde681c414d\") " pod="openshift-ingress/e2e-noten-3c2e6c-openshift-default-849cf5c868-txrmb" Sep 11 16:14:05.162621 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:05.162455 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e28ddf9-c591-47d2-964b-8fde681c414d-istiod-ca-cert\") pod \"e2e-noten-3c2e6c-openshift-default-849cf5c868-txrmb\" (UID: \"9e28ddf9-c591-47d2-964b-8fde681c414d\") " pod="openshift-ingress/e2e-noten-3c2e6c-openshift-default-849cf5c868-txrmb" Sep 11 16:14:05.263766 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:05.263736 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e28ddf9-c591-47d2-964b-8fde681c414d-istio-podinfo\") pod \"e2e-noten-3c2e6c-openshift-default-849cf5c868-txrmb\" (UID: \"9e28ddf9-c591-47d2-964b-8fde681c414d\") " pod="openshift-ingress/e2e-noten-3c2e6c-openshift-default-849cf5c868-txrmb" Sep 11 16:14:05.263927 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:05.263782 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e28ddf9-c591-47d2-964b-8fde681c414d-istio-envoy\") pod \"e2e-noten-3c2e6c-openshift-default-849cf5c868-txrmb\" (UID: \"9e28ddf9-c591-47d2-964b-8fde681c414d\") " pod="openshift-ingress/e2e-noten-3c2e6c-openshift-default-849cf5c868-txrmb" Sep 11 16:14:05.263927 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:05.263820 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e28ddf9-c591-47d2-964b-8fde681c414d-workload-certs\") pod \"e2e-noten-3c2e6c-openshift-default-849cf5c868-txrmb\" (UID: \"9e28ddf9-c591-47d2-964b-8fde681c414d\") " pod="openshift-ingress/e2e-noten-3c2e6c-openshift-default-849cf5c868-txrmb" Sep 11 16:14:05.263927 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:05.263841 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e28ddf9-c591-47d2-964b-8fde681c414d-istiod-ca-cert\") pod \"e2e-noten-3c2e6c-openshift-default-849cf5c868-txrmb\" (UID: \"9e28ddf9-c591-47d2-964b-8fde681c414d\") " pod="openshift-ingress/e2e-noten-3c2e6c-openshift-default-849cf5c868-txrmb" Sep 11 16:14:05.264111 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:05.263939 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e28ddf9-c591-47d2-964b-8fde681c414d-workload-socket\") pod \"e2e-noten-3c2e6c-openshift-default-849cf5c868-txrmb\" (UID: \"9e28ddf9-c591-47d2-964b-8fde681c414d\") " pod="openshift-ingress/e2e-noten-3c2e6c-openshift-default-849cf5c868-txrmb" Sep 11 16:14:05.264111 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:05.264011 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e28ddf9-c591-47d2-964b-8fde681c414d-istio-token\") pod \"e2e-noten-3c2e6c-openshift-default-849cf5c868-txrmb\" (UID: \"9e28ddf9-c591-47d2-964b-8fde681c414d\") " pod="openshift-ingress/e2e-noten-3c2e6c-openshift-default-849cf5c868-txrmb" Sep 11 16:14:05.264111 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:05.264049 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e28ddf9-c591-47d2-964b-8fde681c414d-istio-data\") pod \"e2e-noten-3c2e6c-openshift-default-849cf5c868-txrmb\" (UID: \"9e28ddf9-c591-47d2-964b-8fde681c414d\") " pod="openshift-ingress/e2e-noten-3c2e6c-openshift-default-849cf5c868-txrmb" Sep 11 16:14:05.264111 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:05.264070 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dxkb\" (UniqueName: \"kubernetes.io/projected/9e28ddf9-c591-47d2-964b-8fde681c414d-kube-api-access-4dxkb\") pod \"e2e-noten-3c2e6c-openshift-default-849cf5c868-txrmb\" (UID: \"9e28ddf9-c591-47d2-964b-8fde681c414d\") " pod="openshift-ingress/e2e-noten-3c2e6c-openshift-default-849cf5c868-txrmb" Sep 11 16:14:05.264307 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:05.264141 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e28ddf9-c591-47d2-964b-8fde681c414d-credential-socket\") pod \"e2e-noten-3c2e6c-openshift-default-849cf5c868-txrmb\" (UID: \"9e28ddf9-c591-47d2-964b-8fde681c414d\") " pod="openshift-ingress/e2e-noten-3c2e6c-openshift-default-849cf5c868-txrmb" Sep 11 16:14:05.264500 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:05.264478 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e28ddf9-c591-47d2-964b-8fde681c414d-istio-data\") pod \"e2e-noten-3c2e6c-openshift-default-849cf5c868-txrmb\" (UID: \"9e28ddf9-c591-47d2-964b-8fde681c414d\") " pod="openshift-ingress/e2e-noten-3c2e6c-openshift-default-849cf5c868-txrmb" Sep 11 16:14:05.264613 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:05.264578 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e28ddf9-c591-47d2-964b-8fde681c414d-workload-socket\") pod \"e2e-noten-3c2e6c-openshift-default-849cf5c868-txrmb\" (UID: \"9e28ddf9-c591-47d2-964b-8fde681c414d\") " pod="openshift-ingress/e2e-noten-3c2e6c-openshift-default-849cf5c868-txrmb" Sep 11 16:14:05.264673 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:05.264622 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e28ddf9-c591-47d2-964b-8fde681c414d-credential-socket\") pod \"e2e-noten-3c2e6c-openshift-default-849cf5c868-txrmb\" (UID: \"9e28ddf9-c591-47d2-964b-8fde681c414d\") " pod="openshift-ingress/e2e-noten-3c2e6c-openshift-default-849cf5c868-txrmb" Sep 11 16:14:05.264673 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:05.264650 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e28ddf9-c591-47d2-964b-8fde681c414d-workload-certs\") pod \"e2e-noten-3c2e6c-openshift-default-849cf5c868-txrmb\" (UID: \"9e28ddf9-c591-47d2-964b-8fde681c414d\") " pod="openshift-ingress/e2e-noten-3c2e6c-openshift-default-849cf5c868-txrmb" Sep 11 16:14:05.264959 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:05.264942 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e28ddf9-c591-47d2-964b-8fde681c414d-istiod-ca-cert\") pod \"e2e-noten-3c2e6c-openshift-default-849cf5c868-txrmb\" (UID: \"9e28ddf9-c591-47d2-964b-8fde681c414d\") " pod="openshift-ingress/e2e-noten-3c2e6c-openshift-default-849cf5c868-txrmb" Sep 11 16:14:05.266291 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:05.266270 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e28ddf9-c591-47d2-964b-8fde681c414d-istio-podinfo\") pod \"e2e-noten-3c2e6c-openshift-default-849cf5c868-txrmb\" (UID: \"9e28ddf9-c591-47d2-964b-8fde681c414d\") " pod="openshift-ingress/e2e-noten-3c2e6c-openshift-default-849cf5c868-txrmb" Sep 11 16:14:05.266460 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:05.266444 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e28ddf9-c591-47d2-964b-8fde681c414d-istio-envoy\") pod \"e2e-noten-3c2e6c-openshift-default-849cf5c868-txrmb\" (UID: \"9e28ddf9-c591-47d2-964b-8fde681c414d\") " pod="openshift-ingress/e2e-noten-3c2e6c-openshift-default-849cf5c868-txrmb" Sep 11 16:14:05.272826 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:05.272798 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e28ddf9-c591-47d2-964b-8fde681c414d-istio-token\") pod \"e2e-noten-3c2e6c-openshift-default-849cf5c868-txrmb\" (UID: \"9e28ddf9-c591-47d2-964b-8fde681c414d\") " pod="openshift-ingress/e2e-noten-3c2e6c-openshift-default-849cf5c868-txrmb" Sep 11 16:14:05.272949 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:05.272930 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dxkb\" (UniqueName: \"kubernetes.io/projected/9e28ddf9-c591-47d2-964b-8fde681c414d-kube-api-access-4dxkb\") pod \"e2e-noten-3c2e6c-openshift-default-849cf5c868-txrmb\" (UID: \"9e28ddf9-c591-47d2-964b-8fde681c414d\") " pod="openshift-ingress/e2e-noten-3c2e6c-openshift-default-849cf5c868-txrmb" Sep 11 16:14:05.374415 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:05.374382 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-3c2e6c-openshift-default-849cf5c868-txrmb" Sep 11 16:14:05.670683 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:05.670653 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-3c2e6c-openshift-default-849cf5c868-txrmb"] Sep 11 16:14:05.670947 ip-10-0-136-96 kubenswrapper[2566]: W0911 16:14:05.670659 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e28ddf9_c591_47d2_964b_8fde681c414d.slice/crio-8bc68839044d68f295dbd3f7a00d04ec6e1d767d29c7e4a94ea97276b6261d54 WatchSource:0}: Error finding container 8bc68839044d68f295dbd3f7a00d04ec6e1d767d29c7e4a94ea97276b6261d54: Status 404 returned error can't find the container with id 8bc68839044d68f295dbd3f7a00d04ec6e1d767d29c7e4a94ea97276b6261d54 Sep 11 16:14:05.673076 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:05.673042 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 16:14:05.673158 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:05.673105 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 16:14:05.673158 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:05.673134 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 16:14:05.968509 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:05.968408 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-3c2e6c-openshift-default-849cf5c868-txrmb" event={"ID":"9e28ddf9-c591-47d2-964b-8fde681c414d","Type":"ContainerStarted","Data":"5b0ab917902a60b29cc8d4ce75c6bf4af223111a15318391a351ec3094c65195"} Sep 11 16:14:05.968509 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:05.968479 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-3c2e6c-openshift-default-849cf5c868-txrmb" event={"ID":"9e28ddf9-c591-47d2-964b-8fde681c414d","Type":"ContainerStarted","Data":"8bc68839044d68f295dbd3f7a00d04ec6e1d767d29c7e4a94ea97276b6261d54"} Sep 11 16:14:06.013640 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:06.013579 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-3c2e6c-openshift-default-849cf5c868-txrmb" podStartSLOduration=1.013558691 podStartE2EDuration="1.013558691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:14:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:14:06.009774027 +0000 UTC m=+1153.224607932" watchObservedRunningTime="2026-09-11 16:14:06.013558691 +0000 UTC m=+1153.228392598" Sep 11 16:14:06.374748 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:06.374714 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-3c2e6c-openshift-default-849cf5c868-txrmb" Sep 11 16:14:06.380334 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:06.380305 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-3c2e6c-openshift-default-849cf5c868-txrmb" Sep 11 16:14:06.974501 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:06.974471 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-3c2e6c-openshift-default-849cf5c868-txrmb" Sep 11 16:14:06.975408 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:06.975390 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-3c2e6c-openshift-default-849cf5c868-txrmb" Sep 11 16:14:09.517223 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:09.517193 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-dacdbe-openshift-default-885dc786d-cjdg8" Sep 11 16:14:09.610840 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:09.610744 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7ee410e-6e10-49e4-af3a-c826b06ec137-istio-data\") pod \"a7ee410e-6e10-49e4-af3a-c826b06ec137\" (UID: \"a7ee410e-6e10-49e4-af3a-c826b06ec137\") " Sep 11 16:14:09.610840 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:09.610831 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7ee410e-6e10-49e4-af3a-c826b06ec137-workload-certs\") pod \"a7ee410e-6e10-49e4-af3a-c826b06ec137\" (UID: \"a7ee410e-6e10-49e4-af3a-c826b06ec137\") " Sep 11 16:14:09.611105 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:09.610857 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7ee410e-6e10-49e4-af3a-c826b06ec137-credential-socket\") pod \"a7ee410e-6e10-49e4-af3a-c826b06ec137\" (UID: \"a7ee410e-6e10-49e4-af3a-c826b06ec137\") " Sep 11 16:14:09.611105 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:09.610929 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7ee410e-6e10-49e4-af3a-c826b06ec137-istio-envoy\") pod \"a7ee410e-6e10-49e4-af3a-c826b06ec137\" (UID: \"a7ee410e-6e10-49e4-af3a-c826b06ec137\") " Sep 11 16:14:09.611105 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:09.611006 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7ee410e-6e10-49e4-af3a-c826b06ec137-istio-token\") pod \"a7ee410e-6e10-49e4-af3a-c826b06ec137\" (UID: \"a7ee410e-6e10-49e4-af3a-c826b06ec137\") " Sep 11 16:14:09.611105 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:09.611039 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7ee410e-6e10-49e4-af3a-c826b06ec137-istiod-ca-cert\") pod \"a7ee410e-6e10-49e4-af3a-c826b06ec137\" (UID: \"a7ee410e-6e10-49e4-af3a-c826b06ec137\") " Sep 11 16:14:09.611105 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:09.611073 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7ee410e-6e10-49e4-af3a-c826b06ec137-workload-socket\") pod \"a7ee410e-6e10-49e4-af3a-c826b06ec137\" (UID: \"a7ee410e-6e10-49e4-af3a-c826b06ec137\") " Sep 11 16:14:09.611375 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:09.611113 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7ee410e-6e10-49e4-af3a-c826b06ec137-istio-data" (OuterVolumeSpecName: "istio-data") pod "a7ee410e-6e10-49e4-af3a-c826b06ec137" (UID: "a7ee410e-6e10-49e4-af3a-c826b06ec137"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 16:14:09.611375 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:09.611184 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7ee410e-6e10-49e4-af3a-c826b06ec137-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a7ee410e-6e10-49e4-af3a-c826b06ec137" (UID: "a7ee410e-6e10-49e4-af3a-c826b06ec137"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 16:14:09.611375 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:09.611248 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7ee410e-6e10-49e4-af3a-c826b06ec137-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a7ee410e-6e10-49e4-af3a-c826b06ec137" (UID: "a7ee410e-6e10-49e4-af3a-c826b06ec137"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 16:14:09.611375 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:09.611333 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7ee410e-6e10-49e4-af3a-c826b06ec137-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a7ee410e-6e10-49e4-af3a-c826b06ec137" (UID: "a7ee410e-6e10-49e4-af3a-c826b06ec137"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 16:14:09.611605 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:09.611419 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7ee410e-6e10-49e4-af3a-c826b06ec137-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a7ee410e-6e10-49e4-af3a-c826b06ec137" (UID: "a7ee410e-6e10-49e4-af3a-c826b06ec137"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 16:14:09.611693 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:09.611669 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7ee410e-6e10-49e4-af3a-c826b06ec137-istio-podinfo\") pod \"a7ee410e-6e10-49e4-af3a-c826b06ec137\" (UID: \"a7ee410e-6e10-49e4-af3a-c826b06ec137\") " Sep 11 16:14:09.611759 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:09.611727 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w8dgm\" (UniqueName: \"kubernetes.io/projected/a7ee410e-6e10-49e4-af3a-c826b06ec137-kube-api-access-w8dgm\") pod \"a7ee410e-6e10-49e4-af3a-c826b06ec137\" (UID: \"a7ee410e-6e10-49e4-af3a-c826b06ec137\") " Sep 11 16:14:09.612128 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:09.612102 2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7ee410e-6e10-49e4-af3a-c826b06ec137-istiod-ca-cert\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:14:09.612128 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:09.612129 2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7ee410e-6e10-49e4-af3a-c826b06ec137-workload-socket\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:14:09.612307 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:09.612143 2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7ee410e-6e10-49e4-af3a-c826b06ec137-istio-data\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:14:09.612307 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:09.612159 2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7ee410e-6e10-49e4-af3a-c826b06ec137-workload-certs\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:14:09.612307 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:09.612171 2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7ee410e-6e10-49e4-af3a-c826b06ec137-credential-socket\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:14:09.613342 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:09.613312 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7ee410e-6e10-49e4-af3a-c826b06ec137-istio-token" (OuterVolumeSpecName: "istio-token") pod "a7ee410e-6e10-49e4-af3a-c826b06ec137" (UID: "a7ee410e-6e10-49e4-af3a-c826b06ec137"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 16:14:09.613451 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:09.613400 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a7ee410e-6e10-49e4-af3a-c826b06ec137-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a7ee410e-6e10-49e4-af3a-c826b06ec137" (UID: "a7ee410e-6e10-49e4-af3a-c826b06ec137"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 16:14:09.613451 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:09.613411 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7ee410e-6e10-49e4-af3a-c826b06ec137-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a7ee410e-6e10-49e4-af3a-c826b06ec137" (UID: "a7ee410e-6e10-49e4-af3a-c826b06ec137"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 16:14:09.613671 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:09.613650 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7ee410e-6e10-49e4-af3a-c826b06ec137-kube-api-access-w8dgm" (OuterVolumeSpecName: "kube-api-access-w8dgm") pod "a7ee410e-6e10-49e4-af3a-c826b06ec137" (UID: "a7ee410e-6e10-49e4-af3a-c826b06ec137"). InnerVolumeSpecName "kube-api-access-w8dgm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 16:14:09.713308 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:09.713261 2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7ee410e-6e10-49e4-af3a-c826b06ec137-istio-envoy\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:14:09.713308 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:09.713292 2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7ee410e-6e10-49e4-af3a-c826b06ec137-istio-token\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:14:09.713308 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:09.713305 2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7ee410e-6e10-49e4-af3a-c826b06ec137-istio-podinfo\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:14:09.713308 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:09.713317 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8dgm\" (UniqueName: \"kubernetes.io/projected/a7ee410e-6e10-49e4-af3a-c826b06ec137-kube-api-access-w8dgm\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:14:09.996157 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:09.996062 2566 generic.go:356] "Generic (PLEG): container finished" podID="a7ee410e-6e10-49e4-af3a-c826b06ec137" containerID="d41fa0f75ced236c2e3d9612dfb2ecb17ee34a0553543127a05b5965610aaf22" exitCode=0 Sep 11 16:14:09.996157 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:09.996115 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-dacdbe-openshift-default-885dc786d-cjdg8" event={"ID":"a7ee410e-6e10-49e4-af3a-c826b06ec137","Type":"ContainerDied","Data":"d41fa0f75ced236c2e3d9612dfb2ecb17ee34a0553543127a05b5965610aaf22"} Sep 11 16:14:09.996157 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:09.996121 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-dacdbe-openshift-default-885dc786d-cjdg8" Sep 11 16:14:09.996157 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:09.996157 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-dacdbe-openshift-default-885dc786d-cjdg8" event={"ID":"a7ee410e-6e10-49e4-af3a-c826b06ec137","Type":"ContainerDied","Data":"484b10d01418bb68bb152dfbc0b58e02d6d3bfb2f032bcb13c5221cc36d66618"} Sep 11 16:14:09.996476 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:09.996179 2566 scope.go:117] "RemoveContainer" containerID="d41fa0f75ced236c2e3d9612dfb2ecb17ee34a0553543127a05b5965610aaf22" Sep 11 16:14:10.006434 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:10.006415 2566 scope.go:117] "RemoveContainer" containerID="79fbb574951f008dd2ac020baeba850a78e99d758ee86d7f5e9ce8fced6f52a8" Sep 11 16:14:10.015487 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:10.015460 2566 scope.go:117] "RemoveContainer" containerID="d41fa0f75ced236c2e3d9612dfb2ecb17ee34a0553543127a05b5965610aaf22" Sep 11 16:14:10.015797 ip-10-0-136-96 kubenswrapper[2566]: E0911 16:14:10.015776 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d41fa0f75ced236c2e3d9612dfb2ecb17ee34a0553543127a05b5965610aaf22\": container with ID starting with d41fa0f75ced236c2e3d9612dfb2ecb17ee34a0553543127a05b5965610aaf22 not found: ID does not exist" containerID="d41fa0f75ced236c2e3d9612dfb2ecb17ee34a0553543127a05b5965610aaf22" Sep 11 16:14:10.015865 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:10.015807 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d41fa0f75ced236c2e3d9612dfb2ecb17ee34a0553543127a05b5965610aaf22"} err="failed to get container status \"d41fa0f75ced236c2e3d9612dfb2ecb17ee34a0553543127a05b5965610aaf22\": rpc error: code = NotFound desc = could not find container \"d41fa0f75ced236c2e3d9612dfb2ecb17ee34a0553543127a05b5965610aaf22\": container with ID starting with d41fa0f75ced236c2e3d9612dfb2ecb17ee34a0553543127a05b5965610aaf22 not found: ID does not exist" Sep 11 16:14:10.015865 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:10.015825 2566 scope.go:117] "RemoveContainer" containerID="79fbb574951f008dd2ac020baeba850a78e99d758ee86d7f5e9ce8fced6f52a8" Sep 11 16:14:10.016109 ip-10-0-136-96 kubenswrapper[2566]: E0911 16:14:10.016090 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79fbb574951f008dd2ac020baeba850a78e99d758ee86d7f5e9ce8fced6f52a8\": container with ID starting with 79fbb574951f008dd2ac020baeba850a78e99d758ee86d7f5e9ce8fced6f52a8 not found: ID does not exist" containerID="79fbb574951f008dd2ac020baeba850a78e99d758ee86d7f5e9ce8fced6f52a8" Sep 11 16:14:10.016164 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:10.016116 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79fbb574951f008dd2ac020baeba850a78e99d758ee86d7f5e9ce8fced6f52a8"} err="failed to get container status \"79fbb574951f008dd2ac020baeba850a78e99d758ee86d7f5e9ce8fced6f52a8\": rpc error: code = NotFound desc = could not find container \"79fbb574951f008dd2ac020baeba850a78e99d758ee86d7f5e9ce8fced6f52a8\": container with ID starting with 79fbb574951f008dd2ac020baeba850a78e99d758ee86d7f5e9ce8fced6f52a8 not found: ID does not exist" Sep 11 16:14:10.021135 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:10.021111 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-dacdbe-openshift-default-885dc786d-cjdg8"] Sep 11 16:14:10.027737 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:10.027704 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-dacdbe-openshift-default-885dc786d-cjdg8"] Sep 11 16:14:11.239531 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:11.239487 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7ee410e-6e10-49e4-af3a-c826b06ec137" path="/var/lib/kubelet/pods/a7ee410e-6e10-49e4-af3a-c826b06ec137/volumes" Sep 11 16:14:15.131467 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:15.131428 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dx4xn"] Sep 11 16:14:26.122896 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:26.122859 2566 generic.go:356] "Generic (PLEG): container finished" podID="0c8db87c-c55f-4c6a-91ba-e45267d81a93" containerID="e7617fb86e9d6ad73b57c0475e60bb41173dc5dd7cb86d1a72695ce92cd84d12" exitCode=0 Sep 11 16:14:26.123361 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:26.122936 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-faa154ea-openshift-default-6db9786fc6-48297" event={"ID":"0c8db87c-c55f-4c6a-91ba-e45267d81a93","Type":"ContainerDied","Data":"e7617fb86e9d6ad73b57c0475e60bb41173dc5dd7cb86d1a72695ce92cd84d12"} Sep 11 16:14:26.130830 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:26.124853 2566 scope.go:117] "RemoveContainer" containerID="e7617fb86e9d6ad73b57c0475e60bb41173dc5dd7cb86d1a72695ce92cd84d12" Sep 11 16:14:26.130830 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:26.129287 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 16:14:26.130830 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:26.129426 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 16:14:26.130830 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:26.129524 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 16:14:27.053148 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:27.053097 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dx4xn"] Sep 11 16:14:27.131132 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:27.131100 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-faa154ea-openshift-default-6db9786fc6-48297" event={"ID":"0c8db87c-c55f-4c6a-91ba-e45267d81a93","Type":"ContainerStarted","Data":"c910ec2466cfaca1397664bcb49691bf25f8d05cfc821f8aa7c50970679c644a"} Sep 11 16:14:28.000611 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:28.000569 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-faa154ea-openshift-default-6db9786fc6-48297" Sep 11 16:14:28.000874 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:28.000850 2566 patch_prober.go:28] interesting pod/e2e-mt-faa154ea-openshift-default-6db9786fc6-48297 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Sep 11 16:14:28.000928 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:28.000903 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-faa154ea-openshift-default-6db9786fc6-48297" podUID="0c8db87c-c55f-4c6a-91ba-e45267d81a93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Sep 11 16:14:29.000687 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:29.000648 2566 patch_prober.go:28] interesting pod/e2e-mt-faa154ea-openshift-default-6db9786fc6-48297 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Sep 11 16:14:29.001184 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:29.000715 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-faa154ea-openshift-default-6db9786fc6-48297" podUID="0c8db87c-c55f-4c6a-91ba-e45267d81a93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Sep 11 16:14:30.001305 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:30.001266 2566 patch_prober.go:28] interesting pod/e2e-mt-faa154ea-openshift-default-6db9786fc6-48297 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Sep 11 16:14:30.001791 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:30.001333 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-faa154ea-openshift-default-6db9786fc6-48297" podUID="0c8db87c-c55f-4c6a-91ba-e45267d81a93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Sep 11 16:14:31.001220 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:31.001181 2566 patch_prober.go:28] interesting pod/e2e-mt-faa154ea-openshift-default-6db9786fc6-48297 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Sep 11 16:14:31.001426 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:31.001289 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-faa154ea-openshift-default-6db9786fc6-48297" podUID="0c8db87c-c55f-4c6a-91ba-e45267d81a93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Sep 11 16:14:32.000947 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:32.000912 2566 patch_prober.go:28] interesting pod/e2e-mt-faa154ea-openshift-default-6db9786fc6-48297 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Sep 11 16:14:32.001147 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:32.000986 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-faa154ea-openshift-default-6db9786fc6-48297" podUID="0c8db87c-c55f-4c6a-91ba-e45267d81a93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Sep 11 16:14:33.005313 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:33.005281 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-faa154ea-openshift-default-6db9786fc6-48297" Sep 11 16:14:33.005815 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:33.005792 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-faa154ea-openshift-default-6db9786fc6-48297" Sep 11 16:14:33.006587 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:33.006564 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-faa154ea-openshift-default-6db9786fc6-48297" Sep 11 16:14:38.746769 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:38.746729 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-faa154ea-kn55h"] Sep 11 16:14:38.747275 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:38.747209 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7ee410e-6e10-49e4-af3a-c826b06ec137" containerName="istio-proxy" Sep 11 16:14:38.747275 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:38.747223 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7ee410e-6e10-49e4-af3a-c826b06ec137" containerName="istio-proxy" Sep 11 16:14:38.747275 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:38.747235 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7ee410e-6e10-49e4-af3a-c826b06ec137" containerName="istio-proxy" Sep 11 16:14:38.747275 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:38.747241 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7ee410e-6e10-49e4-af3a-c826b06ec137" containerName="istio-proxy" Sep 11 16:14:38.747476 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:38.747315 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7ee410e-6e10-49e4-af3a-c826b06ec137" containerName="istio-proxy" Sep 11 16:14:38.747476 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:38.747323 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7ee410e-6e10-49e4-af3a-c826b06ec137" containerName="istio-proxy" Sep 11 16:14:38.751903 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:38.751883 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-faa154ea-kn55h" Sep 11 16:14:38.755790 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:38.755771 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qbzdr\"" Sep 11 16:14:38.756141 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:38.756122 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 16:14:38.762264 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:38.762242 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-faa154ea-kn55h"] Sep 11 16:14:38.906060 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:38.906030 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2955c9f2-cbc1-46f0-9267-e0919e13fe4e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-faa154ea-kn55h\" (UID: \"2955c9f2-cbc1-46f0-9267-e0919e13fe4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-faa154ea-kn55h" Sep 11 16:14:39.006824 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:39.006734 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2955c9f2-cbc1-46f0-9267-e0919e13fe4e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-faa154ea-kn55h\" (UID: \"2955c9f2-cbc1-46f0-9267-e0919e13fe4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-faa154ea-kn55h" Sep 11 16:14:39.007357 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:39.007339 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2955c9f2-cbc1-46f0-9267-e0919e13fe4e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-faa154ea-kn55h\" (UID: \"2955c9f2-cbc1-46f0-9267-e0919e13fe4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-faa154ea-kn55h" Sep 11 16:14:39.063827 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:39.063784 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-faa154ea-kn55h" Sep 11 16:14:39.356651 ip-10-0-136-96 kubenswrapper[2566]: W0911 16:14:39.356610 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2955c9f2_cbc1_46f0_9267_e0919e13fe4e.slice/crio-343934ef71bd8806069a5b76196806ba6efe65ea8f668b508744ed2a37753c0e WatchSource:0}: Error finding container 343934ef71bd8806069a5b76196806ba6efe65ea8f668b508744ed2a37753c0e: Status 404 returned error can't find the container with id 343934ef71bd8806069a5b76196806ba6efe65ea8f668b508744ed2a37753c0e Sep 11 16:14:39.358445 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:39.358369 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-faa154ea-kn55h"] Sep 11 16:14:40.225284 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:40.225247 2566 generic.go:356] "Generic (PLEG): container finished" podID="2955c9f2-cbc1-46f0-9267-e0919e13fe4e" containerID="80454128ccbaa3f58fedd9fc8560c2d8f9b00a008fcea6d5f53bdc64b6f3c416" exitCode=0 Sep 11 16:14:40.225666 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:40.225335 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-faa154ea-kn55h" event={"ID":"2955c9f2-cbc1-46f0-9267-e0919e13fe4e","Type":"ContainerDied","Data":"80454128ccbaa3f58fedd9fc8560c2d8f9b00a008fcea6d5f53bdc64b6f3c416"} Sep 11 16:14:40.225666 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:40.225393 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-faa154ea-kn55h" event={"ID":"2955c9f2-cbc1-46f0-9267-e0919e13fe4e","Type":"ContainerStarted","Data":"343934ef71bd8806069a5b76196806ba6efe65ea8f668b508744ed2a37753c0e"} Sep 11 16:14:41.372400 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:41.372375 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-faa154ea-kn55h" Sep 11 16:14:41.533376 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:41.533338 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2955c9f2-cbc1-46f0-9267-e0919e13fe4e-maas-api-service-ca\") pod \"2955c9f2-cbc1-46f0-9267-e0919e13fe4e\" (UID: \"2955c9f2-cbc1-46f0-9267-e0919e13fe4e\") " Sep 11 16:14:41.533745 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:41.533711 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2955c9f2-cbc1-46f0-9267-e0919e13fe4e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2955c9f2-cbc1-46f0-9267-e0919e13fe4e" (UID: "2955c9f2-cbc1-46f0-9267-e0919e13fe4e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 16:14:41.635240 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:41.635207 2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2955c9f2-cbc1-46f0-9267-e0919e13fe4e-maas-api-service-ca\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:14:42.239667 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:42.239634 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-faa154ea-kn55h" Sep 11 16:14:42.239853 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:42.239670 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-faa154ea-kn55h" event={"ID":"2955c9f2-cbc1-46f0-9267-e0919e13fe4e","Type":"ContainerDied","Data":"343934ef71bd8806069a5b76196806ba6efe65ea8f668b508744ed2a37753c0e"} Sep 11 16:14:42.239853 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:42.239725 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="343934ef71bd8806069a5b76196806ba6efe65ea8f668b508744ed2a37753c0e" Sep 11 16:14:42.511199 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:42.511114 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dx4xn"] Sep 11 16:14:49.660432 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:49.660396 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-faa154ea-kn55h"] Sep 11 16:14:49.669506 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:49.669472 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-faa154ea-kn55h"] Sep 11 16:14:51.239878 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:51.239841 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2955c9f2-cbc1-46f0-9267-e0919e13fe4e" path="/var/lib/kubelet/pods/2955c9f2-cbc1-46f0-9267-e0919e13fe4e/volumes" Sep 11 16:14:53.372990 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:53.372937 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-xjvl5_7542ad77-c817-4370-91de-3b08db264753/console-operator/2.log" Sep 11 16:14:53.375565 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:53.375547 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-xjvl5_7542ad77-c817-4370-91de-3b08db264753/console-operator/2.log" Sep 11 16:14:53.379468 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:53.379448 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tgq5c_981d7d1e-f81f-48fc-8962-7831bc5c2b27/ovn-acl-logging/0.log" Sep 11 16:14:53.382095 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:53.382079 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tgq5c_981d7d1e-f81f-48fc-8962-7831bc5c2b27/ovn-acl-logging/0.log" Sep 11 16:14:55.439418 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:55.439386 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dx4xn"] Sep 11 16:14:56.031288 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:56.031249 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8e3121-m6q8q"] Sep 11 16:14:56.036016 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:56.035959 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2955c9f2-cbc1-46f0-9267-e0919e13fe4e" containerName="revoke-keys" Sep 11 16:14:56.036251 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:56.036238 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="2955c9f2-cbc1-46f0-9267-e0919e13fe4e" containerName="revoke-keys" Sep 11 16:14:56.036604 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:56.036582 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="2955c9f2-cbc1-46f0-9267-e0919e13fe4e" containerName="revoke-keys" Sep 11 16:14:56.041994 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:56.041959 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8e3121-m6q8q" Sep 11 16:14:56.045183 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:56.045141 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 16:14:56.045578 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:56.045554 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qbzdr\"" Sep 11 16:14:56.049955 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:56.049927 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8e3121-m6q8q"] Sep 11 16:14:56.076174 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:56.076141 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9973186-448e-4f04-92e8-aae31e05d891-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-8e3121-m6q8q\" (UID: \"e9973186-448e-4f04-92e8-aae31e05d891\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8e3121-m6q8q" Sep 11 16:14:56.177263 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:56.177223 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9973186-448e-4f04-92e8-aae31e05d891-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-8e3121-m6q8q\" (UID: \"e9973186-448e-4f04-92e8-aae31e05d891\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8e3121-m6q8q" Sep 11 16:14:56.177852 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:56.177834 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9973186-448e-4f04-92e8-aae31e05d891-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-8e3121-m6q8q\" (UID: \"e9973186-448e-4f04-92e8-aae31e05d891\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8e3121-m6q8q" Sep 11 16:14:56.359278 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:56.359198 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8e3121-m6q8q" Sep 11 16:14:56.658845 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:56.658821 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8e3121-m6q8q"] Sep 11 16:14:56.659805 ip-10-0-136-96 kubenswrapper[2566]: W0911 16:14:56.659777 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9973186_448e_4f04_92e8_aae31e05d891.slice/crio-b4aa1afac5699ef78a86534b26195e820e22094b2b41c57dc0270bc747bd93b7 WatchSource:0}: Error finding container b4aa1afac5699ef78a86534b26195e820e22094b2b41c57dc0270bc747bd93b7: Status 404 returned error can't find the container with id b4aa1afac5699ef78a86534b26195e820e22094b2b41c57dc0270bc747bd93b7 Sep 11 16:14:56.910462 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:56.910371 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-3c2e6c-openshift-default-849cf5c868-txrmb"] Sep 11 16:14:56.910645 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:56.910607 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-3c2e6c-openshift-default-849cf5c868-txrmb" podUID="9e28ddf9-c591-47d2-964b-8fde681c414d" containerName="istio-proxy" containerID="cri-o://5b0ab917902a60b29cc8d4ce75c6bf4af223111a15318391a351ec3094c65195" gracePeriod=30 Sep 11 16:14:57.337189 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:57.337133 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8e3121-m6q8q" event={"ID":"e9973186-448e-4f04-92e8-aae31e05d891","Type":"ContainerStarted","Data":"eb9814ef703d781325f86402c443afe91023349369de8d2b1ba7f99239e9fc2b"} Sep 11 16:14:57.337379 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:57.337214 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8e3121-m6q8q" event={"ID":"e9973186-448e-4f04-92e8-aae31e05d891","Type":"ContainerStarted","Data":"b4aa1afac5699ef78a86534b26195e820e22094b2b41c57dc0270bc747bd93b7"} Sep 11 16:14:57.358404 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:57.358347 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8e3121-m6q8q" podStartSLOduration=1.3583323840000001 podStartE2EDuration="1.358332384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:14:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:14:57.355439705 +0000 UTC m=+1204.570273611" watchObservedRunningTime="2026-09-11 16:14:57.358332384 +0000 UTC m=+1204.573166289" Sep 11 16:14:58.146617 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:58.146580 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-faa154ea-openshift-default-6db9786fc6-48297"] Sep 11 16:14:58.147059 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:58.146807 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-faa154ea-openshift-default-6db9786fc6-48297" podUID="0c8db87c-c55f-4c6a-91ba-e45267d81a93" containerName="istio-proxy" containerID="cri-o://c910ec2466cfaca1397664bcb49691bf25f8d05cfc821f8aa7c50970679c644a" gracePeriod=30 Sep 11 16:14:58.346162 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:58.346125 2566 generic.go:356] "Generic (PLEG): container finished" podID="e9973186-448e-4f04-92e8-aae31e05d891" containerID="eb9814ef703d781325f86402c443afe91023349369de8d2b1ba7f99239e9fc2b" exitCode=0 Sep 11 16:14:58.346350 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:58.346207 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8e3121-m6q8q" event={"ID":"e9973186-448e-4f04-92e8-aae31e05d891","Type":"ContainerDied","Data":"eb9814ef703d781325f86402c443afe91023349369de8d2b1ba7f99239e9fc2b"} Sep 11 16:14:59.486710 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:59.486686 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8e3121-m6q8q" Sep 11 16:14:59.511060 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:59.511032 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9973186-448e-4f04-92e8-aae31e05d891-maas-api-service-ca\") pod \"e9973186-448e-4f04-92e8-aae31e05d891\" (UID: \"e9973186-448e-4f04-92e8-aae31e05d891\") " Sep 11 16:14:59.511374 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:59.511353 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9973186-448e-4f04-92e8-aae31e05d891-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e9973186-448e-4f04-92e8-aae31e05d891" (UID: "e9973186-448e-4f04-92e8-aae31e05d891"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 16:14:59.612650 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:14:59.612557 2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9973186-448e-4f04-92e8-aae31e05d891-maas-api-service-ca\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:15:00.152866 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:00.152826 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29819055-pck9z"] Sep 11 16:15:00.153316 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:00.153300 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9973186-448e-4f04-92e8-aae31e05d891" containerName="revoke-keys" Sep 11 16:15:00.153358 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:00.153319 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9973186-448e-4f04-92e8-aae31e05d891" containerName="revoke-keys" Sep 11 16:15:00.153416 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:00.153407 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9973186-448e-4f04-92e8-aae31e05d891" containerName="revoke-keys" Sep 11 16:15:00.156519 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:00.156501 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8e3121-29819055-vsdcm"] Sep 11 16:15:00.156645 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:00.156617 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819055-pck9z" Sep 11 16:15:00.159780 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:00.159763 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8e3121-29819055-vsdcm" Sep 11 16:15:00.167704 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:00.167675 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819055-pck9z" Sep 11 16:15:00.169109 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:00.169085 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29819055-pck9z"] Sep 11 16:15:00.175589 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:00.175566 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8e3121-29819055-vsdcm" Sep 11 16:15:00.180826 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:00.180786 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8e3121-29819055-vsdcm"] Sep 11 16:15:00.359625 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:00.359593 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8e3121-m6q8q" Sep 11 16:15:00.359791 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:00.359628 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8e3121-m6q8q" event={"ID":"e9973186-448e-4f04-92e8-aae31e05d891","Type":"ContainerDied","Data":"b4aa1afac5699ef78a86534b26195e820e22094b2b41c57dc0270bc747bd93b7"} Sep 11 16:15:00.359791 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:00.359669 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b4aa1afac5699ef78a86534b26195e820e22094b2b41c57dc0270bc747bd93b7" Sep 11 16:15:00.460624 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:00.460595 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29819055-pck9z"] Sep 11 16:15:00.462067 ip-10-0-136-96 kubenswrapper[2566]: W0911 16:15:00.462026 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d419112_60b9_4f74_ac53_8cdb8565d6f4.slice/crio-f1a012af078979ea333add9fa017a9982d8fc0203d9d722da417b5847c1bd850 WatchSource:0}: Error finding container f1a012af078979ea333add9fa017a9982d8fc0203d9d722da417b5847c1bd850: Status 404 returned error can't find the container with id f1a012af078979ea333add9fa017a9982d8fc0203d9d722da417b5847c1bd850 Sep 11 16:15:00.482994 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:00.482942 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8e3121-29819055-vsdcm"] Sep 11 16:15:00.484707 ip-10-0-136-96 kubenswrapper[2566]: W0911 16:15:00.484685 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc879bcf5_0a2e_44cf_a951_51fc1da22fe9.slice/crio-6582d0f62b6efd6cef1f9ad62931f9a785f6de77d93a88cd84a002fa07abc09d WatchSource:0}: Error finding container 6582d0f62b6efd6cef1f9ad62931f9a785f6de77d93a88cd84a002fa07abc09d: Status 404 returned error can't find the container with id 6582d0f62b6efd6cef1f9ad62931f9a785f6de77d93a88cd84a002fa07abc09d Sep 11 16:15:01.367362 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:01.367335 2566 generic.go:356] "Generic (PLEG): container finished" podID="2d419112-60b9-4f74-ac53-8cdb8565d6f4" containerID="ccb036309429e38f29019c32cbdc9c3eb5c533e54c472df19608af1db64db4a6" exitCode=0 Sep 11 16:15:01.367745 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:01.367418 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819055-pck9z" event={"ID":"2d419112-60b9-4f74-ac53-8cdb8565d6f4","Type":"ContainerDied","Data":"ccb036309429e38f29019c32cbdc9c3eb5c533e54c472df19608af1db64db4a6"} Sep 11 16:15:01.367745 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:01.367471 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819055-pck9z" event={"ID":"2d419112-60b9-4f74-ac53-8cdb8565d6f4","Type":"ContainerStarted","Data":"f1a012af078979ea333add9fa017a9982d8fc0203d9d722da417b5847c1bd850"} Sep 11 16:15:01.369097 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:01.369069 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8e3121-29819055-vsdcm" event={"ID":"c879bcf5-0a2e-44cf-a951-51fc1da22fe9","Type":"ContainerStarted","Data":"fe5a9c6fc6250e35c083608b8da61b6195704837be358658312064d0e854fca8"} Sep 11 16:15:01.369229 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:01.369112 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8e3121-29819055-vsdcm" event={"ID":"c879bcf5-0a2e-44cf-a951-51fc1da22fe9","Type":"ContainerStarted","Data":"6582d0f62b6efd6cef1f9ad62931f9a785f6de77d93a88cd84a002fa07abc09d"} Sep 11 16:15:01.399546 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:01.399498 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8e3121-29819055-vsdcm" podStartSLOduration=1.399482773 podStartE2EDuration="1.399482773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:15:01.397133008 +0000 UTC m=+1208.611966914" watchObservedRunningTime="2026-09-11 16:15:01.399482773 +0000 UTC m=+1208.614316677" Sep 11 16:15:02.068112 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:02.068077 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dx4xn"] Sep 11 16:15:02.167402 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:02.167374 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-3c2e6c-openshift-default-849cf5c868-txrmb" Sep 11 16:15:02.237454 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:02.237380 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e28ddf9-c591-47d2-964b-8fde681c414d-istiod-ca-cert\") pod \"9e28ddf9-c591-47d2-964b-8fde681c414d\" (UID: \"9e28ddf9-c591-47d2-964b-8fde681c414d\") " Sep 11 16:15:02.237454 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:02.237416 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e28ddf9-c591-47d2-964b-8fde681c414d-istio-podinfo\") pod \"9e28ddf9-c591-47d2-964b-8fde681c414d\" (UID: \"9e28ddf9-c591-47d2-964b-8fde681c414d\") " Sep 11 16:15:02.237454 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:02.237436 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e28ddf9-c591-47d2-964b-8fde681c414d-workload-socket\") pod \"9e28ddf9-c591-47d2-964b-8fde681c414d\" (UID: \"9e28ddf9-c591-47d2-964b-8fde681c414d\") " Sep 11 16:15:02.237668 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:02.237568 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e28ddf9-c591-47d2-964b-8fde681c414d-istio-envoy\") pod \"9e28ddf9-c591-47d2-964b-8fde681c414d\" (UID: \"9e28ddf9-c591-47d2-964b-8fde681c414d\") " Sep 11 16:15:02.237668 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:02.237614 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e28ddf9-c591-47d2-964b-8fde681c414d-credential-socket\") pod \"9e28ddf9-c591-47d2-964b-8fde681c414d\" (UID: \"9e28ddf9-c591-47d2-964b-8fde681c414d\") " Sep 11 16:15:02.237668 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:02.237644 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e28ddf9-c591-47d2-964b-8fde681c414d-istio-data\") pod \"9e28ddf9-c591-47d2-964b-8fde681c414d\" (UID: \"9e28ddf9-c591-47d2-964b-8fde681c414d\") " Sep 11 16:15:02.237822 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:02.237673 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e28ddf9-c591-47d2-964b-8fde681c414d-istio-token\") pod \"9e28ddf9-c591-47d2-964b-8fde681c414d\" (UID: \"9e28ddf9-c591-47d2-964b-8fde681c414d\") " Sep 11 16:15:02.237822 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:02.237713 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e28ddf9-c591-47d2-964b-8fde681c414d-workload-certs\") pod \"9e28ddf9-c591-47d2-964b-8fde681c414d\" (UID: \"9e28ddf9-c591-47d2-964b-8fde681c414d\") " Sep 11 16:15:02.237822 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:02.237725 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e28ddf9-c591-47d2-964b-8fde681c414d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9e28ddf9-c591-47d2-964b-8fde681c414d" (UID: "9e28ddf9-c591-47d2-964b-8fde681c414d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 16:15:02.237822 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:02.237759 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4dxkb\" (UniqueName: \"kubernetes.io/projected/9e28ddf9-c591-47d2-964b-8fde681c414d-kube-api-access-4dxkb\") pod \"9e28ddf9-c591-47d2-964b-8fde681c414d\" (UID: \"9e28ddf9-c591-47d2-964b-8fde681c414d\") " Sep 11 16:15:02.237822 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:02.237800 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e28ddf9-c591-47d2-964b-8fde681c414d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9e28ddf9-c591-47d2-964b-8fde681c414d" (UID: "9e28ddf9-c591-47d2-964b-8fde681c414d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 16:15:02.238102 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:02.237946 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e28ddf9-c591-47d2-964b-8fde681c414d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9e28ddf9-c591-47d2-964b-8fde681c414d" (UID: "9e28ddf9-c591-47d2-964b-8fde681c414d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 16:15:02.238102 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:02.237961 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e28ddf9-c591-47d2-964b-8fde681c414d-istio-data" (OuterVolumeSpecName: "istio-data") pod "9e28ddf9-c591-47d2-964b-8fde681c414d" (UID: "9e28ddf9-c591-47d2-964b-8fde681c414d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 16:15:02.238259 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:02.238226 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e28ddf9-c591-47d2-964b-8fde681c414d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9e28ddf9-c591-47d2-964b-8fde681c414d" (UID: "9e28ddf9-c591-47d2-964b-8fde681c414d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 16:15:02.238330 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:02.238270 2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e28ddf9-c591-47d2-964b-8fde681c414d-credential-socket\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:15:02.238330 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:02.238293 2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e28ddf9-c591-47d2-964b-8fde681c414d-istio-data\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:15:02.238330 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:02.238308 2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e28ddf9-c591-47d2-964b-8fde681c414d-istiod-ca-cert\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:15:02.238330 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:02.238322 2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e28ddf9-c591-47d2-964b-8fde681c414d-workload-socket\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:15:02.239909 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:02.239885 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e28ddf9-c591-47d2-964b-8fde681c414d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9e28ddf9-c591-47d2-964b-8fde681c414d" (UID: "9e28ddf9-c591-47d2-964b-8fde681c414d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 16:15:02.240120 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:02.240092 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9e28ddf9-c591-47d2-964b-8fde681c414d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9e28ddf9-c591-47d2-964b-8fde681c414d" (UID: "9e28ddf9-c591-47d2-964b-8fde681c414d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 16:15:02.240219 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:02.240138 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e28ddf9-c591-47d2-964b-8fde681c414d-istio-token" (OuterVolumeSpecName: "istio-token") pod "9e28ddf9-c591-47d2-964b-8fde681c414d" (UID: "9e28ddf9-c591-47d2-964b-8fde681c414d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 16:15:02.240358 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:02.240344 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e28ddf9-c591-47d2-964b-8fde681c414d-kube-api-access-4dxkb" (OuterVolumeSpecName: "kube-api-access-4dxkb") pod "9e28ddf9-c591-47d2-964b-8fde681c414d" (UID: "9e28ddf9-c591-47d2-964b-8fde681c414d"). InnerVolumeSpecName "kube-api-access-4dxkb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 16:15:02.339175 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:02.339130 2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e28ddf9-c591-47d2-964b-8fde681c414d-istio-envoy\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:15:02.339175 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:02.339170 2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e28ddf9-c591-47d2-964b-8fde681c414d-istio-token\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:15:02.339175 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:02.339179 2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e28ddf9-c591-47d2-964b-8fde681c414d-workload-certs\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:15:02.339175 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:02.339189 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dxkb\" (UniqueName: \"kubernetes.io/projected/9e28ddf9-c591-47d2-964b-8fde681c414d-kube-api-access-4dxkb\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:15:02.339514 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:02.339197 2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e28ddf9-c591-47d2-964b-8fde681c414d-istio-podinfo\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:15:02.376562 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:02.376531 2566 generic.go:356] "Generic (PLEG): container finished" podID="9e28ddf9-c591-47d2-964b-8fde681c414d" containerID="5b0ab917902a60b29cc8d4ce75c6bf4af223111a15318391a351ec3094c65195" exitCode=0 Sep 11 16:15:02.377056 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:02.376587 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-3c2e6c-openshift-default-849cf5c868-txrmb" Sep 11 16:15:02.377056 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:02.376616 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-3c2e6c-openshift-default-849cf5c868-txrmb" event={"ID":"9e28ddf9-c591-47d2-964b-8fde681c414d","Type":"ContainerDied","Data":"5b0ab917902a60b29cc8d4ce75c6bf4af223111a15318391a351ec3094c65195"} Sep 11 16:15:02.377056 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:02.376680 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-3c2e6c-openshift-default-849cf5c868-txrmb" event={"ID":"9e28ddf9-c591-47d2-964b-8fde681c414d","Type":"ContainerDied","Data":"8bc68839044d68f295dbd3f7a00d04ec6e1d767d29c7e4a94ea97276b6261d54"} Sep 11 16:15:02.377056 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:02.376712 2566 scope.go:117] "RemoveContainer" containerID="5b0ab917902a60b29cc8d4ce75c6bf4af223111a15318391a351ec3094c65195" Sep 11 16:15:02.378715 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:02.378678 2566 generic.go:356] "Generic (PLEG): container finished" podID="c879bcf5-0a2e-44cf-a951-51fc1da22fe9" containerID="fe5a9c6fc6250e35c083608b8da61b6195704837be358658312064d0e854fca8" exitCode=0 Sep 11 16:15:02.378838 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:02.378734 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8e3121-29819055-vsdcm" event={"ID":"c879bcf5-0a2e-44cf-a951-51fc1da22fe9","Type":"ContainerDied","Data":"fe5a9c6fc6250e35c083608b8da61b6195704837be358658312064d0e854fca8"} Sep 11 16:15:02.387648 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:02.387630 2566 scope.go:117] "RemoveContainer" containerID="5b0ab917902a60b29cc8d4ce75c6bf4af223111a15318391a351ec3094c65195" Sep 11 16:15:02.387931 ip-10-0-136-96 kubenswrapper[2566]: E0911 16:15:02.387913 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b0ab917902a60b29cc8d4ce75c6bf4af223111a15318391a351ec3094c65195\": container with ID starting with 5b0ab917902a60b29cc8d4ce75c6bf4af223111a15318391a351ec3094c65195 not found: ID does not exist" containerID="5b0ab917902a60b29cc8d4ce75c6bf4af223111a15318391a351ec3094c65195" Sep 11 16:15:02.388013 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:02.387940 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b0ab917902a60b29cc8d4ce75c6bf4af223111a15318391a351ec3094c65195"} err="failed to get container status \"5b0ab917902a60b29cc8d4ce75c6bf4af223111a15318391a351ec3094c65195\": rpc error: code = NotFound desc = could not find container \"5b0ab917902a60b29cc8d4ce75c6bf4af223111a15318391a351ec3094c65195\": container with ID starting with 5b0ab917902a60b29cc8d4ce75c6bf4af223111a15318391a351ec3094c65195 not found: ID does not exist" Sep 11 16:15:02.424252 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:02.424220 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-3c2e6c-openshift-default-849cf5c868-txrmb"] Sep 11 16:15:02.427638 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:02.427614 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-3c2e6c-openshift-default-849cf5c868-txrmb"] Sep 11 16:15:02.513453 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:02.513427 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819055-pck9z" Sep 11 16:15:03.008445 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:03.008407 2566 patch_prober.go:28] interesting pod/e2e-mt-faa154ea-openshift-default-6db9786fc6-48297 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 11 16:15:03.008609 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:03.008480 2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-faa154ea-openshift-default-6db9786fc6-48297" podUID="0c8db87c-c55f-4c6a-91ba-e45267d81a93" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 11 16:15:03.240156 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:03.240127 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9e28ddf9-c591-47d2-964b-8fde681c414d" path="/var/lib/kubelet/pods/9e28ddf9-c591-47d2-964b-8fde681c414d/volumes" Sep 11 16:15:03.389546 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:03.389511 2566 generic.go:356] "Generic (PLEG): container finished" podID="0c8db87c-c55f-4c6a-91ba-e45267d81a93" containerID="c910ec2466cfaca1397664bcb49691bf25f8d05cfc821f8aa7c50970679c644a" exitCode=0 Sep 11 16:15:03.389919 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:03.389579 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-faa154ea-openshift-default-6db9786fc6-48297" event={"ID":"0c8db87c-c55f-4c6a-91ba-e45267d81a93","Type":"ContainerDied","Data":"c910ec2466cfaca1397664bcb49691bf25f8d05cfc821f8aa7c50970679c644a"} Sep 11 16:15:03.389919 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:03.389658 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-faa154ea-openshift-default-6db9786fc6-48297" event={"ID":"0c8db87c-c55f-4c6a-91ba-e45267d81a93","Type":"ContainerDied","Data":"0a19bf01098097ef5ab859441bb62379eb8aad487edccbc42c3e461b9c6c27ed"} Sep 11 16:15:03.389919 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:03.389676 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0a19bf01098097ef5ab859441bb62379eb8aad487edccbc42c3e461b9c6c27ed" Sep 11 16:15:03.389919 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:03.389691 2566 scope.go:117] "RemoveContainer" containerID="e7617fb86e9d6ad73b57c0475e60bb41173dc5dd7cb86d1a72695ce92cd84d12" Sep 11 16:15:03.392308 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:03.392279 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819055-pck9z" event={"ID":"2d419112-60b9-4f74-ac53-8cdb8565d6f4","Type":"ContainerDied","Data":"f1a012af078979ea333add9fa017a9982d8fc0203d9d722da417b5847c1bd850"} Sep 11 16:15:03.392308 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:03.392305 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819055-pck9z" Sep 11 16:15:03.392437 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:03.392333 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f1a012af078979ea333add9fa017a9982d8fc0203d9d722da417b5847c1bd850" Sep 11 16:15:03.405181 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:03.405134 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-faa154ea-openshift-default-6db9786fc6-48297" Sep 11 16:15:03.451307 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:03.451277 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c8db87c-c55f-4c6a-91ba-e45267d81a93-workload-socket\") pod \"0c8db87c-c55f-4c6a-91ba-e45267d81a93\" (UID: \"0c8db87c-c55f-4c6a-91ba-e45267d81a93\") " Sep 11 16:15:03.451484 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:03.451325 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c8db87c-c55f-4c6a-91ba-e45267d81a93-istio-token\") pod \"0c8db87c-c55f-4c6a-91ba-e45267d81a93\" (UID: \"0c8db87c-c55f-4c6a-91ba-e45267d81a93\") " Sep 11 16:15:03.451484 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:03.451364 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c8db87c-c55f-4c6a-91ba-e45267d81a93-istio-podinfo\") pod \"0c8db87c-c55f-4c6a-91ba-e45267d81a93\" (UID: \"0c8db87c-c55f-4c6a-91ba-e45267d81a93\") " Sep 11 16:15:03.451484 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:03.451441 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rcz82\" (UniqueName: \"kubernetes.io/projected/0c8db87c-c55f-4c6a-91ba-e45267d81a93-kube-api-access-rcz82\") pod \"0c8db87c-c55f-4c6a-91ba-e45267d81a93\" (UID: \"0c8db87c-c55f-4c6a-91ba-e45267d81a93\") " Sep 11 16:15:03.451660 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:03.451500 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c8db87c-c55f-4c6a-91ba-e45267d81a93-istio-data\") pod \"0c8db87c-c55f-4c6a-91ba-e45267d81a93\" (UID: \"0c8db87c-c55f-4c6a-91ba-e45267d81a93\") " Sep 11 16:15:03.451660 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:03.451553 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c8db87c-c55f-4c6a-91ba-e45267d81a93-credential-socket\") pod \"0c8db87c-c55f-4c6a-91ba-e45267d81a93\" (UID: \"0c8db87c-c55f-4c6a-91ba-e45267d81a93\") " Sep 11 16:15:03.451660 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:03.451564 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c8db87c-c55f-4c6a-91ba-e45267d81a93-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0c8db87c-c55f-4c6a-91ba-e45267d81a93" (UID: "0c8db87c-c55f-4c6a-91ba-e45267d81a93"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 16:15:03.451660 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:03.451588 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c8db87c-c55f-4c6a-91ba-e45267d81a93-istio-envoy\") pod \"0c8db87c-c55f-4c6a-91ba-e45267d81a93\" (UID: \"0c8db87c-c55f-4c6a-91ba-e45267d81a93\") " Sep 11 16:15:03.451660 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:03.451620 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c8db87c-c55f-4c6a-91ba-e45267d81a93-istiod-ca-cert\") pod \"0c8db87c-c55f-4c6a-91ba-e45267d81a93\" (UID: \"0c8db87c-c55f-4c6a-91ba-e45267d81a93\") " Sep 11 16:15:03.451660 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:03.451650 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c8db87c-c55f-4c6a-91ba-e45267d81a93-workload-certs\") pod \"0c8db87c-c55f-4c6a-91ba-e45267d81a93\" (UID: \"0c8db87c-c55f-4c6a-91ba-e45267d81a93\") " Sep 11 16:15:03.451932 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:03.451816 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c8db87c-c55f-4c6a-91ba-e45267d81a93-istio-data" (OuterVolumeSpecName: "istio-data") pod "0c8db87c-c55f-4c6a-91ba-e45267d81a93" (UID: "0c8db87c-c55f-4c6a-91ba-e45267d81a93"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 16:15:03.452327 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:03.452055 2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c8db87c-c55f-4c6a-91ba-e45267d81a93-istio-data\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:15:03.452327 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:03.452077 2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c8db87c-c55f-4c6a-91ba-e45267d81a93-workload-socket\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:15:03.452327 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:03.452278 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c8db87c-c55f-4c6a-91ba-e45267d81a93-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0c8db87c-c55f-4c6a-91ba-e45267d81a93" (UID: "0c8db87c-c55f-4c6a-91ba-e45267d81a93"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 16:15:03.452327 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:03.452198 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c8db87c-c55f-4c6a-91ba-e45267d81a93-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0c8db87c-c55f-4c6a-91ba-e45267d81a93" (UID: "0c8db87c-c55f-4c6a-91ba-e45267d81a93"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 16:15:03.452327 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:03.452287 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c8db87c-c55f-4c6a-91ba-e45267d81a93-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0c8db87c-c55f-4c6a-91ba-e45267d81a93" (UID: "0c8db87c-c55f-4c6a-91ba-e45267d81a93"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 16:15:03.455042 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:03.455014 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c8db87c-c55f-4c6a-91ba-e45267d81a93-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0c8db87c-c55f-4c6a-91ba-e45267d81a93" (UID: "0c8db87c-c55f-4c6a-91ba-e45267d81a93"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 16:15:03.455142 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:03.455043 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c8db87c-c55f-4c6a-91ba-e45267d81a93-istio-token" (OuterVolumeSpecName: "istio-token") pod "0c8db87c-c55f-4c6a-91ba-e45267d81a93" (UID: "0c8db87c-c55f-4c6a-91ba-e45267d81a93"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 16:15:03.455142 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:03.455122 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0c8db87c-c55f-4c6a-91ba-e45267d81a93-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0c8db87c-c55f-4c6a-91ba-e45267d81a93" (UID: "0c8db87c-c55f-4c6a-91ba-e45267d81a93"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 16:15:03.455142 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:03.455120 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c8db87c-c55f-4c6a-91ba-e45267d81a93-kube-api-access-rcz82" (OuterVolumeSpecName: "kube-api-access-rcz82") pod "0c8db87c-c55f-4c6a-91ba-e45267d81a93" (UID: "0c8db87c-c55f-4c6a-91ba-e45267d81a93"). InnerVolumeSpecName "kube-api-access-rcz82". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 16:15:03.544113 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:03.544088 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8e3121-29819055-vsdcm" Sep 11 16:15:03.553298 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:03.553275 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rcz82\" (UniqueName: \"kubernetes.io/projected/0c8db87c-c55f-4c6a-91ba-e45267d81a93-kube-api-access-rcz82\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:15:03.553298 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:03.553299 2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c8db87c-c55f-4c6a-91ba-e45267d81a93-credential-socket\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:15:03.553431 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:03.553310 2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c8db87c-c55f-4c6a-91ba-e45267d81a93-istio-envoy\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:15:03.553431 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:03.553321 2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c8db87c-c55f-4c6a-91ba-e45267d81a93-istiod-ca-cert\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:15:03.553431 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:03.553329 2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c8db87c-c55f-4c6a-91ba-e45267d81a93-workload-certs\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:15:03.553431 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:03.553337 2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c8db87c-c55f-4c6a-91ba-e45267d81a93-istio-token\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:15:03.553431 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:03.553345 2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c8db87c-c55f-4c6a-91ba-e45267d81a93-istio-podinfo\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:15:04.399472 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:04.399443 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8e3121-29819055-vsdcm" Sep 11 16:15:04.399858 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:04.399482 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8e3121-29819055-vsdcm" event={"ID":"c879bcf5-0a2e-44cf-a951-51fc1da22fe9","Type":"ContainerDied","Data":"6582d0f62b6efd6cef1f9ad62931f9a785f6de77d93a88cd84a002fa07abc09d"} Sep 11 16:15:04.399858 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:04.399528 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6582d0f62b6efd6cef1f9ad62931f9a785f6de77d93a88cd84a002fa07abc09d" Sep 11 16:15:04.400948 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:04.400927 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-faa154ea-openshift-default-6db9786fc6-48297" Sep 11 16:15:04.454618 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:04.454580 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-faa154ea-openshift-default-6db9786fc6-48297"] Sep 11 16:15:04.461143 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:04.461115 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-faa154ea-openshift-default-6db9786fc6-48297"] Sep 11 16:15:05.239132 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:05.239100 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c8db87c-c55f-4c6a-91ba-e45267d81a93" path="/var/lib/kubelet/pods/0c8db87c-c55f-4c6a-91ba-e45267d81a93/volumes" Sep 11 16:15:06.725702 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:06.725664 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8e3121-29819055-vsdcm"] Sep 11 16:15:06.728938 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:06.728912 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8e3121-29819055-vsdcm"] Sep 11 16:15:07.042785 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:07.042744 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8e3121-m6q8q"] Sep 11 16:15:07.045406 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:07.045382 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8e3121-m6q8q"] Sep 11 16:15:07.239038 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:07.239000 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c879bcf5-0a2e-44cf-a951-51fc1da22fe9" path="/var/lib/kubelet/pods/c879bcf5-0a2e-44cf-a951-51fc1da22fe9/volumes" Sep 11 16:15:07.239345 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:07.239330 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9973186-448e-4f04-92e8-aae31e05d891" path="/var/lib/kubelet/pods/e9973186-448e-4f04-92e8-aae31e05d891/volumes" Sep 11 16:15:10.014602 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:10.014563 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dx4xn"] Sep 11 16:15:14.029846 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:14.029811 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-f7c21d-openshift-default-857f5cd77b-8phc5"] Sep 11 16:15:14.030249 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:14.030236 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c8db87c-c55f-4c6a-91ba-e45267d81a93" containerName="istio-proxy" Sep 11 16:15:14.030249 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:14.030248 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c8db87c-c55f-4c6a-91ba-e45267d81a93" containerName="istio-proxy" Sep 11 16:15:14.030324 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:14.030267 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e28ddf9-c591-47d2-964b-8fde681c414d" containerName="istio-proxy" Sep 11 16:15:14.030324 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:14.030272 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e28ddf9-c591-47d2-964b-8fde681c414d" containerName="istio-proxy" Sep 11 16:15:14.030324 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:14.030281 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d419112-60b9-4f74-ac53-8cdb8565d6f4" containerName="cleanup" Sep 11 16:15:14.030324 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:14.030287 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d419112-60b9-4f74-ac53-8cdb8565d6f4" containerName="cleanup" Sep 11 16:15:14.030324 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:14.030298 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c879bcf5-0a2e-44cf-a951-51fc1da22fe9" containerName="cleanup" Sep 11 16:15:14.030324 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:14.030304 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="c879bcf5-0a2e-44cf-a951-51fc1da22fe9" containerName="cleanup" Sep 11 16:15:14.030509 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:14.030370 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e28ddf9-c591-47d2-964b-8fde681c414d" containerName="istio-proxy" Sep 11 16:15:14.030509 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:14.030380 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d419112-60b9-4f74-ac53-8cdb8565d6f4" containerName="cleanup" Sep 11 16:15:14.030509 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:14.030386 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="c879bcf5-0a2e-44cf-a951-51fc1da22fe9" containerName="cleanup" Sep 11 16:15:14.030509 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:14.030394 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c8db87c-c55f-4c6a-91ba-e45267d81a93" containerName="istio-proxy" Sep 11 16:15:14.030509 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:14.030400 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c8db87c-c55f-4c6a-91ba-e45267d81a93" containerName="istio-proxy" Sep 11 16:15:14.034693 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:14.034661 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-f7c21d-openshift-default-857f5cd77b-8phc5" Sep 11 16:15:14.040300 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:14.040269 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-f7c21d-openshift-default-dockercfg-hfsjj\"" Sep 11 16:15:14.049734 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:14.049706 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-f7c21d-openshift-default-857f5cd77b-8phc5"] Sep 11 16:15:14.150435 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:14.150402 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b78c6245-c285-479b-a463-42d55ce25b74-workload-socket\") pod \"e2e-shared-a-w2-f7c21d-openshift-default-857f5cd77b-8phc5\" (UID: \"b78c6245-c285-479b-a463-42d55ce25b74\") " pod="openshift-ingress/e2e-shared-a-w2-f7c21d-openshift-default-857f5cd77b-8phc5" Sep 11 16:15:14.150435 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:14.150437 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b78c6245-c285-479b-a463-42d55ce25b74-istio-envoy\") pod \"e2e-shared-a-w2-f7c21d-openshift-default-857f5cd77b-8phc5\" (UID: \"b78c6245-c285-479b-a463-42d55ce25b74\") " pod="openshift-ingress/e2e-shared-a-w2-f7c21d-openshift-default-857f5cd77b-8phc5" Sep 11 16:15:14.150673 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:14.150491 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xtxl\" (UniqueName: \"kubernetes.io/projected/b78c6245-c285-479b-a463-42d55ce25b74-kube-api-access-8xtxl\") pod \"e2e-shared-a-w2-f7c21d-openshift-default-857f5cd77b-8phc5\" (UID: \"b78c6245-c285-479b-a463-42d55ce25b74\") " pod="openshift-ingress/e2e-shared-a-w2-f7c21d-openshift-default-857f5cd77b-8phc5" Sep 11 16:15:14.150673 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:14.150530 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b78c6245-c285-479b-a463-42d55ce25b74-istio-token\") pod \"e2e-shared-a-w2-f7c21d-openshift-default-857f5cd77b-8phc5\" (UID: \"b78c6245-c285-479b-a463-42d55ce25b74\") " pod="openshift-ingress/e2e-shared-a-w2-f7c21d-openshift-default-857f5cd77b-8phc5" Sep 11 16:15:14.150673 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:14.150570 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b78c6245-c285-479b-a463-42d55ce25b74-credential-socket\") pod \"e2e-shared-a-w2-f7c21d-openshift-default-857f5cd77b-8phc5\" (UID: \"b78c6245-c285-479b-a463-42d55ce25b74\") " pod="openshift-ingress/e2e-shared-a-w2-f7c21d-openshift-default-857f5cd77b-8phc5" Sep 11 16:15:14.150673 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:14.150589 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b78c6245-c285-479b-a463-42d55ce25b74-istiod-ca-cert\") pod \"e2e-shared-a-w2-f7c21d-openshift-default-857f5cd77b-8phc5\" (UID: \"b78c6245-c285-479b-a463-42d55ce25b74\") " pod="openshift-ingress/e2e-shared-a-w2-f7c21d-openshift-default-857f5cd77b-8phc5" Sep 11 16:15:14.150673 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:14.150610 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b78c6245-c285-479b-a463-42d55ce25b74-workload-certs\") pod \"e2e-shared-a-w2-f7c21d-openshift-default-857f5cd77b-8phc5\" (UID: \"b78c6245-c285-479b-a463-42d55ce25b74\") " pod="openshift-ingress/e2e-shared-a-w2-f7c21d-openshift-default-857f5cd77b-8phc5" Sep 11 16:15:14.150894 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:14.150677 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b78c6245-c285-479b-a463-42d55ce25b74-istio-data\") pod \"e2e-shared-a-w2-f7c21d-openshift-default-857f5cd77b-8phc5\" (UID: \"b78c6245-c285-479b-a463-42d55ce25b74\") " pod="openshift-ingress/e2e-shared-a-w2-f7c21d-openshift-default-857f5cd77b-8phc5" Sep 11 16:15:14.150894 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:14.150703 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b78c6245-c285-479b-a463-42d55ce25b74-istio-podinfo\") pod \"e2e-shared-a-w2-f7c21d-openshift-default-857f5cd77b-8phc5\" (UID: \"b78c6245-c285-479b-a463-42d55ce25b74\") " pod="openshift-ingress/e2e-shared-a-w2-f7c21d-openshift-default-857f5cd77b-8phc5" Sep 11 16:15:14.251290 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:14.251241 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b78c6245-c285-479b-a463-42d55ce25b74-istio-data\") pod \"e2e-shared-a-w2-f7c21d-openshift-default-857f5cd77b-8phc5\" (UID: \"b78c6245-c285-479b-a463-42d55ce25b74\") " pod="openshift-ingress/e2e-shared-a-w2-f7c21d-openshift-default-857f5cd77b-8phc5" Sep 11 16:15:14.251290 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:14.251290 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b78c6245-c285-479b-a463-42d55ce25b74-istio-podinfo\") pod \"e2e-shared-a-w2-f7c21d-openshift-default-857f5cd77b-8phc5\" (UID: \"b78c6245-c285-479b-a463-42d55ce25b74\") " pod="openshift-ingress/e2e-shared-a-w2-f7c21d-openshift-default-857f5cd77b-8phc5" Sep 11 16:15:14.251560 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:14.251346 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b78c6245-c285-479b-a463-42d55ce25b74-workload-socket\") pod \"e2e-shared-a-w2-f7c21d-openshift-default-857f5cd77b-8phc5\" (UID: \"b78c6245-c285-479b-a463-42d55ce25b74\") " pod="openshift-ingress/e2e-shared-a-w2-f7c21d-openshift-default-857f5cd77b-8phc5" Sep 11 16:15:14.251560 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:14.251366 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b78c6245-c285-479b-a463-42d55ce25b74-istio-envoy\") pod \"e2e-shared-a-w2-f7c21d-openshift-default-857f5cd77b-8phc5\" (UID: \"b78c6245-c285-479b-a463-42d55ce25b74\") " pod="openshift-ingress/e2e-shared-a-w2-f7c21d-openshift-default-857f5cd77b-8phc5" Sep 11 16:15:14.251560 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:14.251403 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xtxl\" (UniqueName: \"kubernetes.io/projected/b78c6245-c285-479b-a463-42d55ce25b74-kube-api-access-8xtxl\") pod \"e2e-shared-a-w2-f7c21d-openshift-default-857f5cd77b-8phc5\" (UID: \"b78c6245-c285-479b-a463-42d55ce25b74\") " pod="openshift-ingress/e2e-shared-a-w2-f7c21d-openshift-default-857f5cd77b-8phc5" Sep 11 16:15:14.251560 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:14.251439 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b78c6245-c285-479b-a463-42d55ce25b74-istio-token\") pod \"e2e-shared-a-w2-f7c21d-openshift-default-857f5cd77b-8phc5\" (UID: \"b78c6245-c285-479b-a463-42d55ce25b74\") " pod="openshift-ingress/e2e-shared-a-w2-f7c21d-openshift-default-857f5cd77b-8phc5" Sep 11 16:15:14.251560 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:14.251485 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b78c6245-c285-479b-a463-42d55ce25b74-credential-socket\") pod \"e2e-shared-a-w2-f7c21d-openshift-default-857f5cd77b-8phc5\" (UID: \"b78c6245-c285-479b-a463-42d55ce25b74\") " pod="openshift-ingress/e2e-shared-a-w2-f7c21d-openshift-default-857f5cd77b-8phc5" Sep 11 16:15:14.251560 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:14.251512 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b78c6245-c285-479b-a463-42d55ce25b74-istiod-ca-cert\") pod \"e2e-shared-a-w2-f7c21d-openshift-default-857f5cd77b-8phc5\" (UID: \"b78c6245-c285-479b-a463-42d55ce25b74\") " pod="openshift-ingress/e2e-shared-a-w2-f7c21d-openshift-default-857f5cd77b-8phc5" Sep 11 16:15:14.251560 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:14.251544 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b78c6245-c285-479b-a463-42d55ce25b74-workload-certs\") pod \"e2e-shared-a-w2-f7c21d-openshift-default-857f5cd77b-8phc5\" (UID: \"b78c6245-c285-479b-a463-42d55ce25b74\") " pod="openshift-ingress/e2e-shared-a-w2-f7c21d-openshift-default-857f5cd77b-8phc5" Sep 11 16:15:14.252011 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:14.251808 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b78c6245-c285-479b-a463-42d55ce25b74-workload-socket\") pod \"e2e-shared-a-w2-f7c21d-openshift-default-857f5cd77b-8phc5\" (UID: \"b78c6245-c285-479b-a463-42d55ce25b74\") " pod="openshift-ingress/e2e-shared-a-w2-f7c21d-openshift-default-857f5cd77b-8phc5" Sep 11 16:15:14.252011 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:14.251908 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b78c6245-c285-479b-a463-42d55ce25b74-credential-socket\") pod \"e2e-shared-a-w2-f7c21d-openshift-default-857f5cd77b-8phc5\" (UID: \"b78c6245-c285-479b-a463-42d55ce25b74\") " pod="openshift-ingress/e2e-shared-a-w2-f7c21d-openshift-default-857f5cd77b-8phc5" Sep 11 16:15:14.252011 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:14.251919 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b78c6245-c285-479b-a463-42d55ce25b74-istio-data\") pod \"e2e-shared-a-w2-f7c21d-openshift-default-857f5cd77b-8phc5\" (UID: \"b78c6245-c285-479b-a463-42d55ce25b74\") " pod="openshift-ingress/e2e-shared-a-w2-f7c21d-openshift-default-857f5cd77b-8phc5" Sep 11 16:15:14.252011 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:14.251939 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b78c6245-c285-479b-a463-42d55ce25b74-workload-certs\") pod \"e2e-shared-a-w2-f7c21d-openshift-default-857f5cd77b-8phc5\" (UID: \"b78c6245-c285-479b-a463-42d55ce25b74\") " pod="openshift-ingress/e2e-shared-a-w2-f7c21d-openshift-default-857f5cd77b-8phc5" Sep 11 16:15:14.252353 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:14.252334 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b78c6245-c285-479b-a463-42d55ce25b74-istiod-ca-cert\") pod \"e2e-shared-a-w2-f7c21d-openshift-default-857f5cd77b-8phc5\" (UID: \"b78c6245-c285-479b-a463-42d55ce25b74\") " pod="openshift-ingress/e2e-shared-a-w2-f7c21d-openshift-default-857f5cd77b-8phc5" Sep 11 16:15:14.253836 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:14.253810 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b78c6245-c285-479b-a463-42d55ce25b74-istio-envoy\") pod \"e2e-shared-a-w2-f7c21d-openshift-default-857f5cd77b-8phc5\" (UID: \"b78c6245-c285-479b-a463-42d55ce25b74\") " pod="openshift-ingress/e2e-shared-a-w2-f7c21d-openshift-default-857f5cd77b-8phc5" Sep 11 16:15:14.254020 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:14.254002 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b78c6245-c285-479b-a463-42d55ce25b74-istio-podinfo\") pod \"e2e-shared-a-w2-f7c21d-openshift-default-857f5cd77b-8phc5\" (UID: \"b78c6245-c285-479b-a463-42d55ce25b74\") " pod="openshift-ingress/e2e-shared-a-w2-f7c21d-openshift-default-857f5cd77b-8phc5" Sep 11 16:15:14.260293 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:14.260265 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b78c6245-c285-479b-a463-42d55ce25b74-istio-token\") pod \"e2e-shared-a-w2-f7c21d-openshift-default-857f5cd77b-8phc5\" (UID: \"b78c6245-c285-479b-a463-42d55ce25b74\") " pod="openshift-ingress/e2e-shared-a-w2-f7c21d-openshift-default-857f5cd77b-8phc5" Sep 11 16:15:14.260754 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:14.260731 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xtxl\" (UniqueName: \"kubernetes.io/projected/b78c6245-c285-479b-a463-42d55ce25b74-kube-api-access-8xtxl\") pod \"e2e-shared-a-w2-f7c21d-openshift-default-857f5cd77b-8phc5\" (UID: \"b78c6245-c285-479b-a463-42d55ce25b74\") " pod="openshift-ingress/e2e-shared-a-w2-f7c21d-openshift-default-857f5cd77b-8phc5" Sep 11 16:15:14.345903 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:14.345817 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-f7c21d-openshift-default-857f5cd77b-8phc5" Sep 11 16:15:14.636236 ip-10-0-136-96 kubenswrapper[2566]: W0911 16:15:14.636199 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb78c6245_c285_479b_a463_42d55ce25b74.slice/crio-120e39fb87a1c08ca5ca7136985ad81999e618a19c1e0204c365862bf744d44a WatchSource:0}: Error finding container 120e39fb87a1c08ca5ca7136985ad81999e618a19c1e0204c365862bf744d44a: Status 404 returned error can't find the container with id 120e39fb87a1c08ca5ca7136985ad81999e618a19c1e0204c365862bf744d44a Sep 11 16:15:14.636698 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:14.636671 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-f7c21d-openshift-default-857f5cd77b-8phc5"] Sep 11 16:15:14.638726 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:14.638688 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 16:15:14.638815 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:14.638770 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 16:15:14.638867 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:14.638816 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 16:15:14.867234 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:14.867196 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-8e3121-openshift-default-7f6ffd865d-vbjmp"] Sep 11 16:15:14.867451 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:14.867410 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-8e3121-openshift-default-7f6ffd865d-vbjmp" podUID="96800483-9c43-4f65-b7ae-a46b5a7d35b7" containerName="istio-proxy" containerID="cri-o://227e4736d0c7482ecc2421407ce9fabf242e373c74b2ef273586d5e88cfd6b49" gracePeriod=30 Sep 11 16:15:15.475710 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:15.475681 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-f7c21d-openshift-default-857f5cd77b-8phc5" event={"ID":"b78c6245-c285-479b-a463-42d55ce25b74","Type":"ContainerStarted","Data":"a4855cdb754886d540398e0300ee96f9d5a9f44587c49318fa62a1db4df3f4a7"} Sep 11 16:15:15.476134 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:15.475729 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-f7c21d-openshift-default-857f5cd77b-8phc5" event={"ID":"b78c6245-c285-479b-a463-42d55ce25b74","Type":"ContainerStarted","Data":"120e39fb87a1c08ca5ca7136985ad81999e618a19c1e0204c365862bf744d44a"} Sep 11 16:15:15.500381 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:15.500328 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-f7c21d-openshift-default-857f5cd77b-8phc5" podStartSLOduration=1.500311821 podStartE2EDuration="1.500311821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:15:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:15:15.498066141 +0000 UTC m=+1222.712900046" watchObservedRunningTime="2026-09-11 16:15:15.500311821 +0000 UTC m=+1222.715145726" Sep 11 16:15:16.346050 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:16.346007 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-f7c21d-openshift-default-857f5cd77b-8phc5" Sep 11 16:15:16.351362 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:16.351336 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-f7c21d-openshift-default-857f5cd77b-8phc5" Sep 11 16:15:16.482152 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:16.482119 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-f7c21d-openshift-default-857f5cd77b-8phc5" Sep 11 16:15:16.483244 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:16.483219 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-f7c21d-openshift-default-857f5cd77b-8phc5" Sep 11 16:15:20.125508 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:20.125478 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8e3121-openshift-default-7f6ffd865d-vbjmp" Sep 11 16:15:20.215653 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:20.215564 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96800483-9c43-4f65-b7ae-a46b5a7d35b7-istio-podinfo\") pod \"96800483-9c43-4f65-b7ae-a46b5a7d35b7\" (UID: \"96800483-9c43-4f65-b7ae-a46b5a7d35b7\") " Sep 11 16:15:20.215653 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:20.215614 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96800483-9c43-4f65-b7ae-a46b5a7d35b7-workload-socket\") pod \"96800483-9c43-4f65-b7ae-a46b5a7d35b7\" (UID: \"96800483-9c43-4f65-b7ae-a46b5a7d35b7\") " Sep 11 16:15:20.215653 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:20.215642 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96800483-9c43-4f65-b7ae-a46b5a7d35b7-istio-token\") pod \"96800483-9c43-4f65-b7ae-a46b5a7d35b7\" (UID: \"96800483-9c43-4f65-b7ae-a46b5a7d35b7\") " Sep 11 16:15:20.215920 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:20.215687 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96800483-9c43-4f65-b7ae-a46b5a7d35b7-istio-data\") pod \"96800483-9c43-4f65-b7ae-a46b5a7d35b7\" (UID: \"96800483-9c43-4f65-b7ae-a46b5a7d35b7\") " Sep 11 16:15:20.215920 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:20.215709 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96800483-9c43-4f65-b7ae-a46b5a7d35b7-workload-certs\") pod \"96800483-9c43-4f65-b7ae-a46b5a7d35b7\" (UID: \"96800483-9c43-4f65-b7ae-a46b5a7d35b7\") " Sep 11 16:15:20.215920 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:20.215764 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96800483-9c43-4f65-b7ae-a46b5a7d35b7-credential-socket\") pod \"96800483-9c43-4f65-b7ae-a46b5a7d35b7\" (UID: \"96800483-9c43-4f65-b7ae-a46b5a7d35b7\") " Sep 11 16:15:20.215920 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:20.215822 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96800483-9c43-4f65-b7ae-a46b5a7d35b7-istio-envoy\") pod \"96800483-9c43-4f65-b7ae-a46b5a7d35b7\" (UID: \"96800483-9c43-4f65-b7ae-a46b5a7d35b7\") " Sep 11 16:15:20.215920 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:20.215853 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96800483-9c43-4f65-b7ae-a46b5a7d35b7-istiod-ca-cert\") pod \"96800483-9c43-4f65-b7ae-a46b5a7d35b7\" (UID: \"96800483-9c43-4f65-b7ae-a46b5a7d35b7\") " Sep 11 16:15:20.215920 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:20.215877 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s5ffk\" (UniqueName: \"kubernetes.io/projected/96800483-9c43-4f65-b7ae-a46b5a7d35b7-kube-api-access-s5ffk\") pod \"96800483-9c43-4f65-b7ae-a46b5a7d35b7\" (UID: \"96800483-9c43-4f65-b7ae-a46b5a7d35b7\") " Sep 11 16:15:20.216257 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:20.215941 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96800483-9c43-4f65-b7ae-a46b5a7d35b7-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "96800483-9c43-4f65-b7ae-a46b5a7d35b7" (UID: "96800483-9c43-4f65-b7ae-a46b5a7d35b7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 16:15:20.216257 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:20.216054 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96800483-9c43-4f65-b7ae-a46b5a7d35b7-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "96800483-9c43-4f65-b7ae-a46b5a7d35b7" (UID: "96800483-9c43-4f65-b7ae-a46b5a7d35b7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 16:15:20.216257 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:20.216068 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96800483-9c43-4f65-b7ae-a46b5a7d35b7-istio-data" (OuterVolumeSpecName: "istio-data") pod "96800483-9c43-4f65-b7ae-a46b5a7d35b7" (UID: "96800483-9c43-4f65-b7ae-a46b5a7d35b7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 16:15:20.216257 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:20.216089 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96800483-9c43-4f65-b7ae-a46b5a7d35b7-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "96800483-9c43-4f65-b7ae-a46b5a7d35b7" (UID: "96800483-9c43-4f65-b7ae-a46b5a7d35b7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 16:15:20.216470 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:20.216308 2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96800483-9c43-4f65-b7ae-a46b5a7d35b7-workload-socket\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:15:20.216470 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:20.216330 2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96800483-9c43-4f65-b7ae-a46b5a7d35b7-istio-data\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:15:20.216470 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:20.216345 2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96800483-9c43-4f65-b7ae-a46b5a7d35b7-workload-certs\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:15:20.216470 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:20.216357 2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96800483-9c43-4f65-b7ae-a46b5a7d35b7-credential-socket\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:15:20.216470 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:20.216394 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96800483-9c43-4f65-b7ae-a46b5a7d35b7-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "96800483-9c43-4f65-b7ae-a46b5a7d35b7" (UID: "96800483-9c43-4f65-b7ae-a46b5a7d35b7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 16:15:20.217899 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:20.217872 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/96800483-9c43-4f65-b7ae-a46b5a7d35b7-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "96800483-9c43-4f65-b7ae-a46b5a7d35b7" (UID: "96800483-9c43-4f65-b7ae-a46b5a7d35b7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 16:15:20.218029 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:20.217908 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96800483-9c43-4f65-b7ae-a46b5a7d35b7-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "96800483-9c43-4f65-b7ae-a46b5a7d35b7" (UID: "96800483-9c43-4f65-b7ae-a46b5a7d35b7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 16:15:20.218165 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:20.218148 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96800483-9c43-4f65-b7ae-a46b5a7d35b7-istio-token" (OuterVolumeSpecName: "istio-token") pod "96800483-9c43-4f65-b7ae-a46b5a7d35b7" (UID: "96800483-9c43-4f65-b7ae-a46b5a7d35b7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 16:15:20.218209 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:20.218168 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96800483-9c43-4f65-b7ae-a46b5a7d35b7-kube-api-access-s5ffk" (OuterVolumeSpecName: "kube-api-access-s5ffk") pod "96800483-9c43-4f65-b7ae-a46b5a7d35b7" (UID: "96800483-9c43-4f65-b7ae-a46b5a7d35b7"). InnerVolumeSpecName "kube-api-access-s5ffk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 16:15:20.317560 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:20.317524 2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96800483-9c43-4f65-b7ae-a46b5a7d35b7-istio-envoy\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:15:20.317560 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:20.317556 2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96800483-9c43-4f65-b7ae-a46b5a7d35b7-istiod-ca-cert\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:15:20.317560 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:20.317566 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5ffk\" (UniqueName: \"kubernetes.io/projected/96800483-9c43-4f65-b7ae-a46b5a7d35b7-kube-api-access-s5ffk\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:15:20.317795 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:20.317574 2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96800483-9c43-4f65-b7ae-a46b5a7d35b7-istio-podinfo\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:15:20.317795 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:20.317583 2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96800483-9c43-4f65-b7ae-a46b5a7d35b7-istio-token\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:15:20.509163 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:20.509069 2566 generic.go:356] "Generic (PLEG): container finished" podID="96800483-9c43-4f65-b7ae-a46b5a7d35b7" containerID="227e4736d0c7482ecc2421407ce9fabf242e373c74b2ef273586d5e88cfd6b49" exitCode=0 Sep 11 16:15:20.509163 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:20.509120 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8e3121-openshift-default-7f6ffd865d-vbjmp" Sep 11 16:15:20.509404 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:20.509157 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8e3121-openshift-default-7f6ffd865d-vbjmp" event={"ID":"96800483-9c43-4f65-b7ae-a46b5a7d35b7","Type":"ContainerDied","Data":"227e4736d0c7482ecc2421407ce9fabf242e373c74b2ef273586d5e88cfd6b49"} Sep 11 16:15:20.509404 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:20.509214 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8e3121-openshift-default-7f6ffd865d-vbjmp" event={"ID":"96800483-9c43-4f65-b7ae-a46b5a7d35b7","Type":"ContainerDied","Data":"05b6ce85583b9d60218f4e73111033c98d9e72f69ea085eb86077c647ca8c8aa"} Sep 11 16:15:20.509404 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:20.509244 2566 scope.go:117] "RemoveContainer" containerID="227e4736d0c7482ecc2421407ce9fabf242e373c74b2ef273586d5e88cfd6b49" Sep 11 16:15:20.519795 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:20.519779 2566 scope.go:117] "RemoveContainer" containerID="7e91796dff1eebcc4d261b64ebc8b043bec53348f548846eddf543ffa770ce4e" Sep 11 16:15:20.529413 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:20.529395 2566 scope.go:117] "RemoveContainer" containerID="227e4736d0c7482ecc2421407ce9fabf242e373c74b2ef273586d5e88cfd6b49" Sep 11 16:15:20.529682 ip-10-0-136-96 kubenswrapper[2566]: E0911 16:15:20.529665 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"227e4736d0c7482ecc2421407ce9fabf242e373c74b2ef273586d5e88cfd6b49\": container with ID starting with 227e4736d0c7482ecc2421407ce9fabf242e373c74b2ef273586d5e88cfd6b49 not found: ID does not exist" containerID="227e4736d0c7482ecc2421407ce9fabf242e373c74b2ef273586d5e88cfd6b49" Sep 11 16:15:20.529729 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:20.529690 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"227e4736d0c7482ecc2421407ce9fabf242e373c74b2ef273586d5e88cfd6b49"} err="failed to get container status \"227e4736d0c7482ecc2421407ce9fabf242e373c74b2ef273586d5e88cfd6b49\": rpc error: code = NotFound desc = could not find container \"227e4736d0c7482ecc2421407ce9fabf242e373c74b2ef273586d5e88cfd6b49\": container with ID starting with 227e4736d0c7482ecc2421407ce9fabf242e373c74b2ef273586d5e88cfd6b49 not found: ID does not exist" Sep 11 16:15:20.529729 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:20.529707 2566 scope.go:117] "RemoveContainer" containerID="7e91796dff1eebcc4d261b64ebc8b043bec53348f548846eddf543ffa770ce4e" Sep 11 16:15:20.530008 ip-10-0-136-96 kubenswrapper[2566]: E0911 16:15:20.529951 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e91796dff1eebcc4d261b64ebc8b043bec53348f548846eddf543ffa770ce4e\": container with ID starting with 7e91796dff1eebcc4d261b64ebc8b043bec53348f548846eddf543ffa770ce4e not found: ID does not exist" containerID="7e91796dff1eebcc4d261b64ebc8b043bec53348f548846eddf543ffa770ce4e" Sep 11 16:15:20.530008 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:20.529995 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e91796dff1eebcc4d261b64ebc8b043bec53348f548846eddf543ffa770ce4e"} err="failed to get container status \"7e91796dff1eebcc4d261b64ebc8b043bec53348f548846eddf543ffa770ce4e\": rpc error: code = NotFound desc = could not find container \"7e91796dff1eebcc4d261b64ebc8b043bec53348f548846eddf543ffa770ce4e\": container with ID starting with 7e91796dff1eebcc4d261b64ebc8b043bec53348f548846eddf543ffa770ce4e not found: ID does not exist" Sep 11 16:15:20.536580 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:20.536556 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-8e3121-openshift-default-7f6ffd865d-vbjmp"] Sep 11 16:15:20.541212 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:20.541191 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-8e3121-openshift-default-7f6ffd865d-vbjmp"] Sep 11 16:15:21.239142 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:21.239111 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="96800483-9c43-4f65-b7ae-a46b5a7d35b7" path="/var/lib/kubelet/pods/96800483-9c43-4f65-b7ae-a46b5a7d35b7/volumes" Sep 11 16:15:23.719998 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:23.719950 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dx4xn"] Sep 11 16:15:31.028325 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:31.028282 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dx4xn"] Sep 11 16:15:32.523155 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:32.523113 2566 patch_prober.go:28] interesting pod/e2e-shared-a-w2-f7c21d-openshift-default-857f5cd77b-8phc5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 11 16:15:32.523515 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:32.523193 2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-f7c21d-openshift-default-857f5cd77b-8phc5" podUID="b78c6245-c285-479b-a463-42d55ce25b74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 11 16:15:32.852058 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:32.851948 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-2b1bd5-openshift-default-687f66dffb-jdh2v"] Sep 11 16:15:32.852260 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:32.852235 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-2b1bd5-openshift-default-687f66dffb-jdh2v" podUID="c311820f-99df-4371-8967-d5518a1e299b" containerName="istio-proxy" containerID="cri-o://5cb2622d6ae86291b94297e89ff86797e65bb74ca9d7c7b38e99a4f375dbbd24" gracePeriod=30 Sep 11 16:15:33.659403 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:33.659372 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-59f108-openshift-default-6cc7dd95cd-c4pmd"] Sep 11 16:15:33.660846 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:33.659803 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96800483-9c43-4f65-b7ae-a46b5a7d35b7" containerName="istio-proxy" Sep 11 16:15:33.660846 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:33.659816 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="96800483-9c43-4f65-b7ae-a46b5a7d35b7" containerName="istio-proxy" Sep 11 16:15:33.660846 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:33.659824 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96800483-9c43-4f65-b7ae-a46b5a7d35b7" containerName="istio-proxy" Sep 11 16:15:33.660846 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:33.659830 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="96800483-9c43-4f65-b7ae-a46b5a7d35b7" containerName="istio-proxy" Sep 11 16:15:33.660846 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:33.659845 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c8db87c-c55f-4c6a-91ba-e45267d81a93" containerName="istio-proxy" Sep 11 16:15:33.660846 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:33.659851 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c8db87c-c55f-4c6a-91ba-e45267d81a93" containerName="istio-proxy" Sep 11 16:15:33.660846 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:33.659915 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="96800483-9c43-4f65-b7ae-a46b5a7d35b7" containerName="istio-proxy" Sep 11 16:15:33.660846 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:33.659927 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="96800483-9c43-4f65-b7ae-a46b5a7d35b7" containerName="istio-proxy" Sep 11 16:15:33.661981 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:33.661953 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-59f108-openshift-default-6cc7dd95cd-c4pmd" Sep 11 16:15:33.665599 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:33.665579 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-59f108-openshift-default-dockercfg-j7vmk\"" Sep 11 16:15:33.688903 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:33.688871 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-59f108-openshift-default-6cc7dd95cd-c4pmd"] Sep 11 16:15:33.748230 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:33.748203 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8716cbf6-b788-4cf2-adb4-e4b4e400a725-istio-token\") pod \"e2e-shared-a-w3-59f108-openshift-default-6cc7dd95cd-c4pmd\" (UID: \"8716cbf6-b788-4cf2-adb4-e4b4e400a725\") " pod="openshift-ingress/e2e-shared-a-w3-59f108-openshift-default-6cc7dd95cd-c4pmd" Sep 11 16:15:33.748559 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:33.748270 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tx7ng\" (UniqueName: \"kubernetes.io/projected/8716cbf6-b788-4cf2-adb4-e4b4e400a725-kube-api-access-tx7ng\") pod \"e2e-shared-a-w3-59f108-openshift-default-6cc7dd95cd-c4pmd\" (UID: \"8716cbf6-b788-4cf2-adb4-e4b4e400a725\") " pod="openshift-ingress/e2e-shared-a-w3-59f108-openshift-default-6cc7dd95cd-c4pmd" Sep 11 16:15:33.748559 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:33.748314 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8716cbf6-b788-4cf2-adb4-e4b4e400a725-istio-envoy\") pod \"e2e-shared-a-w3-59f108-openshift-default-6cc7dd95cd-c4pmd\" (UID: \"8716cbf6-b788-4cf2-adb4-e4b4e400a725\") " pod="openshift-ingress/e2e-shared-a-w3-59f108-openshift-default-6cc7dd95cd-c4pmd" Sep 11 16:15:33.748559 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:33.748337 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8716cbf6-b788-4cf2-adb4-e4b4e400a725-istio-podinfo\") pod \"e2e-shared-a-w3-59f108-openshift-default-6cc7dd95cd-c4pmd\" (UID: \"8716cbf6-b788-4cf2-adb4-e4b4e400a725\") " pod="openshift-ingress/e2e-shared-a-w3-59f108-openshift-default-6cc7dd95cd-c4pmd" Sep 11 16:15:33.748559 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:33.748420 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8716cbf6-b788-4cf2-adb4-e4b4e400a725-workload-certs\") pod \"e2e-shared-a-w3-59f108-openshift-default-6cc7dd95cd-c4pmd\" (UID: \"8716cbf6-b788-4cf2-adb4-e4b4e400a725\") " pod="openshift-ingress/e2e-shared-a-w3-59f108-openshift-default-6cc7dd95cd-c4pmd" Sep 11 16:15:33.748559 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:33.748464 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8716cbf6-b788-4cf2-adb4-e4b4e400a725-istiod-ca-cert\") pod \"e2e-shared-a-w3-59f108-openshift-default-6cc7dd95cd-c4pmd\" (UID: \"8716cbf6-b788-4cf2-adb4-e4b4e400a725\") " pod="openshift-ingress/e2e-shared-a-w3-59f108-openshift-default-6cc7dd95cd-c4pmd" Sep 11 16:15:33.748559 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:33.748508 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8716cbf6-b788-4cf2-adb4-e4b4e400a725-istio-data\") pod \"e2e-shared-a-w3-59f108-openshift-default-6cc7dd95cd-c4pmd\" (UID: \"8716cbf6-b788-4cf2-adb4-e4b4e400a725\") " pod="openshift-ingress/e2e-shared-a-w3-59f108-openshift-default-6cc7dd95cd-c4pmd" Sep 11 16:15:33.748559 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:33.748550 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8716cbf6-b788-4cf2-adb4-e4b4e400a725-credential-socket\") pod \"e2e-shared-a-w3-59f108-openshift-default-6cc7dd95cd-c4pmd\" (UID: \"8716cbf6-b788-4cf2-adb4-e4b4e400a725\") " pod="openshift-ingress/e2e-shared-a-w3-59f108-openshift-default-6cc7dd95cd-c4pmd" Sep 11 16:15:33.749809 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:33.748957 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8716cbf6-b788-4cf2-adb4-e4b4e400a725-workload-socket\") pod \"e2e-shared-a-w3-59f108-openshift-default-6cc7dd95cd-c4pmd\" (UID: \"8716cbf6-b788-4cf2-adb4-e4b4e400a725\") " pod="openshift-ingress/e2e-shared-a-w3-59f108-openshift-default-6cc7dd95cd-c4pmd" Sep 11 16:15:33.850289 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:33.850254 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tx7ng\" (UniqueName: \"kubernetes.io/projected/8716cbf6-b788-4cf2-adb4-e4b4e400a725-kube-api-access-tx7ng\") pod \"e2e-shared-a-w3-59f108-openshift-default-6cc7dd95cd-c4pmd\" (UID: \"8716cbf6-b788-4cf2-adb4-e4b4e400a725\") " pod="openshift-ingress/e2e-shared-a-w3-59f108-openshift-default-6cc7dd95cd-c4pmd" Sep 11 16:15:33.850467 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:33.850294 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8716cbf6-b788-4cf2-adb4-e4b4e400a725-istio-envoy\") pod \"e2e-shared-a-w3-59f108-openshift-default-6cc7dd95cd-c4pmd\" (UID: \"8716cbf6-b788-4cf2-adb4-e4b4e400a725\") " pod="openshift-ingress/e2e-shared-a-w3-59f108-openshift-default-6cc7dd95cd-c4pmd" Sep 11 16:15:33.850467 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:33.850314 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8716cbf6-b788-4cf2-adb4-e4b4e400a725-istio-podinfo\") pod \"e2e-shared-a-w3-59f108-openshift-default-6cc7dd95cd-c4pmd\" (UID: \"8716cbf6-b788-4cf2-adb4-e4b4e400a725\") " pod="openshift-ingress/e2e-shared-a-w3-59f108-openshift-default-6cc7dd95cd-c4pmd" Sep 11 16:15:33.850467 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:33.850335 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8716cbf6-b788-4cf2-adb4-e4b4e400a725-workload-certs\") pod \"e2e-shared-a-w3-59f108-openshift-default-6cc7dd95cd-c4pmd\" (UID: \"8716cbf6-b788-4cf2-adb4-e4b4e400a725\") " pod="openshift-ingress/e2e-shared-a-w3-59f108-openshift-default-6cc7dd95cd-c4pmd" Sep 11 16:15:33.850467 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:33.850357 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8716cbf6-b788-4cf2-adb4-e4b4e400a725-istiod-ca-cert\") pod \"e2e-shared-a-w3-59f108-openshift-default-6cc7dd95cd-c4pmd\" (UID: \"8716cbf6-b788-4cf2-adb4-e4b4e400a725\") " pod="openshift-ingress/e2e-shared-a-w3-59f108-openshift-default-6cc7dd95cd-c4pmd" Sep 11 16:15:33.850467 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:33.850389 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8716cbf6-b788-4cf2-adb4-e4b4e400a725-istio-data\") pod \"e2e-shared-a-w3-59f108-openshift-default-6cc7dd95cd-c4pmd\" (UID: \"8716cbf6-b788-4cf2-adb4-e4b4e400a725\") " pod="openshift-ingress/e2e-shared-a-w3-59f108-openshift-default-6cc7dd95cd-c4pmd" Sep 11 16:15:33.850467 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:33.850407 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8716cbf6-b788-4cf2-adb4-e4b4e400a725-credential-socket\") pod \"e2e-shared-a-w3-59f108-openshift-default-6cc7dd95cd-c4pmd\" (UID: \"8716cbf6-b788-4cf2-adb4-e4b4e400a725\") " pod="openshift-ingress/e2e-shared-a-w3-59f108-openshift-default-6cc7dd95cd-c4pmd" Sep 11 16:15:33.850845 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:33.850472 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8716cbf6-b788-4cf2-adb4-e4b4e400a725-workload-socket\") pod \"e2e-shared-a-w3-59f108-openshift-default-6cc7dd95cd-c4pmd\" (UID: \"8716cbf6-b788-4cf2-adb4-e4b4e400a725\") " pod="openshift-ingress/e2e-shared-a-w3-59f108-openshift-default-6cc7dd95cd-c4pmd" Sep 11 16:15:33.850845 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:33.850511 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8716cbf6-b788-4cf2-adb4-e4b4e400a725-istio-token\") pod \"e2e-shared-a-w3-59f108-openshift-default-6cc7dd95cd-c4pmd\" (UID: \"8716cbf6-b788-4cf2-adb4-e4b4e400a725\") " pod="openshift-ingress/e2e-shared-a-w3-59f108-openshift-default-6cc7dd95cd-c4pmd" Sep 11 16:15:33.850845 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:33.850786 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8716cbf6-b788-4cf2-adb4-e4b4e400a725-workload-certs\") pod \"e2e-shared-a-w3-59f108-openshift-default-6cc7dd95cd-c4pmd\" (UID: \"8716cbf6-b788-4cf2-adb4-e4b4e400a725\") " pod="openshift-ingress/e2e-shared-a-w3-59f108-openshift-default-6cc7dd95cd-c4pmd" Sep 11 16:15:33.851032 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:33.850902 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8716cbf6-b788-4cf2-adb4-e4b4e400a725-istio-data\") pod \"e2e-shared-a-w3-59f108-openshift-default-6cc7dd95cd-c4pmd\" (UID: \"8716cbf6-b788-4cf2-adb4-e4b4e400a725\") " pod="openshift-ingress/e2e-shared-a-w3-59f108-openshift-default-6cc7dd95cd-c4pmd" Sep 11 16:15:33.851032 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:33.850913 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8716cbf6-b788-4cf2-adb4-e4b4e400a725-workload-socket\") pod \"e2e-shared-a-w3-59f108-openshift-default-6cc7dd95cd-c4pmd\" (UID: \"8716cbf6-b788-4cf2-adb4-e4b4e400a725\") " pod="openshift-ingress/e2e-shared-a-w3-59f108-openshift-default-6cc7dd95cd-c4pmd" Sep 11 16:15:33.851154 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:33.851053 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8716cbf6-b788-4cf2-adb4-e4b4e400a725-credential-socket\") pod \"e2e-shared-a-w3-59f108-openshift-default-6cc7dd95cd-c4pmd\" (UID: \"8716cbf6-b788-4cf2-adb4-e4b4e400a725\") " pod="openshift-ingress/e2e-shared-a-w3-59f108-openshift-default-6cc7dd95cd-c4pmd" Sep 11 16:15:33.851230 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:33.851205 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8716cbf6-b788-4cf2-adb4-e4b4e400a725-istiod-ca-cert\") pod \"e2e-shared-a-w3-59f108-openshift-default-6cc7dd95cd-c4pmd\" (UID: \"8716cbf6-b788-4cf2-adb4-e4b4e400a725\") " pod="openshift-ingress/e2e-shared-a-w3-59f108-openshift-default-6cc7dd95cd-c4pmd" Sep 11 16:15:33.853068 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:33.853043 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8716cbf6-b788-4cf2-adb4-e4b4e400a725-istio-envoy\") pod \"e2e-shared-a-w3-59f108-openshift-default-6cc7dd95cd-c4pmd\" (UID: \"8716cbf6-b788-4cf2-adb4-e4b4e400a725\") " pod="openshift-ingress/e2e-shared-a-w3-59f108-openshift-default-6cc7dd95cd-c4pmd" Sep 11 16:15:33.853267 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:33.853249 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8716cbf6-b788-4cf2-adb4-e4b4e400a725-istio-podinfo\") pod \"e2e-shared-a-w3-59f108-openshift-default-6cc7dd95cd-c4pmd\" (UID: \"8716cbf6-b788-4cf2-adb4-e4b4e400a725\") " pod="openshift-ingress/e2e-shared-a-w3-59f108-openshift-default-6cc7dd95cd-c4pmd" Sep 11 16:15:33.861387 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:33.861353 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8716cbf6-b788-4cf2-adb4-e4b4e400a725-istio-token\") pod \"e2e-shared-a-w3-59f108-openshift-default-6cc7dd95cd-c4pmd\" (UID: \"8716cbf6-b788-4cf2-adb4-e4b4e400a725\") " pod="openshift-ingress/e2e-shared-a-w3-59f108-openshift-default-6cc7dd95cd-c4pmd" Sep 11 16:15:33.861783 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:33.861759 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tx7ng\" (UniqueName: \"kubernetes.io/projected/8716cbf6-b788-4cf2-adb4-e4b4e400a725-kube-api-access-tx7ng\") pod \"e2e-shared-a-w3-59f108-openshift-default-6cc7dd95cd-c4pmd\" (UID: \"8716cbf6-b788-4cf2-adb4-e4b4e400a725\") " pod="openshift-ingress/e2e-shared-a-w3-59f108-openshift-default-6cc7dd95cd-c4pmd" Sep 11 16:15:33.974642 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:33.974550 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-59f108-openshift-default-6cc7dd95cd-c4pmd" Sep 11 16:15:34.252308 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:34.252278 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-59f108-openshift-default-6cc7dd95cd-c4pmd"] Sep 11 16:15:34.253894 ip-10-0-136-96 kubenswrapper[2566]: W0911 16:15:34.253867 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8716cbf6_b788_4cf2_adb4_e4b4e400a725.slice/crio-4bdd7be9295cc68512c4d6a6cf54cb121245f99e01c83d7aa6dea403763ce283 WatchSource:0}: Error finding container 4bdd7be9295cc68512c4d6a6cf54cb121245f99e01c83d7aa6dea403763ce283: Status 404 returned error can't find the container with id 4bdd7be9295cc68512c4d6a6cf54cb121245f99e01c83d7aa6dea403763ce283 Sep 11 16:15:34.256038 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:34.256007 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 16:15:34.256110 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:34.256073 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 16:15:34.256110 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:34.256102 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 16:15:34.618050 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:34.618017 2566 generic.go:356] "Generic (PLEG): container finished" podID="b78c6245-c285-479b-a463-42d55ce25b74" containerID="a4855cdb754886d540398e0300ee96f9d5a9f44587c49318fa62a1db4df3f4a7" exitCode=0 Sep 11 16:15:34.618299 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:34.618094 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-f7c21d-openshift-default-857f5cd77b-8phc5" event={"ID":"b78c6245-c285-479b-a463-42d55ce25b74","Type":"ContainerDied","Data":"a4855cdb754886d540398e0300ee96f9d5a9f44587c49318fa62a1db4df3f4a7"} Sep 11 16:15:34.618632 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:34.618611 2566 scope.go:117] "RemoveContainer" containerID="a4855cdb754886d540398e0300ee96f9d5a9f44587c49318fa62a1db4df3f4a7" Sep 11 16:15:34.619787 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:34.619754 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 16:15:34.619886 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:34.619833 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 16:15:34.619886 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:34.619866 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 16:15:34.619998 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:34.619890 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-59f108-openshift-default-6cc7dd95cd-c4pmd" event={"ID":"8716cbf6-b788-4cf2-adb4-e4b4e400a725","Type":"ContainerStarted","Data":"5abd1863c90d2e699da52e79d00277555c75cdacb758af2ddf9613c6a7377d01"} Sep 11 16:15:34.619998 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:34.619935 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-59f108-openshift-default-6cc7dd95cd-c4pmd" event={"ID":"8716cbf6-b788-4cf2-adb4-e4b4e400a725","Type":"ContainerStarted","Data":"4bdd7be9295cc68512c4d6a6cf54cb121245f99e01c83d7aa6dea403763ce283"} Sep 11 16:15:34.680174 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:34.680128 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-59f108-openshift-default-6cc7dd95cd-c4pmd" podStartSLOduration=1.680104945 podStartE2EDuration="1.680104945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:15:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:15:34.676841684 +0000 UTC m=+1241.891675590" watchObservedRunningTime="2026-09-11 16:15:34.680104945 +0000 UTC m=+1241.894938850" Sep 11 16:15:34.975491 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:34.975396 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-59f108-openshift-default-6cc7dd95cd-c4pmd" Sep 11 16:15:35.628289 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:35.628257 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-f7c21d-openshift-default-857f5cd77b-8phc5" event={"ID":"b78c6245-c285-479b-a463-42d55ce25b74","Type":"ContainerStarted","Data":"fd2e910431b4385bbce440937f593040756fd71e123212032c7c50564c2f6872"} Sep 11 16:15:35.981798 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:35.981712 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-59f108-openshift-default-6cc7dd95cd-c4pmd" Sep 11 16:15:36.346298 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:36.346262 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-f7c21d-openshift-default-857f5cd77b-8phc5" Sep 11 16:15:36.346683 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:36.346651 2566 patch_prober.go:28] interesting pod/e2e-shared-a-w2-f7c21d-openshift-default-857f5cd77b-8phc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Sep 11 16:15:36.346815 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:36.346711 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-f7c21d-openshift-default-857f5cd77b-8phc5" podUID="b78c6245-c285-479b-a463-42d55ce25b74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Sep 11 16:15:36.635730 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:36.635629 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-59f108-openshift-default-6cc7dd95cd-c4pmd" Sep 11 16:15:36.636606 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:36.636576 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-59f108-openshift-default-6cc7dd95cd-c4pmd" Sep 11 16:15:37.347441 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:37.347403 2566 patch_prober.go:28] interesting pod/e2e-shared-a-w2-f7c21d-openshift-default-857f5cd77b-8phc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Sep 11 16:15:37.348057 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:37.348016 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-f7c21d-openshift-default-857f5cd77b-8phc5" podUID="b78c6245-c285-479b-a463-42d55ce25b74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Sep 11 16:15:37.877765 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:37.877725 2566 patch_prober.go:28] interesting pod/e2e-inf-b-2b1bd5-openshift-default-687f66dffb-jdh2v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Sep 11 16:15:37.877898 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:37.877800 2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-2b1bd5-openshift-default-687f66dffb-jdh2v" podUID="c311820f-99df-4371-8967-d5518a1e299b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Sep 11 16:15:38.130922 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:38.130868 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2b1bd5-openshift-default-687f66dffb-jdh2v" Sep 11 16:15:38.195516 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:38.195479 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c311820f-99df-4371-8967-d5518a1e299b-istio-token\") pod \"c311820f-99df-4371-8967-d5518a1e299b\" (UID: \"c311820f-99df-4371-8967-d5518a1e299b\") " Sep 11 16:15:38.195687 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:38.195581 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c311820f-99df-4371-8967-d5518a1e299b-istiod-ca-cert\") pod \"c311820f-99df-4371-8967-d5518a1e299b\" (UID: \"c311820f-99df-4371-8967-d5518a1e299b\") " Sep 11 16:15:38.195687 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:38.195612 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c311820f-99df-4371-8967-d5518a1e299b-credential-socket\") pod \"c311820f-99df-4371-8967-d5518a1e299b\" (UID: \"c311820f-99df-4371-8967-d5518a1e299b\") " Sep 11 16:15:38.195687 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:38.195646 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c311820f-99df-4371-8967-d5518a1e299b-workload-socket\") pod \"c311820f-99df-4371-8967-d5518a1e299b\" (UID: \"c311820f-99df-4371-8967-d5518a1e299b\") " Sep 11 16:15:38.195687 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:38.195673 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c311820f-99df-4371-8967-d5518a1e299b-istio-envoy\") pod \"c311820f-99df-4371-8967-d5518a1e299b\" (UID: \"c311820f-99df-4371-8967-d5518a1e299b\") " Sep 11 16:15:38.195920 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:38.195722 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c311820f-99df-4371-8967-d5518a1e299b-istio-podinfo\") pod \"c311820f-99df-4371-8967-d5518a1e299b\" (UID: \"c311820f-99df-4371-8967-d5518a1e299b\") " Sep 11 16:15:38.195920 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:38.195763 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c311820f-99df-4371-8967-d5518a1e299b-workload-certs\") pod \"c311820f-99df-4371-8967-d5518a1e299b\" (UID: \"c311820f-99df-4371-8967-d5518a1e299b\") " Sep 11 16:15:38.195920 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:38.195875 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c311820f-99df-4371-8967-d5518a1e299b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c311820f-99df-4371-8967-d5518a1e299b" (UID: "c311820f-99df-4371-8967-d5518a1e299b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 16:15:38.196106 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:38.195928 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c311820f-99df-4371-8967-d5518a1e299b-istio-data\") pod \"c311820f-99df-4371-8967-d5518a1e299b\" (UID: \"c311820f-99df-4371-8967-d5518a1e299b\") " Sep 11 16:15:38.196106 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:38.195955 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sqpwt\" (UniqueName: \"kubernetes.io/projected/c311820f-99df-4371-8967-d5518a1e299b-kube-api-access-sqpwt\") pod \"c311820f-99df-4371-8967-d5518a1e299b\" (UID: \"c311820f-99df-4371-8967-d5518a1e299b\") " Sep 11 16:15:38.196106 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:38.196031 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c311820f-99df-4371-8967-d5518a1e299b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c311820f-99df-4371-8967-d5518a1e299b" (UID: "c311820f-99df-4371-8967-d5518a1e299b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 16:15:38.196266 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:38.196128 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c311820f-99df-4371-8967-d5518a1e299b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c311820f-99df-4371-8967-d5518a1e299b" (UID: "c311820f-99df-4371-8967-d5518a1e299b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 16:15:38.196266 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:38.196172 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c311820f-99df-4371-8967-d5518a1e299b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c311820f-99df-4371-8967-d5518a1e299b" (UID: "c311820f-99df-4371-8967-d5518a1e299b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 16:15:38.196371 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:38.196329 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c311820f-99df-4371-8967-d5518a1e299b-istio-data" (OuterVolumeSpecName: "istio-data") pod "c311820f-99df-4371-8967-d5518a1e299b" (UID: "c311820f-99df-4371-8967-d5518a1e299b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 16:15:38.196427 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:38.196406 2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c311820f-99df-4371-8967-d5518a1e299b-workload-certs\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:15:38.196482 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:38.196426 2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c311820f-99df-4371-8967-d5518a1e299b-istiod-ca-cert\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:15:38.196482 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:38.196440 2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c311820f-99df-4371-8967-d5518a1e299b-credential-socket\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:15:38.196482 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:38.196456 2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c311820f-99df-4371-8967-d5518a1e299b-workload-socket\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:15:38.197857 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:38.197833 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c311820f-99df-4371-8967-d5518a1e299b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c311820f-99df-4371-8967-d5518a1e299b" (UID: "c311820f-99df-4371-8967-d5518a1e299b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 16:15:38.198193 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:38.198175 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c311820f-99df-4371-8967-d5518a1e299b-istio-token" (OuterVolumeSpecName: "istio-token") pod "c311820f-99df-4371-8967-d5518a1e299b" (UID: "c311820f-99df-4371-8967-d5518a1e299b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 16:15:38.198340 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:38.198319 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c311820f-99df-4371-8967-d5518a1e299b-kube-api-access-sqpwt" (OuterVolumeSpecName: "kube-api-access-sqpwt") pod "c311820f-99df-4371-8967-d5518a1e299b" (UID: "c311820f-99df-4371-8967-d5518a1e299b"). InnerVolumeSpecName "kube-api-access-sqpwt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 16:15:38.198474 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:38.198458 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c311820f-99df-4371-8967-d5518a1e299b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c311820f-99df-4371-8967-d5518a1e299b" (UID: "c311820f-99df-4371-8967-d5518a1e299b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 16:15:38.297242 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:38.297209 2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c311820f-99df-4371-8967-d5518a1e299b-istio-token\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:15:38.297242 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:38.297236 2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c311820f-99df-4371-8967-d5518a1e299b-istio-envoy\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:15:38.297242 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:38.297247 2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c311820f-99df-4371-8967-d5518a1e299b-istio-podinfo\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:15:38.297476 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:38.297255 2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c311820f-99df-4371-8967-d5518a1e299b-istio-data\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:15:38.297476 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:38.297263 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqpwt\" (UniqueName: \"kubernetes.io/projected/c311820f-99df-4371-8967-d5518a1e299b-kube-api-access-sqpwt\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:15:38.346903 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:38.346853 2566 patch_prober.go:28] interesting pod/e2e-shared-a-w2-f7c21d-openshift-default-857f5cd77b-8phc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Sep 11 16:15:38.347099 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:38.346942 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-f7c21d-openshift-default-857f5cd77b-8phc5" podUID="b78c6245-c285-479b-a463-42d55ce25b74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Sep 11 16:15:38.659804 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:38.659758 2566 generic.go:356] "Generic (PLEG): container finished" podID="c311820f-99df-4371-8967-d5518a1e299b" containerID="5cb2622d6ae86291b94297e89ff86797e65bb74ca9d7c7b38e99a4f375dbbd24" exitCode=0 Sep 11 16:15:38.662015 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:38.660428 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2b1bd5-openshift-default-687f66dffb-jdh2v" Sep 11 16:15:38.662015 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:38.660448 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2b1bd5-openshift-default-687f66dffb-jdh2v" event={"ID":"c311820f-99df-4371-8967-d5518a1e299b","Type":"ContainerDied","Data":"5cb2622d6ae86291b94297e89ff86797e65bb74ca9d7c7b38e99a4f375dbbd24"} Sep 11 16:15:38.662015 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:38.661689 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2b1bd5-openshift-default-687f66dffb-jdh2v" event={"ID":"c311820f-99df-4371-8967-d5518a1e299b","Type":"ContainerDied","Data":"bf320695f877cccad43b6a537af43358f43781849675d8e860daf3b42a781ff5"} Sep 11 16:15:38.662015 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:38.661729 2566 scope.go:117] "RemoveContainer" containerID="5cb2622d6ae86291b94297e89ff86797e65bb74ca9d7c7b38e99a4f375dbbd24" Sep 11 16:15:38.673170 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:38.673152 2566 scope.go:117] "RemoveContainer" containerID="0b224b3169b2365e5a3ef2f92a07c4f8b9a14f8b46456b2a50013044873aa8a4" Sep 11 16:15:38.682784 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:38.682764 2566 scope.go:117] "RemoveContainer" containerID="5cb2622d6ae86291b94297e89ff86797e65bb74ca9d7c7b38e99a4f375dbbd24" Sep 11 16:15:38.683117 ip-10-0-136-96 kubenswrapper[2566]: E0911 16:15:38.683094 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cb2622d6ae86291b94297e89ff86797e65bb74ca9d7c7b38e99a4f375dbbd24\": container with ID starting with 5cb2622d6ae86291b94297e89ff86797e65bb74ca9d7c7b38e99a4f375dbbd24 not found: ID does not exist" containerID="5cb2622d6ae86291b94297e89ff86797e65bb74ca9d7c7b38e99a4f375dbbd24" Sep 11 16:15:38.683185 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:38.683127 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cb2622d6ae86291b94297e89ff86797e65bb74ca9d7c7b38e99a4f375dbbd24"} err="failed to get container status \"5cb2622d6ae86291b94297e89ff86797e65bb74ca9d7c7b38e99a4f375dbbd24\": rpc error: code = NotFound desc = could not find container \"5cb2622d6ae86291b94297e89ff86797e65bb74ca9d7c7b38e99a4f375dbbd24\": container with ID starting with 5cb2622d6ae86291b94297e89ff86797e65bb74ca9d7c7b38e99a4f375dbbd24 not found: ID does not exist" Sep 11 16:15:38.683185 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:38.683146 2566 scope.go:117] "RemoveContainer" containerID="0b224b3169b2365e5a3ef2f92a07c4f8b9a14f8b46456b2a50013044873aa8a4" Sep 11 16:15:38.683411 ip-10-0-136-96 kubenswrapper[2566]: E0911 16:15:38.683392 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b224b3169b2365e5a3ef2f92a07c4f8b9a14f8b46456b2a50013044873aa8a4\": container with ID starting with 0b224b3169b2365e5a3ef2f92a07c4f8b9a14f8b46456b2a50013044873aa8a4 not found: ID does not exist" containerID="0b224b3169b2365e5a3ef2f92a07c4f8b9a14f8b46456b2a50013044873aa8a4" Sep 11 16:15:38.683463 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:38.683417 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b224b3169b2365e5a3ef2f92a07c4f8b9a14f8b46456b2a50013044873aa8a4"} err="failed to get container status \"0b224b3169b2365e5a3ef2f92a07c4f8b9a14f8b46456b2a50013044873aa8a4\": rpc error: code = NotFound desc = could not find container \"0b224b3169b2365e5a3ef2f92a07c4f8b9a14f8b46456b2a50013044873aa8a4\": container with ID starting with 0b224b3169b2365e5a3ef2f92a07c4f8b9a14f8b46456b2a50013044873aa8a4 not found: ID does not exist" Sep 11 16:15:38.705696 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:38.705669 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-2b1bd5-openshift-default-687f66dffb-jdh2v"] Sep 11 16:15:38.752199 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:38.752157 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-2b1bd5-openshift-default-687f66dffb-jdh2v"] Sep 11 16:15:39.241010 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:39.240954 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c311820f-99df-4371-8967-d5518a1e299b" path="/var/lib/kubelet/pods/c311820f-99df-4371-8967-d5518a1e299b/volumes" Sep 11 16:15:39.347080 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:39.347043 2566 patch_prober.go:28] interesting pod/e2e-shared-a-w2-f7c21d-openshift-default-857f5cd77b-8phc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Sep 11 16:15:39.347341 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:39.347312 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-f7c21d-openshift-default-857f5cd77b-8phc5" podUID="b78c6245-c285-479b-a463-42d55ce25b74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Sep 11 16:15:40.346472 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:40.346432 2566 patch_prober.go:28] interesting pod/e2e-shared-a-w2-f7c21d-openshift-default-857f5cd77b-8phc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Sep 11 16:15:40.347117 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:40.347075 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-f7c21d-openshift-default-857f5cd77b-8phc5" podUID="b78c6245-c285-479b-a463-42d55ce25b74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Sep 11 16:15:41.352480 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:41.352450 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-f7c21d-openshift-default-857f5cd77b-8phc5" Sep 11 16:15:41.353573 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:41.353547 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-f7c21d-openshift-default-857f5cd77b-8phc5" Sep 11 16:15:41.355026 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:41.355002 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-f7c21d-openshift-default-857f5cd77b-8phc5" Sep 11 16:15:42.865887 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:42.865843 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dx4xn"] Sep 11 16:15:43.282423 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:43.282388 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-320b99-openshift-default-6c5bb8bb78-98z46"] Sep 11 16:15:43.282829 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:43.282813 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c311820f-99df-4371-8967-d5518a1e299b" containerName="istio-proxy" Sep 11 16:15:43.282829 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:43.282830 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="c311820f-99df-4371-8967-d5518a1e299b" containerName="istio-proxy" Sep 11 16:15:43.283005 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:43.282842 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c311820f-99df-4371-8967-d5518a1e299b" containerName="istio-proxy" Sep 11 16:15:43.283005 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:43.282848 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="c311820f-99df-4371-8967-d5518a1e299b" containerName="istio-proxy" Sep 11 16:15:43.283005 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:43.282914 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="c311820f-99df-4371-8967-d5518a1e299b" containerName="istio-proxy" Sep 11 16:15:43.286147 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:43.286125 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-320b99-openshift-default-6c5bb8bb78-98z46" Sep 11 16:15:43.288748 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:43.288729 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-320b99-openshift-default-dockercfg-pbmgk\"" Sep 11 16:15:43.313955 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:43.313929 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-320b99-openshift-default-6c5bb8bb78-98z46"] Sep 11 16:15:43.348915 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:43.348872 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1766158-f57b-4ce5-bad7-026f08295c84-istio-envoy\") pod \"e2e-shared-b-w2-320b99-openshift-default-6c5bb8bb78-98z46\" (UID: \"a1766158-f57b-4ce5-bad7-026f08295c84\") " pod="openshift-ingress/e2e-shared-b-w2-320b99-openshift-default-6c5bb8bb78-98z46" Sep 11 16:15:43.349460 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:43.349428 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1766158-f57b-4ce5-bad7-026f08295c84-istio-token\") pod \"e2e-shared-b-w2-320b99-openshift-default-6c5bb8bb78-98z46\" (UID: \"a1766158-f57b-4ce5-bad7-026f08295c84\") " pod="openshift-ingress/e2e-shared-b-w2-320b99-openshift-default-6c5bb8bb78-98z46" Sep 11 16:15:43.349580 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:43.349505 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1766158-f57b-4ce5-bad7-026f08295c84-istio-data\") pod \"e2e-shared-b-w2-320b99-openshift-default-6c5bb8bb78-98z46\" (UID: \"a1766158-f57b-4ce5-bad7-026f08295c84\") " pod="openshift-ingress/e2e-shared-b-w2-320b99-openshift-default-6c5bb8bb78-98z46" Sep 11 16:15:43.349654 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:43.349583 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1766158-f57b-4ce5-bad7-026f08295c84-istiod-ca-cert\") pod \"e2e-shared-b-w2-320b99-openshift-default-6c5bb8bb78-98z46\" (UID: \"a1766158-f57b-4ce5-bad7-026f08295c84\") " pod="openshift-ingress/e2e-shared-b-w2-320b99-openshift-default-6c5bb8bb78-98z46" Sep 11 16:15:43.349716 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:43.349671 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1766158-f57b-4ce5-bad7-026f08295c84-workload-socket\") pod \"e2e-shared-b-w2-320b99-openshift-default-6c5bb8bb78-98z46\" (UID: \"a1766158-f57b-4ce5-bad7-026f08295c84\") " pod="openshift-ingress/e2e-shared-b-w2-320b99-openshift-default-6c5bb8bb78-98z46" Sep 11 16:15:43.349828 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:43.349803 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1766158-f57b-4ce5-bad7-026f08295c84-workload-certs\") pod \"e2e-shared-b-w2-320b99-openshift-default-6c5bb8bb78-98z46\" (UID: \"a1766158-f57b-4ce5-bad7-026f08295c84\") " pod="openshift-ingress/e2e-shared-b-w2-320b99-openshift-default-6c5bb8bb78-98z46" Sep 11 16:15:43.349958 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:43.349933 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1766158-f57b-4ce5-bad7-026f08295c84-istio-podinfo\") pod \"e2e-shared-b-w2-320b99-openshift-default-6c5bb8bb78-98z46\" (UID: \"a1766158-f57b-4ce5-bad7-026f08295c84\") " pod="openshift-ingress/e2e-shared-b-w2-320b99-openshift-default-6c5bb8bb78-98z46" Sep 11 16:15:43.350040 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:43.350025 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxj5z\" (UniqueName: \"kubernetes.io/projected/a1766158-f57b-4ce5-bad7-026f08295c84-kube-api-access-xxj5z\") pod \"e2e-shared-b-w2-320b99-openshift-default-6c5bb8bb78-98z46\" (UID: \"a1766158-f57b-4ce5-bad7-026f08295c84\") " pod="openshift-ingress/e2e-shared-b-w2-320b99-openshift-default-6c5bb8bb78-98z46" Sep 11 16:15:43.350079 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:43.350055 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1766158-f57b-4ce5-bad7-026f08295c84-credential-socket\") pod \"e2e-shared-b-w2-320b99-openshift-default-6c5bb8bb78-98z46\" (UID: \"a1766158-f57b-4ce5-bad7-026f08295c84\") " pod="openshift-ingress/e2e-shared-b-w2-320b99-openshift-default-6c5bb8bb78-98z46" Sep 11 16:15:43.451488 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:43.451452 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1766158-f57b-4ce5-bad7-026f08295c84-istio-data\") pod \"e2e-shared-b-w2-320b99-openshift-default-6c5bb8bb78-98z46\" (UID: \"a1766158-f57b-4ce5-bad7-026f08295c84\") " pod="openshift-ingress/e2e-shared-b-w2-320b99-openshift-default-6c5bb8bb78-98z46" Sep 11 16:15:43.451725 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:43.451513 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1766158-f57b-4ce5-bad7-026f08295c84-istiod-ca-cert\") pod \"e2e-shared-b-w2-320b99-openshift-default-6c5bb8bb78-98z46\" (UID: \"a1766158-f57b-4ce5-bad7-026f08295c84\") " pod="openshift-ingress/e2e-shared-b-w2-320b99-openshift-default-6c5bb8bb78-98z46" Sep 11 16:15:43.451725 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:43.451555 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1766158-f57b-4ce5-bad7-026f08295c84-workload-socket\") pod \"e2e-shared-b-w2-320b99-openshift-default-6c5bb8bb78-98z46\" (UID: \"a1766158-f57b-4ce5-bad7-026f08295c84\") " pod="openshift-ingress/e2e-shared-b-w2-320b99-openshift-default-6c5bb8bb78-98z46" Sep 11 16:15:43.451725 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:43.451584 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1766158-f57b-4ce5-bad7-026f08295c84-workload-certs\") pod \"e2e-shared-b-w2-320b99-openshift-default-6c5bb8bb78-98z46\" (UID: \"a1766158-f57b-4ce5-bad7-026f08295c84\") " pod="openshift-ingress/e2e-shared-b-w2-320b99-openshift-default-6c5bb8bb78-98z46" Sep 11 16:15:43.451725 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:43.451611 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1766158-f57b-4ce5-bad7-026f08295c84-istio-podinfo\") pod \"e2e-shared-b-w2-320b99-openshift-default-6c5bb8bb78-98z46\" (UID: \"a1766158-f57b-4ce5-bad7-026f08295c84\") " pod="openshift-ingress/e2e-shared-b-w2-320b99-openshift-default-6c5bb8bb78-98z46" Sep 11 16:15:43.451725 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:43.451642 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxj5z\" (UniqueName: \"kubernetes.io/projected/a1766158-f57b-4ce5-bad7-026f08295c84-kube-api-access-xxj5z\") pod \"e2e-shared-b-w2-320b99-openshift-default-6c5bb8bb78-98z46\" (UID: \"a1766158-f57b-4ce5-bad7-026f08295c84\") " pod="openshift-ingress/e2e-shared-b-w2-320b99-openshift-default-6c5bb8bb78-98z46" Sep 11 16:15:43.451725 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:43.451674 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1766158-f57b-4ce5-bad7-026f08295c84-credential-socket\") pod \"e2e-shared-b-w2-320b99-openshift-default-6c5bb8bb78-98z46\" (UID: \"a1766158-f57b-4ce5-bad7-026f08295c84\") " pod="openshift-ingress/e2e-shared-b-w2-320b99-openshift-default-6c5bb8bb78-98z46" Sep 11 16:15:43.452079 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:43.451761 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1766158-f57b-4ce5-bad7-026f08295c84-istio-envoy\") pod \"e2e-shared-b-w2-320b99-openshift-default-6c5bb8bb78-98z46\" (UID: \"a1766158-f57b-4ce5-bad7-026f08295c84\") " pod="openshift-ingress/e2e-shared-b-w2-320b99-openshift-default-6c5bb8bb78-98z46" Sep 11 16:15:43.452079 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:43.451801 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1766158-f57b-4ce5-bad7-026f08295c84-istio-token\") pod \"e2e-shared-b-w2-320b99-openshift-default-6c5bb8bb78-98z46\" (UID: \"a1766158-f57b-4ce5-bad7-026f08295c84\") " pod="openshift-ingress/e2e-shared-b-w2-320b99-openshift-default-6c5bb8bb78-98z46" Sep 11 16:15:43.452079 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:43.451930 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1766158-f57b-4ce5-bad7-026f08295c84-istio-data\") pod \"e2e-shared-b-w2-320b99-openshift-default-6c5bb8bb78-98z46\" (UID: \"a1766158-f57b-4ce5-bad7-026f08295c84\") " pod="openshift-ingress/e2e-shared-b-w2-320b99-openshift-default-6c5bb8bb78-98z46" Sep 11 16:15:43.452663 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:43.452639 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1766158-f57b-4ce5-bad7-026f08295c84-credential-socket\") pod \"e2e-shared-b-w2-320b99-openshift-default-6c5bb8bb78-98z46\" (UID: \"a1766158-f57b-4ce5-bad7-026f08295c84\") " pod="openshift-ingress/e2e-shared-b-w2-320b99-openshift-default-6c5bb8bb78-98z46" Sep 11 16:15:43.452986 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:43.452949 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1766158-f57b-4ce5-bad7-026f08295c84-workload-socket\") pod \"e2e-shared-b-w2-320b99-openshift-default-6c5bb8bb78-98z46\" (UID: \"a1766158-f57b-4ce5-bad7-026f08295c84\") " pod="openshift-ingress/e2e-shared-b-w2-320b99-openshift-default-6c5bb8bb78-98z46" Sep 11 16:15:43.453185 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:43.453164 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1766158-f57b-4ce5-bad7-026f08295c84-workload-certs\") pod \"e2e-shared-b-w2-320b99-openshift-default-6c5bb8bb78-98z46\" (UID: \"a1766158-f57b-4ce5-bad7-026f08295c84\") " pod="openshift-ingress/e2e-shared-b-w2-320b99-openshift-default-6c5bb8bb78-98z46" Sep 11 16:15:43.453286 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:43.453262 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1766158-f57b-4ce5-bad7-026f08295c84-istiod-ca-cert\") pod \"e2e-shared-b-w2-320b99-openshift-default-6c5bb8bb78-98z46\" (UID: \"a1766158-f57b-4ce5-bad7-026f08295c84\") " pod="openshift-ingress/e2e-shared-b-w2-320b99-openshift-default-6c5bb8bb78-98z46" Sep 11 16:15:43.455350 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:43.454870 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1766158-f57b-4ce5-bad7-026f08295c84-istio-podinfo\") pod \"e2e-shared-b-w2-320b99-openshift-default-6c5bb8bb78-98z46\" (UID: \"a1766158-f57b-4ce5-bad7-026f08295c84\") " pod="openshift-ingress/e2e-shared-b-w2-320b99-openshift-default-6c5bb8bb78-98z46" Sep 11 16:15:43.455350 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:43.455218 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1766158-f57b-4ce5-bad7-026f08295c84-istio-envoy\") pod \"e2e-shared-b-w2-320b99-openshift-default-6c5bb8bb78-98z46\" (UID: \"a1766158-f57b-4ce5-bad7-026f08295c84\") " pod="openshift-ingress/e2e-shared-b-w2-320b99-openshift-default-6c5bb8bb78-98z46" Sep 11 16:15:43.471195 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:43.471125 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxj5z\" (UniqueName: \"kubernetes.io/projected/a1766158-f57b-4ce5-bad7-026f08295c84-kube-api-access-xxj5z\") pod \"e2e-shared-b-w2-320b99-openshift-default-6c5bb8bb78-98z46\" (UID: \"a1766158-f57b-4ce5-bad7-026f08295c84\") " pod="openshift-ingress/e2e-shared-b-w2-320b99-openshift-default-6c5bb8bb78-98z46" Sep 11 16:15:43.473160 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:43.473130 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1766158-f57b-4ce5-bad7-026f08295c84-istio-token\") pod \"e2e-shared-b-w2-320b99-openshift-default-6c5bb8bb78-98z46\" (UID: \"a1766158-f57b-4ce5-bad7-026f08295c84\") " pod="openshift-ingress/e2e-shared-b-w2-320b99-openshift-default-6c5bb8bb78-98z46" Sep 11 16:15:43.597093 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:43.597010 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-320b99-openshift-default-6c5bb8bb78-98z46" Sep 11 16:15:43.894941 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:43.894914 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-320b99-openshift-default-6c5bb8bb78-98z46"] Sep 11 16:15:43.896729 ip-10-0-136-96 kubenswrapper[2566]: W0911 16:15:43.896702 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1766158_f57b_4ce5_bad7_026f08295c84.slice/crio-0c1e77431d5d44aae036577ae3ee66f3a7cf1403e15d72d595b3cc1925e6b6f3 WatchSource:0}: Error finding container 0c1e77431d5d44aae036577ae3ee66f3a7cf1403e15d72d595b3cc1925e6b6f3: Status 404 returned error can't find the container with id 0c1e77431d5d44aae036577ae3ee66f3a7cf1403e15d72d595b3cc1925e6b6f3 Sep 11 16:15:43.899056 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:43.899020 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 16:15:43.899120 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:43.899105 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 16:15:43.899184 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:43.899168 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 16:15:44.706735 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:44.706699 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-320b99-openshift-default-6c5bb8bb78-98z46" event={"ID":"a1766158-f57b-4ce5-bad7-026f08295c84","Type":"ContainerStarted","Data":"c1b271e16934bbdd41dd463a1c5c75c8241484a68a13bc9596097a393edf6351"} Sep 11 16:15:44.706933 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:44.706754 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-320b99-openshift-default-6c5bb8bb78-98z46" event={"ID":"a1766158-f57b-4ce5-bad7-026f08295c84","Type":"ContainerStarted","Data":"0c1e77431d5d44aae036577ae3ee66f3a7cf1403e15d72d595b3cc1925e6b6f3"} Sep 11 16:15:44.740763 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:44.740679 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-320b99-openshift-default-6c5bb8bb78-98z46" podStartSLOduration=1.7406570810000002 podStartE2EDuration="1.740657081s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:15:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:15:44.738271609 +0000 UTC m=+1251.953105515" watchObservedRunningTime="2026-09-11 16:15:44.740657081 +0000 UTC m=+1251.955490987" Sep 11 16:15:45.597844 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:45.597763 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-320b99-openshift-default-6c5bb8bb78-98z46" Sep 11 16:15:45.602766 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:45.602743 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-320b99-openshift-default-6c5bb8bb78-98z46" Sep 11 16:15:45.713672 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:45.713643 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-320b99-openshift-default-6c5bb8bb78-98z46" Sep 11 16:15:45.714773 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:45.714753 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-320b99-openshift-default-6c5bb8bb78-98z46" Sep 11 16:15:52.179475 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:52.179441 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-1e0e69-openshift-default-6f9dd57774-zczpr"] Sep 11 16:15:52.179953 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:52.179940 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="c311820f-99df-4371-8967-d5518a1e299b" containerName="istio-proxy" Sep 11 16:15:52.182031 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:52.182015 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-1e0e69-openshift-default-6f9dd57774-zczpr" Sep 11 16:15:52.187070 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:52.187040 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-1e0e69-openshift-default-dockercfg-jvvd9\"" Sep 11 16:15:52.216389 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:52.216354 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-1e0e69-openshift-default-6f9dd57774-zczpr"] Sep 11 16:15:52.243538 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:52.243499 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50f58a6c-7361-4eb4-af63-e25efa7396ef-credential-socket\") pod \"e2e-shared-b-w3-1e0e69-openshift-default-6f9dd57774-zczpr\" (UID: \"50f58a6c-7361-4eb4-af63-e25efa7396ef\") " pod="openshift-ingress/e2e-shared-b-w3-1e0e69-openshift-default-6f9dd57774-zczpr" Sep 11 16:15:52.243708 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:52.243546 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50f58a6c-7361-4eb4-af63-e25efa7396ef-istio-data\") pod \"e2e-shared-b-w3-1e0e69-openshift-default-6f9dd57774-zczpr\" (UID: \"50f58a6c-7361-4eb4-af63-e25efa7396ef\") " pod="openshift-ingress/e2e-shared-b-w3-1e0e69-openshift-default-6f9dd57774-zczpr" Sep 11 16:15:52.243708 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:52.243650 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50f58a6c-7361-4eb4-af63-e25efa7396ef-workload-certs\") pod \"e2e-shared-b-w3-1e0e69-openshift-default-6f9dd57774-zczpr\" (UID: \"50f58a6c-7361-4eb4-af63-e25efa7396ef\") " pod="openshift-ingress/e2e-shared-b-w3-1e0e69-openshift-default-6f9dd57774-zczpr" Sep 11 16:15:52.243803 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:52.243710 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50f58a6c-7361-4eb4-af63-e25efa7396ef-istio-podinfo\") pod \"e2e-shared-b-w3-1e0e69-openshift-default-6f9dd57774-zczpr\" (UID: \"50f58a6c-7361-4eb4-af63-e25efa7396ef\") " pod="openshift-ingress/e2e-shared-b-w3-1e0e69-openshift-default-6f9dd57774-zczpr" Sep 11 16:15:52.243803 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:52.243766 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50f58a6c-7361-4eb4-af63-e25efa7396ef-istiod-ca-cert\") pod \"e2e-shared-b-w3-1e0e69-openshift-default-6f9dd57774-zczpr\" (UID: \"50f58a6c-7361-4eb4-af63-e25efa7396ef\") " pod="openshift-ingress/e2e-shared-b-w3-1e0e69-openshift-default-6f9dd57774-zczpr" Sep 11 16:15:52.243888 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:52.243825 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50f58a6c-7361-4eb4-af63-e25efa7396ef-workload-socket\") pod \"e2e-shared-b-w3-1e0e69-openshift-default-6f9dd57774-zczpr\" (UID: \"50f58a6c-7361-4eb4-af63-e25efa7396ef\") " pod="openshift-ingress/e2e-shared-b-w3-1e0e69-openshift-default-6f9dd57774-zczpr" Sep 11 16:15:52.243888 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:52.243854 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50f58a6c-7361-4eb4-af63-e25efa7396ef-istio-envoy\") pod \"e2e-shared-b-w3-1e0e69-openshift-default-6f9dd57774-zczpr\" (UID: \"50f58a6c-7361-4eb4-af63-e25efa7396ef\") " pod="openshift-ingress/e2e-shared-b-w3-1e0e69-openshift-default-6f9dd57774-zczpr" Sep 11 16:15:52.243996 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:52.243887 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50f58a6c-7361-4eb4-af63-e25efa7396ef-istio-token\") pod \"e2e-shared-b-w3-1e0e69-openshift-default-6f9dd57774-zczpr\" (UID: \"50f58a6c-7361-4eb4-af63-e25efa7396ef\") " pod="openshift-ingress/e2e-shared-b-w3-1e0e69-openshift-default-6f9dd57774-zczpr" Sep 11 16:15:52.243996 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:52.243922 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wks9f\" (UniqueName: \"kubernetes.io/projected/50f58a6c-7361-4eb4-af63-e25efa7396ef-kube-api-access-wks9f\") pod \"e2e-shared-b-w3-1e0e69-openshift-default-6f9dd57774-zczpr\" (UID: \"50f58a6c-7361-4eb4-af63-e25efa7396ef\") " pod="openshift-ingress/e2e-shared-b-w3-1e0e69-openshift-default-6f9dd57774-zczpr" Sep 11 16:15:52.345838 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:52.345804 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50f58a6c-7361-4eb4-af63-e25efa7396ef-istio-podinfo\") pod \"e2e-shared-b-w3-1e0e69-openshift-default-6f9dd57774-zczpr\" (UID: \"50f58a6c-7361-4eb4-af63-e25efa7396ef\") " pod="openshift-ingress/e2e-shared-b-w3-1e0e69-openshift-default-6f9dd57774-zczpr" Sep 11 16:15:52.345838 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:52.345852 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50f58a6c-7361-4eb4-af63-e25efa7396ef-istiod-ca-cert\") pod \"e2e-shared-b-w3-1e0e69-openshift-default-6f9dd57774-zczpr\" (UID: \"50f58a6c-7361-4eb4-af63-e25efa7396ef\") " pod="openshift-ingress/e2e-shared-b-w3-1e0e69-openshift-default-6f9dd57774-zczpr" Sep 11 16:15:52.346102 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:52.345881 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50f58a6c-7361-4eb4-af63-e25efa7396ef-workload-socket\") pod \"e2e-shared-b-w3-1e0e69-openshift-default-6f9dd57774-zczpr\" (UID: \"50f58a6c-7361-4eb4-af63-e25efa7396ef\") " pod="openshift-ingress/e2e-shared-b-w3-1e0e69-openshift-default-6f9dd57774-zczpr" Sep 11 16:15:52.346102 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:52.345909 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50f58a6c-7361-4eb4-af63-e25efa7396ef-istio-envoy\") pod \"e2e-shared-b-w3-1e0e69-openshift-default-6f9dd57774-zczpr\" (UID: \"50f58a6c-7361-4eb4-af63-e25efa7396ef\") " pod="openshift-ingress/e2e-shared-b-w3-1e0e69-openshift-default-6f9dd57774-zczpr" Sep 11 16:15:52.346102 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:52.345930 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50f58a6c-7361-4eb4-af63-e25efa7396ef-istio-token\") pod \"e2e-shared-b-w3-1e0e69-openshift-default-6f9dd57774-zczpr\" (UID: \"50f58a6c-7361-4eb4-af63-e25efa7396ef\") " pod="openshift-ingress/e2e-shared-b-w3-1e0e69-openshift-default-6f9dd57774-zczpr" Sep 11 16:15:52.346102 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:52.345954 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wks9f\" (UniqueName: \"kubernetes.io/projected/50f58a6c-7361-4eb4-af63-e25efa7396ef-kube-api-access-wks9f\") pod \"e2e-shared-b-w3-1e0e69-openshift-default-6f9dd57774-zczpr\" (UID: \"50f58a6c-7361-4eb4-af63-e25efa7396ef\") " pod="openshift-ingress/e2e-shared-b-w3-1e0e69-openshift-default-6f9dd57774-zczpr" Sep 11 16:15:52.346102 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:52.346052 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50f58a6c-7361-4eb4-af63-e25efa7396ef-credential-socket\") pod \"e2e-shared-b-w3-1e0e69-openshift-default-6f9dd57774-zczpr\" (UID: \"50f58a6c-7361-4eb4-af63-e25efa7396ef\") " pod="openshift-ingress/e2e-shared-b-w3-1e0e69-openshift-default-6f9dd57774-zczpr" Sep 11 16:15:52.346102 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:52.346084 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50f58a6c-7361-4eb4-af63-e25efa7396ef-istio-data\") pod \"e2e-shared-b-w3-1e0e69-openshift-default-6f9dd57774-zczpr\" (UID: \"50f58a6c-7361-4eb4-af63-e25efa7396ef\") " pod="openshift-ingress/e2e-shared-b-w3-1e0e69-openshift-default-6f9dd57774-zczpr" Sep 11 16:15:52.346455 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:52.346134 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50f58a6c-7361-4eb4-af63-e25efa7396ef-workload-certs\") pod \"e2e-shared-b-w3-1e0e69-openshift-default-6f9dd57774-zczpr\" (UID: \"50f58a6c-7361-4eb4-af63-e25efa7396ef\") " pod="openshift-ingress/e2e-shared-b-w3-1e0e69-openshift-default-6f9dd57774-zczpr" Sep 11 16:15:52.346455 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:52.346319 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50f58a6c-7361-4eb4-af63-e25efa7396ef-workload-socket\") pod \"e2e-shared-b-w3-1e0e69-openshift-default-6f9dd57774-zczpr\" (UID: \"50f58a6c-7361-4eb4-af63-e25efa7396ef\") " pod="openshift-ingress/e2e-shared-b-w3-1e0e69-openshift-default-6f9dd57774-zczpr" Sep 11 16:15:52.346562 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:52.346491 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50f58a6c-7361-4eb4-af63-e25efa7396ef-workload-certs\") pod \"e2e-shared-b-w3-1e0e69-openshift-default-6f9dd57774-zczpr\" (UID: \"50f58a6c-7361-4eb4-af63-e25efa7396ef\") " pod="openshift-ingress/e2e-shared-b-w3-1e0e69-openshift-default-6f9dd57774-zczpr" Sep 11 16:15:52.346562 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:52.346516 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50f58a6c-7361-4eb4-af63-e25efa7396ef-credential-socket\") pod \"e2e-shared-b-w3-1e0e69-openshift-default-6f9dd57774-zczpr\" (UID: \"50f58a6c-7361-4eb4-af63-e25efa7396ef\") " pod="openshift-ingress/e2e-shared-b-w3-1e0e69-openshift-default-6f9dd57774-zczpr" Sep 11 16:15:52.346686 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:52.346584 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50f58a6c-7361-4eb4-af63-e25efa7396ef-istio-data\") pod \"e2e-shared-b-w3-1e0e69-openshift-default-6f9dd57774-zczpr\" (UID: \"50f58a6c-7361-4eb4-af63-e25efa7396ef\") " pod="openshift-ingress/e2e-shared-b-w3-1e0e69-openshift-default-6f9dd57774-zczpr" Sep 11 16:15:52.347193 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:52.347165 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50f58a6c-7361-4eb4-af63-e25efa7396ef-istiod-ca-cert\") pod \"e2e-shared-b-w3-1e0e69-openshift-default-6f9dd57774-zczpr\" (UID: \"50f58a6c-7361-4eb4-af63-e25efa7396ef\") " pod="openshift-ingress/e2e-shared-b-w3-1e0e69-openshift-default-6f9dd57774-zczpr" Sep 11 16:15:52.348493 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:52.348470 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50f58a6c-7361-4eb4-af63-e25efa7396ef-istio-envoy\") pod \"e2e-shared-b-w3-1e0e69-openshift-default-6f9dd57774-zczpr\" (UID: \"50f58a6c-7361-4eb4-af63-e25efa7396ef\") " pod="openshift-ingress/e2e-shared-b-w3-1e0e69-openshift-default-6f9dd57774-zczpr" Sep 11 16:15:52.348578 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:52.348499 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50f58a6c-7361-4eb4-af63-e25efa7396ef-istio-podinfo\") pod \"e2e-shared-b-w3-1e0e69-openshift-default-6f9dd57774-zczpr\" (UID: \"50f58a6c-7361-4eb4-af63-e25efa7396ef\") " pod="openshift-ingress/e2e-shared-b-w3-1e0e69-openshift-default-6f9dd57774-zczpr" Sep 11 16:15:52.359119 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:52.359094 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50f58a6c-7361-4eb4-af63-e25efa7396ef-istio-token\") pod \"e2e-shared-b-w3-1e0e69-openshift-default-6f9dd57774-zczpr\" (UID: \"50f58a6c-7361-4eb4-af63-e25efa7396ef\") " pod="openshift-ingress/e2e-shared-b-w3-1e0e69-openshift-default-6f9dd57774-zczpr" Sep 11 16:15:52.359230 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:52.359118 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wks9f\" (UniqueName: \"kubernetes.io/projected/50f58a6c-7361-4eb4-af63-e25efa7396ef-kube-api-access-wks9f\") pod \"e2e-shared-b-w3-1e0e69-openshift-default-6f9dd57774-zczpr\" (UID: \"50f58a6c-7361-4eb4-af63-e25efa7396ef\") " pod="openshift-ingress/e2e-shared-b-w3-1e0e69-openshift-default-6f9dd57774-zczpr" Sep 11 16:15:52.495663 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:52.495577 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-1e0e69-openshift-default-6f9dd57774-zczpr" Sep 11 16:15:52.677148 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:52.677111 2566 patch_prober.go:28] interesting pod/e2e-shared-a-w3-59f108-openshift-default-6cc7dd95cd-c4pmd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 11 16:15:52.677315 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:52.677207 2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-59f108-openshift-default-6cc7dd95cd-c4pmd" podUID="8716cbf6-b788-4cf2-adb4-e4b4e400a725" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 11 16:15:52.781956 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:52.781926 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-1e0e69-openshift-default-6f9dd57774-zczpr"] Sep 11 16:15:52.783007 ip-10-0-136-96 kubenswrapper[2566]: W0911 16:15:52.782947 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50f58a6c_7361_4eb4_af63_e25efa7396ef.slice/crio-917fd2e0904ebfa073158ca5d0c15d626332968e8402caf68a43e144417d3b34 WatchSource:0}: Error finding container 917fd2e0904ebfa073158ca5d0c15d626332968e8402caf68a43e144417d3b34: Status 404 returned error can't find the container with id 917fd2e0904ebfa073158ca5d0c15d626332968e8402caf68a43e144417d3b34 Sep 11 16:15:52.785606 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:52.785562 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 16:15:52.785676 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:52.785658 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 16:15:52.785726 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:52.785709 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 16:15:53.774441 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:53.774405 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-1e0e69-openshift-default-6f9dd57774-zczpr" event={"ID":"50f58a6c-7361-4eb4-af63-e25efa7396ef","Type":"ContainerStarted","Data":"d5f1ad0b220f6acad7f87dcb7b1afa38c354a1409a50df0c822eb2608e61aa2e"} Sep 11 16:15:53.774944 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:53.774473 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-1e0e69-openshift-default-6f9dd57774-zczpr" event={"ID":"50f58a6c-7361-4eb4-af63-e25efa7396ef","Type":"ContainerStarted","Data":"917fd2e0904ebfa073158ca5d0c15d626332968e8402caf68a43e144417d3b34"} Sep 11 16:15:53.776047 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:53.776027 2566 generic.go:356] "Generic (PLEG): container finished" podID="8716cbf6-b788-4cf2-adb4-e4b4e400a725" containerID="5abd1863c90d2e699da52e79d00277555c75cdacb758af2ddf9613c6a7377d01" exitCode=0 Sep 11 16:15:53.776136 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:53.776076 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-59f108-openshift-default-6cc7dd95cd-c4pmd" event={"ID":"8716cbf6-b788-4cf2-adb4-e4b4e400a725","Type":"ContainerDied","Data":"5abd1863c90d2e699da52e79d00277555c75cdacb758af2ddf9613c6a7377d01"} Sep 11 16:15:53.776439 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:53.776425 2566 scope.go:117] "RemoveContainer" containerID="5abd1863c90d2e699da52e79d00277555c75cdacb758af2ddf9613c6a7377d01" Sep 11 16:15:53.777432 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:53.777408 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 16:15:53.777482 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:53.777473 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 16:15:53.777525 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:53.777503 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 16:15:53.810898 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:53.810852 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-1e0e69-openshift-default-6f9dd57774-zczpr" podStartSLOduration=1.810835786 podStartE2EDuration="1.810835786s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:15:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:15:53.807902286 +0000 UTC m=+1261.022736219" watchObservedRunningTime="2026-09-11 16:15:53.810835786 +0000 UTC m=+1261.025669691" Sep 11 16:15:54.496368 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:54.496327 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-1e0e69-openshift-default-6f9dd57774-zczpr" Sep 11 16:15:54.501336 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:54.501311 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-1e0e69-openshift-default-6f9dd57774-zczpr" Sep 11 16:15:54.785417 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:54.785382 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-59f108-openshift-default-6cc7dd95cd-c4pmd" event={"ID":"8716cbf6-b788-4cf2-adb4-e4b4e400a725","Type":"ContainerStarted","Data":"da04bd17b9a23ac0a08b84cf1311967b69dad3b929c7902a6d6f33a04ff7b355"} Sep 11 16:15:54.785889 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:54.785738 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-1e0e69-openshift-default-6f9dd57774-zczpr" Sep 11 16:15:54.786634 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:54.786615 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-1e0e69-openshift-default-6f9dd57774-zczpr" Sep 11 16:15:54.974783 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:54.974744 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-59f108-openshift-default-6cc7dd95cd-c4pmd" Sep 11 16:15:54.975000 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:54.974962 2566 patch_prober.go:28] interesting pod/e2e-shared-a-w3-59f108-openshift-default-6cc7dd95cd-c4pmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Sep 11 16:15:54.975087 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:54.975034 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-59f108-openshift-default-6cc7dd95cd-c4pmd" podUID="8716cbf6-b788-4cf2-adb4-e4b4e400a725" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Sep 11 16:15:55.265998 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:55.265680 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dx4xn"] Sep 11 16:15:55.975249 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:55.975204 2566 patch_prober.go:28] interesting pod/e2e-shared-a-w3-59f108-openshift-default-6cc7dd95cd-c4pmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Sep 11 16:15:55.975650 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:55.975276 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-59f108-openshift-default-6cc7dd95cd-c4pmd" podUID="8716cbf6-b788-4cf2-adb4-e4b4e400a725" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Sep 11 16:15:56.975618 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:56.975568 2566 patch_prober.go:28] interesting pod/e2e-shared-a-w3-59f108-openshift-default-6cc7dd95cd-c4pmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Sep 11 16:15:56.976143 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:56.975641 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-59f108-openshift-default-6cc7dd95cd-c4pmd" podUID="8716cbf6-b788-4cf2-adb4-e4b4e400a725" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Sep 11 16:15:57.975808 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:57.975759 2566 patch_prober.go:28] interesting pod/e2e-shared-a-w3-59f108-openshift-default-6cc7dd95cd-c4pmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Sep 11 16:15:57.976358 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:57.975840 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-59f108-openshift-default-6cc7dd95cd-c4pmd" podUID="8716cbf6-b788-4cf2-adb4-e4b4e400a725" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Sep 11 16:15:58.975719 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:58.975681 2566 patch_prober.go:28] interesting pod/e2e-shared-a-w3-59f108-openshift-default-6cc7dd95cd-c4pmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Sep 11 16:15:58.976053 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:15:58.976020 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-59f108-openshift-default-6cc7dd95cd-c4pmd" podUID="8716cbf6-b788-4cf2-adb4-e4b4e400a725" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Sep 11 16:16:00.009177 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:00.009141 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-59f108-openshift-default-6cc7dd95cd-c4pmd" Sep 11 16:16:00.009675 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:00.009596 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-59f108-openshift-default-6cc7dd95cd-c4pmd" Sep 11 16:16:00.010868 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:00.010847 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-59f108-openshift-default-6cc7dd95cd-c4pmd" Sep 11 16:16:05.875918 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:05.875884 2566 generic.go:356] "Generic (PLEG): container finished" podID="a1766158-f57b-4ce5-bad7-026f08295c84" containerID="c1b271e16934bbdd41dd463a1c5c75c8241484a68a13bc9596097a393edf6351" exitCode=0 Sep 11 16:16:05.876353 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:05.875959 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-320b99-openshift-default-6c5bb8bb78-98z46" event={"ID":"a1766158-f57b-4ce5-bad7-026f08295c84","Type":"ContainerDied","Data":"c1b271e16934bbdd41dd463a1c5c75c8241484a68a13bc9596097a393edf6351"} Sep 11 16:16:05.876436 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:05.876417 2566 scope.go:117] "RemoveContainer" containerID="c1b271e16934bbdd41dd463a1c5c75c8241484a68a13bc9596097a393edf6351" Sep 11 16:16:05.877704 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:05.877675 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 16:16:05.877770 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:05.877749 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 16:16:05.877810 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:05.877784 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 16:16:06.327345 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:06.327307 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dx4xn"] Sep 11 16:16:06.889801 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:06.889771 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-320b99-openshift-default-6c5bb8bb78-98z46" event={"ID":"a1766158-f57b-4ce5-bad7-026f08295c84","Type":"ContainerStarted","Data":"4a09f22563b21079215869903ee522cfaff9ad5d1f0dc83317a286eba0fdf37c"} Sep 11 16:16:07.597745 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:07.597700 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-320b99-openshift-default-6c5bb8bb78-98z46" Sep 11 16:16:07.598033 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:07.598006 2566 patch_prober.go:28] interesting pod/e2e-shared-b-w2-320b99-openshift-default-6c5bb8bb78-98z46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Sep 11 16:16:07.598102 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:07.598081 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-320b99-openshift-default-6c5bb8bb78-98z46" podUID="a1766158-f57b-4ce5-bad7-026f08295c84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Sep 11 16:16:08.597587 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:08.597548 2566 patch_prober.go:28] interesting pod/e2e-shared-b-w2-320b99-openshift-default-6c5bb8bb78-98z46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Sep 11 16:16:08.598003 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:08.597612 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-320b99-openshift-default-6c5bb8bb78-98z46" podUID="a1766158-f57b-4ce5-bad7-026f08295c84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Sep 11 16:16:09.598339 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:09.598294 2566 patch_prober.go:28] interesting pod/e2e-shared-b-w2-320b99-openshift-default-6c5bb8bb78-98z46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Sep 11 16:16:09.598724 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:09.598378 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-320b99-openshift-default-6c5bb8bb78-98z46" podUID="a1766158-f57b-4ce5-bad7-026f08295c84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Sep 11 16:16:10.598276 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:10.598240 2566 patch_prober.go:28] interesting pod/e2e-shared-b-w2-320b99-openshift-default-6c5bb8bb78-98z46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Sep 11 16:16:10.598483 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:10.598305 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-320b99-openshift-default-6c5bb8bb78-98z46" podUID="a1766158-f57b-4ce5-bad7-026f08295c84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Sep 11 16:16:10.827187 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:10.827148 2566 patch_prober.go:28] interesting pod/e2e-shared-b-w3-1e0e69-openshift-default-6f9dd57774-zczpr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 11 16:16:10.827355 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:10.827212 2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-1e0e69-openshift-default-6f9dd57774-zczpr" podUID="50f58a6c-7361-4eb4-af63-e25efa7396ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 11 16:16:11.598052 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:11.598011 2566 patch_prober.go:28] interesting pod/e2e-shared-b-w2-320b99-openshift-default-6c5bb8bb78-98z46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Sep 11 16:16:11.598301 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:11.598083 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-320b99-openshift-default-6c5bb8bb78-98z46" podUID="a1766158-f57b-4ce5-bad7-026f08295c84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Sep 11 16:16:12.602230 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:12.602194 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-320b99-openshift-default-6c5bb8bb78-98z46" Sep 11 16:16:12.602631 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:12.602542 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-320b99-openshift-default-6c5bb8bb78-98z46" Sep 11 16:16:12.603387 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:12.603369 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-320b99-openshift-default-6c5bb8bb78-98z46" Sep 11 16:16:12.936425 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:12.936346 2566 generic.go:356] "Generic (PLEG): container finished" podID="50f58a6c-7361-4eb4-af63-e25efa7396ef" containerID="d5f1ad0b220f6acad7f87dcb7b1afa38c354a1409a50df0c822eb2608e61aa2e" exitCode=0 Sep 11 16:16:12.936578 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:12.936420 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-1e0e69-openshift-default-6f9dd57774-zczpr" event={"ID":"50f58a6c-7361-4eb4-af63-e25efa7396ef","Type":"ContainerDied","Data":"d5f1ad0b220f6acad7f87dcb7b1afa38c354a1409a50df0c822eb2608e61aa2e"} Sep 11 16:16:12.936913 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:12.936896 2566 scope.go:117] "RemoveContainer" containerID="d5f1ad0b220f6acad7f87dcb7b1afa38c354a1409a50df0c822eb2608e61aa2e" Sep 11 16:16:12.938038 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:12.938009 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 16:16:12.938113 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:12.938096 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 16:16:12.938157 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:12.938126 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 16:16:13.951088 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:13.951035 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-1e0e69-openshift-default-6f9dd57774-zczpr" event={"ID":"50f58a6c-7361-4eb4-af63-e25efa7396ef","Type":"ContainerStarted","Data":"e69249c3f963a732066628d15b1c44cecaee22daed8a1e7a4b83854bba5466ba"} Sep 11 16:16:14.495700 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:14.495666 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-1e0e69-openshift-default-6f9dd57774-zczpr" Sep 11 16:16:14.496071 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:14.496048 2566 patch_prober.go:28] interesting pod/e2e-shared-b-w3-1e0e69-openshift-default-6f9dd57774-zczpr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Sep 11 16:16:14.496126 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:14.496103 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-1e0e69-openshift-default-6f9dd57774-zczpr" podUID="50f58a6c-7361-4eb4-af63-e25efa7396ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Sep 11 16:16:15.496852 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:15.496814 2566 patch_prober.go:28] interesting pod/e2e-shared-b-w3-1e0e69-openshift-default-6f9dd57774-zczpr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Sep 11 16:16:15.497300 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:15.496891 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-1e0e69-openshift-default-6f9dd57774-zczpr" podUID="50f58a6c-7361-4eb4-af63-e25efa7396ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Sep 11 16:16:16.496392 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:16.496351 2566 patch_prober.go:28] interesting pod/e2e-shared-b-w3-1e0e69-openshift-default-6f9dd57774-zczpr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Sep 11 16:16:16.496613 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:16.496413 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-1e0e69-openshift-default-6f9dd57774-zczpr" podUID="50f58a6c-7361-4eb4-af63-e25efa7396ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Sep 11 16:16:17.496061 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:17.496022 2566 patch_prober.go:28] interesting pod/e2e-shared-b-w3-1e0e69-openshift-default-6f9dd57774-zczpr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Sep 11 16:16:17.496466 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:17.496091 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-1e0e69-openshift-default-6f9dd57774-zczpr" podUID="50f58a6c-7361-4eb4-af63-e25efa7396ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Sep 11 16:16:17.700369 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:17.700331 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7092-6d45a3"] Sep 11 16:16:17.709437 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:17.709407 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7092-6d45a3" Sep 11 16:16:17.712325 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:17.712294 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-vcw5k\"" Sep 11 16:16:17.714362 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:17.714334 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7092-6d45a3"] Sep 11 16:16:17.733146 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:17.733103 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgmw2\" (UniqueName: \"kubernetes.io/projected/d7cf2979-5759-4264-95d0-3c92e1776914-kube-api-access-xgmw2\") pod \"test-curl-7092-6d45a3\" (UID: \"d7cf2979-5759-4264-95d0-3c92e1776914\") " pod="opendatahub/test-curl-7092-6d45a3" Sep 11 16:16:17.834576 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:17.834533 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgmw2\" (UniqueName: \"kubernetes.io/projected/d7cf2979-5759-4264-95d0-3c92e1776914-kube-api-access-xgmw2\") pod \"test-curl-7092-6d45a3\" (UID: \"d7cf2979-5759-4264-95d0-3c92e1776914\") " pod="opendatahub/test-curl-7092-6d45a3" Sep 11 16:16:17.851095 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:17.851041 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgmw2\" (UniqueName: \"kubernetes.io/projected/d7cf2979-5759-4264-95d0-3c92e1776914-kube-api-access-xgmw2\") pod \"test-curl-7092-6d45a3\" (UID: \"d7cf2979-5759-4264-95d0-3c92e1776914\") " pod="opendatahub/test-curl-7092-6d45a3" Sep 11 16:16:18.023198 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:18.023159 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7092-6d45a3" Sep 11 16:16:18.343044 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:18.337927 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7092-6d45a3"] Sep 11 16:16:18.496887 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:18.496845 2566 patch_prober.go:28] interesting pod/e2e-shared-b-w3-1e0e69-openshift-default-6f9dd57774-zczpr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Sep 11 16:16:18.497294 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:18.496910 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-1e0e69-openshift-default-6f9dd57774-zczpr" podUID="50f58a6c-7361-4eb4-af63-e25efa7396ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Sep 11 16:16:18.989575 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:18.989538 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7092-6d45a3" event={"ID":"d7cf2979-5759-4264-95d0-3c92e1776914","Type":"ContainerStarted","Data":"c961c407ddfd4b117775b1c252bbfe59154b377f4015796ac230c33b0749ecd3"} Sep 11 16:16:18.989768 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:18.989591 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7092-6d45a3" event={"ID":"d7cf2979-5759-4264-95d0-3c92e1776914","Type":"ContainerStarted","Data":"9a5713a771976f00b2975bb1285125a647d1ccdc846ecba3fb90dc3bbbc3cc7b"} Sep 11 16:16:19.014594 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:19.014541 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7092-6d45a3" podStartSLOduration=2.014521463 podStartE2EDuration="2.014521463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:16:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:16:19.012272924 +0000 UTC m=+1286.227106830" watchObservedRunningTime="2026-09-11 16:16:19.014521463 +0000 UTC m=+1286.229355367" Sep 11 16:16:19.499555 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:19.499526 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-1e0e69-openshift-default-6f9dd57774-zczpr" Sep 11 16:16:19.499920 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:19.499808 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-1e0e69-openshift-default-6f9dd57774-zczpr" Sep 11 16:16:19.500592 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:19.500565 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-1e0e69-openshift-default-6f9dd57774-zczpr" Sep 11 16:16:19.878080 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:19.878043 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7092-6d45a3"] Sep 11 16:16:19.887060 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:19.887025 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7092-6d45a3"] Sep 11 16:16:19.996473 ip-10-0-136-96 kubenswrapper[2566]: E0911 16:16:19.996445 2566 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7092-6d45a3" container="test-curl-7092-6d45a3" Sep 11 16:16:20.016919 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:20.016887 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7092-7903ff"] Sep 11 16:16:20.017320 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:20.017307 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7cf2979-5759-4264-95d0-3c92e1776914" containerName="test-curl-7092-6d45a3" Sep 11 16:16:20.017381 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:20.017322 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7cf2979-5759-4264-95d0-3c92e1776914" containerName="test-curl-7092-6d45a3" Sep 11 16:16:20.017420 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:20.017413 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7cf2979-5759-4264-95d0-3c92e1776914" containerName="test-curl-7092-6d45a3" Sep 11 16:16:20.019738 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:20.019722 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7092-7903ff" Sep 11 16:16:20.030143 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:20.030119 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7092-7903ff"] Sep 11 16:16:20.075461 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:20.075431 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rdgm\" (UniqueName: \"kubernetes.io/projected/cd0fc716-151f-4f0a-8a53-e3f6eb5be43e-kube-api-access-9rdgm\") pod \"test-curl-7092-7903ff\" (UID: \"cd0fc716-151f-4f0a-8a53-e3f6eb5be43e\") " pod="opendatahub/test-curl-7092-7903ff" Sep 11 16:16:20.177276 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:20.177183 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rdgm\" (UniqueName: \"kubernetes.io/projected/cd0fc716-151f-4f0a-8a53-e3f6eb5be43e-kube-api-access-9rdgm\") pod \"test-curl-7092-7903ff\" (UID: \"cd0fc716-151f-4f0a-8a53-e3f6eb5be43e\") " pod="opendatahub/test-curl-7092-7903ff" Sep 11 16:16:20.189558 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:20.189515 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rdgm\" (UniqueName: \"kubernetes.io/projected/cd0fc716-151f-4f0a-8a53-e3f6eb5be43e-kube-api-access-9rdgm\") pod \"test-curl-7092-7903ff\" (UID: \"cd0fc716-151f-4f0a-8a53-e3f6eb5be43e\") " pod="opendatahub/test-curl-7092-7903ff" Sep 11 16:16:20.331072 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:20.331034 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7092-7903ff" Sep 11 16:16:20.611405 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:20.611381 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7092-7903ff"] Sep 11 16:16:20.612725 ip-10-0-136-96 kubenswrapper[2566]: W0911 16:16:20.612700 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcd0fc716_151f_4f0a_8a53_e3f6eb5be43e.slice/crio-841b3e9c835bb1f8a83b8f23d241692c4b92437ecc51634df8d392c3120d2da4 WatchSource:0}: Error finding container 841b3e9c835bb1f8a83b8f23d241692c4b92437ecc51634df8d392c3120d2da4: Status 404 returned error can't find the container with id 841b3e9c835bb1f8a83b8f23d241692c4b92437ecc51634df8d392c3120d2da4 Sep 11 16:16:21.004255 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:21.004162 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7092-7903ff" event={"ID":"cd0fc716-151f-4f0a-8a53-e3f6eb5be43e","Type":"ContainerStarted","Data":"d04f1d4ee92bc40e31d81be3ad1edb2b2e1a112416bfee514aa5bc23f76ca341"} Sep 11 16:16:21.004255 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:21.004211 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7092-7903ff" event={"ID":"cd0fc716-151f-4f0a-8a53-e3f6eb5be43e","Type":"ContainerStarted","Data":"841b3e9c835bb1f8a83b8f23d241692c4b92437ecc51634df8d392c3120d2da4"} Sep 11 16:16:21.004614 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:21.004580 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7092-6d45a3" podUID="d7cf2979-5759-4264-95d0-3c92e1776914" containerName="test-curl-7092-6d45a3" containerID="cri-o://c961c407ddfd4b117775b1c252bbfe59154b377f4015796ac230c33b0749ecd3" gracePeriod=2 Sep 11 16:16:21.023257 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:21.023203 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7092-7903ff" podStartSLOduration=2.023184114 podStartE2EDuration="2.023184114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:16:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:16:21.021370178 +0000 UTC m=+1288.236204083" watchObservedRunningTime="2026-09-11 16:16:21.023184114 +0000 UTC m=+1288.238018021" Sep 11 16:16:21.023900 ip-10-0-136-96 kubenswrapper[2566]: E0911 16:16:21.023875 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7092-6d45a3\" is forbidden: User \"system:node:ip-10-0-136-96.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-96.ec2.internal' and this object" podUID="d7cf2979-5759-4264-95d0-3c92e1776914" pod="opendatahub/test-curl-7092-6d45a3" Sep 11 16:16:21.818570 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:21.818526 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7092-7903ff"] Sep 11 16:16:21.824873 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:21.824843 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7092-7903ff"] Sep 11 16:16:21.960147 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:21.960110 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7092-a044f0"] Sep 11 16:16:21.960519 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:21.960506 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd0fc716-151f-4f0a-8a53-e3f6eb5be43e" containerName="test-curl-7092-7903ff" Sep 11 16:16:21.960584 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:21.960520 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd0fc716-151f-4f0a-8a53-e3f6eb5be43e" containerName="test-curl-7092-7903ff" Sep 11 16:16:21.960638 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:21.960611 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd0fc716-151f-4f0a-8a53-e3f6eb5be43e" containerName="test-curl-7092-7903ff" Sep 11 16:16:21.962686 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:21.962670 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7092-a044f0" Sep 11 16:16:21.978210 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:21.978186 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7092-a044f0"] Sep 11 16:16:22.011249 ip-10-0-136-96 kubenswrapper[2566]: E0911 16:16:22.011220 2566 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7092-7903ff" container="test-curl-7092-7903ff" Sep 11 16:16:22.097137 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:22.097051 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6sfq\" (UniqueName: \"kubernetes.io/projected/519075bc-43a1-4519-85e4-4bd194a0d85e-kube-api-access-p6sfq\") pod \"test-curl-7092-a044f0\" (UID: \"519075bc-43a1-4519-85e4-4bd194a0d85e\") " pod="opendatahub/test-curl-7092-a044f0" Sep 11 16:16:22.198267 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:22.198230 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6sfq\" (UniqueName: \"kubernetes.io/projected/519075bc-43a1-4519-85e4-4bd194a0d85e-kube-api-access-p6sfq\") pod \"test-curl-7092-a044f0\" (UID: \"519075bc-43a1-4519-85e4-4bd194a0d85e\") " pod="opendatahub/test-curl-7092-a044f0" Sep 11 16:16:22.206383 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:22.206353 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6sfq\" (UniqueName: \"kubernetes.io/projected/519075bc-43a1-4519-85e4-4bd194a0d85e-kube-api-access-p6sfq\") pod \"test-curl-7092-a044f0\" (UID: \"519075bc-43a1-4519-85e4-4bd194a0d85e\") " pod="opendatahub/test-curl-7092-a044f0" Sep 11 16:16:22.271481 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:22.271445 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7092-a044f0" Sep 11 16:16:22.548297 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:22.548269 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7092-a044f0"] Sep 11 16:16:22.550216 ip-10-0-136-96 kubenswrapper[2566]: W0911 16:16:22.550183 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod519075bc_43a1_4519_85e4_4bd194a0d85e.slice/crio-83e6341aa865b27a02279473fbd28794a0738ed022d56973aab7fae481c0ee09 WatchSource:0}: Error finding container 83e6341aa865b27a02279473fbd28794a0738ed022d56973aab7fae481c0ee09: Status 404 returned error can't find the container with id 83e6341aa865b27a02279473fbd28794a0738ed022d56973aab7fae481c0ee09 Sep 11 16:16:23.019315 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:23.019285 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7092-a044f0" event={"ID":"519075bc-43a1-4519-85e4-4bd194a0d85e","Type":"ContainerStarted","Data":"2c9dc5e525e350ceef749be5d95aa412c4e0c340867401c722a0be8fe39876b8"} Sep 11 16:16:23.019673 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:23.019325 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7092-a044f0" event={"ID":"519075bc-43a1-4519-85e4-4bd194a0d85e","Type":"ContainerStarted","Data":"83e6341aa865b27a02279473fbd28794a0738ed022d56973aab7fae481c0ee09"} Sep 11 16:16:23.019673 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:23.019469 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7092-7903ff" podUID="cd0fc716-151f-4f0a-8a53-e3f6eb5be43e" containerName="test-curl-7092-7903ff" containerID="cri-o://d04f1d4ee92bc40e31d81be3ad1edb2b2e1a112416bfee514aa5bc23f76ca341" gracePeriod=2 Sep 11 16:16:23.039028 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:23.038951 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7092-a044f0" podStartSLOduration=2.038931821 podStartE2EDuration="2.038931821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:16:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:16:23.036577677 +0000 UTC m=+1290.251411581" watchObservedRunningTime="2026-09-11 16:16:23.038931821 +0000 UTC m=+1290.253765730" Sep 11 16:16:23.039466 ip-10-0-136-96 kubenswrapper[2566]: E0911 16:16:23.039442 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7092-7903ff\" is forbidden: User \"system:node:ip-10-0-136-96.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-96.ec2.internal' and this object" podUID="cd0fc716-151f-4f0a-8a53-e3f6eb5be43e" pod="opendatahub/test-curl-7092-7903ff" Sep 11 16:16:23.153863 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:23.153840 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7092-6d45a3" Sep 11 16:16:23.157292 ip-10-0-136-96 kubenswrapper[2566]: E0911 16:16:23.157260 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7092-7903ff\" is forbidden: User \"system:node:ip-10-0-136-96.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-96.ec2.internal' and this object" podUID="cd0fc716-151f-4f0a-8a53-e3f6eb5be43e" pod="opendatahub/test-curl-7092-7903ff" Sep 11 16:16:23.210779 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:23.210747 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xgmw2\" (UniqueName: \"kubernetes.io/projected/d7cf2979-5759-4264-95d0-3c92e1776914-kube-api-access-xgmw2\") pod \"d7cf2979-5759-4264-95d0-3c92e1776914\" (UID: \"d7cf2979-5759-4264-95d0-3c92e1776914\") " Sep 11 16:16:23.212958 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:23.212927 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7cf2979-5759-4264-95d0-3c92e1776914-kube-api-access-xgmw2" (OuterVolumeSpecName: "kube-api-access-xgmw2") pod "d7cf2979-5759-4264-95d0-3c92e1776914" (UID: "d7cf2979-5759-4264-95d0-3c92e1776914"). InnerVolumeSpecName "kube-api-access-xgmw2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 16:16:23.238739 ip-10-0-136-96 kubenswrapper[2566]: E0911 16:16:23.238707 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7092-7903ff\" is forbidden: User \"system:node:ip-10-0-136-96.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-96.ec2.internal' and this object" podUID="cd0fc716-151f-4f0a-8a53-e3f6eb5be43e" pod="opendatahub/test-curl-7092-7903ff" Sep 11 16:16:23.239844 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:23.239822 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d7cf2979-5759-4264-95d0-3c92e1776914" path="/var/lib/kubelet/pods/d7cf2979-5759-4264-95d0-3c92e1776914/volumes" Sep 11 16:16:23.311760 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:23.311729 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgmw2\" (UniqueName: \"kubernetes.io/projected/d7cf2979-5759-4264-95d0-3c92e1776914-kube-api-access-xgmw2\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:16:23.679584 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:23.679468 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7092-a044f0"] Sep 11 16:16:23.683078 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:23.683047 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7092-a044f0"] Sep 11 16:16:23.805081 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:23.805044 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7092-544547"] Sep 11 16:16:23.805517 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:23.805504 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="519075bc-43a1-4519-85e4-4bd194a0d85e" containerName="test-curl-7092-a044f0" Sep 11 16:16:23.805567 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:23.805519 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="519075bc-43a1-4519-85e4-4bd194a0d85e" containerName="test-curl-7092-a044f0" Sep 11 16:16:23.805610 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:23.805595 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="519075bc-43a1-4519-85e4-4bd194a0d85e" containerName="test-curl-7092-a044f0" Sep 11 16:16:23.807672 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:23.807656 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7092-544547" Sep 11 16:16:23.816356 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:23.816334 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7092-544547"] Sep 11 16:16:23.917952 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:23.917913 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfv4d\" (UniqueName: \"kubernetes.io/projected/c498b4cb-bfa7-46e0-9c5a-148ae22b2f7f-kube-api-access-tfv4d\") pod \"test-curl-7092-544547\" (UID: \"c498b4cb-bfa7-46e0-9c5a-148ae22b2f7f\") " pod="opendatahub/test-curl-7092-544547" Sep 11 16:16:24.018840 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:24.018810 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfv4d\" (UniqueName: \"kubernetes.io/projected/c498b4cb-bfa7-46e0-9c5a-148ae22b2f7f-kube-api-access-tfv4d\") pod \"test-curl-7092-544547\" (UID: \"c498b4cb-bfa7-46e0-9c5a-148ae22b2f7f\") " pod="opendatahub/test-curl-7092-544547" Sep 11 16:16:24.026652 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:24.026622 2566 generic.go:356] "Generic (PLEG): container finished" podID="d7cf2979-5759-4264-95d0-3c92e1776914" containerID="c961c407ddfd4b117775b1c252bbfe59154b377f4015796ac230c33b0749ecd3" exitCode=137 Sep 11 16:16:24.027151 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:24.026663 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7092-6d45a3" Sep 11 16:16:24.027151 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:24.026719 2566 scope.go:117] "RemoveContainer" containerID="c961c407ddfd4b117775b1c252bbfe59154b377f4015796ac230c33b0749ecd3" Sep 11 16:16:24.027151 ip-10-0-136-96 kubenswrapper[2566]: E0911 16:16:24.027039 2566 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7092-a044f0" container="test-curl-7092-a044f0" Sep 11 16:16:24.029041 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:24.029016 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfv4d\" (UniqueName: \"kubernetes.io/projected/c498b4cb-bfa7-46e0-9c5a-148ae22b2f7f-kube-api-access-tfv4d\") pod \"test-curl-7092-544547\" (UID: \"c498b4cb-bfa7-46e0-9c5a-148ae22b2f7f\") " pod="opendatahub/test-curl-7092-544547" Sep 11 16:16:24.045253 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:24.045228 2566 scope.go:117] "RemoveContainer" containerID="c961c407ddfd4b117775b1c252bbfe59154b377f4015796ac230c33b0749ecd3" Sep 11 16:16:24.045545 ip-10-0-136-96 kubenswrapper[2566]: E0911 16:16:24.045525 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c961c407ddfd4b117775b1c252bbfe59154b377f4015796ac230c33b0749ecd3\": container with ID starting with c961c407ddfd4b117775b1c252bbfe59154b377f4015796ac230c33b0749ecd3 not found: ID does not exist" containerID="c961c407ddfd4b117775b1c252bbfe59154b377f4015796ac230c33b0749ecd3" Sep 11 16:16:24.045623 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:24.045557 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c961c407ddfd4b117775b1c252bbfe59154b377f4015796ac230c33b0749ecd3"} err="failed to get container status \"c961c407ddfd4b117775b1c252bbfe59154b377f4015796ac230c33b0749ecd3\": rpc error: code = NotFound desc = could not find container \"c961c407ddfd4b117775b1c252bbfe59154b377f4015796ac230c33b0749ecd3\": container with ID starting with c961c407ddfd4b117775b1c252bbfe59154b377f4015796ac230c33b0749ecd3 not found: ID does not exist" Sep 11 16:16:24.119062 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:24.119023 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7092-544547" Sep 11 16:16:24.394365 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:24.394272 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7092-544547"] Sep 11 16:16:24.397507 ip-10-0-136-96 kubenswrapper[2566]: W0911 16:16:24.397479 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc498b4cb_bfa7_46e0_9c5a_148ae22b2f7f.slice/crio-b816ec83332d40a6b691f17508abf4fd2a374c2336943679ff3c14deb087f10b WatchSource:0}: Error finding container b816ec83332d40a6b691f17508abf4fd2a374c2336943679ff3c14deb087f10b: Status 404 returned error can't find the container with id b816ec83332d40a6b691f17508abf4fd2a374c2336943679ff3c14deb087f10b Sep 11 16:16:25.036548 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:25.036503 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7092-544547" event={"ID":"c498b4cb-bfa7-46e0-9c5a-148ae22b2f7f","Type":"ContainerStarted","Data":"85512a16bed5951044a1cb5f98d15e2b139335a5d5131827db96b864ee92006d"} Sep 11 16:16:25.036908 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:25.036572 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7092-544547" event={"ID":"c498b4cb-bfa7-46e0-9c5a-148ae22b2f7f","Type":"ContainerStarted","Data":"b816ec83332d40a6b691f17508abf4fd2a374c2336943679ff3c14deb087f10b"} Sep 11 16:16:25.037904 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:25.037878 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7092-a044f0" podUID="519075bc-43a1-4519-85e4-4bd194a0d85e" containerName="test-curl-7092-a044f0" containerID="cri-o://2c9dc5e525e350ceef749be5d95aa412c4e0c340867401c722a0be8fe39876b8" gracePeriod=2 Sep 11 16:16:25.060812 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:25.060745 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7092-544547" podStartSLOduration=2.060724931 podStartE2EDuration="2.060724931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:16:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:16:25.060533097 +0000 UTC m=+1292.275367002" watchObservedRunningTime="2026-09-11 16:16:25.060724931 +0000 UTC m=+1292.275558836" Sep 11 16:16:25.062650 ip-10-0-136-96 kubenswrapper[2566]: E0911 16:16:25.062617 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7092-a044f0\" is forbidden: User \"system:node:ip-10-0-136-96.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-96.ec2.internal' and this object" podUID="519075bc-43a1-4519-85e4-4bd194a0d85e" pod="opendatahub/test-curl-7092-a044f0" Sep 11 16:16:25.173269 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:25.173240 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7092-7903ff" Sep 11 16:16:25.175073 ip-10-0-136-96 kubenswrapper[2566]: E0911 16:16:25.175047 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7092-a044f0\" is forbidden: User \"system:node:ip-10-0-136-96.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-96.ec2.internal' and this object" podUID="519075bc-43a1-4519-85e4-4bd194a0d85e" pod="opendatahub/test-curl-7092-a044f0" Sep 11 16:16:25.334035 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:25.333946 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9rdgm\" (UniqueName: \"kubernetes.io/projected/cd0fc716-151f-4f0a-8a53-e3f6eb5be43e-kube-api-access-9rdgm\") pod \"cd0fc716-151f-4f0a-8a53-e3f6eb5be43e\" (UID: \"cd0fc716-151f-4f0a-8a53-e3f6eb5be43e\") " Sep 11 16:16:25.335819 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:25.335790 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd0fc716-151f-4f0a-8a53-e3f6eb5be43e-kube-api-access-9rdgm" (OuterVolumeSpecName: "kube-api-access-9rdgm") pod "cd0fc716-151f-4f0a-8a53-e3f6eb5be43e" (UID: "cd0fc716-151f-4f0a-8a53-e3f6eb5be43e"). InnerVolumeSpecName "kube-api-access-9rdgm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 16:16:25.434910 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:25.434871 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rdgm\" (UniqueName: \"kubernetes.io/projected/cd0fc716-151f-4f0a-8a53-e3f6eb5be43e-kube-api-access-9rdgm\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:16:25.699432 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:25.699345 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7092-544547"] Sep 11 16:16:25.704481 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:25.704453 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7092-544547"] Sep 11 16:16:26.045873 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:26.045839 2566 generic.go:356] "Generic (PLEG): container finished" podID="cd0fc716-151f-4f0a-8a53-e3f6eb5be43e" containerID="d04f1d4ee92bc40e31d81be3ad1edb2b2e1a112416bfee514aa5bc23f76ca341" exitCode=137 Sep 11 16:16:26.045873 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:26.045876 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7092-7903ff" Sep 11 16:16:26.046353 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:26.045934 2566 scope.go:117] "RemoveContainer" containerID="d04f1d4ee92bc40e31d81be3ad1edb2b2e1a112416bfee514aa5bc23f76ca341" Sep 11 16:16:26.046353 ip-10-0-136-96 kubenswrapper[2566]: E0911 16:16:26.046179 2566 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7092-544547" container="test-curl-7092-544547" Sep 11 16:16:26.055965 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:26.055947 2566 scope.go:117] "RemoveContainer" containerID="d04f1d4ee92bc40e31d81be3ad1edb2b2e1a112416bfee514aa5bc23f76ca341" Sep 11 16:16:26.056269 ip-10-0-136-96 kubenswrapper[2566]: E0911 16:16:26.056251 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d04f1d4ee92bc40e31d81be3ad1edb2b2e1a112416bfee514aa5bc23f76ca341\": container with ID starting with d04f1d4ee92bc40e31d81be3ad1edb2b2e1a112416bfee514aa5bc23f76ca341 not found: ID does not exist" containerID="d04f1d4ee92bc40e31d81be3ad1edb2b2e1a112416bfee514aa5bc23f76ca341" Sep 11 16:16:26.056322 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:26.056279 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d04f1d4ee92bc40e31d81be3ad1edb2b2e1a112416bfee514aa5bc23f76ca341"} err="failed to get container status \"d04f1d4ee92bc40e31d81be3ad1edb2b2e1a112416bfee514aa5bc23f76ca341\": rpc error: code = NotFound desc = could not find container \"d04f1d4ee92bc40e31d81be3ad1edb2b2e1a112416bfee514aa5bc23f76ca341\": container with ID starting with d04f1d4ee92bc40e31d81be3ad1edb2b2e1a112416bfee514aa5bc23f76ca341 not found: ID does not exist" Sep 11 16:16:26.766507 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:26.766472 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-9da1d032-openshift-default-c7f5b6668-9czjm"] Sep 11 16:16:26.766992 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:26.766959 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c498b4cb-bfa7-46e0-9c5a-148ae22b2f7f" containerName="test-curl-7092-544547" Sep 11 16:16:26.767081 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:26.766994 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="c498b4cb-bfa7-46e0-9c5a-148ae22b2f7f" containerName="test-curl-7092-544547" Sep 11 16:16:26.767162 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:26.767137 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="c498b4cb-bfa7-46e0-9c5a-148ae22b2f7f" containerName="test-curl-7092-544547" Sep 11 16:16:26.770214 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:26.770195 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9da1d032-openshift-default-c7f5b6668-9czjm" Sep 11 16:16:26.772813 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:26.772796 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-9da1d032-openshift-default-dockercfg-x9tdg\"" Sep 11 16:16:26.796763 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:26.796738 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-9da1d032-openshift-default-c7f5b6668-9czjm"] Sep 11 16:16:26.848575 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:26.848530 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb9ecb75-970d-4c35-adc0-0f14c1b649a0-istio-podinfo\") pod \"e2e-mt-9da1d032-openshift-default-c7f5b6668-9czjm\" (UID: \"fb9ecb75-970d-4c35-adc0-0f14c1b649a0\") " pod="openshift-ingress/e2e-mt-9da1d032-openshift-default-c7f5b6668-9czjm" Sep 11 16:16:26.848851 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:26.848832 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb9ecb75-970d-4c35-adc0-0f14c1b649a0-istiod-ca-cert\") pod \"e2e-mt-9da1d032-openshift-default-c7f5b6668-9czjm\" (UID: \"fb9ecb75-970d-4c35-adc0-0f14c1b649a0\") " pod="openshift-ingress/e2e-mt-9da1d032-openshift-default-c7f5b6668-9czjm" Sep 11 16:16:26.849022 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:26.849006 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb9ecb75-970d-4c35-adc0-0f14c1b649a0-workload-socket\") pod \"e2e-mt-9da1d032-openshift-default-c7f5b6668-9czjm\" (UID: \"fb9ecb75-970d-4c35-adc0-0f14c1b649a0\") " pod="openshift-ingress/e2e-mt-9da1d032-openshift-default-c7f5b6668-9czjm" Sep 11 16:16:26.849233 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:26.849218 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb9ecb75-970d-4c35-adc0-0f14c1b649a0-istio-envoy\") pod \"e2e-mt-9da1d032-openshift-default-c7f5b6668-9czjm\" (UID: \"fb9ecb75-970d-4c35-adc0-0f14c1b649a0\") " pod="openshift-ingress/e2e-mt-9da1d032-openshift-default-c7f5b6668-9czjm" Sep 11 16:16:26.849423 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:26.849406 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb9ecb75-970d-4c35-adc0-0f14c1b649a0-credential-socket\") pod \"e2e-mt-9da1d032-openshift-default-c7f5b6668-9czjm\" (UID: \"fb9ecb75-970d-4c35-adc0-0f14c1b649a0\") " pod="openshift-ingress/e2e-mt-9da1d032-openshift-default-c7f5b6668-9czjm" Sep 11 16:16:26.849600 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:26.849584 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb9ecb75-970d-4c35-adc0-0f14c1b649a0-workload-certs\") pod \"e2e-mt-9da1d032-openshift-default-c7f5b6668-9czjm\" (UID: \"fb9ecb75-970d-4c35-adc0-0f14c1b649a0\") " pod="openshift-ingress/e2e-mt-9da1d032-openshift-default-c7f5b6668-9czjm" Sep 11 16:16:26.849740 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:26.849723 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpbqj\" (UniqueName: \"kubernetes.io/projected/fb9ecb75-970d-4c35-adc0-0f14c1b649a0-kube-api-access-zpbqj\") pod \"e2e-mt-9da1d032-openshift-default-c7f5b6668-9czjm\" (UID: \"fb9ecb75-970d-4c35-adc0-0f14c1b649a0\") " pod="openshift-ingress/e2e-mt-9da1d032-openshift-default-c7f5b6668-9czjm" Sep 11 16:16:26.849918 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:26.849901 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb9ecb75-970d-4c35-adc0-0f14c1b649a0-istio-data\") pod \"e2e-mt-9da1d032-openshift-default-c7f5b6668-9czjm\" (UID: \"fb9ecb75-970d-4c35-adc0-0f14c1b649a0\") " pod="openshift-ingress/e2e-mt-9da1d032-openshift-default-c7f5b6668-9czjm" Sep 11 16:16:26.850082 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:26.850067 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb9ecb75-970d-4c35-adc0-0f14c1b649a0-istio-token\") pod \"e2e-mt-9da1d032-openshift-default-c7f5b6668-9czjm\" (UID: \"fb9ecb75-970d-4c35-adc0-0f14c1b649a0\") " pod="openshift-ingress/e2e-mt-9da1d032-openshift-default-c7f5b6668-9czjm" Sep 11 16:16:26.951768 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:26.951733 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb9ecb75-970d-4c35-adc0-0f14c1b649a0-istio-data\") pod \"e2e-mt-9da1d032-openshift-default-c7f5b6668-9czjm\" (UID: \"fb9ecb75-970d-4c35-adc0-0f14c1b649a0\") " pod="openshift-ingress/e2e-mt-9da1d032-openshift-default-c7f5b6668-9czjm" Sep 11 16:16:26.951768 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:26.951776 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb9ecb75-970d-4c35-adc0-0f14c1b649a0-istio-token\") pod \"e2e-mt-9da1d032-openshift-default-c7f5b6668-9czjm\" (UID: \"fb9ecb75-970d-4c35-adc0-0f14c1b649a0\") " pod="openshift-ingress/e2e-mt-9da1d032-openshift-default-c7f5b6668-9czjm" Sep 11 16:16:26.952071 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:26.951842 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb9ecb75-970d-4c35-adc0-0f14c1b649a0-istio-podinfo\") pod \"e2e-mt-9da1d032-openshift-default-c7f5b6668-9czjm\" (UID: \"fb9ecb75-970d-4c35-adc0-0f14c1b649a0\") " pod="openshift-ingress/e2e-mt-9da1d032-openshift-default-c7f5b6668-9czjm" Sep 11 16:16:26.952071 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:26.951865 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb9ecb75-970d-4c35-adc0-0f14c1b649a0-istiod-ca-cert\") pod \"e2e-mt-9da1d032-openshift-default-c7f5b6668-9czjm\" (UID: \"fb9ecb75-970d-4c35-adc0-0f14c1b649a0\") " pod="openshift-ingress/e2e-mt-9da1d032-openshift-default-c7f5b6668-9czjm" Sep 11 16:16:26.952071 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:26.952030 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb9ecb75-970d-4c35-adc0-0f14c1b649a0-workload-socket\") pod \"e2e-mt-9da1d032-openshift-default-c7f5b6668-9czjm\" (UID: \"fb9ecb75-970d-4c35-adc0-0f14c1b649a0\") " pod="openshift-ingress/e2e-mt-9da1d032-openshift-default-c7f5b6668-9czjm" Sep 11 16:16:26.952253 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:26.952101 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb9ecb75-970d-4c35-adc0-0f14c1b649a0-istio-envoy\") pod \"e2e-mt-9da1d032-openshift-default-c7f5b6668-9czjm\" (UID: \"fb9ecb75-970d-4c35-adc0-0f14c1b649a0\") " pod="openshift-ingress/e2e-mt-9da1d032-openshift-default-c7f5b6668-9czjm" Sep 11 16:16:26.952253 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:26.952168 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb9ecb75-970d-4c35-adc0-0f14c1b649a0-credential-socket\") pod \"e2e-mt-9da1d032-openshift-default-c7f5b6668-9czjm\" (UID: \"fb9ecb75-970d-4c35-adc0-0f14c1b649a0\") " pod="openshift-ingress/e2e-mt-9da1d032-openshift-default-c7f5b6668-9czjm" Sep 11 16:16:26.952253 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:26.952214 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb9ecb75-970d-4c35-adc0-0f14c1b649a0-workload-certs\") pod \"e2e-mt-9da1d032-openshift-default-c7f5b6668-9czjm\" (UID: \"fb9ecb75-970d-4c35-adc0-0f14c1b649a0\") " pod="openshift-ingress/e2e-mt-9da1d032-openshift-default-c7f5b6668-9czjm" Sep 11 16:16:26.952253 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:26.952248 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpbqj\" (UniqueName: \"kubernetes.io/projected/fb9ecb75-970d-4c35-adc0-0f14c1b649a0-kube-api-access-zpbqj\") pod \"e2e-mt-9da1d032-openshift-default-c7f5b6668-9czjm\" (UID: \"fb9ecb75-970d-4c35-adc0-0f14c1b649a0\") " pod="openshift-ingress/e2e-mt-9da1d032-openshift-default-c7f5b6668-9czjm" Sep 11 16:16:26.952474 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:26.952215 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb9ecb75-970d-4c35-adc0-0f14c1b649a0-istio-data\") pod \"e2e-mt-9da1d032-openshift-default-c7f5b6668-9czjm\" (UID: \"fb9ecb75-970d-4c35-adc0-0f14c1b649a0\") " pod="openshift-ingress/e2e-mt-9da1d032-openshift-default-c7f5b6668-9czjm" Sep 11 16:16:26.952474 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:26.952363 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb9ecb75-970d-4c35-adc0-0f14c1b649a0-workload-socket\") pod \"e2e-mt-9da1d032-openshift-default-c7f5b6668-9czjm\" (UID: \"fb9ecb75-970d-4c35-adc0-0f14c1b649a0\") " pod="openshift-ingress/e2e-mt-9da1d032-openshift-default-c7f5b6668-9czjm" Sep 11 16:16:26.952628 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:26.952593 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb9ecb75-970d-4c35-adc0-0f14c1b649a0-credential-socket\") pod \"e2e-mt-9da1d032-openshift-default-c7f5b6668-9czjm\" (UID: \"fb9ecb75-970d-4c35-adc0-0f14c1b649a0\") " pod="openshift-ingress/e2e-mt-9da1d032-openshift-default-c7f5b6668-9czjm" Sep 11 16:16:26.952628 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:26.952619 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb9ecb75-970d-4c35-adc0-0f14c1b649a0-istiod-ca-cert\") pod \"e2e-mt-9da1d032-openshift-default-c7f5b6668-9czjm\" (UID: \"fb9ecb75-970d-4c35-adc0-0f14c1b649a0\") " pod="openshift-ingress/e2e-mt-9da1d032-openshift-default-c7f5b6668-9czjm" Sep 11 16:16:26.952829 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:26.952720 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb9ecb75-970d-4c35-adc0-0f14c1b649a0-workload-certs\") pod \"e2e-mt-9da1d032-openshift-default-c7f5b6668-9czjm\" (UID: \"fb9ecb75-970d-4c35-adc0-0f14c1b649a0\") " pod="openshift-ingress/e2e-mt-9da1d032-openshift-default-c7f5b6668-9czjm" Sep 11 16:16:26.954391 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:26.954367 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb9ecb75-970d-4c35-adc0-0f14c1b649a0-istio-envoy\") pod \"e2e-mt-9da1d032-openshift-default-c7f5b6668-9czjm\" (UID: \"fb9ecb75-970d-4c35-adc0-0f14c1b649a0\") " pod="openshift-ingress/e2e-mt-9da1d032-openshift-default-c7f5b6668-9czjm" Sep 11 16:16:26.954550 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:26.954534 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb9ecb75-970d-4c35-adc0-0f14c1b649a0-istio-podinfo\") pod \"e2e-mt-9da1d032-openshift-default-c7f5b6668-9czjm\" (UID: \"fb9ecb75-970d-4c35-adc0-0f14c1b649a0\") " pod="openshift-ingress/e2e-mt-9da1d032-openshift-default-c7f5b6668-9czjm" Sep 11 16:16:26.971905 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:26.971883 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb9ecb75-970d-4c35-adc0-0f14c1b649a0-istio-token\") pod \"e2e-mt-9da1d032-openshift-default-c7f5b6668-9czjm\" (UID: \"fb9ecb75-970d-4c35-adc0-0f14c1b649a0\") " pod="openshift-ingress/e2e-mt-9da1d032-openshift-default-c7f5b6668-9czjm" Sep 11 16:16:26.974910 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:26.974891 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpbqj\" (UniqueName: \"kubernetes.io/projected/fb9ecb75-970d-4c35-adc0-0f14c1b649a0-kube-api-access-zpbqj\") pod \"e2e-mt-9da1d032-openshift-default-c7f5b6668-9czjm\" (UID: \"fb9ecb75-970d-4c35-adc0-0f14c1b649a0\") " pod="openshift-ingress/e2e-mt-9da1d032-openshift-default-c7f5b6668-9czjm" Sep 11 16:16:27.055204 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:27.055170 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7092-544547" podUID="c498b4cb-bfa7-46e0-9c5a-148ae22b2f7f" containerName="test-curl-7092-544547" containerID="cri-o://85512a16bed5951044a1cb5f98d15e2b139335a5d5131827db96b864ee92006d" gracePeriod=2 Sep 11 16:16:27.060013 ip-10-0-136-96 kubenswrapper[2566]: E0911 16:16:27.059965 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7092-544547\" is forbidden: User \"system:node:ip-10-0-136-96.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-96.ec2.internal' and this object" podUID="c498b4cb-bfa7-46e0-9c5a-148ae22b2f7f" pod="opendatahub/test-curl-7092-544547" Sep 11 16:16:27.080694 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:27.080663 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9da1d032-openshift-default-c7f5b6668-9czjm" Sep 11 16:16:27.198192 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:27.198166 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7092-a044f0" Sep 11 16:16:27.200963 ip-10-0-136-96 kubenswrapper[2566]: E0911 16:16:27.200931 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7092-544547\" is forbidden: User \"system:node:ip-10-0-136-96.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-96.ec2.internal' and this object" podUID="c498b4cb-bfa7-46e0-9c5a-148ae22b2f7f" pod="opendatahub/test-curl-7092-544547" Sep 11 16:16:27.240393 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:27.240362 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd0fc716-151f-4f0a-8a53-e3f6eb5be43e" path="/var/lib/kubelet/pods/cd0fc716-151f-4f0a-8a53-e3f6eb5be43e/volumes" Sep 11 16:16:27.356734 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:27.356651 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p6sfq\" (UniqueName: \"kubernetes.io/projected/519075bc-43a1-4519-85e4-4bd194a0d85e-kube-api-access-p6sfq\") pod \"519075bc-43a1-4519-85e4-4bd194a0d85e\" (UID: \"519075bc-43a1-4519-85e4-4bd194a0d85e\") " Sep 11 16:16:27.358879 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:27.358848 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/519075bc-43a1-4519-85e4-4bd194a0d85e-kube-api-access-p6sfq" (OuterVolumeSpecName: "kube-api-access-p6sfq") pod "519075bc-43a1-4519-85e4-4bd194a0d85e" (UID: "519075bc-43a1-4519-85e4-4bd194a0d85e"). InnerVolumeSpecName "kube-api-access-p6sfq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 16:16:27.375053 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:27.375030 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-9da1d032-openshift-default-c7f5b6668-9czjm"] Sep 11 16:16:27.376955 ip-10-0-136-96 kubenswrapper[2566]: W0911 16:16:27.376928 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb9ecb75_970d_4c35_adc0_0f14c1b649a0.slice/crio-bcc27f87d2b63eb31a82d93ad0c8a7fc29f4aaef102c78704056a5ba0484df47 WatchSource:0}: Error finding container bcc27f87d2b63eb31a82d93ad0c8a7fc29f4aaef102c78704056a5ba0484df47: Status 404 returned error can't find the container with id bcc27f87d2b63eb31a82d93ad0c8a7fc29f4aaef102c78704056a5ba0484df47 Sep 11 16:16:27.379452 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:27.379423 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 16:16:27.379529 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:27.379488 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 16:16:27.379529 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:27.379516 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 16:16:27.457795 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:27.457768 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6sfq\" (UniqueName: \"kubernetes.io/projected/519075bc-43a1-4519-85e4-4bd194a0d85e-kube-api-access-p6sfq\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:16:28.063753 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:28.063719 2566 generic.go:356] "Generic (PLEG): container finished" podID="519075bc-43a1-4519-85e4-4bd194a0d85e" containerID="2c9dc5e525e350ceef749be5d95aa412c4e0c340867401c722a0be8fe39876b8" exitCode=137 Sep 11 16:16:28.064224 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:28.063783 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7092-a044f0" Sep 11 16:16:28.064224 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:28.063823 2566 scope.go:117] "RemoveContainer" containerID="2c9dc5e525e350ceef749be5d95aa412c4e0c340867401c722a0be8fe39876b8" Sep 11 16:16:28.065802 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:28.065777 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9da1d032-openshift-default-c7f5b6668-9czjm" event={"ID":"fb9ecb75-970d-4c35-adc0-0f14c1b649a0","Type":"ContainerStarted","Data":"322a69f6d050d06dcbced073c6b55a64e5ead8be4ae5fa2c387f4e3e45e68bff"} Sep 11 16:16:28.065891 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:28.065839 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9da1d032-openshift-default-c7f5b6668-9czjm" event={"ID":"fb9ecb75-970d-4c35-adc0-0f14c1b649a0","Type":"ContainerStarted","Data":"bcc27f87d2b63eb31a82d93ad0c8a7fc29f4aaef102c78704056a5ba0484df47"} Sep 11 16:16:28.074944 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:28.074923 2566 scope.go:117] "RemoveContainer" containerID="2c9dc5e525e350ceef749be5d95aa412c4e0c340867401c722a0be8fe39876b8" Sep 11 16:16:28.075261 ip-10-0-136-96 kubenswrapper[2566]: E0911 16:16:28.075242 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c9dc5e525e350ceef749be5d95aa412c4e0c340867401c722a0be8fe39876b8\": container with ID starting with 2c9dc5e525e350ceef749be5d95aa412c4e0c340867401c722a0be8fe39876b8 not found: ID does not exist" containerID="2c9dc5e525e350ceef749be5d95aa412c4e0c340867401c722a0be8fe39876b8" Sep 11 16:16:28.075328 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:28.075274 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c9dc5e525e350ceef749be5d95aa412c4e0c340867401c722a0be8fe39876b8"} err="failed to get container status \"2c9dc5e525e350ceef749be5d95aa412c4e0c340867401c722a0be8fe39876b8\": rpc error: code = NotFound desc = could not find container \"2c9dc5e525e350ceef749be5d95aa412c4e0c340867401c722a0be8fe39876b8\": container with ID starting with 2c9dc5e525e350ceef749be5d95aa412c4e0c340867401c722a0be8fe39876b8 not found: ID does not exist" Sep 11 16:16:28.081492 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:28.081471 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-9da1d032-openshift-default-c7f5b6668-9czjm" Sep 11 16:16:28.086590 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:28.086570 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-9da1d032-openshift-default-c7f5b6668-9czjm" Sep 11 16:16:28.097201 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:28.097150 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-9da1d032-openshift-default-c7f5b6668-9czjm" podStartSLOduration=2.097132633 podStartE2EDuration="2.097132633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:16:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:16:28.094671971 +0000 UTC m=+1295.309505875" watchObservedRunningTime="2026-09-11 16:16:28.097132633 +0000 UTC m=+1295.311966545" Sep 11 16:16:29.081734 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:29.081704 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-9da1d032-openshift-default-c7f5b6668-9czjm" Sep 11 16:16:29.082669 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:29.082645 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-9da1d032-openshift-default-c7f5b6668-9czjm" Sep 11 16:16:29.245096 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:29.245066 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="519075bc-43a1-4519-85e4-4bd194a0d85e" path="/var/lib/kubelet/pods/519075bc-43a1-4519-85e4-4bd194a0d85e/volumes" Sep 11 16:16:29.247668 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:29.247650 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7092-544547" Sep 11 16:16:29.382144 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:29.382065 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tfv4d\" (UniqueName: \"kubernetes.io/projected/c498b4cb-bfa7-46e0-9c5a-148ae22b2f7f-kube-api-access-tfv4d\") pod \"c498b4cb-bfa7-46e0-9c5a-148ae22b2f7f\" (UID: \"c498b4cb-bfa7-46e0-9c5a-148ae22b2f7f\") " Sep 11 16:16:29.384155 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:29.384128 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c498b4cb-bfa7-46e0-9c5a-148ae22b2f7f-kube-api-access-tfv4d" (OuterVolumeSpecName: "kube-api-access-tfv4d") pod "c498b4cb-bfa7-46e0-9c5a-148ae22b2f7f" (UID: "c498b4cb-bfa7-46e0-9c5a-148ae22b2f7f"). InnerVolumeSpecName "kube-api-access-tfv4d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 16:16:29.482926 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:29.482895 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfv4d\" (UniqueName: \"kubernetes.io/projected/c498b4cb-bfa7-46e0-9c5a-148ae22b2f7f-kube-api-access-tfv4d\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:16:30.089458 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:30.089425 2566 generic.go:356] "Generic (PLEG): container finished" podID="c498b4cb-bfa7-46e0-9c5a-148ae22b2f7f" containerID="85512a16bed5951044a1cb5f98d15e2b139335a5d5131827db96b864ee92006d" exitCode=137 Sep 11 16:16:30.089895 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:30.089471 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7092-544547" Sep 11 16:16:30.089895 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:30.089520 2566 scope.go:117] "RemoveContainer" containerID="85512a16bed5951044a1cb5f98d15e2b139335a5d5131827db96b864ee92006d" Sep 11 16:16:30.102288 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:30.102149 2566 scope.go:117] "RemoveContainer" containerID="85512a16bed5951044a1cb5f98d15e2b139335a5d5131827db96b864ee92006d" Sep 11 16:16:30.102445 ip-10-0-136-96 kubenswrapper[2566]: E0911 16:16:30.102428 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85512a16bed5951044a1cb5f98d15e2b139335a5d5131827db96b864ee92006d\": container with ID starting with 85512a16bed5951044a1cb5f98d15e2b139335a5d5131827db96b864ee92006d not found: ID does not exist" containerID="85512a16bed5951044a1cb5f98d15e2b139335a5d5131827db96b864ee92006d" Sep 11 16:16:30.102488 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:30.102455 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85512a16bed5951044a1cb5f98d15e2b139335a5d5131827db96b864ee92006d"} err="failed to get container status \"85512a16bed5951044a1cb5f98d15e2b139335a5d5131827db96b864ee92006d\": rpc error: code = NotFound desc = could not find container \"85512a16bed5951044a1cb5f98d15e2b139335a5d5131827db96b864ee92006d\": container with ID starting with 85512a16bed5951044a1cb5f98d15e2b139335a5d5131827db96b864ee92006d not found: ID does not exist" Sep 11 16:16:30.102606 ip-10-0-136-96 kubenswrapper[2566]: E0911 16:16:30.102585 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7092-544547\" is forbidden: User \"system:node:ip-10-0-136-96.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-96.ec2.internal' and this object" podUID="c498b4cb-bfa7-46e0-9c5a-148ae22b2f7f" pod="opendatahub/test-curl-7092-544547" Sep 11 16:16:31.240947 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:31.240854 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c498b4cb-bfa7-46e0-9c5a-148ae22b2f7f" path="/var/lib/kubelet/pods/c498b4cb-bfa7-46e0-9c5a-148ae22b2f7f/volumes" Sep 11 16:16:39.433917 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:39.433880 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dx4xn"] Sep 11 16:16:50.234552 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:50.234511 2566 generic.go:356] "Generic (PLEG): container finished" podID="fb9ecb75-970d-4c35-adc0-0f14c1b649a0" containerID="322a69f6d050d06dcbced073c6b55a64e5ead8be4ae5fa2c387f4e3e45e68bff" exitCode=0 Sep 11 16:16:50.235106 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:50.234567 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9da1d032-openshift-default-c7f5b6668-9czjm" event={"ID":"fb9ecb75-970d-4c35-adc0-0f14c1b649a0","Type":"ContainerDied","Data":"322a69f6d050d06dcbced073c6b55a64e5ead8be4ae5fa2c387f4e3e45e68bff"} Sep 11 16:16:50.235180 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:50.235158 2566 scope.go:117] "RemoveContainer" containerID="322a69f6d050d06dcbced073c6b55a64e5ead8be4ae5fa2c387f4e3e45e68bff" Sep 11 16:16:50.236414 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:50.236380 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 16:16:50.236487 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:50.236470 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 16:16:50.236594 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:50.236514 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 16:16:51.252209 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:51.251156 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9da1d032-openshift-default-c7f5b6668-9czjm" event={"ID":"fb9ecb75-970d-4c35-adc0-0f14c1b649a0","Type":"ContainerStarted","Data":"73cc562d1d8f8e2920ed7a31c2457de7f0e716a7ea82b2a54385eb8f215d5ec7"} Sep 11 16:16:52.081594 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:52.081555 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-9da1d032-openshift-default-c7f5b6668-9czjm" Sep 11 16:16:52.081856 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:52.081833 2566 patch_prober.go:28] interesting pod/e2e-mt-9da1d032-openshift-default-c7f5b6668-9czjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Sep 11 16:16:52.081919 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:52.081882 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9da1d032-openshift-default-c7f5b6668-9czjm" podUID="fb9ecb75-970d-4c35-adc0-0f14c1b649a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Sep 11 16:16:53.081071 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:53.081036 2566 patch_prober.go:28] interesting pod/e2e-mt-9da1d032-openshift-default-c7f5b6668-9czjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Sep 11 16:16:53.081443 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:53.081095 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9da1d032-openshift-default-c7f5b6668-9czjm" podUID="fb9ecb75-970d-4c35-adc0-0f14c1b649a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Sep 11 16:16:53.128954 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:53.128920 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dx4xn"] Sep 11 16:16:53.475326 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:53.475235 2566 scope.go:117] "RemoveContainer" containerID="cec3bd2c980a438e8c1168eea8b5560a310203ffaaf7efb273f5d3a626ecdfc1" Sep 11 16:16:53.486039 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:53.486006 2566 scope.go:117] "RemoveContainer" containerID="ec9248bc34432f1a2b283acba1cece731b054aead8be06bedc56e92c59284dea" Sep 11 16:16:54.082106 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:54.082068 2566 patch_prober.go:28] interesting pod/e2e-mt-9da1d032-openshift-default-c7f5b6668-9czjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Sep 11 16:16:54.082496 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:54.082126 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9da1d032-openshift-default-c7f5b6668-9czjm" podUID="fb9ecb75-970d-4c35-adc0-0f14c1b649a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Sep 11 16:16:55.081081 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:55.081040 2566 patch_prober.go:28] interesting pod/e2e-mt-9da1d032-openshift-default-c7f5b6668-9czjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Sep 11 16:16:55.081259 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:55.081105 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9da1d032-openshift-default-c7f5b6668-9czjm" podUID="fb9ecb75-970d-4c35-adc0-0f14c1b649a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Sep 11 16:16:56.081567 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:56.081530 2566 patch_prober.go:28] interesting pod/e2e-mt-9da1d032-openshift-default-c7f5b6668-9czjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Sep 11 16:16:56.081943 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:56.081586 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9da1d032-openshift-default-c7f5b6668-9czjm" podUID="fb9ecb75-970d-4c35-adc0-0f14c1b649a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Sep 11 16:16:57.086105 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:57.086073 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-9da1d032-openshift-default-c7f5b6668-9czjm" Sep 11 16:16:57.086482 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:57.086326 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-9da1d032-openshift-default-c7f5b6668-9czjm" Sep 11 16:16:57.086994 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:16:57.086965 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-9da1d032-openshift-default-c7f5b6668-9czjm" Sep 11 16:17:08.237728 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:17:08.237689 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dx4xn"] Sep 11 16:17:22.249767 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:17:22.249729 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dx4xn"] Sep 11 16:17:38.923290 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:17:38.923249 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dx4xn"] Sep 11 16:17:44.692205 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:17:44.692165 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9da1d032-qcckf"] Sep 11 16:17:44.694874 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:17:44.694856 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9da1d032-qcckf" Sep 11 16:17:44.698429 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:17:44.698297 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qbzdr\"" Sep 11 16:17:44.708261 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:17:44.708232 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 16:17:44.713332 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:17:44.713308 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9da1d032-qcckf"] Sep 11 16:17:44.761587 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:17:44.761553 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c58fe3e-a6b9-4c47-99d1-e7e5d8414188-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-9da1d032-qcckf\" (UID: \"8c58fe3e-a6b9-4c47-99d1-e7e5d8414188\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9da1d032-qcckf" Sep 11 16:17:44.862219 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:17:44.862185 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c58fe3e-a6b9-4c47-99d1-e7e5d8414188-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-9da1d032-qcckf\" (UID: \"8c58fe3e-a6b9-4c47-99d1-e7e5d8414188\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9da1d032-qcckf" Sep 11 16:17:44.862818 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:17:44.862797 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c58fe3e-a6b9-4c47-99d1-e7e5d8414188-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-9da1d032-qcckf\" (UID: \"8c58fe3e-a6b9-4c47-99d1-e7e5d8414188\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9da1d032-qcckf" Sep 11 16:17:45.008742 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:17:45.008653 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9da1d032-qcckf" Sep 11 16:17:45.288198 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:17:45.288170 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9da1d032-qcckf"] Sep 11 16:17:45.289762 ip-10-0-136-96 kubenswrapper[2566]: W0911 16:17:45.289736 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c58fe3e_a6b9_4c47_99d1_e7e5d8414188.slice/crio-514e2aad5b5ea0cf08a3703efb70f50c3b191cf3af86dd69070b5a87225195cf WatchSource:0}: Error finding container 514e2aad5b5ea0cf08a3703efb70f50c3b191cf3af86dd69070b5a87225195cf: Status 404 returned error can't find the container with id 514e2aad5b5ea0cf08a3703efb70f50c3b191cf3af86dd69070b5a87225195cf Sep 11 16:17:45.618829 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:17:45.618782 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9da1d032-qcckf" event={"ID":"8c58fe3e-a6b9-4c47-99d1-e7e5d8414188","Type":"ContainerStarted","Data":"805db22686b8379ada2eafbc372d32ec0f48ce68f9b30c1bc2bdcd7b9f05e41a"} Sep 11 16:17:45.619034 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:17:45.618847 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9da1d032-qcckf" event={"ID":"8c58fe3e-a6b9-4c47-99d1-e7e5d8414188","Type":"ContainerStarted","Data":"514e2aad5b5ea0cf08a3703efb70f50c3b191cf3af86dd69070b5a87225195cf"} Sep 11 16:17:45.637075 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:17:45.637029 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9da1d032-qcckf" podStartSLOduration=1.637013474 podStartE2EDuration="1.637013474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:17:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:17:45.635177714 +0000 UTC m=+1372.850011618" watchObservedRunningTime="2026-09-11 16:17:45.637013474 +0000 UTC m=+1372.851847383" Sep 11 16:17:46.626966 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:17:46.626932 2566 generic.go:356] "Generic (PLEG): container finished" podID="8c58fe3e-a6b9-4c47-99d1-e7e5d8414188" containerID="805db22686b8379ada2eafbc372d32ec0f48ce68f9b30c1bc2bdcd7b9f05e41a" exitCode=0 Sep 11 16:17:46.627359 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:17:46.627007 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9da1d032-qcckf" event={"ID":"8c58fe3e-a6b9-4c47-99d1-e7e5d8414188","Type":"ContainerDied","Data":"805db22686b8379ada2eafbc372d32ec0f48ce68f9b30c1bc2bdcd7b9f05e41a"} Sep 11 16:17:47.770265 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:17:47.770238 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9da1d032-qcckf" Sep 11 16:17:47.789647 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:17:47.789607 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c58fe3e-a6b9-4c47-99d1-e7e5d8414188-maas-api-service-ca\") pod \"8c58fe3e-a6b9-4c47-99d1-e7e5d8414188\" (UID: \"8c58fe3e-a6b9-4c47-99d1-e7e5d8414188\") " Sep 11 16:17:47.790006 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:17:47.789942 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c58fe3e-a6b9-4c47-99d1-e7e5d8414188-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8c58fe3e-a6b9-4c47-99d1-e7e5d8414188" (UID: "8c58fe3e-a6b9-4c47-99d1-e7e5d8414188"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 16:17:47.790222 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:17:47.790122 2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c58fe3e-a6b9-4c47-99d1-e7e5d8414188-maas-api-service-ca\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:17:48.642153 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:17:48.642117 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9da1d032-qcckf" Sep 11 16:17:48.642355 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:17:48.642118 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9da1d032-qcckf" event={"ID":"8c58fe3e-a6b9-4c47-99d1-e7e5d8414188","Type":"ContainerDied","Data":"514e2aad5b5ea0cf08a3703efb70f50c3b191cf3af86dd69070b5a87225195cf"} Sep 11 16:17:48.642355 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:17:48.642275 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="514e2aad5b5ea0cf08a3703efb70f50c3b191cf3af86dd69070b5a87225195cf" Sep 11 16:17:53.536834 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:17:53.536801 2566 scope.go:117] "RemoveContainer" containerID="5e2fc177a46b80d225fc692497577b5e9df4d44333c5b04d881972b697449a59" Sep 11 16:17:55.441578 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:17:55.441541 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dx4xn"] Sep 11 16:18:01.513043 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:18:01.513008 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9da1d032-qcckf"] Sep 11 16:18:01.516386 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:18:01.516361 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9da1d032-qcckf"] Sep 11 16:18:03.239300 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:18:03.239260 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c58fe3e-a6b9-4c47-99d1-e7e5d8414188" path="/var/lib/kubelet/pods/8c58fe3e-a6b9-4c47-99d1-e7e5d8414188/volumes" Sep 11 16:18:10.233320 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:18:10.233282 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dx4xn"] Sep 11 16:18:26.641520 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:18:26.641482 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dx4xn"] Sep 11 16:18:42.139788 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:18:42.139745 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dx4xn"] Sep 11 16:18:53.564772 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:18:53.564689 2566 scope.go:117] "RemoveContainer" containerID="84791abd3445f9ddf2ddb9fb510dc853e7d22e89dba0e9af34797562b85c3512" Sep 11 16:18:53.574308 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:18:53.574286 2566 scope.go:117] "RemoveContainer" containerID="9d5c419a420bd1a2f356ba495dbbf7db4a7d0de1ea013c82ac81ea0c50223511" Sep 11 16:19:00.025696 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:19:00.025655 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dx4xn"] Sep 11 16:19:00.930596 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:19:00.930559 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-9da1d032-openshift-default-c7f5b6668-9czjm"] Sep 11 16:19:00.930863 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:19:00.930838 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-9da1d032-openshift-default-c7f5b6668-9czjm" podUID="fb9ecb75-970d-4c35-adc0-0f14c1b649a0" containerName="istio-proxy" containerID="cri-o://73cc562d1d8f8e2920ed7a31c2457de7f0e716a7ea82b2a54385eb8f215d5ec7" gracePeriod=30 Sep 11 16:19:06.153827 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:19:06.153793 2566 generic.go:356] "Generic (PLEG): container finished" podID="fb9ecb75-970d-4c35-adc0-0f14c1b649a0" containerID="73cc562d1d8f8e2920ed7a31c2457de7f0e716a7ea82b2a54385eb8f215d5ec7" exitCode=0 Sep 11 16:19:06.154334 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:19:06.153866 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9da1d032-openshift-default-c7f5b6668-9czjm" event={"ID":"fb9ecb75-970d-4c35-adc0-0f14c1b649a0","Type":"ContainerDied","Data":"73cc562d1d8f8e2920ed7a31c2457de7f0e716a7ea82b2a54385eb8f215d5ec7"} Sep 11 16:19:06.154334 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:19:06.153929 2566 scope.go:117] "RemoveContainer" containerID="322a69f6d050d06dcbced073c6b55a64e5ead8be4ae5fa2c387f4e3e45e68bff" Sep 11 16:19:06.188026 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:19:06.188004 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9da1d032-openshift-default-c7f5b6668-9czjm" Sep 11 16:19:06.285680 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:19:06.285654 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb9ecb75-970d-4c35-adc0-0f14c1b649a0-istio-data\") pod \"fb9ecb75-970d-4c35-adc0-0f14c1b649a0\" (UID: \"fb9ecb75-970d-4c35-adc0-0f14c1b649a0\") " Sep 11 16:19:06.285856 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:19:06.285702 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zpbqj\" (UniqueName: \"kubernetes.io/projected/fb9ecb75-970d-4c35-adc0-0f14c1b649a0-kube-api-access-zpbqj\") pod \"fb9ecb75-970d-4c35-adc0-0f14c1b649a0\" (UID: \"fb9ecb75-970d-4c35-adc0-0f14c1b649a0\") " Sep 11 16:19:06.285856 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:19:06.285757 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb9ecb75-970d-4c35-adc0-0f14c1b649a0-workload-socket\") pod \"fb9ecb75-970d-4c35-adc0-0f14c1b649a0\" (UID: \"fb9ecb75-970d-4c35-adc0-0f14c1b649a0\") " Sep 11 16:19:06.285856 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:19:06.285774 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb9ecb75-970d-4c35-adc0-0f14c1b649a0-istio-token\") pod \"fb9ecb75-970d-4c35-adc0-0f14c1b649a0\" (UID: \"fb9ecb75-970d-4c35-adc0-0f14c1b649a0\") " Sep 11 16:19:06.285856 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:19:06.285802 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb9ecb75-970d-4c35-adc0-0f14c1b649a0-credential-socket\") pod \"fb9ecb75-970d-4c35-adc0-0f14c1b649a0\" (UID: \"fb9ecb75-970d-4c35-adc0-0f14c1b649a0\") " Sep 11 16:19:06.286124 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:19:06.285886 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb9ecb75-970d-4c35-adc0-0f14c1b649a0-istio-podinfo\") pod \"fb9ecb75-970d-4c35-adc0-0f14c1b649a0\" (UID: \"fb9ecb75-970d-4c35-adc0-0f14c1b649a0\") " Sep 11 16:19:06.286124 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:19:06.285910 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb9ecb75-970d-4c35-adc0-0f14c1b649a0-workload-certs\") pod \"fb9ecb75-970d-4c35-adc0-0f14c1b649a0\" (UID: \"fb9ecb75-970d-4c35-adc0-0f14c1b649a0\") " Sep 11 16:19:06.286124 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:19:06.285963 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb9ecb75-970d-4c35-adc0-0f14c1b649a0-istiod-ca-cert\") pod \"fb9ecb75-970d-4c35-adc0-0f14c1b649a0\" (UID: \"fb9ecb75-970d-4c35-adc0-0f14c1b649a0\") " Sep 11 16:19:06.286124 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:19:06.285964 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb9ecb75-970d-4c35-adc0-0f14c1b649a0-istio-data" (OuterVolumeSpecName: "istio-data") pod "fb9ecb75-970d-4c35-adc0-0f14c1b649a0" (UID: "fb9ecb75-970d-4c35-adc0-0f14c1b649a0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 16:19:06.286124 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:19:06.286036 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb9ecb75-970d-4c35-adc0-0f14c1b649a0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "fb9ecb75-970d-4c35-adc0-0f14c1b649a0" (UID: "fb9ecb75-970d-4c35-adc0-0f14c1b649a0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 16:19:06.286124 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:19:06.286057 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb9ecb75-970d-4c35-adc0-0f14c1b649a0-istio-envoy\") pod \"fb9ecb75-970d-4c35-adc0-0f14c1b649a0\" (UID: \"fb9ecb75-970d-4c35-adc0-0f14c1b649a0\") " Sep 11 16:19:06.286433 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:19:06.286264 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb9ecb75-970d-4c35-adc0-0f14c1b649a0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "fb9ecb75-970d-4c35-adc0-0f14c1b649a0" (UID: "fb9ecb75-970d-4c35-adc0-0f14c1b649a0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 16:19:06.286433 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:19:06.286282 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb9ecb75-970d-4c35-adc0-0f14c1b649a0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "fb9ecb75-970d-4c35-adc0-0f14c1b649a0" (UID: "fb9ecb75-970d-4c35-adc0-0f14c1b649a0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 16:19:06.286433 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:19:06.286349 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb9ecb75-970d-4c35-adc0-0f14c1b649a0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "fb9ecb75-970d-4c35-adc0-0f14c1b649a0" (UID: "fb9ecb75-970d-4c35-adc0-0f14c1b649a0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 16:19:06.286574 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:19:06.286514 2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb9ecb75-970d-4c35-adc0-0f14c1b649a0-istio-data\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:19:06.286574 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:19:06.286533 2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb9ecb75-970d-4c35-adc0-0f14c1b649a0-workload-socket\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:19:06.286574 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:19:06.286546 2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb9ecb75-970d-4c35-adc0-0f14c1b649a0-credential-socket\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:19:06.286574 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:19:06.286560 2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb9ecb75-970d-4c35-adc0-0f14c1b649a0-workload-certs\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:19:06.286574 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:19:06.286573 2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb9ecb75-970d-4c35-adc0-0f14c1b649a0-istiod-ca-cert\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:19:06.288108 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:19:06.288076 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fb9ecb75-970d-4c35-adc0-0f14c1b649a0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "fb9ecb75-970d-4c35-adc0-0f14c1b649a0" (UID: "fb9ecb75-970d-4c35-adc0-0f14c1b649a0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 16:19:06.288212 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:19:06.288175 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb9ecb75-970d-4c35-adc0-0f14c1b649a0-kube-api-access-zpbqj" (OuterVolumeSpecName: "kube-api-access-zpbqj") pod "fb9ecb75-970d-4c35-adc0-0f14c1b649a0" (UID: "fb9ecb75-970d-4c35-adc0-0f14c1b649a0"). InnerVolumeSpecName "kube-api-access-zpbqj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 16:19:06.288466 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:19:06.288451 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb9ecb75-970d-4c35-adc0-0f14c1b649a0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "fb9ecb75-970d-4c35-adc0-0f14c1b649a0" (UID: "fb9ecb75-970d-4c35-adc0-0f14c1b649a0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 16:19:06.288513 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:19:06.288485 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb9ecb75-970d-4c35-adc0-0f14c1b649a0-istio-token" (OuterVolumeSpecName: "istio-token") pod "fb9ecb75-970d-4c35-adc0-0f14c1b649a0" (UID: "fb9ecb75-970d-4c35-adc0-0f14c1b649a0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 16:19:06.387716 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:19:06.387682 2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb9ecb75-970d-4c35-adc0-0f14c1b649a0-istio-podinfo\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:19:06.387716 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:19:06.387712 2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb9ecb75-970d-4c35-adc0-0f14c1b649a0-istio-envoy\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:19:06.387716 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:19:06.387723 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpbqj\" (UniqueName: \"kubernetes.io/projected/fb9ecb75-970d-4c35-adc0-0f14c1b649a0-kube-api-access-zpbqj\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:19:06.387957 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:19:06.387732 2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb9ecb75-970d-4c35-adc0-0f14c1b649a0-istio-token\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:19:07.161897 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:19:07.161867 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9da1d032-openshift-default-c7f5b6668-9czjm" Sep 11 16:19:07.162386 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:19:07.161914 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9da1d032-openshift-default-c7f5b6668-9czjm" event={"ID":"fb9ecb75-970d-4c35-adc0-0f14c1b649a0","Type":"ContainerDied","Data":"bcc27f87d2b63eb31a82d93ad0c8a7fc29f4aaef102c78704056a5ba0484df47"} Sep 11 16:19:07.162386 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:19:07.161998 2566 scope.go:117] "RemoveContainer" containerID="73cc562d1d8f8e2920ed7a31c2457de7f0e716a7ea82b2a54385eb8f215d5ec7" Sep 11 16:19:07.195010 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:19:07.194959 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-9da1d032-openshift-default-c7f5b6668-9czjm"] Sep 11 16:19:07.200069 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:19:07.200031 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-9da1d032-openshift-default-c7f5b6668-9czjm"] Sep 11 16:19:07.241600 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:19:07.241567 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb9ecb75-970d-4c35-adc0-0f14c1b649a0" path="/var/lib/kubelet/pods/fb9ecb75-970d-4c35-adc0-0f14c1b649a0/volumes" Sep 11 16:19:15.314490 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:19:15.314448 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dx4xn"] Sep 11 16:19:30.635115 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:19:30.635079 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dx4xn"] Sep 11 16:19:33.354396 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:19:33.354359 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7095-33e756"] Sep 11 16:19:33.354922 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:19:33.354823 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c58fe3e-a6b9-4c47-99d1-e7e5d8414188" containerName="revoke-keys" Sep 11 16:19:33.354922 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:19:33.354834 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c58fe3e-a6b9-4c47-99d1-e7e5d8414188" containerName="revoke-keys" Sep 11 16:19:33.354922 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:19:33.354849 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb9ecb75-970d-4c35-adc0-0f14c1b649a0" containerName="istio-proxy" Sep 11 16:19:33.354922 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:19:33.354854 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb9ecb75-970d-4c35-adc0-0f14c1b649a0" containerName="istio-proxy" Sep 11 16:19:33.354922 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:19:33.354861 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb9ecb75-970d-4c35-adc0-0f14c1b649a0" containerName="istio-proxy" Sep 11 16:19:33.354922 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:19:33.354868 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb9ecb75-970d-4c35-adc0-0f14c1b649a0" containerName="istio-proxy" Sep 11 16:19:33.355162 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:19:33.354948 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb9ecb75-970d-4c35-adc0-0f14c1b649a0" containerName="istio-proxy" Sep 11 16:19:33.355162 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:19:33.354957 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb9ecb75-970d-4c35-adc0-0f14c1b649a0" containerName="istio-proxy" Sep 11 16:19:33.355162 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:19:33.354963 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c58fe3e-a6b9-4c47-99d1-e7e5d8414188" containerName="revoke-keys" Sep 11 16:19:33.357133 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:19:33.357117 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7095-33e756" Sep 11 16:19:33.360764 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:19:33.360737 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-vcw5k\"" Sep 11 16:19:33.376409 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:19:33.376387 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7095-33e756"] Sep 11 16:19:33.469554 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:19:33.469501 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7j45l\" (UniqueName: \"kubernetes.io/projected/b9f2cb71-c1c8-4bab-a824-2fac42e1e39b-kube-api-access-7j45l\") pod \"test-curl-7095-33e756\" (UID: \"b9f2cb71-c1c8-4bab-a824-2fac42e1e39b\") " pod="opendatahub/test-curl-7095-33e756" Sep 11 16:19:33.570859 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:19:33.570827 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7j45l\" (UniqueName: \"kubernetes.io/projected/b9f2cb71-c1c8-4bab-a824-2fac42e1e39b-kube-api-access-7j45l\") pod \"test-curl-7095-33e756\" (UID: \"b9f2cb71-c1c8-4bab-a824-2fac42e1e39b\") " pod="opendatahub/test-curl-7095-33e756" Sep 11 16:19:33.581600 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:19:33.581568 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7j45l\" (UniqueName: \"kubernetes.io/projected/b9f2cb71-c1c8-4bab-a824-2fac42e1e39b-kube-api-access-7j45l\") pod \"test-curl-7095-33e756\" (UID: \"b9f2cb71-c1c8-4bab-a824-2fac42e1e39b\") " pod="opendatahub/test-curl-7095-33e756" Sep 11 16:19:33.666403 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:19:33.666319 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7095-33e756" Sep 11 16:19:33.949725 ip-10-0-136-96 kubenswrapper[2566]: W0911 16:19:33.949695 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb9f2cb71_c1c8_4bab_a824_2fac42e1e39b.slice/crio-91e4689f54033d1fe45de4f10d4b0ea9aaae90416d1d491bae61cd5359bad565 WatchSource:0}: Error finding container 91e4689f54033d1fe45de4f10d4b0ea9aaae90416d1d491bae61cd5359bad565: Status 404 returned error can't find the container with id 91e4689f54033d1fe45de4f10d4b0ea9aaae90416d1d491bae61cd5359bad565 Sep 11 16:19:33.951146 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:19:33.951119 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7095-33e756"] Sep 11 16:19:34.342195 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:19:34.342158 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7095-33e756" event={"ID":"b9f2cb71-c1c8-4bab-a824-2fac42e1e39b","Type":"ContainerStarted","Data":"1ca26a891df4114c9d27d7b96d3eb950bd420f3468fa0505bc8de7ae3033124c"} Sep 11 16:19:34.342356 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:19:34.342210 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7095-33e756" event={"ID":"b9f2cb71-c1c8-4bab-a824-2fac42e1e39b","Type":"ContainerStarted","Data":"91e4689f54033d1fe45de4f10d4b0ea9aaae90416d1d491bae61cd5359bad565"} Sep 11 16:19:34.361809 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:19:34.361751 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7095-33e756" podStartSLOduration=1.36173426 podStartE2EDuration="1.36173426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:19:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:19:34.358869899 +0000 UTC m=+1481.573703805" watchObservedRunningTime="2026-09-11 16:19:34.36173426 +0000 UTC m=+1481.576568165" Sep 11 16:19:35.002598 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:19:35.002563 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7095-33e756"] Sep 11 16:19:35.009629 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:19:35.009601 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7095-33e756"] Sep 11 16:19:35.348164 ip-10-0-136-96 kubenswrapper[2566]: E0911 16:19:35.348135 2566 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7095-33e756" container="test-curl-7095-33e756" Sep 11 16:19:36.354755 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:19:36.354690 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7095-33e756" podUID="b9f2cb71-c1c8-4bab-a824-2fac42e1e39b" containerName="test-curl-7095-33e756" containerID="cri-o://1ca26a891df4114c9d27d7b96d3eb950bd420f3468fa0505bc8de7ae3033124c" gracePeriod=2 Sep 11 16:19:38.526486 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:19:38.526462 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7095-33e756" Sep 11 16:19:38.617125 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:19:38.617043 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7j45l\" (UniqueName: \"kubernetes.io/projected/b9f2cb71-c1c8-4bab-a824-2fac42e1e39b-kube-api-access-7j45l\") pod \"b9f2cb71-c1c8-4bab-a824-2fac42e1e39b\" (UID: \"b9f2cb71-c1c8-4bab-a824-2fac42e1e39b\") " Sep 11 16:19:38.619150 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:19:38.619127 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9f2cb71-c1c8-4bab-a824-2fac42e1e39b-kube-api-access-7j45l" (OuterVolumeSpecName: "kube-api-access-7j45l") pod "b9f2cb71-c1c8-4bab-a824-2fac42e1e39b" (UID: "b9f2cb71-c1c8-4bab-a824-2fac42e1e39b"). InnerVolumeSpecName "kube-api-access-7j45l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 16:19:38.718757 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:19:38.718722 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7j45l\" (UniqueName: \"kubernetes.io/projected/b9f2cb71-c1c8-4bab-a824-2fac42e1e39b-kube-api-access-7j45l\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:19:39.239666 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:19:39.239633 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9f2cb71-c1c8-4bab-a824-2fac42e1e39b" path="/var/lib/kubelet/pods/b9f2cb71-c1c8-4bab-a824-2fac42e1e39b/volumes" Sep 11 16:19:39.376187 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:19:39.376151 2566 generic.go:356] "Generic (PLEG): container finished" podID="b9f2cb71-c1c8-4bab-a824-2fac42e1e39b" containerID="1ca26a891df4114c9d27d7b96d3eb950bd420f3468fa0505bc8de7ae3033124c" exitCode=137 Sep 11 16:19:39.376365 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:19:39.376197 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7095-33e756" Sep 11 16:19:39.376365 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:19:39.376253 2566 scope.go:117] "RemoveContainer" containerID="1ca26a891df4114c9d27d7b96d3eb950bd420f3468fa0505bc8de7ae3033124c" Sep 11 16:19:39.385870 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:19:39.385835 2566 scope.go:117] "RemoveContainer" containerID="1ca26a891df4114c9d27d7b96d3eb950bd420f3468fa0505bc8de7ae3033124c" Sep 11 16:19:39.386205 ip-10-0-136-96 kubenswrapper[2566]: E0911 16:19:39.386184 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ca26a891df4114c9d27d7b96d3eb950bd420f3468fa0505bc8de7ae3033124c\": container with ID starting with 1ca26a891df4114c9d27d7b96d3eb950bd420f3468fa0505bc8de7ae3033124c not found: ID does not exist" containerID="1ca26a891df4114c9d27d7b96d3eb950bd420f3468fa0505bc8de7ae3033124c" Sep 11 16:19:39.386284 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:19:39.386213 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ca26a891df4114c9d27d7b96d3eb950bd420f3468fa0505bc8de7ae3033124c"} err="failed to get container status \"1ca26a891df4114c9d27d7b96d3eb950bd420f3468fa0505bc8de7ae3033124c\": rpc error: code = NotFound desc = could not find container \"1ca26a891df4114c9d27d7b96d3eb950bd420f3468fa0505bc8de7ae3033124c\": container with ID starting with 1ca26a891df4114c9d27d7b96d3eb950bd420f3468fa0505bc8de7ae3033124c not found: ID does not exist" Sep 11 16:19:41.318124 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:19:41.318088 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dx4xn"] Sep 11 16:19:51.534731 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:19:51.534687 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dx4xn"] Sep 11 16:19:53.422256 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:19:53.422228 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-xjvl5_7542ad77-c817-4370-91de-3b08db264753/console-operator/2.log" Sep 11 16:19:53.424549 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:19:53.424528 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-xjvl5_7542ad77-c817-4370-91de-3b08db264753/console-operator/2.log" Sep 11 16:19:53.429619 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:19:53.429595 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tgq5c_981d7d1e-f81f-48fc-8962-7831bc5c2b27/ovn-acl-logging/0.log" Sep 11 16:19:53.431636 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:19:53.431617 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tgq5c_981d7d1e-f81f-48fc-8962-7831bc5c2b27/ovn-acl-logging/0.log" Sep 11 16:19:53.601555 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:19:53.601522 2566 scope.go:117] "RemoveContainer" containerID="1484f8ab357b22383e38dca1786d2f8739188d9e78729cc93c6baafe7ac482c2" Sep 11 16:19:58.737988 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:19:58.737935 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-c98185-openshift-default-7c7758cdbd-8vq8j"] Sep 11 16:19:58.738557 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:19:58.738485 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9f2cb71-c1c8-4bab-a824-2fac42e1e39b" containerName="test-curl-7095-33e756" Sep 11 16:19:58.738557 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:19:58.738506 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9f2cb71-c1c8-4bab-a824-2fac42e1e39b" containerName="test-curl-7095-33e756" Sep 11 16:19:58.738681 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:19:58.738637 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9f2cb71-c1c8-4bab-a824-2fac42e1e39b" containerName="test-curl-7095-33e756" Sep 11 16:19:58.741715 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:19:58.741697 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c98185-openshift-default-7c7758cdbd-8vq8j" Sep 11 16:19:58.745251 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:19:58.745232 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-c98185-openshift-default-dockercfg-7jklt\"" Sep 11 16:19:58.763836 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:19:58.763809 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-c98185-openshift-default-7c7758cdbd-8vq8j"] Sep 11 16:19:58.830064 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:19:58.830025 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b262660d-18f1-4ec8-963d-60383e08d57d-credential-socket\") pod \"e2e-api-a-c98185-openshift-default-7c7758cdbd-8vq8j\" (UID: \"b262660d-18f1-4ec8-963d-60383e08d57d\") " pod="openshift-ingress/e2e-api-a-c98185-openshift-default-7c7758cdbd-8vq8j" Sep 11 16:19:58.830357 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:19:58.830337 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bgsk\" (UniqueName: \"kubernetes.io/projected/b262660d-18f1-4ec8-963d-60383e08d57d-kube-api-access-6bgsk\") pod \"e2e-api-a-c98185-openshift-default-7c7758cdbd-8vq8j\" (UID: \"b262660d-18f1-4ec8-963d-60383e08d57d\") " pod="openshift-ingress/e2e-api-a-c98185-openshift-default-7c7758cdbd-8vq8j" Sep 11 16:19:58.830485 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:19:58.830469 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b262660d-18f1-4ec8-963d-60383e08d57d-istio-podinfo\") pod \"e2e-api-a-c98185-openshift-default-7c7758cdbd-8vq8j\" (UID: \"b262660d-18f1-4ec8-963d-60383e08d57d\") " pod="openshift-ingress/e2e-api-a-c98185-openshift-default-7c7758cdbd-8vq8j" Sep 11 16:19:58.830610 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:19:58.830588 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b262660d-18f1-4ec8-963d-60383e08d57d-workload-certs\") pod \"e2e-api-a-c98185-openshift-default-7c7758cdbd-8vq8j\" (UID: \"b262660d-18f1-4ec8-963d-60383e08d57d\") " pod="openshift-ingress/e2e-api-a-c98185-openshift-default-7c7758cdbd-8vq8j" Sep 11 16:19:58.830759 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:19:58.830650 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b262660d-18f1-4ec8-963d-60383e08d57d-istio-envoy\") pod \"e2e-api-a-c98185-openshift-default-7c7758cdbd-8vq8j\" (UID: \"b262660d-18f1-4ec8-963d-60383e08d57d\") " pod="openshift-ingress/e2e-api-a-c98185-openshift-default-7c7758cdbd-8vq8j" Sep 11 16:19:58.830759 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:19:58.830683 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b262660d-18f1-4ec8-963d-60383e08d57d-istio-token\") pod \"e2e-api-a-c98185-openshift-default-7c7758cdbd-8vq8j\" (UID: \"b262660d-18f1-4ec8-963d-60383e08d57d\") " pod="openshift-ingress/e2e-api-a-c98185-openshift-default-7c7758cdbd-8vq8j" Sep 11 16:19:58.830759 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:19:58.830711 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b262660d-18f1-4ec8-963d-60383e08d57d-workload-socket\") pod \"e2e-api-a-c98185-openshift-default-7c7758cdbd-8vq8j\" (UID: \"b262660d-18f1-4ec8-963d-60383e08d57d\") " pod="openshift-ingress/e2e-api-a-c98185-openshift-default-7c7758cdbd-8vq8j" Sep 11 16:19:58.830930 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:19:58.830801 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b262660d-18f1-4ec8-963d-60383e08d57d-istio-data\") pod \"e2e-api-a-c98185-openshift-default-7c7758cdbd-8vq8j\" (UID: \"b262660d-18f1-4ec8-963d-60383e08d57d\") " pod="openshift-ingress/e2e-api-a-c98185-openshift-default-7c7758cdbd-8vq8j" Sep 11 16:19:58.830930 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:19:58.830826 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b262660d-18f1-4ec8-963d-60383e08d57d-istiod-ca-cert\") pod \"e2e-api-a-c98185-openshift-default-7c7758cdbd-8vq8j\" (UID: \"b262660d-18f1-4ec8-963d-60383e08d57d\") " pod="openshift-ingress/e2e-api-a-c98185-openshift-default-7c7758cdbd-8vq8j" Sep 11 16:19:58.931674 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:19:58.931635 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b262660d-18f1-4ec8-963d-60383e08d57d-workload-certs\") pod \"e2e-api-a-c98185-openshift-default-7c7758cdbd-8vq8j\" (UID: \"b262660d-18f1-4ec8-963d-60383e08d57d\") " pod="openshift-ingress/e2e-api-a-c98185-openshift-default-7c7758cdbd-8vq8j" Sep 11 16:19:58.931836 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:19:58.931699 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b262660d-18f1-4ec8-963d-60383e08d57d-istio-envoy\") pod \"e2e-api-a-c98185-openshift-default-7c7758cdbd-8vq8j\" (UID: \"b262660d-18f1-4ec8-963d-60383e08d57d\") " pod="openshift-ingress/e2e-api-a-c98185-openshift-default-7c7758cdbd-8vq8j" Sep 11 16:19:58.931836 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:19:58.931731 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b262660d-18f1-4ec8-963d-60383e08d57d-istio-token\") pod \"e2e-api-a-c98185-openshift-default-7c7758cdbd-8vq8j\" (UID: \"b262660d-18f1-4ec8-963d-60383e08d57d\") " pod="openshift-ingress/e2e-api-a-c98185-openshift-default-7c7758cdbd-8vq8j" Sep 11 16:19:58.931836 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:19:58.931754 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b262660d-18f1-4ec8-963d-60383e08d57d-workload-socket\") pod \"e2e-api-a-c98185-openshift-default-7c7758cdbd-8vq8j\" (UID: \"b262660d-18f1-4ec8-963d-60383e08d57d\") " pod="openshift-ingress/e2e-api-a-c98185-openshift-default-7c7758cdbd-8vq8j" Sep 11 16:19:58.931836 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:19:58.931829 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b262660d-18f1-4ec8-963d-60383e08d57d-istio-data\") pod \"e2e-api-a-c98185-openshift-default-7c7758cdbd-8vq8j\" (UID: \"b262660d-18f1-4ec8-963d-60383e08d57d\") " pod="openshift-ingress/e2e-api-a-c98185-openshift-default-7c7758cdbd-8vq8j" Sep 11 16:19:58.932102 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:19:58.931853 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b262660d-18f1-4ec8-963d-60383e08d57d-istiod-ca-cert\") pod \"e2e-api-a-c98185-openshift-default-7c7758cdbd-8vq8j\" (UID: \"b262660d-18f1-4ec8-963d-60383e08d57d\") " pod="openshift-ingress/e2e-api-a-c98185-openshift-default-7c7758cdbd-8vq8j" Sep 11 16:19:58.932102 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:19:58.931918 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b262660d-18f1-4ec8-963d-60383e08d57d-credential-socket\") pod \"e2e-api-a-c98185-openshift-default-7c7758cdbd-8vq8j\" (UID: \"b262660d-18f1-4ec8-963d-60383e08d57d\") " pod="openshift-ingress/e2e-api-a-c98185-openshift-default-7c7758cdbd-8vq8j" Sep 11 16:19:58.932102 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:19:58.932006 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bgsk\" (UniqueName: \"kubernetes.io/projected/b262660d-18f1-4ec8-963d-60383e08d57d-kube-api-access-6bgsk\") pod \"e2e-api-a-c98185-openshift-default-7c7758cdbd-8vq8j\" (UID: \"b262660d-18f1-4ec8-963d-60383e08d57d\") " pod="openshift-ingress/e2e-api-a-c98185-openshift-default-7c7758cdbd-8vq8j" Sep 11 16:19:58.932102 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:19:58.932035 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b262660d-18f1-4ec8-963d-60383e08d57d-istio-podinfo\") pod \"e2e-api-a-c98185-openshift-default-7c7758cdbd-8vq8j\" (UID: \"b262660d-18f1-4ec8-963d-60383e08d57d\") " pod="openshift-ingress/e2e-api-a-c98185-openshift-default-7c7758cdbd-8vq8j" Sep 11 16:19:58.932308 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:19:58.932127 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b262660d-18f1-4ec8-963d-60383e08d57d-workload-certs\") pod \"e2e-api-a-c98185-openshift-default-7c7758cdbd-8vq8j\" (UID: \"b262660d-18f1-4ec8-963d-60383e08d57d\") " pod="openshift-ingress/e2e-api-a-c98185-openshift-default-7c7758cdbd-8vq8j" Sep 11 16:19:58.932308 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:19:58.932174 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b262660d-18f1-4ec8-963d-60383e08d57d-workload-socket\") pod \"e2e-api-a-c98185-openshift-default-7c7758cdbd-8vq8j\" (UID: \"b262660d-18f1-4ec8-963d-60383e08d57d\") " pod="openshift-ingress/e2e-api-a-c98185-openshift-default-7c7758cdbd-8vq8j" Sep 11 16:19:58.932308 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:19:58.932217 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b262660d-18f1-4ec8-963d-60383e08d57d-istio-data\") pod \"e2e-api-a-c98185-openshift-default-7c7758cdbd-8vq8j\" (UID: \"b262660d-18f1-4ec8-963d-60383e08d57d\") " pod="openshift-ingress/e2e-api-a-c98185-openshift-default-7c7758cdbd-8vq8j" Sep 11 16:19:58.932479 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:19:58.932314 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b262660d-18f1-4ec8-963d-60383e08d57d-credential-socket\") pod \"e2e-api-a-c98185-openshift-default-7c7758cdbd-8vq8j\" (UID: \"b262660d-18f1-4ec8-963d-60383e08d57d\") " pod="openshift-ingress/e2e-api-a-c98185-openshift-default-7c7758cdbd-8vq8j" Sep 11 16:19:58.932912 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:19:58.932888 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b262660d-18f1-4ec8-963d-60383e08d57d-istiod-ca-cert\") pod \"e2e-api-a-c98185-openshift-default-7c7758cdbd-8vq8j\" (UID: \"b262660d-18f1-4ec8-963d-60383e08d57d\") " pod="openshift-ingress/e2e-api-a-c98185-openshift-default-7c7758cdbd-8vq8j" Sep 11 16:19:58.934300 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:19:58.934282 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b262660d-18f1-4ec8-963d-60383e08d57d-istio-envoy\") pod \"e2e-api-a-c98185-openshift-default-7c7758cdbd-8vq8j\" (UID: \"b262660d-18f1-4ec8-963d-60383e08d57d\") " pod="openshift-ingress/e2e-api-a-c98185-openshift-default-7c7758cdbd-8vq8j" Sep 11 16:19:58.934569 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:19:58.934550 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b262660d-18f1-4ec8-963d-60383e08d57d-istio-podinfo\") pod \"e2e-api-a-c98185-openshift-default-7c7758cdbd-8vq8j\" (UID: \"b262660d-18f1-4ec8-963d-60383e08d57d\") " pod="openshift-ingress/e2e-api-a-c98185-openshift-default-7c7758cdbd-8vq8j" Sep 11 16:19:58.940416 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:19:58.940394 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b262660d-18f1-4ec8-963d-60383e08d57d-istio-token\") pod \"e2e-api-a-c98185-openshift-default-7c7758cdbd-8vq8j\" (UID: \"b262660d-18f1-4ec8-963d-60383e08d57d\") " pod="openshift-ingress/e2e-api-a-c98185-openshift-default-7c7758cdbd-8vq8j" Sep 11 16:19:58.943280 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:19:58.943260 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bgsk\" (UniqueName: \"kubernetes.io/projected/b262660d-18f1-4ec8-963d-60383e08d57d-kube-api-access-6bgsk\") pod \"e2e-api-a-c98185-openshift-default-7c7758cdbd-8vq8j\" (UID: \"b262660d-18f1-4ec8-963d-60383e08d57d\") " pod="openshift-ingress/e2e-api-a-c98185-openshift-default-7c7758cdbd-8vq8j" Sep 11 16:19:59.051944 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:19:59.051898 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c98185-openshift-default-7c7758cdbd-8vq8j" Sep 11 16:19:59.346703 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:19:59.346680 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-c98185-openshift-default-7c7758cdbd-8vq8j"] Sep 11 16:19:59.348403 ip-10-0-136-96 kubenswrapper[2566]: W0911 16:19:59.348374 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb262660d_18f1_4ec8_963d_60383e08d57d.slice/crio-2eaa5f5b646cd701086fc3b14f1d65bacfa16617b13235e414a5222b7c580c98 WatchSource:0}: Error finding container 2eaa5f5b646cd701086fc3b14f1d65bacfa16617b13235e414a5222b7c580c98: Status 404 returned error can't find the container with id 2eaa5f5b646cd701086fc3b14f1d65bacfa16617b13235e414a5222b7c580c98 Sep 11 16:19:59.351031 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:19:59.350955 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 16:19:59.351116 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:19:59.351082 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 16:19:59.351156 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:19:59.351124 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 16:19:59.520059 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:19:59.520020 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c98185-openshift-default-7c7758cdbd-8vq8j" event={"ID":"b262660d-18f1-4ec8-963d-60383e08d57d","Type":"ContainerStarted","Data":"29be6889492e012374cee158ecd80ca8c2f62d653ec711bb3e8c77d264ba136d"} Sep 11 16:19:59.520267 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:19:59.520096 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c98185-openshift-default-7c7758cdbd-8vq8j" event={"ID":"b262660d-18f1-4ec8-963d-60383e08d57d","Type":"ContainerStarted","Data":"2eaa5f5b646cd701086fc3b14f1d65bacfa16617b13235e414a5222b7c580c98"} Sep 11 16:19:59.543542 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:19:59.543490 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-c98185-openshift-default-7c7758cdbd-8vq8j" podStartSLOduration=1.543472676 podStartE2EDuration="1.543472676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:19:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:19:59.540759902 +0000 UTC m=+1506.755593808" watchObservedRunningTime="2026-09-11 16:19:59.543472676 +0000 UTC m=+1506.758306580" Sep 11 16:20:00.052337 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:20:00.052294 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-c98185-openshift-default-7c7758cdbd-8vq8j" Sep 11 16:20:00.057873 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:20:00.057847 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-c98185-openshift-default-7c7758cdbd-8vq8j" Sep 11 16:20:00.526854 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:20:00.526825 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-c98185-openshift-default-7c7758cdbd-8vq8j" Sep 11 16:20:00.527916 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:20:00.527897 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-c98185-openshift-default-7c7758cdbd-8vq8j" Sep 11 16:20:10.828882 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:20:10.828845 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dx4xn"] Sep 11 16:20:20.659012 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:20:20.658950 2566 generic.go:356] "Generic (PLEG): container finished" podID="b262660d-18f1-4ec8-963d-60383e08d57d" containerID="29be6889492e012374cee158ecd80ca8c2f62d653ec711bb3e8c77d264ba136d" exitCode=0 Sep 11 16:20:20.659385 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:20:20.659034 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c98185-openshift-default-7c7758cdbd-8vq8j" event={"ID":"b262660d-18f1-4ec8-963d-60383e08d57d","Type":"ContainerDied","Data":"29be6889492e012374cee158ecd80ca8c2f62d653ec711bb3e8c77d264ba136d"} Sep 11 16:20:20.659482 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:20:20.659468 2566 scope.go:117] "RemoveContainer" containerID="29be6889492e012374cee158ecd80ca8c2f62d653ec711bb3e8c77d264ba136d" Sep 11 16:20:20.660619 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:20:20.660586 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 16:20:20.660667 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:20:20.660658 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 16:20:20.660703 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:20:20.660689 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 16:20:21.669225 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:20:21.669192 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c98185-openshift-default-7c7758cdbd-8vq8j" event={"ID":"b262660d-18f1-4ec8-963d-60383e08d57d","Type":"ContainerStarted","Data":"b5719d033a4e4a393f13696bda1b0c8d93e11c3dd6b3a203027d7b4da318d76c"} Sep 11 16:20:22.053469 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:20:22.053423 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-c98185-openshift-default-7c7758cdbd-8vq8j" Sep 11 16:20:22.053698 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:20:22.053576 2566 patch_prober.go:28] interesting pod/e2e-api-a-c98185-openshift-default-7c7758cdbd-8vq8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Sep 11 16:20:22.053698 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:20:22.053625 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c98185-openshift-default-7c7758cdbd-8vq8j" podUID="b262660d-18f1-4ec8-963d-60383e08d57d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Sep 11 16:20:23.053233 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:20:23.053194 2566 patch_prober.go:28] interesting pod/e2e-api-a-c98185-openshift-default-7c7758cdbd-8vq8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Sep 11 16:20:23.053827 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:20:23.053794 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c98185-openshift-default-7c7758cdbd-8vq8j" podUID="b262660d-18f1-4ec8-963d-60383e08d57d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Sep 11 16:20:24.053029 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:20:24.052963 2566 patch_prober.go:28] interesting pod/e2e-api-a-c98185-openshift-default-7c7758cdbd-8vq8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Sep 11 16:20:24.053250 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:20:24.053060 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c98185-openshift-default-7c7758cdbd-8vq8j" podUID="b262660d-18f1-4ec8-963d-60383e08d57d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Sep 11 16:20:25.052902 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:20:25.052854 2566 patch_prober.go:28] interesting pod/e2e-api-a-c98185-openshift-default-7c7758cdbd-8vq8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Sep 11 16:20:25.053137 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:20:25.052931 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c98185-openshift-default-7c7758cdbd-8vq8j" podUID="b262660d-18f1-4ec8-963d-60383e08d57d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Sep 11 16:20:26.053415 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:20:26.053365 2566 patch_prober.go:28] interesting pod/e2e-api-a-c98185-openshift-default-7c7758cdbd-8vq8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Sep 11 16:20:26.053919 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:20:26.053441 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c98185-openshift-default-7c7758cdbd-8vq8j" podUID="b262660d-18f1-4ec8-963d-60383e08d57d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Sep 11 16:20:27.056711 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:20:27.056678 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-c98185-openshift-default-7c7758cdbd-8vq8j" Sep 11 16:20:27.057161 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:20:27.057086 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-c98185-openshift-default-7c7758cdbd-8vq8j" Sep 11 16:20:27.057663 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:20:27.057646 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-c98185-openshift-default-7c7758cdbd-8vq8j" Sep 11 16:20:32.980158 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:20:32.980120 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-bf0614-openshift-default-76d57574c6-4mwfx"] Sep 11 16:20:32.984094 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:20:32.984074 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-bf0614-openshift-default-76d57574c6-4mwfx" Sep 11 16:20:32.988473 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:20:32.988451 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-bf0614-openshift-default-dockercfg-bb4rt\"" Sep 11 16:20:33.012127 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:20:33.012095 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-bf0614-openshift-default-76d57574c6-4mwfx"] Sep 11 16:20:33.092715 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:20:33.092676 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4721eb83-aef7-40dd-845f-92d07e1012f0-istio-podinfo\") pod \"e2e-api-b-bf0614-openshift-default-76d57574c6-4mwfx\" (UID: \"4721eb83-aef7-40dd-845f-92d07e1012f0\") " pod="openshift-ingress/e2e-api-b-bf0614-openshift-default-76d57574c6-4mwfx" Sep 11 16:20:33.092715 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:20:33.092723 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4721eb83-aef7-40dd-845f-92d07e1012f0-credential-socket\") pod \"e2e-api-b-bf0614-openshift-default-76d57574c6-4mwfx\" (UID: \"4721eb83-aef7-40dd-845f-92d07e1012f0\") " pod="openshift-ingress/e2e-api-b-bf0614-openshift-default-76d57574c6-4mwfx" Sep 11 16:20:33.093034 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:20:33.092755 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4721eb83-aef7-40dd-845f-92d07e1012f0-istio-data\") pod \"e2e-api-b-bf0614-openshift-default-76d57574c6-4mwfx\" (UID: \"4721eb83-aef7-40dd-845f-92d07e1012f0\") " pod="openshift-ingress/e2e-api-b-bf0614-openshift-default-76d57574c6-4mwfx" Sep 11 16:20:33.093034 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:20:33.092805 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4721eb83-aef7-40dd-845f-92d07e1012f0-istio-envoy\") pod \"e2e-api-b-bf0614-openshift-default-76d57574c6-4mwfx\" (UID: \"4721eb83-aef7-40dd-845f-92d07e1012f0\") " pod="openshift-ingress/e2e-api-b-bf0614-openshift-default-76d57574c6-4mwfx" Sep 11 16:20:33.093034 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:20:33.092847 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4721eb83-aef7-40dd-845f-92d07e1012f0-workload-socket\") pod \"e2e-api-b-bf0614-openshift-default-76d57574c6-4mwfx\" (UID: \"4721eb83-aef7-40dd-845f-92d07e1012f0\") " pod="openshift-ingress/e2e-api-b-bf0614-openshift-default-76d57574c6-4mwfx" Sep 11 16:20:33.093034 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:20:33.092862 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4721eb83-aef7-40dd-845f-92d07e1012f0-workload-certs\") pod \"e2e-api-b-bf0614-openshift-default-76d57574c6-4mwfx\" (UID: \"4721eb83-aef7-40dd-845f-92d07e1012f0\") " pod="openshift-ingress/e2e-api-b-bf0614-openshift-default-76d57574c6-4mwfx" Sep 11 16:20:33.093034 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:20:33.092889 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4721eb83-aef7-40dd-845f-92d07e1012f0-istiod-ca-cert\") pod \"e2e-api-b-bf0614-openshift-default-76d57574c6-4mwfx\" (UID: \"4721eb83-aef7-40dd-845f-92d07e1012f0\") " pod="openshift-ingress/e2e-api-b-bf0614-openshift-default-76d57574c6-4mwfx" Sep 11 16:20:33.093291 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:20:33.093068 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hj9sl\" (UniqueName: \"kubernetes.io/projected/4721eb83-aef7-40dd-845f-92d07e1012f0-kube-api-access-hj9sl\") pod \"e2e-api-b-bf0614-openshift-default-76d57574c6-4mwfx\" (UID: \"4721eb83-aef7-40dd-845f-92d07e1012f0\") " pod="openshift-ingress/e2e-api-b-bf0614-openshift-default-76d57574c6-4mwfx" Sep 11 16:20:33.093291 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:20:33.093133 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4721eb83-aef7-40dd-845f-92d07e1012f0-istio-token\") pod \"e2e-api-b-bf0614-openshift-default-76d57574c6-4mwfx\" (UID: \"4721eb83-aef7-40dd-845f-92d07e1012f0\") " pod="openshift-ingress/e2e-api-b-bf0614-openshift-default-76d57574c6-4mwfx" Sep 11 16:20:33.194311 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:20:33.194274 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4721eb83-aef7-40dd-845f-92d07e1012f0-istio-envoy\") pod \"e2e-api-b-bf0614-openshift-default-76d57574c6-4mwfx\" (UID: \"4721eb83-aef7-40dd-845f-92d07e1012f0\") " pod="openshift-ingress/e2e-api-b-bf0614-openshift-default-76d57574c6-4mwfx" Sep 11 16:20:33.194515 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:20:33.194337 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4721eb83-aef7-40dd-845f-92d07e1012f0-workload-socket\") pod \"e2e-api-b-bf0614-openshift-default-76d57574c6-4mwfx\" (UID: \"4721eb83-aef7-40dd-845f-92d07e1012f0\") " pod="openshift-ingress/e2e-api-b-bf0614-openshift-default-76d57574c6-4mwfx" Sep 11 16:20:33.194515 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:20:33.194357 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4721eb83-aef7-40dd-845f-92d07e1012f0-workload-certs\") pod \"e2e-api-b-bf0614-openshift-default-76d57574c6-4mwfx\" (UID: \"4721eb83-aef7-40dd-845f-92d07e1012f0\") " pod="openshift-ingress/e2e-api-b-bf0614-openshift-default-76d57574c6-4mwfx" Sep 11 16:20:33.194515 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:20:33.194391 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4721eb83-aef7-40dd-845f-92d07e1012f0-istiod-ca-cert\") pod \"e2e-api-b-bf0614-openshift-default-76d57574c6-4mwfx\" (UID: \"4721eb83-aef7-40dd-845f-92d07e1012f0\") " pod="openshift-ingress/e2e-api-b-bf0614-openshift-default-76d57574c6-4mwfx" Sep 11 16:20:33.194683 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:20:33.194518 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hj9sl\" (UniqueName: \"kubernetes.io/projected/4721eb83-aef7-40dd-845f-92d07e1012f0-kube-api-access-hj9sl\") pod \"e2e-api-b-bf0614-openshift-default-76d57574c6-4mwfx\" (UID: \"4721eb83-aef7-40dd-845f-92d07e1012f0\") " pod="openshift-ingress/e2e-api-b-bf0614-openshift-default-76d57574c6-4mwfx" Sep 11 16:20:33.194683 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:20:33.194572 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4721eb83-aef7-40dd-845f-92d07e1012f0-istio-token\") pod \"e2e-api-b-bf0614-openshift-default-76d57574c6-4mwfx\" (UID: \"4721eb83-aef7-40dd-845f-92d07e1012f0\") " pod="openshift-ingress/e2e-api-b-bf0614-openshift-default-76d57574c6-4mwfx" Sep 11 16:20:33.194683 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:20:33.194629 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4721eb83-aef7-40dd-845f-92d07e1012f0-istio-podinfo\") pod \"e2e-api-b-bf0614-openshift-default-76d57574c6-4mwfx\" (UID: \"4721eb83-aef7-40dd-845f-92d07e1012f0\") " pod="openshift-ingress/e2e-api-b-bf0614-openshift-default-76d57574c6-4mwfx" Sep 11 16:20:33.194683 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:20:33.194669 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4721eb83-aef7-40dd-845f-92d07e1012f0-credential-socket\") pod \"e2e-api-b-bf0614-openshift-default-76d57574c6-4mwfx\" (UID: \"4721eb83-aef7-40dd-845f-92d07e1012f0\") " pod="openshift-ingress/e2e-api-b-bf0614-openshift-default-76d57574c6-4mwfx" Sep 11 16:20:33.194845 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:20:33.194704 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4721eb83-aef7-40dd-845f-92d07e1012f0-istio-data\") pod \"e2e-api-b-bf0614-openshift-default-76d57574c6-4mwfx\" (UID: \"4721eb83-aef7-40dd-845f-92d07e1012f0\") " pod="openshift-ingress/e2e-api-b-bf0614-openshift-default-76d57574c6-4mwfx" Sep 11 16:20:33.194845 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:20:33.194791 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4721eb83-aef7-40dd-845f-92d07e1012f0-workload-socket\") pod \"e2e-api-b-bf0614-openshift-default-76d57574c6-4mwfx\" (UID: \"4721eb83-aef7-40dd-845f-92d07e1012f0\") " pod="openshift-ingress/e2e-api-b-bf0614-openshift-default-76d57574c6-4mwfx" Sep 11 16:20:33.195073 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:20:33.195049 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4721eb83-aef7-40dd-845f-92d07e1012f0-workload-certs\") pod \"e2e-api-b-bf0614-openshift-default-76d57574c6-4mwfx\" (UID: \"4721eb83-aef7-40dd-845f-92d07e1012f0\") " pod="openshift-ingress/e2e-api-b-bf0614-openshift-default-76d57574c6-4mwfx" Sep 11 16:20:33.195141 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:20:33.195114 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4721eb83-aef7-40dd-845f-92d07e1012f0-istio-data\") pod \"e2e-api-b-bf0614-openshift-default-76d57574c6-4mwfx\" (UID: \"4721eb83-aef7-40dd-845f-92d07e1012f0\") " pod="openshift-ingress/e2e-api-b-bf0614-openshift-default-76d57574c6-4mwfx" Sep 11 16:20:33.195186 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:20:33.195135 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4721eb83-aef7-40dd-845f-92d07e1012f0-credential-socket\") pod \"e2e-api-b-bf0614-openshift-default-76d57574c6-4mwfx\" (UID: \"4721eb83-aef7-40dd-845f-92d07e1012f0\") " pod="openshift-ingress/e2e-api-b-bf0614-openshift-default-76d57574c6-4mwfx" Sep 11 16:20:33.195228 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:20:33.195214 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4721eb83-aef7-40dd-845f-92d07e1012f0-istiod-ca-cert\") pod \"e2e-api-b-bf0614-openshift-default-76d57574c6-4mwfx\" (UID: \"4721eb83-aef7-40dd-845f-92d07e1012f0\") " pod="openshift-ingress/e2e-api-b-bf0614-openshift-default-76d57574c6-4mwfx" Sep 11 16:20:33.196909 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:20:33.196883 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4721eb83-aef7-40dd-845f-92d07e1012f0-istio-envoy\") pod \"e2e-api-b-bf0614-openshift-default-76d57574c6-4mwfx\" (UID: \"4721eb83-aef7-40dd-845f-92d07e1012f0\") " pod="openshift-ingress/e2e-api-b-bf0614-openshift-default-76d57574c6-4mwfx" Sep 11 16:20:33.197080 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:20:33.197060 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4721eb83-aef7-40dd-845f-92d07e1012f0-istio-podinfo\") pod \"e2e-api-b-bf0614-openshift-default-76d57574c6-4mwfx\" (UID: \"4721eb83-aef7-40dd-845f-92d07e1012f0\") " pod="openshift-ingress/e2e-api-b-bf0614-openshift-default-76d57574c6-4mwfx" Sep 11 16:20:33.209045 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:20:33.209017 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hj9sl\" (UniqueName: \"kubernetes.io/projected/4721eb83-aef7-40dd-845f-92d07e1012f0-kube-api-access-hj9sl\") pod \"e2e-api-b-bf0614-openshift-default-76d57574c6-4mwfx\" (UID: \"4721eb83-aef7-40dd-845f-92d07e1012f0\") " pod="openshift-ingress/e2e-api-b-bf0614-openshift-default-76d57574c6-4mwfx" Sep 11 16:20:33.225339 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:20:33.225305 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4721eb83-aef7-40dd-845f-92d07e1012f0-istio-token\") pod \"e2e-api-b-bf0614-openshift-default-76d57574c6-4mwfx\" (UID: \"4721eb83-aef7-40dd-845f-92d07e1012f0\") " pod="openshift-ingress/e2e-api-b-bf0614-openshift-default-76d57574c6-4mwfx" Sep 11 16:20:33.294462 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:20:33.294422 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-bf0614-openshift-default-76d57574c6-4mwfx" Sep 11 16:20:33.594194 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:20:33.594097 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-bf0614-openshift-default-76d57574c6-4mwfx"] Sep 11 16:20:33.598397 ip-10-0-136-96 kubenswrapper[2566]: W0911 16:20:33.598371 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4721eb83_aef7_40dd_845f_92d07e1012f0.slice/crio-d37de84c6c3fe911b6e26366232860304d0d4ea9acbcc0966665ad70c82ef2fa WatchSource:0}: Error finding container d37de84c6c3fe911b6e26366232860304d0d4ea9acbcc0966665ad70c82ef2fa: Status 404 returned error can't find the container with id d37de84c6c3fe911b6e26366232860304d0d4ea9acbcc0966665ad70c82ef2fa Sep 11 16:20:33.600575 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:20:33.600545 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 16:20:33.600644 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:20:33.600625 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 16:20:33.600683 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:20:33.600654 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 16:20:33.751225 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:20:33.751195 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bf0614-openshift-default-76d57574c6-4mwfx" event={"ID":"4721eb83-aef7-40dd-845f-92d07e1012f0","Type":"ContainerStarted","Data":"b3ab998e549829e50aacab96f123be187ae62ab18a5420374662056dab3ed508"} Sep 11 16:20:33.751398 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:20:33.751252 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bf0614-openshift-default-76d57574c6-4mwfx" event={"ID":"4721eb83-aef7-40dd-845f-92d07e1012f0","Type":"ContainerStarted","Data":"d37de84c6c3fe911b6e26366232860304d0d4ea9acbcc0966665ad70c82ef2fa"} Sep 11 16:20:33.792069 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:20:33.792016 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-bf0614-openshift-default-76d57574c6-4mwfx" podStartSLOduration=1.79199596 podStartE2EDuration="1.79199596s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:20:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:20:33.791959865 +0000 UTC m=+1541.006793771" watchObservedRunningTime="2026-09-11 16:20:33.79199596 +0000 UTC m=+1541.006829857" Sep 11 16:20:34.295590 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:20:34.295552 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-bf0614-openshift-default-76d57574c6-4mwfx" Sep 11 16:20:34.300835 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:20:34.300814 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-bf0614-openshift-default-76d57574c6-4mwfx" Sep 11 16:20:34.758783 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:20:34.758690 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-bf0614-openshift-default-76d57574c6-4mwfx" Sep 11 16:20:34.759796 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:20:34.759778 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-bf0614-openshift-default-76d57574c6-4mwfx" Sep 11 16:20:44.177588 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:20:44.177545 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dx4xn"] Sep 11 16:20:53.633145 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:20:53.633102 2566 scope.go:117] "RemoveContainer" containerID="80454128ccbaa3f58fedd9fc8560c2d8f9b00a008fcea6d5f53bdc64b6f3c416" Sep 11 16:20:53.642908 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:20:53.642885 2566 scope.go:117] "RemoveContainer" containerID="c910ec2466cfaca1397664bcb49691bf25f8d05cfc821f8aa7c50970679c644a" Sep 11 16:20:54.462152 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:20:54.462109 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c98185-jbp2p"] Sep 11 16:20:54.465767 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:20:54.465745 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c98185-jbp2p" Sep 11 16:20:54.468252 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:20:54.468232 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qbzdr\"" Sep 11 16:20:54.469009 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:20:54.468989 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 16:20:54.498387 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:20:54.498348 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c98185-jbp2p"] Sep 11 16:20:54.524301 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:20:54.524258 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6307119-9c36-41f8-a984-5bf1399fe239-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c98185-jbp2p\" (UID: \"c6307119-9c36-41f8-a984-5bf1399fe239\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c98185-jbp2p" Sep 11 16:20:54.625920 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:20:54.625877 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6307119-9c36-41f8-a984-5bf1399fe239-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c98185-jbp2p\" (UID: \"c6307119-9c36-41f8-a984-5bf1399fe239\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c98185-jbp2p" Sep 11 16:20:54.626557 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:20:54.626538 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6307119-9c36-41f8-a984-5bf1399fe239-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c98185-jbp2p\" (UID: \"c6307119-9c36-41f8-a984-5bf1399fe239\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c98185-jbp2p" Sep 11 16:20:54.777349 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:20:54.777318 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c98185-jbp2p" Sep 11 16:20:55.090171 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:20:55.090141 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c98185-jbp2p"] Sep 11 16:20:55.092333 ip-10-0-136-96 kubenswrapper[2566]: W0911 16:20:55.092305 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6307119_9c36_41f8_a984_5bf1399fe239.slice/crio-0c1e97ca0a4381577e159fff45cb704d86396fb6c0374e69f405cb953d7f82ea WatchSource:0}: Error finding container 0c1e97ca0a4381577e159fff45cb704d86396fb6c0374e69f405cb953d7f82ea: Status 404 returned error can't find the container with id 0c1e97ca0a4381577e159fff45cb704d86396fb6c0374e69f405cb953d7f82ea Sep 11 16:20:55.909659 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:20:55.909620 2566 generic.go:356] "Generic (PLEG): container finished" podID="c6307119-9c36-41f8-a984-5bf1399fe239" containerID="1aa50eae5b7519fd265328caad43d92f61bab5f397be4da48b1546a97cd25ed4" exitCode=0 Sep 11 16:20:55.910090 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:20:55.909671 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c98185-jbp2p" event={"ID":"c6307119-9c36-41f8-a984-5bf1399fe239","Type":"ContainerDied","Data":"1aa50eae5b7519fd265328caad43d92f61bab5f397be4da48b1546a97cd25ed4"} Sep 11 16:20:55.910090 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:20:55.909721 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c98185-jbp2p" event={"ID":"c6307119-9c36-41f8-a984-5bf1399fe239","Type":"ContainerStarted","Data":"0c1e97ca0a4381577e159fff45cb704d86396fb6c0374e69f405cb953d7f82ea"} Sep 11 16:20:55.911420 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:20:55.911399 2566 generic.go:356] "Generic (PLEG): container finished" podID="4721eb83-aef7-40dd-845f-92d07e1012f0" containerID="b3ab998e549829e50aacab96f123be187ae62ab18a5420374662056dab3ed508" exitCode=0 Sep 11 16:20:55.911517 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:20:55.911466 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bf0614-openshift-default-76d57574c6-4mwfx" event={"ID":"4721eb83-aef7-40dd-845f-92d07e1012f0","Type":"ContainerDied","Data":"b3ab998e549829e50aacab96f123be187ae62ab18a5420374662056dab3ed508"} Sep 11 16:20:55.911873 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:20:55.911860 2566 scope.go:117] "RemoveContainer" containerID="b3ab998e549829e50aacab96f123be187ae62ab18a5420374662056dab3ed508" Sep 11 16:20:55.913103 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:20:55.913079 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 16:20:55.913159 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:20:55.913142 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 16:20:55.913202 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:20:55.913173 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 16:20:56.920780 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:20:56.920742 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bf0614-openshift-default-76d57574c6-4mwfx" event={"ID":"4721eb83-aef7-40dd-845f-92d07e1012f0","Type":"ContainerStarted","Data":"6e7a538a049a9c18fe14c65731af94f34a22db85b21cb0b5e55b1bd634d7ecad"} Sep 11 16:20:57.099269 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:20:57.099242 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c98185-jbp2p" Sep 11 16:20:57.162167 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:20:57.161399 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6307119-9c36-41f8-a984-5bf1399fe239-maas-api-service-ca\") pod \"c6307119-9c36-41f8-a984-5bf1399fe239\" (UID: \"c6307119-9c36-41f8-a984-5bf1399fe239\") " Sep 11 16:20:57.162570 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:20:57.162524 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6307119-9c36-41f8-a984-5bf1399fe239-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c6307119-9c36-41f8-a984-5bf1399fe239" (UID: "c6307119-9c36-41f8-a984-5bf1399fe239"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 16:20:57.262758 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:20:57.262720 2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6307119-9c36-41f8-a984-5bf1399fe239-maas-api-service-ca\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:20:57.295544 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:20:57.295488 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-bf0614-openshift-default-76d57574c6-4mwfx" Sep 11 16:20:57.295846 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:20:57.295825 2566 patch_prober.go:28] interesting pod/e2e-api-b-bf0614-openshift-default-76d57574c6-4mwfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Sep 11 16:20:57.295901 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:20:57.295876 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-bf0614-openshift-default-76d57574c6-4mwfx" podUID="4721eb83-aef7-40dd-845f-92d07e1012f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Sep 11 16:20:57.928490 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:20:57.928461 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c98185-jbp2p" Sep 11 16:20:57.928921 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:20:57.928492 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c98185-jbp2p" event={"ID":"c6307119-9c36-41f8-a984-5bf1399fe239","Type":"ContainerDied","Data":"0c1e97ca0a4381577e159fff45cb704d86396fb6c0374e69f405cb953d7f82ea"} Sep 11 16:20:57.928921 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:20:57.928540 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0c1e97ca0a4381577e159fff45cb704d86396fb6c0374e69f405cb953d7f82ea" Sep 11 16:20:58.295753 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:20:58.295716 2566 patch_prober.go:28] interesting pod/e2e-api-b-bf0614-openshift-default-76d57574c6-4mwfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Sep 11 16:20:58.295934 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:20:58.295772 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-bf0614-openshift-default-76d57574c6-4mwfx" podUID="4721eb83-aef7-40dd-845f-92d07e1012f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Sep 11 16:20:59.295462 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:20:59.295426 2566 patch_prober.go:28] interesting pod/e2e-api-b-bf0614-openshift-default-76d57574c6-4mwfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Sep 11 16:20:59.295904 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:20:59.295482 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-bf0614-openshift-default-76d57574c6-4mwfx" podUID="4721eb83-aef7-40dd-845f-92d07e1012f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Sep 11 16:21:00.295580 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:00.295541 2566 patch_prober.go:28] interesting pod/e2e-api-b-bf0614-openshift-default-76d57574c6-4mwfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Sep 11 16:21:00.296009 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:00.295597 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-bf0614-openshift-default-76d57574c6-4mwfx" podUID="4721eb83-aef7-40dd-845f-92d07e1012f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Sep 11 16:21:01.295338 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:01.295303 2566 patch_prober.go:28] interesting pod/e2e-api-b-bf0614-openshift-default-76d57574c6-4mwfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Sep 11 16:21:01.295524 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:01.295358 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-bf0614-openshift-default-76d57574c6-4mwfx" podUID="4721eb83-aef7-40dd-845f-92d07e1012f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Sep 11 16:21:02.299615 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:02.299583 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-bf0614-openshift-default-76d57574c6-4mwfx" Sep 11 16:21:02.300098 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:02.299997 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-bf0614-openshift-default-76d57574c6-4mwfx" Sep 11 16:21:02.300795 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:02.300776 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-bf0614-openshift-default-76d57574c6-4mwfx" Sep 11 16:21:04.242829 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:04.242784 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dx4xn"] Sep 11 16:21:08.435634 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:08.435596 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7095-e1dfe3"] Sep 11 16:21:08.436087 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:08.436073 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6307119-9c36-41f8-a984-5bf1399fe239" containerName="revoke-keys" Sep 11 16:21:08.436087 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:08.436088 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6307119-9c36-41f8-a984-5bf1399fe239" containerName="revoke-keys" Sep 11 16:21:08.436174 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:08.436162 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6307119-9c36-41f8-a984-5bf1399fe239" containerName="revoke-keys" Sep 11 16:21:08.444016 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:08.443401 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7095-e1dfe3" Sep 11 16:21:08.447102 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:08.447075 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-vcw5k\"" Sep 11 16:21:08.450376 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:08.450346 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7095-e1dfe3"] Sep 11 16:21:08.584857 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:08.584823 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwh8g\" (UniqueName: \"kubernetes.io/projected/68074b98-5aa2-4c49-beda-77c776656272-kube-api-access-pwh8g\") pod \"test-curl-7095-e1dfe3\" (UID: \"68074b98-5aa2-4c49-beda-77c776656272\") " pod="opendatahub/test-curl-7095-e1dfe3" Sep 11 16:21:08.685706 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:08.685619 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwh8g\" (UniqueName: \"kubernetes.io/projected/68074b98-5aa2-4c49-beda-77c776656272-kube-api-access-pwh8g\") pod \"test-curl-7095-e1dfe3\" (UID: \"68074b98-5aa2-4c49-beda-77c776656272\") " pod="opendatahub/test-curl-7095-e1dfe3" Sep 11 16:21:08.695291 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:08.695263 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwh8g\" (UniqueName: \"kubernetes.io/projected/68074b98-5aa2-4c49-beda-77c776656272-kube-api-access-pwh8g\") pod \"test-curl-7095-e1dfe3\" (UID: \"68074b98-5aa2-4c49-beda-77c776656272\") " pod="opendatahub/test-curl-7095-e1dfe3" Sep 11 16:21:08.756273 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:08.756224 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7095-e1dfe3" Sep 11 16:21:09.038114 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:09.038086 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7095-e1dfe3"] Sep 11 16:21:09.040410 ip-10-0-136-96 kubenswrapper[2566]: W0911 16:21:09.040382 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod68074b98_5aa2_4c49_beda_77c776656272.slice/crio-33d533cb87fe239862a8141afca46ecad640a7052f2c49a538ae38e1cb12fe9d WatchSource:0}: Error finding container 33d533cb87fe239862a8141afca46ecad640a7052f2c49a538ae38e1cb12fe9d: Status 404 returned error can't find the container with id 33d533cb87fe239862a8141afca46ecad640a7052f2c49a538ae38e1cb12fe9d Sep 11 16:21:10.019060 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:10.019017 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7095-e1dfe3" event={"ID":"68074b98-5aa2-4c49-beda-77c776656272","Type":"ContainerStarted","Data":"f7b3fa6361627056490caa22e7cc4b589d9d2c3eb4df9f04f61717d3e48f99f7"} Sep 11 16:21:10.019441 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:10.019070 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7095-e1dfe3" event={"ID":"68074b98-5aa2-4c49-beda-77c776656272","Type":"ContainerStarted","Data":"33d533cb87fe239862a8141afca46ecad640a7052f2c49a538ae38e1cb12fe9d"} Sep 11 16:21:10.057892 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:10.057833 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7095-e1dfe3" podStartSLOduration=2.057816351 podStartE2EDuration="2.057816351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:21:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:21:10.055326631 +0000 UTC m=+1577.270160535" watchObservedRunningTime="2026-09-11 16:21:10.057816351 +0000 UTC m=+1577.272650304" Sep 11 16:21:10.868676 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:10.868638 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7095-e1dfe3"] Sep 11 16:21:10.882810 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:10.882734 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7095-e1dfe3"] Sep 11 16:21:11.027177 ip-10-0-136-96 kubenswrapper[2566]: E0911 16:21:11.027145 2566 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7095-e1dfe3" container="test-curl-7095-e1dfe3" Sep 11 16:21:11.028582 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:11.028553 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7095-fe39fe"] Sep 11 16:21:11.029087 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:11.029070 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68074b98-5aa2-4c49-beda-77c776656272" containerName="test-curl-7095-e1dfe3" Sep 11 16:21:11.029139 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:11.029089 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="68074b98-5aa2-4c49-beda-77c776656272" containerName="test-curl-7095-e1dfe3" Sep 11 16:21:11.029200 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:11.029188 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="68074b98-5aa2-4c49-beda-77c776656272" containerName="test-curl-7095-e1dfe3" Sep 11 16:21:11.031410 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:11.031393 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7095-fe39fe" Sep 11 16:21:11.040072 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:11.040044 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7095-fe39fe"] Sep 11 16:21:11.111028 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:11.110989 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ndss\" (UniqueName: \"kubernetes.io/projected/3d732abd-a686-4604-991b-6d1c1e24296e-kube-api-access-6ndss\") pod \"test-curl-7095-fe39fe\" (UID: \"3d732abd-a686-4604-991b-6d1c1e24296e\") " pod="opendatahub/test-curl-7095-fe39fe" Sep 11 16:21:11.212550 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:11.212455 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ndss\" (UniqueName: \"kubernetes.io/projected/3d732abd-a686-4604-991b-6d1c1e24296e-kube-api-access-6ndss\") pod \"test-curl-7095-fe39fe\" (UID: \"3d732abd-a686-4604-991b-6d1c1e24296e\") " pod="opendatahub/test-curl-7095-fe39fe" Sep 11 16:21:11.229672 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:11.229639 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ndss\" (UniqueName: \"kubernetes.io/projected/3d732abd-a686-4604-991b-6d1c1e24296e-kube-api-access-6ndss\") pod \"test-curl-7095-fe39fe\" (UID: \"3d732abd-a686-4604-991b-6d1c1e24296e\") " pod="opendatahub/test-curl-7095-fe39fe" Sep 11 16:21:11.342251 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:11.342218 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7095-fe39fe" Sep 11 16:21:11.417306 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:11.417238 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c98185-jbp2p"] Sep 11 16:21:11.420160 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:11.420122 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c98185-jbp2p"] Sep 11 16:21:11.629963 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:11.629935 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7095-fe39fe"] Sep 11 16:21:11.632285 ip-10-0-136-96 kubenswrapper[2566]: W0911 16:21:11.632259 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3d732abd_a686_4604_991b_6d1c1e24296e.slice/crio-54d393dbca2b647b44d30e35cbb42a7255d858defb34a75df460d74df7b51fec WatchSource:0}: Error finding container 54d393dbca2b647b44d30e35cbb42a7255d858defb34a75df460d74df7b51fec: Status 404 returned error can't find the container with id 54d393dbca2b647b44d30e35cbb42a7255d858defb34a75df460d74df7b51fec Sep 11 16:21:12.034946 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:12.034911 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7095-fe39fe" event={"ID":"3d732abd-a686-4604-991b-6d1c1e24296e","Type":"ContainerStarted","Data":"689fb42832362a96032746032406e5d271a76ecf90900e955aacb8afdea47898"} Sep 11 16:21:12.034946 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:12.034956 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7095-fe39fe" event={"ID":"3d732abd-a686-4604-991b-6d1c1e24296e","Type":"ContainerStarted","Data":"54d393dbca2b647b44d30e35cbb42a7255d858defb34a75df460d74df7b51fec"} Sep 11 16:21:12.035490 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:12.034946 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7095-e1dfe3" podUID="68074b98-5aa2-4c49-beda-77c776656272" containerName="test-curl-7095-e1dfe3" containerID="cri-o://f7b3fa6361627056490caa22e7cc4b589d9d2c3eb4df9f04f61717d3e48f99f7" gracePeriod=2 Sep 11 16:21:12.062996 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:12.060341 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7095-fe39fe" podStartSLOduration=2.060318437 podStartE2EDuration="2.060318437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:21:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:21:12.057752317 +0000 UTC m=+1579.272586226" watchObservedRunningTime="2026-09-11 16:21:12.060318437 +0000 UTC m=+1579.275152347" Sep 11 16:21:12.698612 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:12.698578 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7095-fe39fe"] Sep 11 16:21:12.702958 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:12.702929 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7095-fe39fe"] Sep 11 16:21:13.040722 ip-10-0-136-96 kubenswrapper[2566]: E0911 16:21:13.040691 2566 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7095-fe39fe" container="test-curl-7095-fe39fe" Sep 11 16:21:13.240538 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:13.240504 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6307119-9c36-41f8-a984-5bf1399fe239" path="/var/lib/kubelet/pods/c6307119-9c36-41f8-a984-5bf1399fe239/volumes" Sep 11 16:21:14.047118 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:14.047080 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7095-fe39fe" podUID="3d732abd-a686-4604-991b-6d1c1e24296e" containerName="test-curl-7095-fe39fe" containerID="cri-o://689fb42832362a96032746032406e5d271a76ecf90900e955aacb8afdea47898" gracePeriod=2 Sep 11 16:21:14.217391 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:14.217364 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7095-e1dfe3" Sep 11 16:21:14.348342 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:14.348227 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pwh8g\" (UniqueName: \"kubernetes.io/projected/68074b98-5aa2-4c49-beda-77c776656272-kube-api-access-pwh8g\") pod \"68074b98-5aa2-4c49-beda-77c776656272\" (UID: \"68074b98-5aa2-4c49-beda-77c776656272\") " Sep 11 16:21:14.350316 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:14.350284 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68074b98-5aa2-4c49-beda-77c776656272-kube-api-access-pwh8g" (OuterVolumeSpecName: "kube-api-access-pwh8g") pod "68074b98-5aa2-4c49-beda-77c776656272" (UID: "68074b98-5aa2-4c49-beda-77c776656272"). InnerVolumeSpecName "kube-api-access-pwh8g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 16:21:14.450241 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:14.450201 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwh8g\" (UniqueName: \"kubernetes.io/projected/68074b98-5aa2-4c49-beda-77c776656272-kube-api-access-pwh8g\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:21:15.054685 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:15.054647 2566 generic.go:356] "Generic (PLEG): container finished" podID="68074b98-5aa2-4c49-beda-77c776656272" containerID="f7b3fa6361627056490caa22e7cc4b589d9d2c3eb4df9f04f61717d3e48f99f7" exitCode=137 Sep 11 16:21:15.054685 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:15.054691 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7095-e1dfe3" Sep 11 16:21:15.055164 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:15.054741 2566 scope.go:117] "RemoveContainer" containerID="f7b3fa6361627056490caa22e7cc4b589d9d2c3eb4df9f04f61717d3e48f99f7" Sep 11 16:21:15.064780 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:15.064731 2566 scope.go:117] "RemoveContainer" containerID="f7b3fa6361627056490caa22e7cc4b589d9d2c3eb4df9f04f61717d3e48f99f7" Sep 11 16:21:15.064936 ip-10-0-136-96 kubenswrapper[2566]: E0911 16:21:15.064904 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7095-e1dfe3\" is forbidden: User \"system:node:ip-10-0-136-96.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-96.ec2.internal' and this object" podUID="68074b98-5aa2-4c49-beda-77c776656272" pod="opendatahub/test-curl-7095-e1dfe3" Sep 11 16:21:15.065103 ip-10-0-136-96 kubenswrapper[2566]: E0911 16:21:15.065082 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7b3fa6361627056490caa22e7cc4b589d9d2c3eb4df9f04f61717d3e48f99f7\": container with ID starting with f7b3fa6361627056490caa22e7cc4b589d9d2c3eb4df9f04f61717d3e48f99f7 not found: ID does not exist" containerID="f7b3fa6361627056490caa22e7cc4b589d9d2c3eb4df9f04f61717d3e48f99f7" Sep 11 16:21:15.065151 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:15.065113 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7b3fa6361627056490caa22e7cc4b589d9d2c3eb4df9f04f61717d3e48f99f7"} err="failed to get container status \"f7b3fa6361627056490caa22e7cc4b589d9d2c3eb4df9f04f61717d3e48f99f7\": rpc error: code = NotFound desc = could not find container \"f7b3fa6361627056490caa22e7cc4b589d9d2c3eb4df9f04f61717d3e48f99f7\": container with ID starting with f7b3fa6361627056490caa22e7cc4b589d9d2c3eb4df9f04f61717d3e48f99f7 not found: ID does not exist" Sep 11 16:21:15.240463 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:15.240428 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="68074b98-5aa2-4c49-beda-77c776656272" path="/var/lib/kubelet/pods/68074b98-5aa2-4c49-beda-77c776656272/volumes" Sep 11 16:21:16.236304 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:16.236278 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7095-fe39fe" Sep 11 16:21:16.368919 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:16.368835 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6ndss\" (UniqueName: \"kubernetes.io/projected/3d732abd-a686-4604-991b-6d1c1e24296e-kube-api-access-6ndss\") pod \"3d732abd-a686-4604-991b-6d1c1e24296e\" (UID: \"3d732abd-a686-4604-991b-6d1c1e24296e\") " Sep 11 16:21:16.371080 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:16.371054 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d732abd-a686-4604-991b-6d1c1e24296e-kube-api-access-6ndss" (OuterVolumeSpecName: "kube-api-access-6ndss") pod "3d732abd-a686-4604-991b-6d1c1e24296e" (UID: "3d732abd-a686-4604-991b-6d1c1e24296e"). InnerVolumeSpecName "kube-api-access-6ndss". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 16:21:16.470576 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:16.470538 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ndss\" (UniqueName: \"kubernetes.io/projected/3d732abd-a686-4604-991b-6d1c1e24296e-kube-api-access-6ndss\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:21:17.072433 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:17.072395 2566 generic.go:356] "Generic (PLEG): container finished" podID="3d732abd-a686-4604-991b-6d1c1e24296e" containerID="689fb42832362a96032746032406e5d271a76ecf90900e955aacb8afdea47898" exitCode=137 Sep 11 16:21:17.072638 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:17.072442 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7095-fe39fe" Sep 11 16:21:17.072638 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:17.072478 2566 scope.go:117] "RemoveContainer" containerID="689fb42832362a96032746032406e5d271a76ecf90900e955aacb8afdea47898" Sep 11 16:21:17.083146 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:17.083125 2566 scope.go:117] "RemoveContainer" containerID="689fb42832362a96032746032406e5d271a76ecf90900e955aacb8afdea47898" Sep 11 16:21:17.083492 ip-10-0-136-96 kubenswrapper[2566]: E0911 16:21:17.083466 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"689fb42832362a96032746032406e5d271a76ecf90900e955aacb8afdea47898\": container with ID starting with 689fb42832362a96032746032406e5d271a76ecf90900e955aacb8afdea47898 not found: ID does not exist" containerID="689fb42832362a96032746032406e5d271a76ecf90900e955aacb8afdea47898" Sep 11 16:21:17.083569 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:17.083502 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"689fb42832362a96032746032406e5d271a76ecf90900e955aacb8afdea47898"} err="failed to get container status \"689fb42832362a96032746032406e5d271a76ecf90900e955aacb8afdea47898\": rpc error: code = NotFound desc = could not find container \"689fb42832362a96032746032406e5d271a76ecf90900e955aacb8afdea47898\": container with ID starting with 689fb42832362a96032746032406e5d271a76ecf90900e955aacb8afdea47898 not found: ID does not exist" Sep 11 16:21:17.083632 ip-10-0-136-96 kubenswrapper[2566]: E0911 16:21:17.083612 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7095-fe39fe\" is forbidden: User \"system:node:ip-10-0-136-96.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-96.ec2.internal' and this object" podUID="3d732abd-a686-4604-991b-6d1c1e24296e" pod="opendatahub/test-curl-7095-fe39fe" Sep 11 16:21:17.240394 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:17.240354 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d732abd-a686-4604-991b-6d1c1e24296e" path="/var/lib/kubelet/pods/3d732abd-a686-4604-991b-6d1c1e24296e/volumes" Sep 11 16:21:19.156434 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:19.156395 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-c98185-openshift-default-7c7758cdbd-8vq8j"] Sep 11 16:21:19.157046 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:19.156658 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-c98185-openshift-default-7c7758cdbd-8vq8j" podUID="b262660d-18f1-4ec8-963d-60383e08d57d" containerName="istio-proxy" containerID="cri-o://b5719d033a4e4a393f13696bda1b0c8d93e11c3dd6b3a203027d7b4da318d76c" gracePeriod=30 Sep 11 16:21:24.415200 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:24.415175 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c98185-openshift-default-7c7758cdbd-8vq8j" Sep 11 16:21:24.450131 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:24.450049 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b262660d-18f1-4ec8-963d-60383e08d57d-workload-socket\") pod \"b262660d-18f1-4ec8-963d-60383e08d57d\" (UID: \"b262660d-18f1-4ec8-963d-60383e08d57d\") " Sep 11 16:21:24.450131 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:24.450106 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b262660d-18f1-4ec8-963d-60383e08d57d-istio-data\") pod \"b262660d-18f1-4ec8-963d-60383e08d57d\" (UID: \"b262660d-18f1-4ec8-963d-60383e08d57d\") " Sep 11 16:21:24.450337 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:24.450136 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b262660d-18f1-4ec8-963d-60383e08d57d-istio-podinfo\") pod \"b262660d-18f1-4ec8-963d-60383e08d57d\" (UID: \"b262660d-18f1-4ec8-963d-60383e08d57d\") " Sep 11 16:21:24.450337 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:24.450176 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b262660d-18f1-4ec8-963d-60383e08d57d-istio-token\") pod \"b262660d-18f1-4ec8-963d-60383e08d57d\" (UID: \"b262660d-18f1-4ec8-963d-60383e08d57d\") " Sep 11 16:21:24.450337 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:24.450222 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b262660d-18f1-4ec8-963d-60383e08d57d-istiod-ca-cert\") pod \"b262660d-18f1-4ec8-963d-60383e08d57d\" (UID: \"b262660d-18f1-4ec8-963d-60383e08d57d\") " Sep 11 16:21:24.450337 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:24.450279 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6bgsk\" (UniqueName: \"kubernetes.io/projected/b262660d-18f1-4ec8-963d-60383e08d57d-kube-api-access-6bgsk\") pod \"b262660d-18f1-4ec8-963d-60383e08d57d\" (UID: \"b262660d-18f1-4ec8-963d-60383e08d57d\") " Sep 11 16:21:24.450337 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:24.450302 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b262660d-18f1-4ec8-963d-60383e08d57d-istio-envoy\") pod \"b262660d-18f1-4ec8-963d-60383e08d57d\" (UID: \"b262660d-18f1-4ec8-963d-60383e08d57d\") " Sep 11 16:21:24.450583 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:24.450358 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b262660d-18f1-4ec8-963d-60383e08d57d-workload-certs\") pod \"b262660d-18f1-4ec8-963d-60383e08d57d\" (UID: \"b262660d-18f1-4ec8-963d-60383e08d57d\") " Sep 11 16:21:24.450583 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:24.450399 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b262660d-18f1-4ec8-963d-60383e08d57d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b262660d-18f1-4ec8-963d-60383e08d57d" (UID: "b262660d-18f1-4ec8-963d-60383e08d57d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 16:21:24.450583 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:24.450422 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b262660d-18f1-4ec8-963d-60383e08d57d-credential-socket\") pod \"b262660d-18f1-4ec8-963d-60383e08d57d\" (UID: \"b262660d-18f1-4ec8-963d-60383e08d57d\") " Sep 11 16:21:24.450583 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:24.450417 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b262660d-18f1-4ec8-963d-60383e08d57d-istio-data" (OuterVolumeSpecName: "istio-data") pod "b262660d-18f1-4ec8-963d-60383e08d57d" (UID: "b262660d-18f1-4ec8-963d-60383e08d57d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 16:21:24.450801 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:24.450653 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b262660d-18f1-4ec8-963d-60383e08d57d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b262660d-18f1-4ec8-963d-60383e08d57d" (UID: "b262660d-18f1-4ec8-963d-60383e08d57d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 16:21:24.450801 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:24.450695 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b262660d-18f1-4ec8-963d-60383e08d57d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b262660d-18f1-4ec8-963d-60383e08d57d" (UID: "b262660d-18f1-4ec8-963d-60383e08d57d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 16:21:24.450801 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:24.450746 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b262660d-18f1-4ec8-963d-60383e08d57d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b262660d-18f1-4ec8-963d-60383e08d57d" (UID: "b262660d-18f1-4ec8-963d-60383e08d57d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 16:21:24.450801 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:24.450796 2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b262660d-18f1-4ec8-963d-60383e08d57d-workload-certs\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:21:24.451052 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:24.450814 2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b262660d-18f1-4ec8-963d-60383e08d57d-credential-socket\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:21:24.451052 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:24.450830 2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b262660d-18f1-4ec8-963d-60383e08d57d-workload-socket\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:21:24.451052 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:24.450842 2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b262660d-18f1-4ec8-963d-60383e08d57d-istio-data\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:21:24.452959 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:24.452933 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b262660d-18f1-4ec8-963d-60383e08d57d-kube-api-access-6bgsk" (OuterVolumeSpecName: "kube-api-access-6bgsk") pod "b262660d-18f1-4ec8-963d-60383e08d57d" (UID: "b262660d-18f1-4ec8-963d-60383e08d57d"). InnerVolumeSpecName "kube-api-access-6bgsk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 16:21:24.452959 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:24.452945 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b262660d-18f1-4ec8-963d-60383e08d57d-istio-token" (OuterVolumeSpecName: "istio-token") pod "b262660d-18f1-4ec8-963d-60383e08d57d" (UID: "b262660d-18f1-4ec8-963d-60383e08d57d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 16:21:24.452959 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:24.452957 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b262660d-18f1-4ec8-963d-60383e08d57d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b262660d-18f1-4ec8-963d-60383e08d57d" (UID: "b262660d-18f1-4ec8-963d-60383e08d57d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 16:21:24.453162 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:24.452946 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b262660d-18f1-4ec8-963d-60383e08d57d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b262660d-18f1-4ec8-963d-60383e08d57d" (UID: "b262660d-18f1-4ec8-963d-60383e08d57d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 16:21:24.552321 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:24.552279 2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b262660d-18f1-4ec8-963d-60383e08d57d-istio-podinfo\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:21:24.552321 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:24.552315 2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b262660d-18f1-4ec8-963d-60383e08d57d-istio-token\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:21:24.552321 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:24.552324 2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b262660d-18f1-4ec8-963d-60383e08d57d-istiod-ca-cert\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:21:24.552592 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:24.552332 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bgsk\" (UniqueName: \"kubernetes.io/projected/b262660d-18f1-4ec8-963d-60383e08d57d-kube-api-access-6bgsk\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:21:24.552592 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:24.552342 2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b262660d-18f1-4ec8-963d-60383e08d57d-istio-envoy\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:21:25.033122 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:25.033086 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dx4xn"] Sep 11 16:21:25.129267 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:25.129233 2566 generic.go:356] "Generic (PLEG): container finished" podID="b262660d-18f1-4ec8-963d-60383e08d57d" containerID="b5719d033a4e4a393f13696bda1b0c8d93e11c3dd6b3a203027d7b4da318d76c" exitCode=0 Sep 11 16:21:25.129460 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:25.129286 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c98185-openshift-default-7c7758cdbd-8vq8j" Sep 11 16:21:25.129460 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:25.129324 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c98185-openshift-default-7c7758cdbd-8vq8j" event={"ID":"b262660d-18f1-4ec8-963d-60383e08d57d","Type":"ContainerDied","Data":"b5719d033a4e4a393f13696bda1b0c8d93e11c3dd6b3a203027d7b4da318d76c"} Sep 11 16:21:25.129460 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:25.129388 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c98185-openshift-default-7c7758cdbd-8vq8j" event={"ID":"b262660d-18f1-4ec8-963d-60383e08d57d","Type":"ContainerDied","Data":"2eaa5f5b646cd701086fc3b14f1d65bacfa16617b13235e414a5222b7c580c98"} Sep 11 16:21:25.129460 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:25.129418 2566 scope.go:117] "RemoveContainer" containerID="b5719d033a4e4a393f13696bda1b0c8d93e11c3dd6b3a203027d7b4da318d76c" Sep 11 16:21:25.139512 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:25.139493 2566 scope.go:117] "RemoveContainer" containerID="29be6889492e012374cee158ecd80ca8c2f62d653ec711bb3e8c77d264ba136d" Sep 11 16:21:25.148700 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:25.148678 2566 scope.go:117] "RemoveContainer" containerID="b5719d033a4e4a393f13696bda1b0c8d93e11c3dd6b3a203027d7b4da318d76c" Sep 11 16:21:25.149038 ip-10-0-136-96 kubenswrapper[2566]: E0911 16:21:25.148963 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5719d033a4e4a393f13696bda1b0c8d93e11c3dd6b3a203027d7b4da318d76c\": container with ID starting with b5719d033a4e4a393f13696bda1b0c8d93e11c3dd6b3a203027d7b4da318d76c not found: ID does not exist" containerID="b5719d033a4e4a393f13696bda1b0c8d93e11c3dd6b3a203027d7b4da318d76c" Sep 11 16:21:25.149038 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:25.149006 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5719d033a4e4a393f13696bda1b0c8d93e11c3dd6b3a203027d7b4da318d76c"} err="failed to get container status \"b5719d033a4e4a393f13696bda1b0c8d93e11c3dd6b3a203027d7b4da318d76c\": rpc error: code = NotFound desc = could not find container \"b5719d033a4e4a393f13696bda1b0c8d93e11c3dd6b3a203027d7b4da318d76c\": container with ID starting with b5719d033a4e4a393f13696bda1b0c8d93e11c3dd6b3a203027d7b4da318d76c not found: ID does not exist" Sep 11 16:21:25.149038 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:25.149024 2566 scope.go:117] "RemoveContainer" containerID="29be6889492e012374cee158ecd80ca8c2f62d653ec711bb3e8c77d264ba136d" Sep 11 16:21:25.149431 ip-10-0-136-96 kubenswrapper[2566]: E0911 16:21:25.149383 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29be6889492e012374cee158ecd80ca8c2f62d653ec711bb3e8c77d264ba136d\": container with ID starting with 29be6889492e012374cee158ecd80ca8c2f62d653ec711bb3e8c77d264ba136d not found: ID does not exist" containerID="29be6889492e012374cee158ecd80ca8c2f62d653ec711bb3e8c77d264ba136d" Sep 11 16:21:25.149527 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:25.149440 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29be6889492e012374cee158ecd80ca8c2f62d653ec711bb3e8c77d264ba136d"} err="failed to get container status \"29be6889492e012374cee158ecd80ca8c2f62d653ec711bb3e8c77d264ba136d\": rpc error: code = NotFound desc = could not find container \"29be6889492e012374cee158ecd80ca8c2f62d653ec711bb3e8c77d264ba136d\": container with ID starting with 29be6889492e012374cee158ecd80ca8c2f62d653ec711bb3e8c77d264ba136d not found: ID does not exist" Sep 11 16:21:25.152533 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:25.152511 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-c98185-openshift-default-7c7758cdbd-8vq8j"] Sep 11 16:21:25.156862 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:25.156840 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-c98185-openshift-default-7c7758cdbd-8vq8j"] Sep 11 16:21:25.240092 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:25.240048 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b262660d-18f1-4ec8-963d-60383e08d57d" path="/var/lib/kubelet/pods/b262660d-18f1-4ec8-963d-60383e08d57d/volumes" Sep 11 16:21:36.308578 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:36.308542 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dx4xn"] Sep 11 16:21:42.534343 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:42.534310 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-bf0614-openshift-default-76d57574c6-4mwfx"] Sep 11 16:21:42.534741 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:42.534536 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-bf0614-openshift-default-76d57574c6-4mwfx" podUID="4721eb83-aef7-40dd-845f-92d07e1012f0" containerName="istio-proxy" containerID="cri-o://6e7a538a049a9c18fe14c65731af94f34a22db85b21cb0b5e55b1bd634d7ecad" gracePeriod=30 Sep 11 16:21:46.626127 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:46.626032 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dx4xn"] Sep 11 16:21:47.302128 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:47.302093 2566 patch_prober.go:28] interesting pod/e2e-api-b-bf0614-openshift-default-76d57574c6-4mwfx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 11 16:21:47.302330 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:47.302168 2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-bf0614-openshift-default-76d57574c6-4mwfx" podUID="4721eb83-aef7-40dd-845f-92d07e1012f0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 11 16:21:47.803242 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:47.803217 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-bf0614-openshift-default-76d57574c6-4mwfx" Sep 11 16:21:47.811384 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:47.811333 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4721eb83-aef7-40dd-845f-92d07e1012f0-istiod-ca-cert\") pod \"4721eb83-aef7-40dd-845f-92d07e1012f0\" (UID: \"4721eb83-aef7-40dd-845f-92d07e1012f0\") " Sep 11 16:21:47.811384 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:47.811371 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4721eb83-aef7-40dd-845f-92d07e1012f0-istio-envoy\") pod \"4721eb83-aef7-40dd-845f-92d07e1012f0\" (UID: \"4721eb83-aef7-40dd-845f-92d07e1012f0\") " Sep 11 16:21:47.811512 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:47.811416 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4721eb83-aef7-40dd-845f-92d07e1012f0-istio-podinfo\") pod \"4721eb83-aef7-40dd-845f-92d07e1012f0\" (UID: \"4721eb83-aef7-40dd-845f-92d07e1012f0\") " Sep 11 16:21:47.811512 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:47.811468 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4721eb83-aef7-40dd-845f-92d07e1012f0-workload-socket\") pod \"4721eb83-aef7-40dd-845f-92d07e1012f0\" (UID: \"4721eb83-aef7-40dd-845f-92d07e1012f0\") " Sep 11 16:21:47.811512 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:47.811507 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4721eb83-aef7-40dd-845f-92d07e1012f0-istio-data\") pod \"4721eb83-aef7-40dd-845f-92d07e1012f0\" (UID: \"4721eb83-aef7-40dd-845f-92d07e1012f0\") " Sep 11 16:21:47.811651 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:47.811538 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4721eb83-aef7-40dd-845f-92d07e1012f0-credential-socket\") pod \"4721eb83-aef7-40dd-845f-92d07e1012f0\" (UID: \"4721eb83-aef7-40dd-845f-92d07e1012f0\") " Sep 11 16:21:47.811651 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:47.811557 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hj9sl\" (UniqueName: \"kubernetes.io/projected/4721eb83-aef7-40dd-845f-92d07e1012f0-kube-api-access-hj9sl\") pod \"4721eb83-aef7-40dd-845f-92d07e1012f0\" (UID: \"4721eb83-aef7-40dd-845f-92d07e1012f0\") " Sep 11 16:21:47.811651 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:47.811577 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4721eb83-aef7-40dd-845f-92d07e1012f0-istio-token\") pod \"4721eb83-aef7-40dd-845f-92d07e1012f0\" (UID: \"4721eb83-aef7-40dd-845f-92d07e1012f0\") " Sep 11 16:21:47.811651 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:47.811602 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4721eb83-aef7-40dd-845f-92d07e1012f0-workload-certs\") pod \"4721eb83-aef7-40dd-845f-92d07e1012f0\" (UID: \"4721eb83-aef7-40dd-845f-92d07e1012f0\") " Sep 11 16:21:47.811849 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:47.811684 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4721eb83-aef7-40dd-845f-92d07e1012f0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4721eb83-aef7-40dd-845f-92d07e1012f0" (UID: "4721eb83-aef7-40dd-845f-92d07e1012f0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 16:21:47.811849 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:47.811824 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4721eb83-aef7-40dd-845f-92d07e1012f0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4721eb83-aef7-40dd-845f-92d07e1012f0" (UID: "4721eb83-aef7-40dd-845f-92d07e1012f0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 16:21:47.811963 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:47.811864 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4721eb83-aef7-40dd-845f-92d07e1012f0-istio-data" (OuterVolumeSpecName: "istio-data") pod "4721eb83-aef7-40dd-845f-92d07e1012f0" (UID: "4721eb83-aef7-40dd-845f-92d07e1012f0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 16:21:47.811963 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:47.811874 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4721eb83-aef7-40dd-845f-92d07e1012f0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4721eb83-aef7-40dd-845f-92d07e1012f0" (UID: "4721eb83-aef7-40dd-845f-92d07e1012f0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 16:21:47.811963 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:47.811900 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4721eb83-aef7-40dd-845f-92d07e1012f0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4721eb83-aef7-40dd-845f-92d07e1012f0" (UID: "4721eb83-aef7-40dd-845f-92d07e1012f0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 16:21:47.812148 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:47.812112 2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4721eb83-aef7-40dd-845f-92d07e1012f0-istiod-ca-cert\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:21:47.812148 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:47.812127 2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4721eb83-aef7-40dd-845f-92d07e1012f0-workload-socket\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:21:47.812148 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:47.812135 2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4721eb83-aef7-40dd-845f-92d07e1012f0-istio-data\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:21:47.812148 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:47.812143 2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4721eb83-aef7-40dd-845f-92d07e1012f0-credential-socket\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:21:47.812312 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:47.812156 2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4721eb83-aef7-40dd-845f-92d07e1012f0-workload-certs\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:21:47.813853 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:47.813832 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4721eb83-aef7-40dd-845f-92d07e1012f0-kube-api-access-hj9sl" (OuterVolumeSpecName: "kube-api-access-hj9sl") pod "4721eb83-aef7-40dd-845f-92d07e1012f0" (UID: "4721eb83-aef7-40dd-845f-92d07e1012f0"). InnerVolumeSpecName "kube-api-access-hj9sl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 16:21:47.813966 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:47.813877 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4721eb83-aef7-40dd-845f-92d07e1012f0-istio-token" (OuterVolumeSpecName: "istio-token") pod "4721eb83-aef7-40dd-845f-92d07e1012f0" (UID: "4721eb83-aef7-40dd-845f-92d07e1012f0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 16:21:47.814151 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:47.814126 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4721eb83-aef7-40dd-845f-92d07e1012f0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4721eb83-aef7-40dd-845f-92d07e1012f0" (UID: "4721eb83-aef7-40dd-845f-92d07e1012f0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 16:21:47.814151 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:47.814128 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4721eb83-aef7-40dd-845f-92d07e1012f0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4721eb83-aef7-40dd-845f-92d07e1012f0" (UID: "4721eb83-aef7-40dd-845f-92d07e1012f0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 16:21:47.913442 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:47.913403 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hj9sl\" (UniqueName: \"kubernetes.io/projected/4721eb83-aef7-40dd-845f-92d07e1012f0-kube-api-access-hj9sl\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:21:47.913442 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:47.913436 2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4721eb83-aef7-40dd-845f-92d07e1012f0-istio-token\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:21:47.913442 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:47.913446 2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4721eb83-aef7-40dd-845f-92d07e1012f0-istio-envoy\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:21:47.913442 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:47.913453 2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4721eb83-aef7-40dd-845f-92d07e1012f0-istio-podinfo\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:21:48.292656 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:48.292623 2566 generic.go:356] "Generic (PLEG): container finished" podID="4721eb83-aef7-40dd-845f-92d07e1012f0" containerID="6e7a538a049a9c18fe14c65731af94f34a22db85b21cb0b5e55b1bd634d7ecad" exitCode=0 Sep 11 16:21:48.292852 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:48.292664 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bf0614-openshift-default-76d57574c6-4mwfx" event={"ID":"4721eb83-aef7-40dd-845f-92d07e1012f0","Type":"ContainerDied","Data":"6e7a538a049a9c18fe14c65731af94f34a22db85b21cb0b5e55b1bd634d7ecad"} Sep 11 16:21:48.292852 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:48.292672 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-bf0614-openshift-default-76d57574c6-4mwfx" Sep 11 16:21:48.292852 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:48.292707 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bf0614-openshift-default-76d57574c6-4mwfx" event={"ID":"4721eb83-aef7-40dd-845f-92d07e1012f0","Type":"ContainerDied","Data":"d37de84c6c3fe911b6e26366232860304d0d4ea9acbcc0966665ad70c82ef2fa"} Sep 11 16:21:48.292852 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:48.292728 2566 scope.go:117] "RemoveContainer" containerID="6e7a538a049a9c18fe14c65731af94f34a22db85b21cb0b5e55b1bd634d7ecad" Sep 11 16:21:48.302997 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:48.302962 2566 scope.go:117] "RemoveContainer" containerID="b3ab998e549829e50aacab96f123be187ae62ab18a5420374662056dab3ed508" Sep 11 16:21:48.311787 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:48.311766 2566 scope.go:117] "RemoveContainer" containerID="6e7a538a049a9c18fe14c65731af94f34a22db85b21cb0b5e55b1bd634d7ecad" Sep 11 16:21:48.312091 ip-10-0-136-96 kubenswrapper[2566]: E0911 16:21:48.312070 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e7a538a049a9c18fe14c65731af94f34a22db85b21cb0b5e55b1bd634d7ecad\": container with ID starting with 6e7a538a049a9c18fe14c65731af94f34a22db85b21cb0b5e55b1bd634d7ecad not found: ID does not exist" containerID="6e7a538a049a9c18fe14c65731af94f34a22db85b21cb0b5e55b1bd634d7ecad" Sep 11 16:21:48.312186 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:48.312103 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e7a538a049a9c18fe14c65731af94f34a22db85b21cb0b5e55b1bd634d7ecad"} err="failed to get container status \"6e7a538a049a9c18fe14c65731af94f34a22db85b21cb0b5e55b1bd634d7ecad\": rpc error: code = NotFound desc = could not find container \"6e7a538a049a9c18fe14c65731af94f34a22db85b21cb0b5e55b1bd634d7ecad\": container with ID starting with 6e7a538a049a9c18fe14c65731af94f34a22db85b21cb0b5e55b1bd634d7ecad not found: ID does not exist" Sep 11 16:21:48.312186 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:48.312127 2566 scope.go:117] "RemoveContainer" containerID="b3ab998e549829e50aacab96f123be187ae62ab18a5420374662056dab3ed508" Sep 11 16:21:48.312487 ip-10-0-136-96 kubenswrapper[2566]: E0911 16:21:48.312467 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3ab998e549829e50aacab96f123be187ae62ab18a5420374662056dab3ed508\": container with ID starting with b3ab998e549829e50aacab96f123be187ae62ab18a5420374662056dab3ed508 not found: ID does not exist" containerID="b3ab998e549829e50aacab96f123be187ae62ab18a5420374662056dab3ed508" Sep 11 16:21:48.312534 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:48.312493 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3ab998e549829e50aacab96f123be187ae62ab18a5420374662056dab3ed508"} err="failed to get container status \"b3ab998e549829e50aacab96f123be187ae62ab18a5420374662056dab3ed508\": rpc error: code = NotFound desc = could not find container \"b3ab998e549829e50aacab96f123be187ae62ab18a5420374662056dab3ed508\": container with ID starting with b3ab998e549829e50aacab96f123be187ae62ab18a5420374662056dab3ed508 not found: ID does not exist" Sep 11 16:21:48.319057 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:48.319033 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-bf0614-openshift-default-76d57574c6-4mwfx"] Sep 11 16:21:48.325179 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:48.325153 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-bf0614-openshift-default-76d57574c6-4mwfx"] Sep 11 16:21:49.240304 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:49.240269 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4721eb83-aef7-40dd-845f-92d07e1012f0" path="/var/lib/kubelet/pods/4721eb83-aef7-40dd-845f-92d07e1012f0/volumes" Sep 11 16:21:53.673954 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:53.673920 2566 scope.go:117] "RemoveContainer" containerID="eb9814ef703d781325f86402c443afe91023349369de8d2b1ba7f99239e9fc2b" Sep 11 16:21:53.686749 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:53.686720 2566 scope.go:117] "RemoveContainer" containerID="fe5a9c6fc6250e35c083608b8da61b6195704837be358658312064d0e854fca8" Sep 11 16:21:59.223383 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:21:59.223344 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dx4xn"] Sep 11 16:22:01.038741 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:01.038706 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-f7c21d-openshift-default-857f5cd77b-8phc5"] Sep 11 16:22:01.039137 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:01.038961 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-f7c21d-openshift-default-857f5cd77b-8phc5" podUID="b78c6245-c285-479b-a463-42d55ce25b74" containerName="istio-proxy" containerID="cri-o://fd2e910431b4385bbce440937f593040756fd71e123212032c7c50564c2f6872" gracePeriod=30 Sep 11 16:22:01.503446 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:01.503369 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-320b99-scxw8"] Sep 11 16:22:01.503793 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:01.503780 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b262660d-18f1-4ec8-963d-60383e08d57d" containerName="istio-proxy" Sep 11 16:22:01.503839 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:01.503795 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b262660d-18f1-4ec8-963d-60383e08d57d" containerName="istio-proxy" Sep 11 16:22:01.503839 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:01.503802 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4721eb83-aef7-40dd-845f-92d07e1012f0" containerName="istio-proxy" Sep 11 16:22:01.503839 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:01.503808 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="4721eb83-aef7-40dd-845f-92d07e1012f0" containerName="istio-proxy" Sep 11 16:22:01.503839 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:01.503832 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d732abd-a686-4604-991b-6d1c1e24296e" containerName="test-curl-7095-fe39fe" Sep 11 16:22:01.503839 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:01.503838 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d732abd-a686-4604-991b-6d1c1e24296e" containerName="test-curl-7095-fe39fe" Sep 11 16:22:01.504027 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:01.503898 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d732abd-a686-4604-991b-6d1c1e24296e" containerName="test-curl-7095-fe39fe" Sep 11 16:22:01.504027 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:01.503908 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="4721eb83-aef7-40dd-845f-92d07e1012f0" containerName="istio-proxy" Sep 11 16:22:01.504027 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:01.503916 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="b262660d-18f1-4ec8-963d-60383e08d57d" containerName="istio-proxy" Sep 11 16:22:01.504027 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:01.503922 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="b262660d-18f1-4ec8-963d-60383e08d57d" containerName="istio-proxy" Sep 11 16:22:01.504027 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:01.503929 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="4721eb83-aef7-40dd-845f-92d07e1012f0" containerName="istio-proxy" Sep 11 16:22:01.507123 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:01.507107 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-320b99-scxw8" Sep 11 16:22:01.515151 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:01.515113 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 16:22:01.515450 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:01.515429 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qbzdr\"" Sep 11 16:22:01.518916 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:01.518894 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-320b99-scxw8"] Sep 11 16:22:01.552207 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:01.552178 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1559b571-c2cc-47f1-b866-ca59ad34f76e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-320b99-scxw8\" (UID: \"1559b571-c2cc-47f1-b866-ca59ad34f76e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-320b99-scxw8" Sep 11 16:22:01.653067 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:01.653027 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1559b571-c2cc-47f1-b866-ca59ad34f76e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-320b99-scxw8\" (UID: \"1559b571-c2cc-47f1-b866-ca59ad34f76e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-320b99-scxw8" Sep 11 16:22:01.653636 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:01.653618 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1559b571-c2cc-47f1-b866-ca59ad34f76e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-320b99-scxw8\" (UID: \"1559b571-c2cc-47f1-b866-ca59ad34f76e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-320b99-scxw8" Sep 11 16:22:01.818273 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:01.818234 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-320b99-scxw8" Sep 11 16:22:02.127380 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:02.127351 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-320b99-scxw8"] Sep 11 16:22:02.129210 ip-10-0-136-96 kubenswrapper[2566]: W0911 16:22:02.129183 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1559b571_c2cc_47f1_b866_ca59ad34f76e.slice/crio-0c6018f2ce2e50594e7c14b19b1c333118470c0f216928a0760071f89b747505 WatchSource:0}: Error finding container 0c6018f2ce2e50594e7c14b19b1c333118470c0f216928a0760071f89b747505: Status 404 returned error can't find the container with id 0c6018f2ce2e50594e7c14b19b1c333118470c0f216928a0760071f89b747505 Sep 11 16:22:02.386869 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:02.386778 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-320b99-scxw8" event={"ID":"1559b571-c2cc-47f1-b866-ca59ad34f76e","Type":"ContainerStarted","Data":"0c6018f2ce2e50594e7c14b19b1c333118470c0f216928a0760071f89b747505"} Sep 11 16:22:03.394573 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:03.394537 2566 generic.go:356] "Generic (PLEG): container finished" podID="1559b571-c2cc-47f1-b866-ca59ad34f76e" containerID="97bf5cb3bec3e2492ad766d066f74c47d463213a447cbb094d90b4e707732bf3" exitCode=0 Sep 11 16:22:03.394954 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:03.394588 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-320b99-scxw8" event={"ID":"1559b571-c2cc-47f1-b866-ca59ad34f76e","Type":"ContainerDied","Data":"97bf5cb3bec3e2492ad766d066f74c47d463213a447cbb094d90b4e707732bf3"} Sep 11 16:22:04.549551 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:04.549520 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-320b99-scxw8" Sep 11 16:22:04.685097 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:04.684996 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1559b571-c2cc-47f1-b866-ca59ad34f76e-maas-api-service-ca\") pod \"1559b571-c2cc-47f1-b866-ca59ad34f76e\" (UID: \"1559b571-c2cc-47f1-b866-ca59ad34f76e\") " Sep 11 16:22:04.685339 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:04.685317 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1559b571-c2cc-47f1-b866-ca59ad34f76e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1559b571-c2cc-47f1-b866-ca59ad34f76e" (UID: "1559b571-c2cc-47f1-b866-ca59ad34f76e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 16:22:04.786015 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:04.785958 2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1559b571-c2cc-47f1-b866-ca59ad34f76e-maas-api-service-ca\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:22:05.417900 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:05.417864 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-320b99-scxw8" Sep 11 16:22:05.418111 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:05.417862 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-320b99-scxw8" event={"ID":"1559b571-c2cc-47f1-b866-ca59ad34f76e","Type":"ContainerDied","Data":"0c6018f2ce2e50594e7c14b19b1c333118470c0f216928a0760071f89b747505"} Sep 11 16:22:05.418111 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:05.418063 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0c6018f2ce2e50594e7c14b19b1c333118470c0f216928a0760071f89b747505" Sep 11 16:22:06.299815 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:06.299788 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-f7c21d-openshift-default-857f5cd77b-8phc5" Sep 11 16:22:06.403277 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:06.403193 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b78c6245-c285-479b-a463-42d55ce25b74-credential-socket\") pod \"b78c6245-c285-479b-a463-42d55ce25b74\" (UID: \"b78c6245-c285-479b-a463-42d55ce25b74\") " Sep 11 16:22:06.403277 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:06.403242 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b78c6245-c285-479b-a463-42d55ce25b74-istio-token\") pod \"b78c6245-c285-479b-a463-42d55ce25b74\" (UID: \"b78c6245-c285-479b-a463-42d55ce25b74\") " Sep 11 16:22:06.403277 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:06.403278 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b78c6245-c285-479b-a463-42d55ce25b74-istio-data\") pod \"b78c6245-c285-479b-a463-42d55ce25b74\" (UID: \"b78c6245-c285-479b-a463-42d55ce25b74\") " Sep 11 16:22:06.403544 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:06.403317 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b78c6245-c285-479b-a463-42d55ce25b74-istio-envoy\") pod \"b78c6245-c285-479b-a463-42d55ce25b74\" (UID: \"b78c6245-c285-479b-a463-42d55ce25b74\") " Sep 11 16:22:06.403544 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:06.403343 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b78c6245-c285-479b-a463-42d55ce25b74-istiod-ca-cert\") pod \"b78c6245-c285-479b-a463-42d55ce25b74\" (UID: \"b78c6245-c285-479b-a463-42d55ce25b74\") " Sep 11 16:22:06.403544 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:06.403410 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b78c6245-c285-479b-a463-42d55ce25b74-workload-certs\") pod \"b78c6245-c285-479b-a463-42d55ce25b74\" (UID: \"b78c6245-c285-479b-a463-42d55ce25b74\") " Sep 11 16:22:06.403544 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:06.403454 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b78c6245-c285-479b-a463-42d55ce25b74-workload-socket\") pod \"b78c6245-c285-479b-a463-42d55ce25b74\" (UID: \"b78c6245-c285-479b-a463-42d55ce25b74\") " Sep 11 16:22:06.403544 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:06.403484 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b78c6245-c285-479b-a463-42d55ce25b74-istio-podinfo\") pod \"b78c6245-c285-479b-a463-42d55ce25b74\" (UID: \"b78c6245-c285-479b-a463-42d55ce25b74\") " Sep 11 16:22:06.403544 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:06.403498 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b78c6245-c285-479b-a463-42d55ce25b74-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b78c6245-c285-479b-a463-42d55ce25b74" (UID: "b78c6245-c285-479b-a463-42d55ce25b74"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 16:22:06.403544 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:06.403525 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8xtxl\" (UniqueName: \"kubernetes.io/projected/b78c6245-c285-479b-a463-42d55ce25b74-kube-api-access-8xtxl\") pod \"b78c6245-c285-479b-a463-42d55ce25b74\" (UID: \"b78c6245-c285-479b-a463-42d55ce25b74\") " Sep 11 16:22:06.403950 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:06.403597 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b78c6245-c285-479b-a463-42d55ce25b74-istio-data" (OuterVolumeSpecName: "istio-data") pod "b78c6245-c285-479b-a463-42d55ce25b74" (UID: "b78c6245-c285-479b-a463-42d55ce25b74"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 16:22:06.403950 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:06.403715 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b78c6245-c285-479b-a463-42d55ce25b74-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b78c6245-c285-479b-a463-42d55ce25b74" (UID: "b78c6245-c285-479b-a463-42d55ce25b74"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 16:22:06.403950 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:06.403792 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b78c6245-c285-479b-a463-42d55ce25b74-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b78c6245-c285-479b-a463-42d55ce25b74" (UID: "b78c6245-c285-479b-a463-42d55ce25b74"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 16:22:06.403950 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:06.403953 2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b78c6245-c285-479b-a463-42d55ce25b74-workload-certs\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:22:06.403950 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:06.403988 2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b78c6245-c285-479b-a463-42d55ce25b74-workload-socket\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:22:06.404268 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:06.404001 2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b78c6245-c285-479b-a463-42d55ce25b74-credential-socket\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:22:06.404268 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:06.404015 2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b78c6245-c285-479b-a463-42d55ce25b74-istio-data\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:22:06.404373 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:06.404307 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b78c6245-c285-479b-a463-42d55ce25b74-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b78c6245-c285-479b-a463-42d55ce25b74" (UID: "b78c6245-c285-479b-a463-42d55ce25b74"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 16:22:06.405626 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:06.405599 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b78c6245-c285-479b-a463-42d55ce25b74-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b78c6245-c285-479b-a463-42d55ce25b74" (UID: "b78c6245-c285-479b-a463-42d55ce25b74"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 16:22:06.405761 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:06.405650 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b78c6245-c285-479b-a463-42d55ce25b74-istio-token" (OuterVolumeSpecName: "istio-token") pod "b78c6245-c285-479b-a463-42d55ce25b74" (UID: "b78c6245-c285-479b-a463-42d55ce25b74"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 16:22:06.406098 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:06.406072 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b78c6245-c285-479b-a463-42d55ce25b74-kube-api-access-8xtxl" (OuterVolumeSpecName: "kube-api-access-8xtxl") pod "b78c6245-c285-479b-a463-42d55ce25b74" (UID: "b78c6245-c285-479b-a463-42d55ce25b74"). InnerVolumeSpecName "kube-api-access-8xtxl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 16:22:06.406200 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:06.406139 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b78c6245-c285-479b-a463-42d55ce25b74-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b78c6245-c285-479b-a463-42d55ce25b74" (UID: "b78c6245-c285-479b-a463-42d55ce25b74"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 16:22:06.425665 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:06.425630 2566 generic.go:356] "Generic (PLEG): container finished" podID="b78c6245-c285-479b-a463-42d55ce25b74" containerID="fd2e910431b4385bbce440937f593040756fd71e123212032c7c50564c2f6872" exitCode=0 Sep 11 16:22:06.425816 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:06.425670 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-f7c21d-openshift-default-857f5cd77b-8phc5" Sep 11 16:22:06.425816 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:06.425719 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-f7c21d-openshift-default-857f5cd77b-8phc5" event={"ID":"b78c6245-c285-479b-a463-42d55ce25b74","Type":"ContainerDied","Data":"fd2e910431b4385bbce440937f593040756fd71e123212032c7c50564c2f6872"} Sep 11 16:22:06.425816 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:06.425788 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-f7c21d-openshift-default-857f5cd77b-8phc5" event={"ID":"b78c6245-c285-479b-a463-42d55ce25b74","Type":"ContainerDied","Data":"120e39fb87a1c08ca5ca7136985ad81999e618a19c1e0204c365862bf744d44a"} Sep 11 16:22:06.425945 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:06.425816 2566 scope.go:117] "RemoveContainer" containerID="fd2e910431b4385bbce440937f593040756fd71e123212032c7c50564c2f6872" Sep 11 16:22:06.436034 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:06.436012 2566 scope.go:117] "RemoveContainer" containerID="a4855cdb754886d540398e0300ee96f9d5a9f44587c49318fa62a1db4df3f4a7" Sep 11 16:22:06.445598 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:06.445577 2566 scope.go:117] "RemoveContainer" containerID="fd2e910431b4385bbce440937f593040756fd71e123212032c7c50564c2f6872" Sep 11 16:22:06.445858 ip-10-0-136-96 kubenswrapper[2566]: E0911 16:22:06.445839 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd2e910431b4385bbce440937f593040756fd71e123212032c7c50564c2f6872\": container with ID starting with fd2e910431b4385bbce440937f593040756fd71e123212032c7c50564c2f6872 not found: ID does not exist" containerID="fd2e910431b4385bbce440937f593040756fd71e123212032c7c50564c2f6872" Sep 11 16:22:06.445918 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:06.445867 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd2e910431b4385bbce440937f593040756fd71e123212032c7c50564c2f6872"} err="failed to get container status \"fd2e910431b4385bbce440937f593040756fd71e123212032c7c50564c2f6872\": rpc error: code = NotFound desc = could not find container \"fd2e910431b4385bbce440937f593040756fd71e123212032c7c50564c2f6872\": container with ID starting with fd2e910431b4385bbce440937f593040756fd71e123212032c7c50564c2f6872 not found: ID does not exist" Sep 11 16:22:06.445918 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:06.445885 2566 scope.go:117] "RemoveContainer" containerID="a4855cdb754886d540398e0300ee96f9d5a9f44587c49318fa62a1db4df3f4a7" Sep 11 16:22:06.446178 ip-10-0-136-96 kubenswrapper[2566]: E0911 16:22:06.446161 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4855cdb754886d540398e0300ee96f9d5a9f44587c49318fa62a1db4df3f4a7\": container with ID starting with a4855cdb754886d540398e0300ee96f9d5a9f44587c49318fa62a1db4df3f4a7 not found: ID does not exist" containerID="a4855cdb754886d540398e0300ee96f9d5a9f44587c49318fa62a1db4df3f4a7" Sep 11 16:22:06.446230 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:06.446180 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4855cdb754886d540398e0300ee96f9d5a9f44587c49318fa62a1db4df3f4a7"} err="failed to get container status \"a4855cdb754886d540398e0300ee96f9d5a9f44587c49318fa62a1db4df3f4a7\": rpc error: code = NotFound desc = could not find container \"a4855cdb754886d540398e0300ee96f9d5a9f44587c49318fa62a1db4df3f4a7\": container with ID starting with a4855cdb754886d540398e0300ee96f9d5a9f44587c49318fa62a1db4df3f4a7 not found: ID does not exist" Sep 11 16:22:06.450314 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:06.450286 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-f7c21d-openshift-default-857f5cd77b-8phc5"] Sep 11 16:22:06.456643 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:06.456611 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-f7c21d-openshift-default-857f5cd77b-8phc5"] Sep 11 16:22:06.505228 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:06.505195 2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b78c6245-c285-479b-a463-42d55ce25b74-istio-podinfo\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:22:06.505228 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:06.505222 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xtxl\" (UniqueName: \"kubernetes.io/projected/b78c6245-c285-479b-a463-42d55ce25b74-kube-api-access-8xtxl\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:22:06.505228 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:06.505232 2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b78c6245-c285-479b-a463-42d55ce25b74-istio-token\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:22:06.505422 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:06.505241 2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b78c6245-c285-479b-a463-42d55ce25b74-istio-envoy\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:22:06.505422 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:06.505251 2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b78c6245-c285-479b-a463-42d55ce25b74-istiod-ca-cert\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:22:07.239422 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:07.239386 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b78c6245-c285-479b-a463-42d55ce25b74" path="/var/lib/kubelet/pods/b78c6245-c285-479b-a463-42d55ce25b74/volumes" Sep 11 16:22:09.219193 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:09.219156 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dx4xn"] Sep 11 16:22:12.372901 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:12.372858 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-320b99-scxw8"] Sep 11 16:22:12.376318 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:12.376285 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-320b99-scxw8"] Sep 11 16:22:13.240140 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:13.240107 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1559b571-c2cc-47f1-b866-ca59ad34f76e" path="/var/lib/kubelet/pods/1559b571-c2cc-47f1-b866-ca59ad34f76e/volumes" Sep 11 16:22:19.984811 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:19.984775 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-320b99-openshift-default-6c5bb8bb78-98z46"] Sep 11 16:22:19.985253 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:19.985021 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-320b99-openshift-default-6c5bb8bb78-98z46" podUID="a1766158-f57b-4ce5-bad7-026f08295c84" containerName="istio-proxy" containerID="cri-o://4a09f22563b21079215869903ee522cfaff9ad5d1f0dc83317a286eba0fdf37c" gracePeriod=30 Sep 11 16:22:23.522085 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:23.522048 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dx4xn"] Sep 11 16:22:25.233741 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:25.233713 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-320b99-openshift-default-6c5bb8bb78-98z46" Sep 11 16:22:25.289840 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:25.289805 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xxj5z\" (UniqueName: \"kubernetes.io/projected/a1766158-f57b-4ce5-bad7-026f08295c84-kube-api-access-xxj5z\") pod \"a1766158-f57b-4ce5-bad7-026f08295c84\" (UID: \"a1766158-f57b-4ce5-bad7-026f08295c84\") " Sep 11 16:22:25.290046 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:25.289856 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1766158-f57b-4ce5-bad7-026f08295c84-istio-podinfo\") pod \"a1766158-f57b-4ce5-bad7-026f08295c84\" (UID: \"a1766158-f57b-4ce5-bad7-026f08295c84\") " Sep 11 16:22:25.290046 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:25.289886 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1766158-f57b-4ce5-bad7-026f08295c84-workload-certs\") pod \"a1766158-f57b-4ce5-bad7-026f08295c84\" (UID: \"a1766158-f57b-4ce5-bad7-026f08295c84\") " Sep 11 16:22:25.290046 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:25.290031 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1766158-f57b-4ce5-bad7-026f08295c84-istiod-ca-cert\") pod \"a1766158-f57b-4ce5-bad7-026f08295c84\" (UID: \"a1766158-f57b-4ce5-bad7-026f08295c84\") " Sep 11 16:22:25.290195 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:25.290081 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1766158-f57b-4ce5-bad7-026f08295c84-credential-socket\") pod \"a1766158-f57b-4ce5-bad7-026f08295c84\" (UID: \"a1766158-f57b-4ce5-bad7-026f08295c84\") " Sep 11 16:22:25.290195 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:25.290116 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1766158-f57b-4ce5-bad7-026f08295c84-istio-envoy\") pod \"a1766158-f57b-4ce5-bad7-026f08295c84\" (UID: \"a1766158-f57b-4ce5-bad7-026f08295c84\") " Sep 11 16:22:25.290195 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:25.290177 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1766158-f57b-4ce5-bad7-026f08295c84-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a1766158-f57b-4ce5-bad7-026f08295c84" (UID: "a1766158-f57b-4ce5-bad7-026f08295c84"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 16:22:25.290362 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:25.290213 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1766158-f57b-4ce5-bad7-026f08295c84-istio-data\") pod \"a1766158-f57b-4ce5-bad7-026f08295c84\" (UID: \"a1766158-f57b-4ce5-bad7-026f08295c84\") " Sep 11 16:22:25.290362 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:25.290250 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1766158-f57b-4ce5-bad7-026f08295c84-istio-token\") pod \"a1766158-f57b-4ce5-bad7-026f08295c84\" (UID: \"a1766158-f57b-4ce5-bad7-026f08295c84\") " Sep 11 16:22:25.290362 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:25.290288 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1766158-f57b-4ce5-bad7-026f08295c84-workload-socket\") pod \"a1766158-f57b-4ce5-bad7-026f08295c84\" (UID: \"a1766158-f57b-4ce5-bad7-026f08295c84\") " Sep 11 16:22:25.290516 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:25.290387 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1766158-f57b-4ce5-bad7-026f08295c84-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a1766158-f57b-4ce5-bad7-026f08295c84" (UID: "a1766158-f57b-4ce5-bad7-026f08295c84"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 16:22:25.290628 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:25.290605 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1766158-f57b-4ce5-bad7-026f08295c84-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a1766158-f57b-4ce5-bad7-026f08295c84" (UID: "a1766158-f57b-4ce5-bad7-026f08295c84"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 16:22:25.290720 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:25.290701 2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1766158-f57b-4ce5-bad7-026f08295c84-credential-socket\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:22:25.290808 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:25.290723 2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1766158-f57b-4ce5-bad7-026f08295c84-workload-socket\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:22:25.290808 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:25.290737 2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1766158-f57b-4ce5-bad7-026f08295c84-workload-certs\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:22:25.290808 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:25.290752 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1766158-f57b-4ce5-bad7-026f08295c84-istio-data" (OuterVolumeSpecName: "istio-data") pod "a1766158-f57b-4ce5-bad7-026f08295c84" (UID: "a1766158-f57b-4ce5-bad7-026f08295c84"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 16:22:25.290992 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:25.290918 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1766158-f57b-4ce5-bad7-026f08295c84-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a1766158-f57b-4ce5-bad7-026f08295c84" (UID: "a1766158-f57b-4ce5-bad7-026f08295c84"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 16:22:25.292135 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:25.292112 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1766158-f57b-4ce5-bad7-026f08295c84-kube-api-access-xxj5z" (OuterVolumeSpecName: "kube-api-access-xxj5z") pod "a1766158-f57b-4ce5-bad7-026f08295c84" (UID: "a1766158-f57b-4ce5-bad7-026f08295c84"). InnerVolumeSpecName "kube-api-access-xxj5z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 16:22:25.292444 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:25.292428 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1766158-f57b-4ce5-bad7-026f08295c84-istio-token" (OuterVolumeSpecName: "istio-token") pod "a1766158-f57b-4ce5-bad7-026f08295c84" (UID: "a1766158-f57b-4ce5-bad7-026f08295c84"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 16:22:25.292579 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:25.292554 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1766158-f57b-4ce5-bad7-026f08295c84-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a1766158-f57b-4ce5-bad7-026f08295c84" (UID: "a1766158-f57b-4ce5-bad7-026f08295c84"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 16:22:25.292637 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:25.292554 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a1766158-f57b-4ce5-bad7-026f08295c84-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a1766158-f57b-4ce5-bad7-026f08295c84" (UID: "a1766158-f57b-4ce5-bad7-026f08295c84"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 16:22:25.391275 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:25.391239 2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1766158-f57b-4ce5-bad7-026f08295c84-istio-envoy\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:22:25.391275 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:25.391270 2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1766158-f57b-4ce5-bad7-026f08295c84-istio-data\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:22:25.391275 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:25.391279 2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1766158-f57b-4ce5-bad7-026f08295c84-istio-token\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:22:25.391523 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:25.391287 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxj5z\" (UniqueName: \"kubernetes.io/projected/a1766158-f57b-4ce5-bad7-026f08295c84-kube-api-access-xxj5z\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:22:25.391523 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:25.391297 2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1766158-f57b-4ce5-bad7-026f08295c84-istio-podinfo\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:22:25.391523 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:25.391305 2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1766158-f57b-4ce5-bad7-026f08295c84-istiod-ca-cert\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:22:25.546355 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:25.546266 2566 generic.go:356] "Generic (PLEG): container finished" podID="a1766158-f57b-4ce5-bad7-026f08295c84" containerID="4a09f22563b21079215869903ee522cfaff9ad5d1f0dc83317a286eba0fdf37c" exitCode=0 Sep 11 16:22:25.546355 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:25.546314 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-320b99-openshift-default-6c5bb8bb78-98z46" event={"ID":"a1766158-f57b-4ce5-bad7-026f08295c84","Type":"ContainerDied","Data":"4a09f22563b21079215869903ee522cfaff9ad5d1f0dc83317a286eba0fdf37c"} Sep 11 16:22:25.546537 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:25.546362 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-320b99-openshift-default-6c5bb8bb78-98z46" event={"ID":"a1766158-f57b-4ce5-bad7-026f08295c84","Type":"ContainerDied","Data":"0c1e77431d5d44aae036577ae3ee66f3a7cf1403e15d72d595b3cc1925e6b6f3"} Sep 11 16:22:25.546537 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:25.546386 2566 scope.go:117] "RemoveContainer" containerID="4a09f22563b21079215869903ee522cfaff9ad5d1f0dc83317a286eba0fdf37c" Sep 11 16:22:25.546537 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:25.546321 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-320b99-openshift-default-6c5bb8bb78-98z46" Sep 11 16:22:25.557471 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:25.557450 2566 scope.go:117] "RemoveContainer" containerID="c1b271e16934bbdd41dd463a1c5c75c8241484a68a13bc9596097a393edf6351" Sep 11 16:22:25.566154 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:25.566133 2566 scope.go:117] "RemoveContainer" containerID="4a09f22563b21079215869903ee522cfaff9ad5d1f0dc83317a286eba0fdf37c" Sep 11 16:22:25.566428 ip-10-0-136-96 kubenswrapper[2566]: E0911 16:22:25.566409 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a09f22563b21079215869903ee522cfaff9ad5d1f0dc83317a286eba0fdf37c\": container with ID starting with 4a09f22563b21079215869903ee522cfaff9ad5d1f0dc83317a286eba0fdf37c not found: ID does not exist" containerID="4a09f22563b21079215869903ee522cfaff9ad5d1f0dc83317a286eba0fdf37c" Sep 11 16:22:25.566474 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:25.566438 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a09f22563b21079215869903ee522cfaff9ad5d1f0dc83317a286eba0fdf37c"} err="failed to get container status \"4a09f22563b21079215869903ee522cfaff9ad5d1f0dc83317a286eba0fdf37c\": rpc error: code = NotFound desc = could not find container \"4a09f22563b21079215869903ee522cfaff9ad5d1f0dc83317a286eba0fdf37c\": container with ID starting with 4a09f22563b21079215869903ee522cfaff9ad5d1f0dc83317a286eba0fdf37c not found: ID does not exist" Sep 11 16:22:25.566474 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:25.566458 2566 scope.go:117] "RemoveContainer" containerID="c1b271e16934bbdd41dd463a1c5c75c8241484a68a13bc9596097a393edf6351" Sep 11 16:22:25.566737 ip-10-0-136-96 kubenswrapper[2566]: E0911 16:22:25.566700 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1b271e16934bbdd41dd463a1c5c75c8241484a68a13bc9596097a393edf6351\": container with ID starting with c1b271e16934bbdd41dd463a1c5c75c8241484a68a13bc9596097a393edf6351 not found: ID does not exist" containerID="c1b271e16934bbdd41dd463a1c5c75c8241484a68a13bc9596097a393edf6351" Sep 11 16:22:25.566869 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:25.566741 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1b271e16934bbdd41dd463a1c5c75c8241484a68a13bc9596097a393edf6351"} err="failed to get container status \"c1b271e16934bbdd41dd463a1c5c75c8241484a68a13bc9596097a393edf6351\": rpc error: code = NotFound desc = could not find container \"c1b271e16934bbdd41dd463a1c5c75c8241484a68a13bc9596097a393edf6351\": container with ID starting with c1b271e16934bbdd41dd463a1c5c75c8241484a68a13bc9596097a393edf6351 not found: ID does not exist" Sep 11 16:22:25.570046 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:25.570025 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-320b99-openshift-default-6c5bb8bb78-98z46"] Sep 11 16:22:25.575085 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:25.575063 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-320b99-openshift-default-6c5bb8bb78-98z46"] Sep 11 16:22:27.240740 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:27.240699 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a1766158-f57b-4ce5-bad7-026f08295c84" path="/var/lib/kubelet/pods/a1766158-f57b-4ce5-bad7-026f08295c84/volumes" Sep 11 16:22:34.653945 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:34.653906 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dx4xn"] Sep 11 16:22:43.396652 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:43.396607 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-f9d303-openshift-default-5754c68888-x9vh2"] Sep 11 16:22:43.397099 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:43.397080 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1766158-f57b-4ce5-bad7-026f08295c84" containerName="istio-proxy" Sep 11 16:22:43.397160 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:43.397104 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1766158-f57b-4ce5-bad7-026f08295c84" containerName="istio-proxy" Sep 11 16:22:43.397160 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:43.397114 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1766158-f57b-4ce5-bad7-026f08295c84" containerName="istio-proxy" Sep 11 16:22:43.397160 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:43.397120 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1766158-f57b-4ce5-bad7-026f08295c84" containerName="istio-proxy" Sep 11 16:22:43.397160 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:43.397125 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1559b571-c2cc-47f1-b866-ca59ad34f76e" containerName="revoke-keys" Sep 11 16:22:43.397160 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:43.397131 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="1559b571-c2cc-47f1-b866-ca59ad34f76e" containerName="revoke-keys" Sep 11 16:22:43.397160 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:43.397140 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b78c6245-c285-479b-a463-42d55ce25b74" containerName="istio-proxy" Sep 11 16:22:43.397160 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:43.397147 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b78c6245-c285-479b-a463-42d55ce25b74" containerName="istio-proxy" Sep 11 16:22:43.397404 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:43.397170 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b262660d-18f1-4ec8-963d-60383e08d57d" containerName="istio-proxy" Sep 11 16:22:43.397404 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:43.397178 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b262660d-18f1-4ec8-963d-60383e08d57d" containerName="istio-proxy" Sep 11 16:22:43.397404 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:43.397187 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4721eb83-aef7-40dd-845f-92d07e1012f0" containerName="istio-proxy" Sep 11 16:22:43.397404 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:43.397195 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="4721eb83-aef7-40dd-845f-92d07e1012f0" containerName="istio-proxy" Sep 11 16:22:43.397404 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:43.397209 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b78c6245-c285-479b-a463-42d55ce25b74" containerName="istio-proxy" Sep 11 16:22:43.397404 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:43.397216 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b78c6245-c285-479b-a463-42d55ce25b74" containerName="istio-proxy" Sep 11 16:22:43.397404 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:43.397340 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="1559b571-c2cc-47f1-b866-ca59ad34f76e" containerName="revoke-keys" Sep 11 16:22:43.397404 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:43.397353 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1766158-f57b-4ce5-bad7-026f08295c84" containerName="istio-proxy" Sep 11 16:22:43.397404 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:43.397365 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1766158-f57b-4ce5-bad7-026f08295c84" containerName="istio-proxy" Sep 11 16:22:43.397404 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:43.397376 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="b78c6245-c285-479b-a463-42d55ce25b74" containerName="istio-proxy" Sep 11 16:22:43.400547 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:43.400530 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f9d303-openshift-default-5754c68888-x9vh2" Sep 11 16:22:43.403642 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:43.403612 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-f9d303-openshift-default-dockercfg-qqtms\"" Sep 11 16:22:43.406380 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:43.406355 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-f9d303-openshift-default-5754c68888-x9vh2"] Sep 11 16:22:43.474122 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:43.474086 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63e70794-d8a3-4a93-af85-6060def3d813-istiod-ca-cert\") pod \"e2e-ipp-a-f9d303-openshift-default-5754c68888-x9vh2\" (UID: \"63e70794-d8a3-4a93-af85-6060def3d813\") " pod="openshift-ingress/e2e-ipp-a-f9d303-openshift-default-5754c68888-x9vh2" Sep 11 16:22:43.474328 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:43.474278 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63e70794-d8a3-4a93-af85-6060def3d813-istio-envoy\") pod \"e2e-ipp-a-f9d303-openshift-default-5754c68888-x9vh2\" (UID: \"63e70794-d8a3-4a93-af85-6060def3d813\") " pod="openshift-ingress/e2e-ipp-a-f9d303-openshift-default-5754c68888-x9vh2" Sep 11 16:22:43.474570 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:43.474379 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63e70794-d8a3-4a93-af85-6060def3d813-credential-socket\") pod \"e2e-ipp-a-f9d303-openshift-default-5754c68888-x9vh2\" (UID: \"63e70794-d8a3-4a93-af85-6060def3d813\") " pod="openshift-ingress/e2e-ipp-a-f9d303-openshift-default-5754c68888-x9vh2" Sep 11 16:22:43.474570 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:43.474414 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63e70794-d8a3-4a93-af85-6060def3d813-istio-podinfo\") pod \"e2e-ipp-a-f9d303-openshift-default-5754c68888-x9vh2\" (UID: \"63e70794-d8a3-4a93-af85-6060def3d813\") " pod="openshift-ingress/e2e-ipp-a-f9d303-openshift-default-5754c68888-x9vh2" Sep 11 16:22:43.474570 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:43.474451 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qngc7\" (UniqueName: \"kubernetes.io/projected/63e70794-d8a3-4a93-af85-6060def3d813-kube-api-access-qngc7\") pod \"e2e-ipp-a-f9d303-openshift-default-5754c68888-x9vh2\" (UID: \"63e70794-d8a3-4a93-af85-6060def3d813\") " pod="openshift-ingress/e2e-ipp-a-f9d303-openshift-default-5754c68888-x9vh2" Sep 11 16:22:43.474570 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:43.474502 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63e70794-d8a3-4a93-af85-6060def3d813-istio-data\") pod \"e2e-ipp-a-f9d303-openshift-default-5754c68888-x9vh2\" (UID: \"63e70794-d8a3-4a93-af85-6060def3d813\") " pod="openshift-ingress/e2e-ipp-a-f9d303-openshift-default-5754c68888-x9vh2" Sep 11 16:22:43.475094 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:43.474589 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63e70794-d8a3-4a93-af85-6060def3d813-workload-socket\") pod \"e2e-ipp-a-f9d303-openshift-default-5754c68888-x9vh2\" (UID: \"63e70794-d8a3-4a93-af85-6060def3d813\") " pod="openshift-ingress/e2e-ipp-a-f9d303-openshift-default-5754c68888-x9vh2" Sep 11 16:22:43.475094 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:43.474616 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63e70794-d8a3-4a93-af85-6060def3d813-workload-certs\") pod \"e2e-ipp-a-f9d303-openshift-default-5754c68888-x9vh2\" (UID: \"63e70794-d8a3-4a93-af85-6060def3d813\") " pod="openshift-ingress/e2e-ipp-a-f9d303-openshift-default-5754c68888-x9vh2" Sep 11 16:22:43.475094 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:43.474647 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63e70794-d8a3-4a93-af85-6060def3d813-istio-token\") pod \"e2e-ipp-a-f9d303-openshift-default-5754c68888-x9vh2\" (UID: \"63e70794-d8a3-4a93-af85-6060def3d813\") " pod="openshift-ingress/e2e-ipp-a-f9d303-openshift-default-5754c68888-x9vh2" Sep 11 16:22:43.576029 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:43.575963 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63e70794-d8a3-4a93-af85-6060def3d813-istio-envoy\") pod \"e2e-ipp-a-f9d303-openshift-default-5754c68888-x9vh2\" (UID: \"63e70794-d8a3-4a93-af85-6060def3d813\") " pod="openshift-ingress/e2e-ipp-a-f9d303-openshift-default-5754c68888-x9vh2" Sep 11 16:22:43.576204 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:43.576084 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63e70794-d8a3-4a93-af85-6060def3d813-credential-socket\") pod \"e2e-ipp-a-f9d303-openshift-default-5754c68888-x9vh2\" (UID: \"63e70794-d8a3-4a93-af85-6060def3d813\") " pod="openshift-ingress/e2e-ipp-a-f9d303-openshift-default-5754c68888-x9vh2" Sep 11 16:22:43.576204 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:43.576107 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63e70794-d8a3-4a93-af85-6060def3d813-istio-podinfo\") pod \"e2e-ipp-a-f9d303-openshift-default-5754c68888-x9vh2\" (UID: \"63e70794-d8a3-4a93-af85-6060def3d813\") " pod="openshift-ingress/e2e-ipp-a-f9d303-openshift-default-5754c68888-x9vh2" Sep 11 16:22:43.576204 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:43.576124 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qngc7\" (UniqueName: \"kubernetes.io/projected/63e70794-d8a3-4a93-af85-6060def3d813-kube-api-access-qngc7\") pod \"e2e-ipp-a-f9d303-openshift-default-5754c68888-x9vh2\" (UID: \"63e70794-d8a3-4a93-af85-6060def3d813\") " pod="openshift-ingress/e2e-ipp-a-f9d303-openshift-default-5754c68888-x9vh2" Sep 11 16:22:43.576204 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:43.576150 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63e70794-d8a3-4a93-af85-6060def3d813-istio-data\") pod \"e2e-ipp-a-f9d303-openshift-default-5754c68888-x9vh2\" (UID: \"63e70794-d8a3-4a93-af85-6060def3d813\") " pod="openshift-ingress/e2e-ipp-a-f9d303-openshift-default-5754c68888-x9vh2" Sep 11 16:22:43.576204 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:43.576196 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63e70794-d8a3-4a93-af85-6060def3d813-workload-socket\") pod \"e2e-ipp-a-f9d303-openshift-default-5754c68888-x9vh2\" (UID: \"63e70794-d8a3-4a93-af85-6060def3d813\") " pod="openshift-ingress/e2e-ipp-a-f9d303-openshift-default-5754c68888-x9vh2" Sep 11 16:22:43.576552 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:43.576221 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63e70794-d8a3-4a93-af85-6060def3d813-workload-certs\") pod \"e2e-ipp-a-f9d303-openshift-default-5754c68888-x9vh2\" (UID: \"63e70794-d8a3-4a93-af85-6060def3d813\") " pod="openshift-ingress/e2e-ipp-a-f9d303-openshift-default-5754c68888-x9vh2" Sep 11 16:22:43.576552 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:43.576254 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63e70794-d8a3-4a93-af85-6060def3d813-istio-token\") pod \"e2e-ipp-a-f9d303-openshift-default-5754c68888-x9vh2\" (UID: \"63e70794-d8a3-4a93-af85-6060def3d813\") " pod="openshift-ingress/e2e-ipp-a-f9d303-openshift-default-5754c68888-x9vh2" Sep 11 16:22:43.576552 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:43.576319 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63e70794-d8a3-4a93-af85-6060def3d813-istiod-ca-cert\") pod \"e2e-ipp-a-f9d303-openshift-default-5754c68888-x9vh2\" (UID: \"63e70794-d8a3-4a93-af85-6060def3d813\") " pod="openshift-ingress/e2e-ipp-a-f9d303-openshift-default-5754c68888-x9vh2" Sep 11 16:22:43.576766 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:43.576738 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63e70794-d8a3-4a93-af85-6060def3d813-credential-socket\") pod \"e2e-ipp-a-f9d303-openshift-default-5754c68888-x9vh2\" (UID: \"63e70794-d8a3-4a93-af85-6060def3d813\") " pod="openshift-ingress/e2e-ipp-a-f9d303-openshift-default-5754c68888-x9vh2" Sep 11 16:22:43.576883 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:43.576852 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63e70794-d8a3-4a93-af85-6060def3d813-workload-socket\") pod \"e2e-ipp-a-f9d303-openshift-default-5754c68888-x9vh2\" (UID: \"63e70794-d8a3-4a93-af85-6060def3d813\") " pod="openshift-ingress/e2e-ipp-a-f9d303-openshift-default-5754c68888-x9vh2" Sep 11 16:22:43.576999 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:43.576916 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63e70794-d8a3-4a93-af85-6060def3d813-istio-data\") pod \"e2e-ipp-a-f9d303-openshift-default-5754c68888-x9vh2\" (UID: \"63e70794-d8a3-4a93-af85-6060def3d813\") " pod="openshift-ingress/e2e-ipp-a-f9d303-openshift-default-5754c68888-x9vh2" Sep 11 16:22:43.576999 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:43.576945 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63e70794-d8a3-4a93-af85-6060def3d813-workload-certs\") pod \"e2e-ipp-a-f9d303-openshift-default-5754c68888-x9vh2\" (UID: \"63e70794-d8a3-4a93-af85-6060def3d813\") " pod="openshift-ingress/e2e-ipp-a-f9d303-openshift-default-5754c68888-x9vh2" Sep 11 16:22:43.577387 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:43.577361 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63e70794-d8a3-4a93-af85-6060def3d813-istiod-ca-cert\") pod \"e2e-ipp-a-f9d303-openshift-default-5754c68888-x9vh2\" (UID: \"63e70794-d8a3-4a93-af85-6060def3d813\") " pod="openshift-ingress/e2e-ipp-a-f9d303-openshift-default-5754c68888-x9vh2" Sep 11 16:22:43.579173 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:43.579149 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63e70794-d8a3-4a93-af85-6060def3d813-istio-envoy\") pod \"e2e-ipp-a-f9d303-openshift-default-5754c68888-x9vh2\" (UID: \"63e70794-d8a3-4a93-af85-6060def3d813\") " pod="openshift-ingress/e2e-ipp-a-f9d303-openshift-default-5754c68888-x9vh2" Sep 11 16:22:43.584395 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:43.584352 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63e70794-d8a3-4a93-af85-6060def3d813-istio-podinfo\") pod \"e2e-ipp-a-f9d303-openshift-default-5754c68888-x9vh2\" (UID: \"63e70794-d8a3-4a93-af85-6060def3d813\") " pod="openshift-ingress/e2e-ipp-a-f9d303-openshift-default-5754c68888-x9vh2" Sep 11 16:22:43.588428 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:43.588401 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63e70794-d8a3-4a93-af85-6060def3d813-istio-token\") pod \"e2e-ipp-a-f9d303-openshift-default-5754c68888-x9vh2\" (UID: \"63e70794-d8a3-4a93-af85-6060def3d813\") " pod="openshift-ingress/e2e-ipp-a-f9d303-openshift-default-5754c68888-x9vh2" Sep 11 16:22:43.589022 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:43.588966 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qngc7\" (UniqueName: \"kubernetes.io/projected/63e70794-d8a3-4a93-af85-6060def3d813-kube-api-access-qngc7\") pod \"e2e-ipp-a-f9d303-openshift-default-5754c68888-x9vh2\" (UID: \"63e70794-d8a3-4a93-af85-6060def3d813\") " pod="openshift-ingress/e2e-ipp-a-f9d303-openshift-default-5754c68888-x9vh2" Sep 11 16:22:43.712418 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:43.712295 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f9d303-openshift-default-5754c68888-x9vh2" Sep 11 16:22:43.996753 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:43.996724 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-f9d303-openshift-default-5754c68888-x9vh2"] Sep 11 16:22:43.999474 ip-10-0-136-96 kubenswrapper[2566]: W0911 16:22:43.999446 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63e70794_d8a3_4a93_af85_6060def3d813.slice/crio-7c0043115e94859070a69f49233184e3fc56b378da6c044efdfca757af57b36a WatchSource:0}: Error finding container 7c0043115e94859070a69f49233184e3fc56b378da6c044efdfca757af57b36a: Status 404 returned error can't find the container with id 7c0043115e94859070a69f49233184e3fc56b378da6c044efdfca757af57b36a Sep 11 16:22:44.001649 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:44.001610 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 16:22:44.001737 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:44.001679 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 16:22:44.001737 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:44.001707 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 16:22:44.324672 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:44.324633 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dx4xn"] Sep 11 16:22:44.663645 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:44.663555 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f9d303-openshift-default-5754c68888-x9vh2" event={"ID":"63e70794-d8a3-4a93-af85-6060def3d813","Type":"ContainerStarted","Data":"905814ea5eede8ccf23d4946ecbaa6e062983fd5f1d9293f588bedd601d3484e"} Sep 11 16:22:44.663645 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:44.663619 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f9d303-openshift-default-5754c68888-x9vh2" event={"ID":"63e70794-d8a3-4a93-af85-6060def3d813","Type":"ContainerStarted","Data":"7c0043115e94859070a69f49233184e3fc56b378da6c044efdfca757af57b36a"} Sep 11 16:22:44.686556 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:44.686503 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-f9d303-openshift-default-5754c68888-x9vh2" podStartSLOduration=1.68648636 podStartE2EDuration="1.68648636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:22:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:22:44.683745431 +0000 UTC m=+1671.898579336" watchObservedRunningTime="2026-09-11 16:22:44.68648636 +0000 UTC m=+1671.901320266" Sep 11 16:22:44.713061 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:44.713023 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-f9d303-openshift-default-5754c68888-x9vh2" Sep 11 16:22:45.717982 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:45.717937 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-f9d303-openshift-default-5754c68888-x9vh2" Sep 11 16:22:46.676375 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:46.676342 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-f9d303-openshift-default-5754c68888-x9vh2" Sep 11 16:22:46.677358 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:46.677334 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-f9d303-openshift-default-5754c68888-x9vh2" Sep 11 16:22:53.825980 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:22:53.825931 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dx4xn"] Sep 11 16:23:02.718197 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:23:02.718147 2566 patch_prober.go:28] interesting pod/e2e-ipp-a-f9d303-openshift-default-5754c68888-x9vh2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 11 16:23:02.718642 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:23:02.718227 2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-f9d303-openshift-default-5754c68888-x9vh2" podUID="63e70794-d8a3-4a93-af85-6060def3d813" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 11 16:23:03.794773 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:23:03.794738 2566 generic.go:356] "Generic (PLEG): container finished" podID="63e70794-d8a3-4a93-af85-6060def3d813" containerID="905814ea5eede8ccf23d4946ecbaa6e062983fd5f1d9293f588bedd601d3484e" exitCode=0 Sep 11 16:23:03.795342 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:23:03.794815 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f9d303-openshift-default-5754c68888-x9vh2" event={"ID":"63e70794-d8a3-4a93-af85-6060def3d813","Type":"ContainerDied","Data":"905814ea5eede8ccf23d4946ecbaa6e062983fd5f1d9293f588bedd601d3484e"} Sep 11 16:23:03.795413 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:23:03.795383 2566 scope.go:117] "RemoveContainer" containerID="905814ea5eede8ccf23d4946ecbaa6e062983fd5f1d9293f588bedd601d3484e" Sep 11 16:23:03.796482 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:23:03.796453 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 16:23:03.796558 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:23:03.796523 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 16:23:03.796558 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:23:03.796554 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 16:23:04.802666 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:23:04.802632 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f9d303-openshift-default-5754c68888-x9vh2" event={"ID":"63e70794-d8a3-4a93-af85-6060def3d813","Type":"ContainerStarted","Data":"b779674b3ecd0095a31783f1259ca299694ea34821011ec3370a4457044d99f2"} Sep 11 16:23:05.712940 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:23:05.712900 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-f9d303-openshift-default-5754c68888-x9vh2" Sep 11 16:23:05.713182 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:23:05.713160 2566 patch_prober.go:28] interesting pod/e2e-ipp-a-f9d303-openshift-default-5754c68888-x9vh2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Sep 11 16:23:05.713232 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:23:05.713213 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f9d303-openshift-default-5754c68888-x9vh2" podUID="63e70794-d8a3-4a93-af85-6060def3d813" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Sep 11 16:23:06.712963 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:23:06.712920 2566 patch_prober.go:28] interesting pod/e2e-ipp-a-f9d303-openshift-default-5754c68888-x9vh2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Sep 11 16:23:06.713481 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:23:06.713019 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f9d303-openshift-default-5754c68888-x9vh2" podUID="63e70794-d8a3-4a93-af85-6060def3d813" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Sep 11 16:23:07.713338 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:23:07.713297 2566 patch_prober.go:28] interesting pod/e2e-ipp-a-f9d303-openshift-default-5754c68888-x9vh2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Sep 11 16:23:07.713753 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:23:07.713366 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f9d303-openshift-default-5754c68888-x9vh2" podUID="63e70794-d8a3-4a93-af85-6060def3d813" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Sep 11 16:23:08.713178 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:23:08.713142 2566 patch_prober.go:28] interesting pod/e2e-ipp-a-f9d303-openshift-default-5754c68888-x9vh2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Sep 11 16:23:08.713373 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:23:08.713199 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f9d303-openshift-default-5754c68888-x9vh2" podUID="63e70794-d8a3-4a93-af85-6060def3d813" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Sep 11 16:23:09.713329 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:23:09.713286 2566 patch_prober.go:28] interesting pod/e2e-ipp-a-f9d303-openshift-default-5754c68888-x9vh2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Sep 11 16:23:09.713543 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:23:09.713358 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f9d303-openshift-default-5754c68888-x9vh2" podUID="63e70794-d8a3-4a93-af85-6060def3d813" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Sep 11 16:23:10.716901 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:23:10.716867 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-f9d303-openshift-default-5754c68888-x9vh2" Sep 11 16:23:10.717371 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:23:10.717351 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-f9d303-openshift-default-5754c68888-x9vh2" Sep 11 16:23:10.718068 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:23:10.718047 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-f9d303-openshift-default-5754c68888-x9vh2" Sep 11 16:23:17.802762 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:23:17.802678 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-c0aa4c-openshift-default-84f54cffb-jxqp5"] Sep 11 16:23:17.803270 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:23:17.803255 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="b78c6245-c285-479b-a463-42d55ce25b74" containerName="istio-proxy" Sep 11 16:23:17.805505 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:23:17.805490 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-c0aa4c-openshift-default-84f54cffb-jxqp5" Sep 11 16:23:17.807771 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:23:17.807752 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-c0aa4c-openshift-default-dockercfg-zfrh5\"" Sep 11 16:23:17.835450 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:23:17.835420 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-c0aa4c-openshift-default-84f54cffb-jxqp5"] Sep 11 16:23:17.897172 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:23:17.897133 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/340a9de4-01a7-4623-9e4b-0a6332bd8b35-istio-data\") pod \"e2e-ipp-b-c0aa4c-openshift-default-84f54cffb-jxqp5\" (UID: \"340a9de4-01a7-4623-9e4b-0a6332bd8b35\") " pod="openshift-ingress/e2e-ipp-b-c0aa4c-openshift-default-84f54cffb-jxqp5" Sep 11 16:23:17.897496 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:23:17.897472 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/340a9de4-01a7-4623-9e4b-0a6332bd8b35-credential-socket\") pod \"e2e-ipp-b-c0aa4c-openshift-default-84f54cffb-jxqp5\" (UID: \"340a9de4-01a7-4623-9e4b-0a6332bd8b35\") " pod="openshift-ingress/e2e-ipp-b-c0aa4c-openshift-default-84f54cffb-jxqp5" Sep 11 16:23:17.897689 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:23:17.897674 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/340a9de4-01a7-4623-9e4b-0a6332bd8b35-istio-token\") pod \"e2e-ipp-b-c0aa4c-openshift-default-84f54cffb-jxqp5\" (UID: \"340a9de4-01a7-4623-9e4b-0a6332bd8b35\") " pod="openshift-ingress/e2e-ipp-b-c0aa4c-openshift-default-84f54cffb-jxqp5" Sep 11 16:23:17.897868 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:23:17.897842 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/340a9de4-01a7-4623-9e4b-0a6332bd8b35-istio-envoy\") pod \"e2e-ipp-b-c0aa4c-openshift-default-84f54cffb-jxqp5\" (UID: \"340a9de4-01a7-4623-9e4b-0a6332bd8b35\") " pod="openshift-ingress/e2e-ipp-b-c0aa4c-openshift-default-84f54cffb-jxqp5" Sep 11 16:23:17.898020 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:23:17.897881 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/340a9de4-01a7-4623-9e4b-0a6332bd8b35-workload-certs\") pod \"e2e-ipp-b-c0aa4c-openshift-default-84f54cffb-jxqp5\" (UID: \"340a9de4-01a7-4623-9e4b-0a6332bd8b35\") " pod="openshift-ingress/e2e-ipp-b-c0aa4c-openshift-default-84f54cffb-jxqp5" Sep 11 16:23:17.898125 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:23:17.897962 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/340a9de4-01a7-4623-9e4b-0a6332bd8b35-istiod-ca-cert\") pod \"e2e-ipp-b-c0aa4c-openshift-default-84f54cffb-jxqp5\" (UID: \"340a9de4-01a7-4623-9e4b-0a6332bd8b35\") " pod="openshift-ingress/e2e-ipp-b-c0aa4c-openshift-default-84f54cffb-jxqp5" Sep 11 16:23:17.898374 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:23:17.898355 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jprtt\" (UniqueName: \"kubernetes.io/projected/340a9de4-01a7-4623-9e4b-0a6332bd8b35-kube-api-access-jprtt\") pod \"e2e-ipp-b-c0aa4c-openshift-default-84f54cffb-jxqp5\" (UID: \"340a9de4-01a7-4623-9e4b-0a6332bd8b35\") " pod="openshift-ingress/e2e-ipp-b-c0aa4c-openshift-default-84f54cffb-jxqp5" Sep 11 16:23:17.898560 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:23:17.898495 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/340a9de4-01a7-4623-9e4b-0a6332bd8b35-istio-podinfo\") pod \"e2e-ipp-b-c0aa4c-openshift-default-84f54cffb-jxqp5\" (UID: \"340a9de4-01a7-4623-9e4b-0a6332bd8b35\") " pod="openshift-ingress/e2e-ipp-b-c0aa4c-openshift-default-84f54cffb-jxqp5" Sep 11 16:23:17.898777 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:23:17.898746 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/340a9de4-01a7-4623-9e4b-0a6332bd8b35-workload-socket\") pod \"e2e-ipp-b-c0aa4c-openshift-default-84f54cffb-jxqp5\" (UID: \"340a9de4-01a7-4623-9e4b-0a6332bd8b35\") " pod="openshift-ingress/e2e-ipp-b-c0aa4c-openshift-default-84f54cffb-jxqp5" Sep 11 16:23:17.999816 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:23:17.999778 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/340a9de4-01a7-4623-9e4b-0a6332bd8b35-workload-socket\") pod \"e2e-ipp-b-c0aa4c-openshift-default-84f54cffb-jxqp5\" (UID: \"340a9de4-01a7-4623-9e4b-0a6332bd8b35\") " pod="openshift-ingress/e2e-ipp-b-c0aa4c-openshift-default-84f54cffb-jxqp5" Sep 11 16:23:17.999816 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:23:17.999820 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/340a9de4-01a7-4623-9e4b-0a6332bd8b35-istio-data\") pod \"e2e-ipp-b-c0aa4c-openshift-default-84f54cffb-jxqp5\" (UID: \"340a9de4-01a7-4623-9e4b-0a6332bd8b35\") " pod="openshift-ingress/e2e-ipp-b-c0aa4c-openshift-default-84f54cffb-jxqp5" Sep 11 16:23:18.000064 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:23:17.999870 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/340a9de4-01a7-4623-9e4b-0a6332bd8b35-credential-socket\") pod \"e2e-ipp-b-c0aa4c-openshift-default-84f54cffb-jxqp5\" (UID: \"340a9de4-01a7-4623-9e4b-0a6332bd8b35\") " pod="openshift-ingress/e2e-ipp-b-c0aa4c-openshift-default-84f54cffb-jxqp5" Sep 11 16:23:18.000196 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:23:18.000171 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/340a9de4-01a7-4623-9e4b-0a6332bd8b35-istio-token\") pod \"e2e-ipp-b-c0aa4c-openshift-default-84f54cffb-jxqp5\" (UID: \"340a9de4-01a7-4623-9e4b-0a6332bd8b35\") " pod="openshift-ingress/e2e-ipp-b-c0aa4c-openshift-default-84f54cffb-jxqp5" Sep 11 16:23:18.000279 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:23:18.000260 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/340a9de4-01a7-4623-9e4b-0a6332bd8b35-istio-envoy\") pod \"e2e-ipp-b-c0aa4c-openshift-default-84f54cffb-jxqp5\" (UID: \"340a9de4-01a7-4623-9e4b-0a6332bd8b35\") " pod="openshift-ingress/e2e-ipp-b-c0aa4c-openshift-default-84f54cffb-jxqp5" Sep 11 16:23:18.000315 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:23:18.000297 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/340a9de4-01a7-4623-9e4b-0a6332bd8b35-workload-certs\") pod \"e2e-ipp-b-c0aa4c-openshift-default-84f54cffb-jxqp5\" (UID: \"340a9de4-01a7-4623-9e4b-0a6332bd8b35\") " pod="openshift-ingress/e2e-ipp-b-c0aa4c-openshift-default-84f54cffb-jxqp5" Sep 11 16:23:18.000365 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:23:18.000307 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/340a9de4-01a7-4623-9e4b-0a6332bd8b35-credential-socket\") pod \"e2e-ipp-b-c0aa4c-openshift-default-84f54cffb-jxqp5\" (UID: \"340a9de4-01a7-4623-9e4b-0a6332bd8b35\") " pod="openshift-ingress/e2e-ipp-b-c0aa4c-openshift-default-84f54cffb-jxqp5" Sep 11 16:23:18.000365 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:23:18.000322 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/340a9de4-01a7-4623-9e4b-0a6332bd8b35-istiod-ca-cert\") pod \"e2e-ipp-b-c0aa4c-openshift-default-84f54cffb-jxqp5\" (UID: \"340a9de4-01a7-4623-9e4b-0a6332bd8b35\") " pod="openshift-ingress/e2e-ipp-b-c0aa4c-openshift-default-84f54cffb-jxqp5" Sep 11 16:23:18.000365 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:23:18.000339 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/340a9de4-01a7-4623-9e4b-0a6332bd8b35-workload-socket\") pod \"e2e-ipp-b-c0aa4c-openshift-default-84f54cffb-jxqp5\" (UID: \"340a9de4-01a7-4623-9e4b-0a6332bd8b35\") " pod="openshift-ingress/e2e-ipp-b-c0aa4c-openshift-default-84f54cffb-jxqp5" Sep 11 16:23:18.000524 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:23:18.000373 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/340a9de4-01a7-4623-9e4b-0a6332bd8b35-istio-data\") pod \"e2e-ipp-b-c0aa4c-openshift-default-84f54cffb-jxqp5\" (UID: \"340a9de4-01a7-4623-9e4b-0a6332bd8b35\") " pod="openshift-ingress/e2e-ipp-b-c0aa4c-openshift-default-84f54cffb-jxqp5" Sep 11 16:23:18.000524 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:23:18.000378 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jprtt\" (UniqueName: \"kubernetes.io/projected/340a9de4-01a7-4623-9e4b-0a6332bd8b35-kube-api-access-jprtt\") pod \"e2e-ipp-b-c0aa4c-openshift-default-84f54cffb-jxqp5\" (UID: \"340a9de4-01a7-4623-9e4b-0a6332bd8b35\") " pod="openshift-ingress/e2e-ipp-b-c0aa4c-openshift-default-84f54cffb-jxqp5" Sep 11 16:23:18.000524 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:23:18.000433 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/340a9de4-01a7-4623-9e4b-0a6332bd8b35-istio-podinfo\") pod \"e2e-ipp-b-c0aa4c-openshift-default-84f54cffb-jxqp5\" (UID: \"340a9de4-01a7-4623-9e4b-0a6332bd8b35\") " pod="openshift-ingress/e2e-ipp-b-c0aa4c-openshift-default-84f54cffb-jxqp5" Sep 11 16:23:18.000734 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:23:18.000713 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/340a9de4-01a7-4623-9e4b-0a6332bd8b35-workload-certs\") pod \"e2e-ipp-b-c0aa4c-openshift-default-84f54cffb-jxqp5\" (UID: \"340a9de4-01a7-4623-9e4b-0a6332bd8b35\") " pod="openshift-ingress/e2e-ipp-b-c0aa4c-openshift-default-84f54cffb-jxqp5" Sep 11 16:23:18.001148 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:23:18.001129 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/340a9de4-01a7-4623-9e4b-0a6332bd8b35-istiod-ca-cert\") pod \"e2e-ipp-b-c0aa4c-openshift-default-84f54cffb-jxqp5\" (UID: \"340a9de4-01a7-4623-9e4b-0a6332bd8b35\") " pod="openshift-ingress/e2e-ipp-b-c0aa4c-openshift-default-84f54cffb-jxqp5" Sep 11 16:23:18.002576 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:23:18.002550 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/340a9de4-01a7-4623-9e4b-0a6332bd8b35-istio-envoy\") pod \"e2e-ipp-b-c0aa4c-openshift-default-84f54cffb-jxqp5\" (UID: \"340a9de4-01a7-4623-9e4b-0a6332bd8b35\") " pod="openshift-ingress/e2e-ipp-b-c0aa4c-openshift-default-84f54cffb-jxqp5" Sep 11 16:23:18.002834 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:23:18.002812 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/340a9de4-01a7-4623-9e4b-0a6332bd8b35-istio-podinfo\") pod \"e2e-ipp-b-c0aa4c-openshift-default-84f54cffb-jxqp5\" (UID: \"340a9de4-01a7-4623-9e4b-0a6332bd8b35\") " pod="openshift-ingress/e2e-ipp-b-c0aa4c-openshift-default-84f54cffb-jxqp5" Sep 11 16:23:18.009315 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:23:18.009289 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/340a9de4-01a7-4623-9e4b-0a6332bd8b35-istio-token\") pod \"e2e-ipp-b-c0aa4c-openshift-default-84f54cffb-jxqp5\" (UID: \"340a9de4-01a7-4623-9e4b-0a6332bd8b35\") " pod="openshift-ingress/e2e-ipp-b-c0aa4c-openshift-default-84f54cffb-jxqp5" Sep 11 16:23:18.009462 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:23:18.009444 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jprtt\" (UniqueName: \"kubernetes.io/projected/340a9de4-01a7-4623-9e4b-0a6332bd8b35-kube-api-access-jprtt\") pod \"e2e-ipp-b-c0aa4c-openshift-default-84f54cffb-jxqp5\" (UID: \"340a9de4-01a7-4623-9e4b-0a6332bd8b35\") " pod="openshift-ingress/e2e-ipp-b-c0aa4c-openshift-default-84f54cffb-jxqp5" Sep 11 16:23:18.115992 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:23:18.115886 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-c0aa4c-openshift-default-84f54cffb-jxqp5" Sep 11 16:23:18.403640 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:23:18.403608 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-c0aa4c-openshift-default-84f54cffb-jxqp5"] Sep 11 16:23:18.404852 ip-10-0-136-96 kubenswrapper[2566]: W0911 16:23:18.404821 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod340a9de4_01a7_4623_9e4b_0a6332bd8b35.slice/crio-07145715bb1f7fc1ead38d76a319974ecd5e429b1e01998372c73a62551702f7 WatchSource:0}: Error finding container 07145715bb1f7fc1ead38d76a319974ecd5e429b1e01998372c73a62551702f7: Status 404 returned error can't find the container with id 07145715bb1f7fc1ead38d76a319974ecd5e429b1e01998372c73a62551702f7 Sep 11 16:23:18.407499 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:23:18.407463 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 16:23:18.407600 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:23:18.407535 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 16:23:18.407600 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:23:18.407563 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 16:23:18.893286 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:23:18.893249 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c0aa4c-openshift-default-84f54cffb-jxqp5" event={"ID":"340a9de4-01a7-4623-9e4b-0a6332bd8b35","Type":"ContainerStarted","Data":"2b952693a70fd3c6ff4611c2ad2b0896c06cf2eb6b9499e39be3969755733ca7"} Sep 11 16:23:18.893683 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:23:18.893335 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c0aa4c-openshift-default-84f54cffb-jxqp5" event={"ID":"340a9de4-01a7-4623-9e4b-0a6332bd8b35","Type":"ContainerStarted","Data":"07145715bb1f7fc1ead38d76a319974ecd5e429b1e01998372c73a62551702f7"} Sep 11 16:23:18.919448 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:23:18.919395 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-c0aa4c-openshift-default-84f54cffb-jxqp5" podStartSLOduration=1.9193789030000001 podStartE2EDuration="1.919378903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:23:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:23:18.918243006 +0000 UTC m=+1706.133076912" watchObservedRunningTime="2026-09-11 16:23:18.919378903 +0000 UTC m=+1706.134212808" Sep 11 16:23:19.116629 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:23:19.116580 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-c0aa4c-openshift-default-84f54cffb-jxqp5" Sep 11 16:23:19.121899 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:23:19.121871 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-c0aa4c-openshift-default-84f54cffb-jxqp5" Sep 11 16:23:19.899137 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:23:19.899106 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-c0aa4c-openshift-default-84f54cffb-jxqp5" Sep 11 16:23:19.900080 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:23:19.900063 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-c0aa4c-openshift-default-84f54cffb-jxqp5" Sep 11 16:23:30.757083 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:23:30.757049 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dx4xn"] Sep 11 16:23:42.037793 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:23:42.037762 2566 generic.go:356] "Generic (PLEG): container finished" podID="340a9de4-01a7-4623-9e4b-0a6332bd8b35" containerID="2b952693a70fd3c6ff4611c2ad2b0896c06cf2eb6b9499e39be3969755733ca7" exitCode=0 Sep 11 16:23:42.038224 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:23:42.037840 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c0aa4c-openshift-default-84f54cffb-jxqp5" event={"ID":"340a9de4-01a7-4623-9e4b-0a6332bd8b35","Type":"ContainerDied","Data":"2b952693a70fd3c6ff4611c2ad2b0896c06cf2eb6b9499e39be3969755733ca7"} Sep 11 16:23:42.038289 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:23:42.038274 2566 scope.go:117] "RemoveContainer" containerID="2b952693a70fd3c6ff4611c2ad2b0896c06cf2eb6b9499e39be3969755733ca7" Sep 11 16:23:42.039458 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:23:42.039433 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 16:23:42.039513 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:23:42.039500 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 16:23:42.039551 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:23:42.039533 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 16:23:43.066849 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:23:43.066809 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c0aa4c-openshift-default-84f54cffb-jxqp5" event={"ID":"340a9de4-01a7-4623-9e4b-0a6332bd8b35","Type":"ContainerStarted","Data":"1e538c11413d344902f7730696d6832bf198aa291e2da57a0a33c9e678e1e01e"} Sep 11 16:23:43.116251 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:23:43.116214 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-c0aa4c-openshift-default-84f54cffb-jxqp5" Sep 11 16:23:43.116444 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:23:43.116410 2566 patch_prober.go:28] interesting pod/e2e-ipp-b-c0aa4c-openshift-default-84f54cffb-jxqp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Sep 11 16:23:43.116521 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:23:43.116480 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c0aa4c-openshift-default-84f54cffb-jxqp5" podUID="340a9de4-01a7-4623-9e4b-0a6332bd8b35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Sep 11 16:23:44.116721 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:23:44.116678 2566 patch_prober.go:28] interesting pod/e2e-ipp-b-c0aa4c-openshift-default-84f54cffb-jxqp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Sep 11 16:23:44.117235 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:23:44.116758 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c0aa4c-openshift-default-84f54cffb-jxqp5" podUID="340a9de4-01a7-4623-9e4b-0a6332bd8b35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Sep 11 16:23:45.116497 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:23:45.116465 2566 patch_prober.go:28] interesting pod/e2e-ipp-b-c0aa4c-openshift-default-84f54cffb-jxqp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Sep 11 16:23:45.116662 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:23:45.116529 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c0aa4c-openshift-default-84f54cffb-jxqp5" podUID="340a9de4-01a7-4623-9e4b-0a6332bd8b35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Sep 11 16:23:46.117237 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:23:46.117200 2566 patch_prober.go:28] interesting pod/e2e-ipp-b-c0aa4c-openshift-default-84f54cffb-jxqp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Sep 11 16:23:46.117625 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:23:46.117269 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c0aa4c-openshift-default-84f54cffb-jxqp5" podUID="340a9de4-01a7-4623-9e4b-0a6332bd8b35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Sep 11 16:23:47.116897 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:23:47.116858 2566 patch_prober.go:28] interesting pod/e2e-ipp-b-c0aa4c-openshift-default-84f54cffb-jxqp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Sep 11 16:23:47.117101 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:23:47.116915 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c0aa4c-openshift-default-84f54cffb-jxqp5" podUID="340a9de4-01a7-4623-9e4b-0a6332bd8b35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Sep 11 16:23:48.116843 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:23:48.116807 2566 patch_prober.go:28] interesting pod/e2e-ipp-b-c0aa4c-openshift-default-84f54cffb-jxqp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Sep 11 16:23:48.117270 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:23:48.116864 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c0aa4c-openshift-default-84f54cffb-jxqp5" podUID="340a9de4-01a7-4623-9e4b-0a6332bd8b35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Sep 11 16:23:49.120314 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:23:49.120284 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-c0aa4c-openshift-default-84f54cffb-jxqp5" Sep 11 16:23:49.120887 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:23:49.120640 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-c0aa4c-openshift-default-84f54cffb-jxqp5" Sep 11 16:23:49.121350 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:23:49.121331 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-c0aa4c-openshift-default-84f54cffb-jxqp5" Sep 11 16:23:53.760083 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:23:53.760049 2566 scope.go:117] "RemoveContainer" containerID="805db22686b8379ada2eafbc372d32ec0f48ce68f9b30c1bc2bdcd7b9f05e41a" Sep 11 16:24:23.160377 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:24:23.160330 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dx4xn"] Sep 11 16:24:36.166032 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:24:36.165992 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dx4xn"] Sep 11 16:24:43.531807 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:24:43.531774 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-214daa-openshift-default-bc446bc94-bmxq6"] Sep 11 16:24:43.535595 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:24:43.535576 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-214daa-openshift-default-bc446bc94-bmxq6" Sep 11 16:24:43.540854 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:24:43.540828 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-214daa-openshift-default-dockercfg-v4l5b\"" Sep 11 16:24:43.591611 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:24:43.591570 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-214daa-openshift-default-bc446bc94-bmxq6"] Sep 11 16:24:43.670367 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:24:43.670326 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0-istio-data\") pod \"e2e-ipp-cleanup-214daa-openshift-default-bc446bc94-bmxq6\" (UID: \"1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0\") " pod="openshift-ingress/e2e-ipp-cleanup-214daa-openshift-default-bc446bc94-bmxq6" Sep 11 16:24:43.670561 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:24:43.670378 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0-istio-token\") pod \"e2e-ipp-cleanup-214daa-openshift-default-bc446bc94-bmxq6\" (UID: \"1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0\") " pod="openshift-ingress/e2e-ipp-cleanup-214daa-openshift-default-bc446bc94-bmxq6" Sep 11 16:24:43.670561 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:24:43.670459 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0-credential-socket\") pod \"e2e-ipp-cleanup-214daa-openshift-default-bc446bc94-bmxq6\" (UID: \"1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0\") " pod="openshift-ingress/e2e-ipp-cleanup-214daa-openshift-default-bc446bc94-bmxq6" Sep 11 16:24:43.670561 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:24:43.670518 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0-istio-podinfo\") pod \"e2e-ipp-cleanup-214daa-openshift-default-bc446bc94-bmxq6\" (UID: \"1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0\") " pod="openshift-ingress/e2e-ipp-cleanup-214daa-openshift-default-bc446bc94-bmxq6" Sep 11 16:24:43.670561 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:24:43.670556 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0-istiod-ca-cert\") pod \"e2e-ipp-cleanup-214daa-openshift-default-bc446bc94-bmxq6\" (UID: \"1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0\") " pod="openshift-ingress/e2e-ipp-cleanup-214daa-openshift-default-bc446bc94-bmxq6" Sep 11 16:24:43.670763 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:24:43.670599 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0-workload-socket\") pod \"e2e-ipp-cleanup-214daa-openshift-default-bc446bc94-bmxq6\" (UID: \"1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0\") " pod="openshift-ingress/e2e-ipp-cleanup-214daa-openshift-default-bc446bc94-bmxq6" Sep 11 16:24:43.670763 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:24:43.670665 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0-istio-envoy\") pod \"e2e-ipp-cleanup-214daa-openshift-default-bc446bc94-bmxq6\" (UID: \"1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0\") " pod="openshift-ingress/e2e-ipp-cleanup-214daa-openshift-default-bc446bc94-bmxq6" Sep 11 16:24:43.670845 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:24:43.670757 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0-workload-certs\") pod \"e2e-ipp-cleanup-214daa-openshift-default-bc446bc94-bmxq6\" (UID: \"1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0\") " pod="openshift-ingress/e2e-ipp-cleanup-214daa-openshift-default-bc446bc94-bmxq6" Sep 11 16:24:43.670845 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:24:43.670787 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2zwj\" (UniqueName: \"kubernetes.io/projected/1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0-kube-api-access-r2zwj\") pod \"e2e-ipp-cleanup-214daa-openshift-default-bc446bc94-bmxq6\" (UID: \"1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0\") " pod="openshift-ingress/e2e-ipp-cleanup-214daa-openshift-default-bc446bc94-bmxq6" Sep 11 16:24:43.771436 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:24:43.771395 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0-istio-token\") pod \"e2e-ipp-cleanup-214daa-openshift-default-bc446bc94-bmxq6\" (UID: \"1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0\") " pod="openshift-ingress/e2e-ipp-cleanup-214daa-openshift-default-bc446bc94-bmxq6" Sep 11 16:24:43.771627 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:24:43.771449 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0-credential-socket\") pod \"e2e-ipp-cleanup-214daa-openshift-default-bc446bc94-bmxq6\" (UID: \"1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0\") " pod="openshift-ingress/e2e-ipp-cleanup-214daa-openshift-default-bc446bc94-bmxq6" Sep 11 16:24:43.771627 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:24:43.771483 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0-istio-podinfo\") pod \"e2e-ipp-cleanup-214daa-openshift-default-bc446bc94-bmxq6\" (UID: \"1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0\") " pod="openshift-ingress/e2e-ipp-cleanup-214daa-openshift-default-bc446bc94-bmxq6" Sep 11 16:24:43.771627 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:24:43.771513 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0-istiod-ca-cert\") pod \"e2e-ipp-cleanup-214daa-openshift-default-bc446bc94-bmxq6\" (UID: \"1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0\") " pod="openshift-ingress/e2e-ipp-cleanup-214daa-openshift-default-bc446bc94-bmxq6" Sep 11 16:24:43.771627 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:24:43.771550 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0-workload-socket\") pod \"e2e-ipp-cleanup-214daa-openshift-default-bc446bc94-bmxq6\" (UID: \"1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0\") " pod="openshift-ingress/e2e-ipp-cleanup-214daa-openshift-default-bc446bc94-bmxq6" Sep 11 16:24:43.771627 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:24:43.771610 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0-istio-envoy\") pod \"e2e-ipp-cleanup-214daa-openshift-default-bc446bc94-bmxq6\" (UID: \"1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0\") " pod="openshift-ingress/e2e-ipp-cleanup-214daa-openshift-default-bc446bc94-bmxq6" Sep 11 16:24:43.771897 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:24:43.771691 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0-workload-certs\") pod \"e2e-ipp-cleanup-214daa-openshift-default-bc446bc94-bmxq6\" (UID: \"1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0\") " pod="openshift-ingress/e2e-ipp-cleanup-214daa-openshift-default-bc446bc94-bmxq6" Sep 11 16:24:43.771897 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:24:43.771727 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2zwj\" (UniqueName: \"kubernetes.io/projected/1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0-kube-api-access-r2zwj\") pod \"e2e-ipp-cleanup-214daa-openshift-default-bc446bc94-bmxq6\" (UID: \"1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0\") " pod="openshift-ingress/e2e-ipp-cleanup-214daa-openshift-default-bc446bc94-bmxq6" Sep 11 16:24:43.771897 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:24:43.771784 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0-istio-data\") pod \"e2e-ipp-cleanup-214daa-openshift-default-bc446bc94-bmxq6\" (UID: \"1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0\") " pod="openshift-ingress/e2e-ipp-cleanup-214daa-openshift-default-bc446bc94-bmxq6" Sep 11 16:24:43.772109 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:24:43.771954 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0-credential-socket\") pod \"e2e-ipp-cleanup-214daa-openshift-default-bc446bc94-bmxq6\" (UID: \"1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0\") " pod="openshift-ingress/e2e-ipp-cleanup-214daa-openshift-default-bc446bc94-bmxq6" Sep 11 16:24:43.772109 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:24:43.772020 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0-workload-socket\") pod \"e2e-ipp-cleanup-214daa-openshift-default-bc446bc94-bmxq6\" (UID: \"1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0\") " pod="openshift-ingress/e2e-ipp-cleanup-214daa-openshift-default-bc446bc94-bmxq6" Sep 11 16:24:43.772109 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:24:43.772084 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0-workload-certs\") pod \"e2e-ipp-cleanup-214daa-openshift-default-bc446bc94-bmxq6\" (UID: \"1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0\") " pod="openshift-ingress/e2e-ipp-cleanup-214daa-openshift-default-bc446bc94-bmxq6" Sep 11 16:24:43.772276 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:24:43.772207 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0-istio-data\") pod \"e2e-ipp-cleanup-214daa-openshift-default-bc446bc94-bmxq6\" (UID: \"1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0\") " pod="openshift-ingress/e2e-ipp-cleanup-214daa-openshift-default-bc446bc94-bmxq6" Sep 11 16:24:43.772440 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:24:43.772419 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0-istiod-ca-cert\") pod \"e2e-ipp-cleanup-214daa-openshift-default-bc446bc94-bmxq6\" (UID: \"1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0\") " pod="openshift-ingress/e2e-ipp-cleanup-214daa-openshift-default-bc446bc94-bmxq6" Sep 11 16:24:43.773962 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:24:43.773943 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0-istio-envoy\") pod \"e2e-ipp-cleanup-214daa-openshift-default-bc446bc94-bmxq6\" (UID: \"1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0\") " pod="openshift-ingress/e2e-ipp-cleanup-214daa-openshift-default-bc446bc94-bmxq6" Sep 11 16:24:43.774405 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:24:43.774379 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0-istio-podinfo\") pod \"e2e-ipp-cleanup-214daa-openshift-default-bc446bc94-bmxq6\" (UID: \"1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0\") " pod="openshift-ingress/e2e-ipp-cleanup-214daa-openshift-default-bc446bc94-bmxq6" Sep 11 16:24:43.794016 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:24:43.793945 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0-istio-token\") pod \"e2e-ipp-cleanup-214daa-openshift-default-bc446bc94-bmxq6\" (UID: \"1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0\") " pod="openshift-ingress/e2e-ipp-cleanup-214daa-openshift-default-bc446bc94-bmxq6" Sep 11 16:24:43.796866 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:24:43.796844 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2zwj\" (UniqueName: \"kubernetes.io/projected/1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0-kube-api-access-r2zwj\") pod \"e2e-ipp-cleanup-214daa-openshift-default-bc446bc94-bmxq6\" (UID: \"1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0\") " pod="openshift-ingress/e2e-ipp-cleanup-214daa-openshift-default-bc446bc94-bmxq6" Sep 11 16:24:43.846407 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:24:43.846373 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-214daa-openshift-default-bc446bc94-bmxq6" Sep 11 16:24:44.139215 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:24:44.139186 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-214daa-openshift-default-bc446bc94-bmxq6"] Sep 11 16:24:44.140840 ip-10-0-136-96 kubenswrapper[2566]: W0911 16:24:44.140816 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ee2db09_9134_4ed6_a4ab_f72aeef8b7e0.slice/crio-ca56ceb7d97867357f7ec6ce71ac0996b56ceadbbbddea0e548b09a0b1c9f9f5 WatchSource:0}: Error finding container ca56ceb7d97867357f7ec6ce71ac0996b56ceadbbbddea0e548b09a0b1c9f9f5: Status 404 returned error can't find the container with id ca56ceb7d97867357f7ec6ce71ac0996b56ceadbbbddea0e548b09a0b1c9f9f5 Sep 11 16:24:44.143206 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:24:44.143174 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 16:24:44.143268 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:24:44.143238 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 16:24:44.143318 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:24:44.143269 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 16:24:44.443300 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:24:44.443210 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-214daa-openshift-default-bc446bc94-bmxq6" event={"ID":"1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0","Type":"ContainerStarted","Data":"bc3f31d7e33333a5ea52652cd32572ffce2ccacbc69631b6474bc43d0c772bc6"} Sep 11 16:24:44.443300 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:24:44.443263 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-214daa-openshift-default-bc446bc94-bmxq6" event={"ID":"1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0","Type":"ContainerStarted","Data":"ca56ceb7d97867357f7ec6ce71ac0996b56ceadbbbddea0e548b09a0b1c9f9f5"} Sep 11 16:24:44.471873 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:24:44.471821 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-214daa-openshift-default-bc446bc94-bmxq6" podStartSLOduration=1.47180362 podStartE2EDuration="1.47180362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:24:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:24:44.467566289 +0000 UTC m=+1791.682400194" watchObservedRunningTime="2026-09-11 16:24:44.47180362 +0000 UTC m=+1791.686637580" Sep 11 16:24:44.847159 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:24:44.847124 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-214daa-openshift-default-bc446bc94-bmxq6" Sep 11 16:24:45.854534 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:24:45.854458 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-214daa-openshift-default-bc446bc94-bmxq6" Sep 11 16:24:46.456465 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:24:46.456430 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-214daa-openshift-default-bc446bc94-bmxq6" Sep 11 16:24:46.457277 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:24:46.457261 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-214daa-openshift-default-bc446bc94-bmxq6" Sep 11 16:24:53.472787 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:24:53.472760 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-xjvl5_7542ad77-c817-4370-91de-3b08db264753/console-operator/2.log" Sep 11 16:24:53.477389 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:24:53.477363 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-xjvl5_7542ad77-c817-4370-91de-3b08db264753/console-operator/2.log" Sep 11 16:24:53.480801 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:24:53.480776 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tgq5c_981d7d1e-f81f-48fc-8962-7831bc5c2b27/ovn-acl-logging/0.log" Sep 11 16:24:53.484797 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:24:53.484780 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tgq5c_981d7d1e-f81f-48fc-8962-7831bc5c2b27/ovn-acl-logging/0.log" Sep 11 16:24:56.255216 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:24:56.255180 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dx4xn"] Sep 11 16:25:06.588936 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:25:06.588902 2566 generic.go:356] "Generic (PLEG): container finished" podID="1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0" containerID="bc3f31d7e33333a5ea52652cd32572ffce2ccacbc69631b6474bc43d0c772bc6" exitCode=0 Sep 11 16:25:06.589473 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:25:06.588965 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-214daa-openshift-default-bc446bc94-bmxq6" event={"ID":"1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0","Type":"ContainerDied","Data":"bc3f31d7e33333a5ea52652cd32572ffce2ccacbc69631b6474bc43d0c772bc6"} Sep 11 16:25:06.589556 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:25:06.589534 2566 scope.go:117] "RemoveContainer" containerID="bc3f31d7e33333a5ea52652cd32572ffce2ccacbc69631b6474bc43d0c772bc6" Sep 11 16:25:06.590759 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:25:06.590729 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 16:25:06.590845 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:25:06.590801 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 16:25:06.590845 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:25:06.590835 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 11 16:25:07.597041 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:25:07.597007 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-214daa-openshift-default-bc446bc94-bmxq6" event={"ID":"1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0","Type":"ContainerStarted","Data":"6572ac967b9d10f8002883fda7896bf320a74b949b04d271d3d08969453a9220"} Sep 11 16:25:07.847671 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:25:07.847091 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-214daa-openshift-default-bc446bc94-bmxq6" Sep 11 16:25:07.848064 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:25:07.848027 2566 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-214daa-openshift-default-bc446bc94-bmxq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Sep 11 16:25:07.848252 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:25:07.848229 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-214daa-openshift-default-bc446bc94-bmxq6" podUID="1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Sep 11 16:25:08.847565 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:25:08.847523 2566 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-214daa-openshift-default-bc446bc94-bmxq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Sep 11 16:25:08.848186 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:25:08.848152 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-214daa-openshift-default-bc446bc94-bmxq6" podUID="1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Sep 11 16:25:09.847442 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:25:09.847399 2566 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-214daa-openshift-default-bc446bc94-bmxq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Sep 11 16:25:09.847762 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:25:09.847732 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-214daa-openshift-default-bc446bc94-bmxq6" podUID="1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Sep 11 16:25:10.847888 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:25:10.847845 2566 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-214daa-openshift-default-bc446bc94-bmxq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Sep 11 16:25:10.848431 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:25:10.847916 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-214daa-openshift-default-bc446bc94-bmxq6" podUID="1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Sep 11 16:25:11.847666 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:25:11.847624 2566 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-214daa-openshift-default-bc446bc94-bmxq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Sep 11 16:25:11.847959 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:25:11.847928 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-214daa-openshift-default-bc446bc94-bmxq6" podUID="1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Sep 11 16:25:12.847568 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:25:12.847526 2566 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-214daa-openshift-default-bc446bc94-bmxq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Sep 11 16:25:12.847899 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:25:12.847866 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-214daa-openshift-default-bc446bc94-bmxq6" podUID="1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Sep 11 16:25:13.851180 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:25:13.851146 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-214daa-openshift-default-bc446bc94-bmxq6" Sep 11 16:25:13.851558 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:25:13.851441 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-214daa-openshift-default-bc446bc94-bmxq6" Sep 11 16:25:13.852289 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:25:13.852271 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-214daa-openshift-default-bc446bc94-bmxq6" Sep 11 16:25:17.528322 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:25:17.528285 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-214daa-h68mh"] Sep 11 16:25:17.532089 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:25:17.532071 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-214daa-h68mh" Sep 11 16:25:17.537961 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:25:17.537930 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 16:25:17.558884 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:25:17.558852 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qbzdr\"" Sep 11 16:25:17.578785 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:25:17.578755 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-214daa-h68mh"] Sep 11 16:25:17.622075 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:25:17.622044 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98ec968c-c6f4-4d59-91b1-82f29dccb4f8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-214daa-h68mh\" (UID: \"98ec968c-c6f4-4d59-91b1-82f29dccb4f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-214daa-h68mh" Sep 11 16:25:17.722775 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:25:17.722733 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98ec968c-c6f4-4d59-91b1-82f29dccb4f8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-214daa-h68mh\" (UID: \"98ec968c-c6f4-4d59-91b1-82f29dccb4f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-214daa-h68mh" Sep 11 16:25:17.723437 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:25:17.723413 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98ec968c-c6f4-4d59-91b1-82f29dccb4f8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-214daa-h68mh\" (UID: \"98ec968c-c6f4-4d59-91b1-82f29dccb4f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-214daa-h68mh" Sep 11 16:25:17.842535 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:25:17.842455 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-214daa-h68mh" Sep 11 16:25:18.141429 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:25:18.141403 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-214daa-h68mh"] Sep 11 16:25:18.143859 ip-10-0-136-96 kubenswrapper[2566]: W0911 16:25:18.143820 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98ec968c_c6f4_4d59_91b1_82f29dccb4f8.slice/crio-274aef9c5a7622be14c56ed18c53bf7155e7e6634a372c454e7c4abf3733376d WatchSource:0}: Error finding container 274aef9c5a7622be14c56ed18c53bf7155e7e6634a372c454e7c4abf3733376d: Status 404 returned error can't find the container with id 274aef9c5a7622be14c56ed18c53bf7155e7e6634a372c454e7c4abf3733376d Sep 11 16:25:18.664982 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:25:18.664944 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-214daa-h68mh" event={"ID":"98ec968c-c6f4-4d59-91b1-82f29dccb4f8","Type":"ContainerStarted","Data":"2aa31cd0227c47ac1f41868a97f7c4c021d74cb9ab292fa54555e5449510e53c"} Sep 11 16:25:18.665379 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:25:18.665012 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-214daa-h68mh" event={"ID":"98ec968c-c6f4-4d59-91b1-82f29dccb4f8","Type":"ContainerStarted","Data":"274aef9c5a7622be14c56ed18c53bf7155e7e6634a372c454e7c4abf3733376d"} Sep 11 16:25:18.685367 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:25:18.685314 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-214daa-h68mh" podStartSLOduration=1.685296517 podStartE2EDuration="1.685296517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:25:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:25:18.684149376 +0000 UTC m=+1825.898983282" watchObservedRunningTime="2026-09-11 16:25:18.685296517 +0000 UTC m=+1825.900130421" Sep 11 16:25:19.673014 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:25:19.672954 2566 generic.go:356] "Generic (PLEG): container finished" podID="98ec968c-c6f4-4d59-91b1-82f29dccb4f8" containerID="2aa31cd0227c47ac1f41868a97f7c4c021d74cb9ab292fa54555e5449510e53c" exitCode=0 Sep 11 16:25:19.673450 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:25:19.673053 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-214daa-h68mh" event={"ID":"98ec968c-c6f4-4d59-91b1-82f29dccb4f8","Type":"ContainerDied","Data":"2aa31cd0227c47ac1f41868a97f7c4c021d74cb9ab292fa54555e5449510e53c"} Sep 11 16:25:20.812782 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:25:20.812759 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-214daa-h68mh" Sep 11 16:25:20.956588 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:25:20.956498 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98ec968c-c6f4-4d59-91b1-82f29dccb4f8-maas-api-service-ca\") pod \"98ec968c-c6f4-4d59-91b1-82f29dccb4f8\" (UID: \"98ec968c-c6f4-4d59-91b1-82f29dccb4f8\") " Sep 11 16:25:20.956850 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:25:20.956827 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98ec968c-c6f4-4d59-91b1-82f29dccb4f8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "98ec968c-c6f4-4d59-91b1-82f29dccb4f8" (UID: "98ec968c-c6f4-4d59-91b1-82f29dccb4f8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 16:25:21.057489 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:25:21.057454 2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98ec968c-c6f4-4d59-91b1-82f29dccb4f8-maas-api-service-ca\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:25:21.686746 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:25:21.686708 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-214daa-h68mh" event={"ID":"98ec968c-c6f4-4d59-91b1-82f29dccb4f8","Type":"ContainerDied","Data":"274aef9c5a7622be14c56ed18c53bf7155e7e6634a372c454e7c4abf3733376d"} Sep 11 16:25:21.686952 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:25:21.686769 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="274aef9c5a7622be14c56ed18c53bf7155e7e6634a372c454e7c4abf3733376d" Sep 11 16:25:21.686952 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:25:21.686720 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-214daa-h68mh" Sep 11 16:25:29.161563 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:25:29.161525 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-214daa-h68mh"] Sep 11 16:25:29.166151 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:25:29.166123 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-214daa-h68mh"] Sep 11 16:25:29.241584 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:25:29.241549 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="98ec968c-c6f4-4d59-91b1-82f29dccb4f8" path="/var/lib/kubelet/pods/98ec968c-c6f4-4d59-91b1-82f29dccb4f8/volumes" Sep 11 16:25:33.727868 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:25:33.727836 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dx4xn"] Sep 11 16:25:35.767083 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:25:35.767042 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-214daa-openshift-default-bc446bc94-bmxq6"] Sep 11 16:25:35.767526 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:25:35.767296 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-214daa-openshift-default-bc446bc94-bmxq6" podUID="1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0" containerName="istio-proxy" containerID="cri-o://6572ac967b9d10f8002883fda7896bf320a74b949b04d271d3d08969453a9220" gracePeriod=30 Sep 11 16:25:39.543248 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:25:39.543214 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f9d303-dsw5z"] Sep 11 16:25:39.543650 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:25:39.543637 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98ec968c-c6f4-4d59-91b1-82f29dccb4f8" containerName="revoke-keys" Sep 11 16:25:39.543695 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:25:39.543651 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="98ec968c-c6f4-4d59-91b1-82f29dccb4f8" containerName="revoke-keys" Sep 11 16:25:39.543741 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:25:39.543732 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="98ec968c-c6f4-4d59-91b1-82f29dccb4f8" containerName="revoke-keys" Sep 11 16:25:39.547229 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:25:39.547213 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f9d303-dsw5z" Sep 11 16:25:39.549950 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:25:39.549929 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 16:25:39.550073 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:25:39.549966 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qbzdr\"" Sep 11 16:25:39.558479 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:25:39.558460 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f9d303-dsw5z"] Sep 11 16:25:39.651349 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:25:39.651300 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a25bef5-7a3e-4271-bc59-95ba530e19ad-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f9d303-dsw5z\" (UID: \"5a25bef5-7a3e-4271-bc59-95ba530e19ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f9d303-dsw5z" Sep 11 16:25:39.752104 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:25:39.752072 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a25bef5-7a3e-4271-bc59-95ba530e19ad-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f9d303-dsw5z\" (UID: \"5a25bef5-7a3e-4271-bc59-95ba530e19ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f9d303-dsw5z" Sep 11 16:25:39.752676 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:25:39.752655 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a25bef5-7a3e-4271-bc59-95ba530e19ad-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f9d303-dsw5z\" (UID: \"5a25bef5-7a3e-4271-bc59-95ba530e19ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f9d303-dsw5z" Sep 11 16:25:39.858935 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:25:39.858831 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f9d303-dsw5z" Sep 11 16:25:40.132896 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:25:40.132863 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f9d303-dsw5z"] Sep 11 16:25:40.134459 ip-10-0-136-96 kubenswrapper[2566]: W0911 16:25:40.134432 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a25bef5_7a3e_4271_bc59_95ba530e19ad.slice/crio-c4820f4e156fd58ee7dc7b3c1665bfb6add54cea98edf442fc3d5c2a2dc2fe21 WatchSource:0}: Error finding container c4820f4e156fd58ee7dc7b3c1665bfb6add54cea98edf442fc3d5c2a2dc2fe21: Status 404 returned error can't find the container with id c4820f4e156fd58ee7dc7b3c1665bfb6add54cea98edf442fc3d5c2a2dc2fe21 Sep 11 16:25:40.806716 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:25:40.806680 2566 generic.go:356] "Generic (PLEG): container finished" podID="5a25bef5-7a3e-4271-bc59-95ba530e19ad" containerID="2331522b9b2006a3412ee128959d745301dcdcee216008e944b19a25ba8054a6" exitCode=0 Sep 11 16:25:40.807107 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:25:40.806762 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f9d303-dsw5z" event={"ID":"5a25bef5-7a3e-4271-bc59-95ba530e19ad","Type":"ContainerDied","Data":"2331522b9b2006a3412ee128959d745301dcdcee216008e944b19a25ba8054a6"} Sep 11 16:25:40.807107 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:25:40.806820 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f9d303-dsw5z" event={"ID":"5a25bef5-7a3e-4271-bc59-95ba530e19ad","Type":"ContainerStarted","Data":"c4820f4e156fd58ee7dc7b3c1665bfb6add54cea98edf442fc3d5c2a2dc2fe21"} Sep 11 16:25:41.017924 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:25:41.017904 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-214daa-openshift-default-bc446bc94-bmxq6" Sep 11 16:25:41.166111 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:25:41.166074 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r2zwj\" (UniqueName: \"kubernetes.io/projected/1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0-kube-api-access-r2zwj\") pod \"1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0\" (UID: \"1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0\") " Sep 11 16:25:41.166111 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:25:41.166115 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0-workload-socket\") pod \"1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0\" (UID: \"1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0\") " Sep 11 16:25:41.166374 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:25:41.166133 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0-istio-token\") pod \"1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0\" (UID: \"1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0\") " Sep 11 16:25:41.166374 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:25:41.166172 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0-istio-podinfo\") pod \"1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0\" (UID: \"1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0\") " Sep 11 16:25:41.166374 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:25:41.166200 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0-istio-data\") pod \"1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0\" (UID: \"1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0\") " Sep 11 16:25:41.166374 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:25:41.166271 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0-credential-socket\") pod \"1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0\" (UID: \"1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0\") " Sep 11 16:25:41.166374 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:25:41.166292 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0-workload-certs\") pod \"1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0\" (UID: \"1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0\") " Sep 11 16:25:41.166374 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:25:41.166353 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0-istio-envoy\") pod \"1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0\" (UID: \"1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0\") " Sep 11 16:25:41.166690 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:25:41.166393 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0-istiod-ca-cert\") pod \"1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0\" (UID: \"1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0\") " Sep 11 16:25:41.166690 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:25:41.166422 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0" (UID: "1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 16:25:41.166690 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:25:41.166653 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0-istio-data" (OuterVolumeSpecName: "istio-data") pod "1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0" (UID: "1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 16:25:41.166690 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:25:41.166663 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0" (UID: "1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 16:25:41.166899 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:25:41.166830 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0" (UID: "1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 16:25:41.166899 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:25:41.166829 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0" (UID: "1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 16:25:41.167042 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:25:41.167000 2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0-istiod-ca-cert\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:25:41.167042 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:25:41.167020 2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0-workload-socket\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:25:41.167042 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:25:41.167033 2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0-istio-data\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:25:41.167203 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:25:41.167047 2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0-credential-socket\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:25:41.167203 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:25:41.167058 2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0-workload-certs\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:25:41.168425 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:25:41.168398 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0-kube-api-access-r2zwj" (OuterVolumeSpecName: "kube-api-access-r2zwj") pod "1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0" (UID: "1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0"). InnerVolumeSpecName "kube-api-access-r2zwj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 16:25:41.168538 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:25:41.168522 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0-istio-token" (OuterVolumeSpecName: "istio-token") pod "1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0" (UID: "1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 16:25:41.168614 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:25:41.168593 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0" (UID: "1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 16:25:41.168757 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:25:41.168735 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0" (UID: "1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 16:25:41.268144 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:25:41.268069 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2zwj\" (UniqueName: \"kubernetes.io/projected/1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0-kube-api-access-r2zwj\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:25:41.268144 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:25:41.268097 2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0-istio-token\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:25:41.268144 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:25:41.268110 2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0-istio-podinfo\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:25:41.268144 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:25:41.268122 2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0-istio-envoy\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:25:41.815129 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:25:41.815095 2566 generic.go:356] "Generic (PLEG): container finished" podID="1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0" containerID="6572ac967b9d10f8002883fda7896bf320a74b949b04d271d3d08969453a9220" exitCode=0 Sep 11 16:25:41.815585 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:25:41.815147 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-214daa-openshift-default-bc446bc94-bmxq6" Sep 11 16:25:41.815585 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:25:41.815174 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-214daa-openshift-default-bc446bc94-bmxq6" event={"ID":"1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0","Type":"ContainerDied","Data":"6572ac967b9d10f8002883fda7896bf320a74b949b04d271d3d08969453a9220"} Sep 11 16:25:41.815585 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:25:41.815226 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-214daa-openshift-default-bc446bc94-bmxq6" event={"ID":"1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0","Type":"ContainerDied","Data":"ca56ceb7d97867357f7ec6ce71ac0996b56ceadbbbddea0e548b09a0b1c9f9f5"} Sep 11 16:25:41.815585 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:25:41.815248 2566 scope.go:117] "RemoveContainer" containerID="6572ac967b9d10f8002883fda7896bf320a74b949b04d271d3d08969453a9220" Sep 11 16:25:41.825888 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:25:41.825857 2566 scope.go:117] "RemoveContainer" containerID="bc3f31d7e33333a5ea52652cd32572ffce2ccacbc69631b6474bc43d0c772bc6" Sep 11 16:25:41.840211 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:25:41.840170 2566 scope.go:117] "RemoveContainer" containerID="6572ac967b9d10f8002883fda7896bf320a74b949b04d271d3d08969453a9220" Sep 11 16:25:41.840702 ip-10-0-136-96 kubenswrapper[2566]: E0911 16:25:41.840678 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6572ac967b9d10f8002883fda7896bf320a74b949b04d271d3d08969453a9220\": container with ID starting with 6572ac967b9d10f8002883fda7896bf320a74b949b04d271d3d08969453a9220 not found: ID does not exist" containerID="6572ac967b9d10f8002883fda7896bf320a74b949b04d271d3d08969453a9220" Sep 11 16:25:41.840826 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:25:41.840710 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6572ac967b9d10f8002883fda7896bf320a74b949b04d271d3d08969453a9220"} err="failed to get container status \"6572ac967b9d10f8002883fda7896bf320a74b949b04d271d3d08969453a9220\": rpc error: code = NotFound desc = could not find container \"6572ac967b9d10f8002883fda7896bf320a74b949b04d271d3d08969453a9220\": container with ID starting with 6572ac967b9d10f8002883fda7896bf320a74b949b04d271d3d08969453a9220 not found: ID does not exist" Sep 11 16:25:41.840826 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:25:41.840728 2566 scope.go:117] "RemoveContainer" containerID="bc3f31d7e33333a5ea52652cd32572ffce2ccacbc69631b6474bc43d0c772bc6" Sep 11 16:25:41.841071 ip-10-0-136-96 kubenswrapper[2566]: E0911 16:25:41.841052 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc3f31d7e33333a5ea52652cd32572ffce2ccacbc69631b6474bc43d0c772bc6\": container with ID starting with bc3f31d7e33333a5ea52652cd32572ffce2ccacbc69631b6474bc43d0c772bc6 not found: ID does not exist" containerID="bc3f31d7e33333a5ea52652cd32572ffce2ccacbc69631b6474bc43d0c772bc6" Sep 11 16:25:41.841144 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:25:41.841076 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc3f31d7e33333a5ea52652cd32572ffce2ccacbc69631b6474bc43d0c772bc6"} err="failed to get container status \"bc3f31d7e33333a5ea52652cd32572ffce2ccacbc69631b6474bc43d0c772bc6\": rpc error: code = NotFound desc = could not find container \"bc3f31d7e33333a5ea52652cd32572ffce2ccacbc69631b6474bc43d0c772bc6\": container with ID starting with bc3f31d7e33333a5ea52652cd32572ffce2ccacbc69631b6474bc43d0c772bc6 not found: ID does not exist" Sep 11 16:25:41.851212 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:25:41.851177 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-214daa-openshift-default-bc446bc94-bmxq6"] Sep 11 16:25:41.854790 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:25:41.854762 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-214daa-openshift-default-bc446bc94-bmxq6"] Sep 11 16:25:41.959359 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:25:41.959334 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f9d303-dsw5z" Sep 11 16:25:42.075293 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:25:42.075207 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a25bef5-7a3e-4271-bc59-95ba530e19ad-maas-api-service-ca\") pod \"5a25bef5-7a3e-4271-bc59-95ba530e19ad\" (UID: \"5a25bef5-7a3e-4271-bc59-95ba530e19ad\") " Sep 11 16:25:42.075625 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:25:42.075605 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a25bef5-7a3e-4271-bc59-95ba530e19ad-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5a25bef5-7a3e-4271-bc59-95ba530e19ad" (UID: "5a25bef5-7a3e-4271-bc59-95ba530e19ad"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 16:25:42.176519 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:25:42.176484 2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a25bef5-7a3e-4271-bc59-95ba530e19ad-maas-api-service-ca\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:25:42.824013 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:25:42.823963 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f9d303-dsw5z" Sep 11 16:25:42.824514 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:25:42.823963 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f9d303-dsw5z" event={"ID":"5a25bef5-7a3e-4271-bc59-95ba530e19ad","Type":"ContainerDied","Data":"c4820f4e156fd58ee7dc7b3c1665bfb6add54cea98edf442fc3d5c2a2dc2fe21"} Sep 11 16:25:42.824514 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:25:42.824114 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c4820f4e156fd58ee7dc7b3c1665bfb6add54cea98edf442fc3d5c2a2dc2fe21" Sep 11 16:25:43.240306 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:25:43.240140 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0" path="/var/lib/kubelet/pods/1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0/volumes" Sep 11 16:25:53.021834 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:25:53.021797 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dx4xn"] Sep 11 16:25:55.721035 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:25:55.720997 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f9d303-dsw5z"] Sep 11 16:25:55.723547 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:25:55.723524 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f9d303-dsw5z"] Sep 11 16:25:57.239591 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:25:57.239557 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a25bef5-7a3e-4271-bc59-95ba530e19ad" path="/var/lib/kubelet/pods/5a25bef5-7a3e-4271-bc59-95ba530e19ad/volumes" Sep 11 16:26:05.718280 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:26:05.718234 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dx4xn"] Sep 11 16:26:44.367259 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:26:44.367220 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-f9d303-openshift-default-5754c68888-x9vh2"] Sep 11 16:26:44.367799 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:26:44.367488 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-f9d303-openshift-default-5754c68888-x9vh2" podUID="63e70794-d8a3-4a93-af85-6060def3d813" containerName="istio-proxy" containerID="cri-o://b779674b3ecd0095a31783f1259ca299694ea34821011ec3370a4457044d99f2" gracePeriod=30 Sep 11 16:26:49.614639 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:26:49.614613 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f9d303-openshift-default-5754c68888-x9vh2" Sep 11 16:26:49.749377 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:26:49.749283 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63e70794-d8a3-4a93-af85-6060def3d813-istiod-ca-cert\") pod \"63e70794-d8a3-4a93-af85-6060def3d813\" (UID: \"63e70794-d8a3-4a93-af85-6060def3d813\") " Sep 11 16:26:49.749377 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:26:49.749333 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63e70794-d8a3-4a93-af85-6060def3d813-workload-socket\") pod \"63e70794-d8a3-4a93-af85-6060def3d813\" (UID: \"63e70794-d8a3-4a93-af85-6060def3d813\") " Sep 11 16:26:49.749585 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:26:49.749416 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qngc7\" (UniqueName: \"kubernetes.io/projected/63e70794-d8a3-4a93-af85-6060def3d813-kube-api-access-qngc7\") pod \"63e70794-d8a3-4a93-af85-6060def3d813\" (UID: \"63e70794-d8a3-4a93-af85-6060def3d813\") " Sep 11 16:26:49.749585 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:26:49.749444 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63e70794-d8a3-4a93-af85-6060def3d813-istio-token\") pod \"63e70794-d8a3-4a93-af85-6060def3d813\" (UID: \"63e70794-d8a3-4a93-af85-6060def3d813\") " Sep 11 16:26:49.749585 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:26:49.749470 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63e70794-d8a3-4a93-af85-6060def3d813-istio-envoy\") pod \"63e70794-d8a3-4a93-af85-6060def3d813\" (UID: \"63e70794-d8a3-4a93-af85-6060def3d813\") " Sep 11 16:26:49.749585 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:26:49.749499 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63e70794-d8a3-4a93-af85-6060def3d813-istio-data\") pod \"63e70794-d8a3-4a93-af85-6060def3d813\" (UID: \"63e70794-d8a3-4a93-af85-6060def3d813\") " Sep 11 16:26:49.749585 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:26:49.749533 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63e70794-d8a3-4a93-af85-6060def3d813-istio-podinfo\") pod \"63e70794-d8a3-4a93-af85-6060def3d813\" (UID: \"63e70794-d8a3-4a93-af85-6060def3d813\") " Sep 11 16:26:49.749855 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:26:49.749586 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63e70794-d8a3-4a93-af85-6060def3d813-credential-socket\") pod \"63e70794-d8a3-4a93-af85-6060def3d813\" (UID: \"63e70794-d8a3-4a93-af85-6060def3d813\") " Sep 11 16:26:49.749855 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:26:49.749626 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63e70794-d8a3-4a93-af85-6060def3d813-workload-certs\") pod \"63e70794-d8a3-4a93-af85-6060def3d813\" (UID: \"63e70794-d8a3-4a93-af85-6060def3d813\") " Sep 11 16:26:49.749855 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:26:49.749655 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63e70794-d8a3-4a93-af85-6060def3d813-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "63e70794-d8a3-4a93-af85-6060def3d813" (UID: "63e70794-d8a3-4a93-af85-6060def3d813"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 16:26:49.749855 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:26:49.749701 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63e70794-d8a3-4a93-af85-6060def3d813-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "63e70794-d8a3-4a93-af85-6060def3d813" (UID: "63e70794-d8a3-4a93-af85-6060def3d813"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 16:26:49.749855 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:26:49.749823 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63e70794-d8a3-4a93-af85-6060def3d813-istio-data" (OuterVolumeSpecName: "istio-data") pod "63e70794-d8a3-4a93-af85-6060def3d813" (UID: "63e70794-d8a3-4a93-af85-6060def3d813"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 16:26:49.750135 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:26:49.749959 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63e70794-d8a3-4a93-af85-6060def3d813-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "63e70794-d8a3-4a93-af85-6060def3d813" (UID: "63e70794-d8a3-4a93-af85-6060def3d813"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 16:26:49.750135 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:26:49.750003 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63e70794-d8a3-4a93-af85-6060def3d813-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "63e70794-d8a3-4a93-af85-6060def3d813" (UID: "63e70794-d8a3-4a93-af85-6060def3d813"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 16:26:49.750135 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:26:49.750047 2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63e70794-d8a3-4a93-af85-6060def3d813-istio-data\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:26:49.750135 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:26:49.750063 2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63e70794-d8a3-4a93-af85-6060def3d813-istiod-ca-cert\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:26:49.750135 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:26:49.750076 2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63e70794-d8a3-4a93-af85-6060def3d813-workload-socket\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:26:49.751807 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:26:49.751784 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63e70794-d8a3-4a93-af85-6060def3d813-istio-token" (OuterVolumeSpecName: "istio-token") pod "63e70794-d8a3-4a93-af85-6060def3d813" (UID: "63e70794-d8a3-4a93-af85-6060def3d813"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 16:26:49.751807 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:26:49.751778 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/63e70794-d8a3-4a93-af85-6060def3d813-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "63e70794-d8a3-4a93-af85-6060def3d813" (UID: "63e70794-d8a3-4a93-af85-6060def3d813"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 16:26:49.752087 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:26:49.752067 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63e70794-d8a3-4a93-af85-6060def3d813-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "63e70794-d8a3-4a93-af85-6060def3d813" (UID: "63e70794-d8a3-4a93-af85-6060def3d813"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 16:26:49.752158 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:26:49.752143 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63e70794-d8a3-4a93-af85-6060def3d813-kube-api-access-qngc7" (OuterVolumeSpecName: "kube-api-access-qngc7") pod "63e70794-d8a3-4a93-af85-6060def3d813" (UID: "63e70794-d8a3-4a93-af85-6060def3d813"). InnerVolumeSpecName "kube-api-access-qngc7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 16:26:49.850944 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:26:49.850907 2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63e70794-d8a3-4a93-af85-6060def3d813-istio-envoy\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:26:49.850944 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:26:49.850941 2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63e70794-d8a3-4a93-af85-6060def3d813-istio-podinfo\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:26:49.850944 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:26:49.850950 2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63e70794-d8a3-4a93-af85-6060def3d813-credential-socket\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:26:49.851204 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:26:49.850960 2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63e70794-d8a3-4a93-af85-6060def3d813-workload-certs\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:26:49.851204 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:26:49.851001 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qngc7\" (UniqueName: \"kubernetes.io/projected/63e70794-d8a3-4a93-af85-6060def3d813-kube-api-access-qngc7\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:26:49.851204 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:26:49.851023 2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63e70794-d8a3-4a93-af85-6060def3d813-istio-token\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:26:50.260537 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:26:50.260498 2566 generic.go:356] "Generic (PLEG): container finished" podID="63e70794-d8a3-4a93-af85-6060def3d813" containerID="b779674b3ecd0095a31783f1259ca299694ea34821011ec3370a4457044d99f2" exitCode=0 Sep 11 16:26:50.260713 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:26:50.260554 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f9d303-openshift-default-5754c68888-x9vh2" Sep 11 16:26:50.260713 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:26:50.260587 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f9d303-openshift-default-5754c68888-x9vh2" event={"ID":"63e70794-d8a3-4a93-af85-6060def3d813","Type":"ContainerDied","Data":"b779674b3ecd0095a31783f1259ca299694ea34821011ec3370a4457044d99f2"} Sep 11 16:26:50.260713 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:26:50.260644 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f9d303-openshift-default-5754c68888-x9vh2" event={"ID":"63e70794-d8a3-4a93-af85-6060def3d813","Type":"ContainerDied","Data":"7c0043115e94859070a69f49233184e3fc56b378da6c044efdfca757af57b36a"} Sep 11 16:26:50.260713 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:26:50.260668 2566 scope.go:117] "RemoveContainer" containerID="b779674b3ecd0095a31783f1259ca299694ea34821011ec3370a4457044d99f2" Sep 11 16:26:50.272833 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:26:50.272810 2566 scope.go:117] "RemoveContainer" containerID="905814ea5eede8ccf23d4946ecbaa6e062983fd5f1d9293f588bedd601d3484e" Sep 11 16:26:50.281849 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:26:50.281830 2566 scope.go:117] "RemoveContainer" containerID="b779674b3ecd0095a31783f1259ca299694ea34821011ec3370a4457044d99f2" Sep 11 16:26:50.282173 ip-10-0-136-96 kubenswrapper[2566]: E0911 16:26:50.282150 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b779674b3ecd0095a31783f1259ca299694ea34821011ec3370a4457044d99f2\": container with ID starting with b779674b3ecd0095a31783f1259ca299694ea34821011ec3370a4457044d99f2 not found: ID does not exist" containerID="b779674b3ecd0095a31783f1259ca299694ea34821011ec3370a4457044d99f2" Sep 11 16:26:50.282247 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:26:50.282186 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b779674b3ecd0095a31783f1259ca299694ea34821011ec3370a4457044d99f2"} err="failed to get container status \"b779674b3ecd0095a31783f1259ca299694ea34821011ec3370a4457044d99f2\": rpc error: code = NotFound desc = could not find container \"b779674b3ecd0095a31783f1259ca299694ea34821011ec3370a4457044d99f2\": container with ID starting with b779674b3ecd0095a31783f1259ca299694ea34821011ec3370a4457044d99f2 not found: ID does not exist" Sep 11 16:26:50.282247 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:26:50.282210 2566 scope.go:117] "RemoveContainer" containerID="905814ea5eede8ccf23d4946ecbaa6e062983fd5f1d9293f588bedd601d3484e" Sep 11 16:26:50.282488 ip-10-0-136-96 kubenswrapper[2566]: E0911 16:26:50.282468 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"905814ea5eede8ccf23d4946ecbaa6e062983fd5f1d9293f588bedd601d3484e\": container with ID starting with 905814ea5eede8ccf23d4946ecbaa6e062983fd5f1d9293f588bedd601d3484e not found: ID does not exist" containerID="905814ea5eede8ccf23d4946ecbaa6e062983fd5f1d9293f588bedd601d3484e" Sep 11 16:26:50.282548 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:26:50.282500 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"905814ea5eede8ccf23d4946ecbaa6e062983fd5f1d9293f588bedd601d3484e"} err="failed to get container status \"905814ea5eede8ccf23d4946ecbaa6e062983fd5f1d9293f588bedd601d3484e\": rpc error: code = NotFound desc = could not find container \"905814ea5eede8ccf23d4946ecbaa6e062983fd5f1d9293f588bedd601d3484e\": container with ID starting with 905814ea5eede8ccf23d4946ecbaa6e062983fd5f1d9293f588bedd601d3484e not found: ID does not exist" Sep 11 16:26:50.283870 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:26:50.283851 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-f9d303-openshift-default-5754c68888-x9vh2"] Sep 11 16:26:50.288673 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:26:50.288652 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-f9d303-openshift-default-5754c68888-x9vh2"] Sep 11 16:26:51.239635 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:26:51.239598 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="63e70794-d8a3-4a93-af85-6060def3d813" path="/var/lib/kubelet/pods/63e70794-d8a3-4a93-af85-6060def3d813/volumes" Sep 11 16:26:58.712299 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:26:58.712265 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dx4xn"] Sep 11 16:27:02.571438 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:02.571402 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-c0aa4c-openshift-default-84f54cffb-jxqp5"] Sep 11 16:27:02.571912 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:02.571663 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-c0aa4c-openshift-default-84f54cffb-jxqp5" podUID="340a9de4-01a7-4623-9e4b-0a6332bd8b35" containerName="istio-proxy" containerID="cri-o://1e538c11413d344902f7730696d6832bf198aa291e2da57a0a33c9e678e1e01e" gracePeriod=30 Sep 11 16:27:04.123022 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:04.122958 2566 patch_prober.go:28] interesting pod/e2e-ipp-b-c0aa4c-openshift-default-84f54cffb-jxqp5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 11 16:27:04.123388 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:04.123042 2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-c0aa4c-openshift-default-84f54cffb-jxqp5" podUID="340a9de4-01a7-4623-9e4b-0a6332bd8b35" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 11 16:27:07.833293 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:07.833229 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-c0aa4c-openshift-default-84f54cffb-jxqp5" Sep 11 16:27:07.940677 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:07.940642 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/340a9de4-01a7-4623-9e4b-0a6332bd8b35-istio-podinfo\") pod \"340a9de4-01a7-4623-9e4b-0a6332bd8b35\" (UID: \"340a9de4-01a7-4623-9e4b-0a6332bd8b35\") " Sep 11 16:27:07.940864 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:07.940713 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/340a9de4-01a7-4623-9e4b-0a6332bd8b35-istio-token\") pod \"340a9de4-01a7-4623-9e4b-0a6332bd8b35\" (UID: \"340a9de4-01a7-4623-9e4b-0a6332bd8b35\") " Sep 11 16:27:07.940864 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:07.940753 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/340a9de4-01a7-4623-9e4b-0a6332bd8b35-istio-data\") pod \"340a9de4-01a7-4623-9e4b-0a6332bd8b35\" (UID: \"340a9de4-01a7-4623-9e4b-0a6332bd8b35\") " Sep 11 16:27:07.940864 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:07.940809 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/340a9de4-01a7-4623-9e4b-0a6332bd8b35-workload-socket\") pod \"340a9de4-01a7-4623-9e4b-0a6332bd8b35\" (UID: \"340a9de4-01a7-4623-9e4b-0a6332bd8b35\") " Sep 11 16:27:07.941042 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:07.940876 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/340a9de4-01a7-4623-9e4b-0a6332bd8b35-istiod-ca-cert\") pod \"340a9de4-01a7-4623-9e4b-0a6332bd8b35\" (UID: \"340a9de4-01a7-4623-9e4b-0a6332bd8b35\") " Sep 11 16:27:07.941042 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:07.940904 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jprtt\" (UniqueName: \"kubernetes.io/projected/340a9de4-01a7-4623-9e4b-0a6332bd8b35-kube-api-access-jprtt\") pod \"340a9de4-01a7-4623-9e4b-0a6332bd8b35\" (UID: \"340a9de4-01a7-4623-9e4b-0a6332bd8b35\") " Sep 11 16:27:07.941145 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:07.941055 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/340a9de4-01a7-4623-9e4b-0a6332bd8b35-credential-socket\") pod \"340a9de4-01a7-4623-9e4b-0a6332bd8b35\" (UID: \"340a9de4-01a7-4623-9e4b-0a6332bd8b35\") " Sep 11 16:27:07.941145 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:07.941121 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/340a9de4-01a7-4623-9e4b-0a6332bd8b35-istio-envoy\") pod \"340a9de4-01a7-4623-9e4b-0a6332bd8b35\" (UID: \"340a9de4-01a7-4623-9e4b-0a6332bd8b35\") " Sep 11 16:27:07.941145 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:07.941120 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/340a9de4-01a7-4623-9e4b-0a6332bd8b35-istio-data" (OuterVolumeSpecName: "istio-data") pod "340a9de4-01a7-4623-9e4b-0a6332bd8b35" (UID: "340a9de4-01a7-4623-9e4b-0a6332bd8b35"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 16:27:07.941298 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:07.941172 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/340a9de4-01a7-4623-9e4b-0a6332bd8b35-workload-certs\") pod \"340a9de4-01a7-4623-9e4b-0a6332bd8b35\" (UID: \"340a9de4-01a7-4623-9e4b-0a6332bd8b35\") " Sep 11 16:27:07.941298 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:07.941192 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/340a9de4-01a7-4623-9e4b-0a6332bd8b35-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "340a9de4-01a7-4623-9e4b-0a6332bd8b35" (UID: "340a9de4-01a7-4623-9e4b-0a6332bd8b35"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 16:27:07.941396 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:07.941327 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/340a9de4-01a7-4623-9e4b-0a6332bd8b35-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "340a9de4-01a7-4623-9e4b-0a6332bd8b35" (UID: "340a9de4-01a7-4623-9e4b-0a6332bd8b35"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 16:27:07.941396 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:07.941360 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/340a9de4-01a7-4623-9e4b-0a6332bd8b35-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "340a9de4-01a7-4623-9e4b-0a6332bd8b35" (UID: "340a9de4-01a7-4623-9e4b-0a6332bd8b35"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 16:27:07.941542 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:07.941508 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/340a9de4-01a7-4623-9e4b-0a6332bd8b35-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "340a9de4-01a7-4623-9e4b-0a6332bd8b35" (UID: "340a9de4-01a7-4623-9e4b-0a6332bd8b35"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 16:27:07.941680 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:07.941577 2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/340a9de4-01a7-4623-9e4b-0a6332bd8b35-istiod-ca-cert\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:27:07.941680 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:07.941591 2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/340a9de4-01a7-4623-9e4b-0a6332bd8b35-credential-socket\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:27:07.941680 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:07.941600 2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/340a9de4-01a7-4623-9e4b-0a6332bd8b35-istio-data\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:27:07.941680 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:07.941609 2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/340a9de4-01a7-4623-9e4b-0a6332bd8b35-workload-socket\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:27:07.943107 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:07.943076 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/340a9de4-01a7-4623-9e4b-0a6332bd8b35-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "340a9de4-01a7-4623-9e4b-0a6332bd8b35" (UID: "340a9de4-01a7-4623-9e4b-0a6332bd8b35"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 16:27:07.943240 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:07.943220 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/340a9de4-01a7-4623-9e4b-0a6332bd8b35-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "340a9de4-01a7-4623-9e4b-0a6332bd8b35" (UID: "340a9de4-01a7-4623-9e4b-0a6332bd8b35"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 16:27:07.943452 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:07.943437 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/340a9de4-01a7-4623-9e4b-0a6332bd8b35-kube-api-access-jprtt" (OuterVolumeSpecName: "kube-api-access-jprtt") pod "340a9de4-01a7-4623-9e4b-0a6332bd8b35" (UID: "340a9de4-01a7-4623-9e4b-0a6332bd8b35"). InnerVolumeSpecName "kube-api-access-jprtt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 16:27:07.943537 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:07.943521 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/340a9de4-01a7-4623-9e4b-0a6332bd8b35-istio-token" (OuterVolumeSpecName: "istio-token") pod "340a9de4-01a7-4623-9e4b-0a6332bd8b35" (UID: "340a9de4-01a7-4623-9e4b-0a6332bd8b35"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 16:27:08.043144 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:08.043106 2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/340a9de4-01a7-4623-9e4b-0a6332bd8b35-istio-envoy\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:27:08.043144 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:08.043138 2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/340a9de4-01a7-4623-9e4b-0a6332bd8b35-workload-certs\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:27:08.043144 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:08.043147 2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/340a9de4-01a7-4623-9e4b-0a6332bd8b35-istio-podinfo\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:27:08.043144 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:08.043156 2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/340a9de4-01a7-4623-9e4b-0a6332bd8b35-istio-token\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:27:08.043438 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:08.043165 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jprtt\" (UniqueName: \"kubernetes.io/projected/340a9de4-01a7-4623-9e4b-0a6332bd8b35-kube-api-access-jprtt\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:27:08.371522 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:08.371487 2566 generic.go:356] "Generic (PLEG): container finished" podID="340a9de4-01a7-4623-9e4b-0a6332bd8b35" containerID="1e538c11413d344902f7730696d6832bf198aa291e2da57a0a33c9e678e1e01e" exitCode=0 Sep 11 16:27:08.371665 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:08.371529 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-c0aa4c-openshift-default-84f54cffb-jxqp5" Sep 11 16:27:08.371665 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:08.371562 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c0aa4c-openshift-default-84f54cffb-jxqp5" event={"ID":"340a9de4-01a7-4623-9e4b-0a6332bd8b35","Type":"ContainerDied","Data":"1e538c11413d344902f7730696d6832bf198aa291e2da57a0a33c9e678e1e01e"} Sep 11 16:27:08.371665 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:08.371617 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c0aa4c-openshift-default-84f54cffb-jxqp5" event={"ID":"340a9de4-01a7-4623-9e4b-0a6332bd8b35","Type":"ContainerDied","Data":"07145715bb1f7fc1ead38d76a319974ecd5e429b1e01998372c73a62551702f7"} Sep 11 16:27:08.371665 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:08.371640 2566 scope.go:117] "RemoveContainer" containerID="1e538c11413d344902f7730696d6832bf198aa291e2da57a0a33c9e678e1e01e" Sep 11 16:27:08.381770 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:08.381750 2566 scope.go:117] "RemoveContainer" containerID="2b952693a70fd3c6ff4611c2ad2b0896c06cf2eb6b9499e39be3969755733ca7" Sep 11 16:27:08.390940 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:08.390919 2566 scope.go:117] "RemoveContainer" containerID="1e538c11413d344902f7730696d6832bf198aa291e2da57a0a33c9e678e1e01e" Sep 11 16:27:08.391251 ip-10-0-136-96 kubenswrapper[2566]: E0911 16:27:08.391229 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e538c11413d344902f7730696d6832bf198aa291e2da57a0a33c9e678e1e01e\": container with ID starting with 1e538c11413d344902f7730696d6832bf198aa291e2da57a0a33c9e678e1e01e not found: ID does not exist" containerID="1e538c11413d344902f7730696d6832bf198aa291e2da57a0a33c9e678e1e01e" Sep 11 16:27:08.391317 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:08.391260 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e538c11413d344902f7730696d6832bf198aa291e2da57a0a33c9e678e1e01e"} err="failed to get container status \"1e538c11413d344902f7730696d6832bf198aa291e2da57a0a33c9e678e1e01e\": rpc error: code = NotFound desc = could not find container \"1e538c11413d344902f7730696d6832bf198aa291e2da57a0a33c9e678e1e01e\": container with ID starting with 1e538c11413d344902f7730696d6832bf198aa291e2da57a0a33c9e678e1e01e not found: ID does not exist" Sep 11 16:27:08.391317 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:08.391279 2566 scope.go:117] "RemoveContainer" containerID="2b952693a70fd3c6ff4611c2ad2b0896c06cf2eb6b9499e39be3969755733ca7" Sep 11 16:27:08.391607 ip-10-0-136-96 kubenswrapper[2566]: E0911 16:27:08.391582 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b952693a70fd3c6ff4611c2ad2b0896c06cf2eb6b9499e39be3969755733ca7\": container with ID starting with 2b952693a70fd3c6ff4611c2ad2b0896c06cf2eb6b9499e39be3969755733ca7 not found: ID does not exist" containerID="2b952693a70fd3c6ff4611c2ad2b0896c06cf2eb6b9499e39be3969755733ca7" Sep 11 16:27:08.391804 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:08.391620 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b952693a70fd3c6ff4611c2ad2b0896c06cf2eb6b9499e39be3969755733ca7"} err="failed to get container status \"2b952693a70fd3c6ff4611c2ad2b0896c06cf2eb6b9499e39be3969755733ca7\": rpc error: code = NotFound desc = could not find container \"2b952693a70fd3c6ff4611c2ad2b0896c06cf2eb6b9499e39be3969755733ca7\": container with ID starting with 2b952693a70fd3c6ff4611c2ad2b0896c06cf2eb6b9499e39be3969755733ca7 not found: ID does not exist" Sep 11 16:27:08.395675 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:08.395651 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-c0aa4c-openshift-default-84f54cffb-jxqp5"] Sep 11 16:27:08.400694 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:08.400672 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-c0aa4c-openshift-default-84f54cffb-jxqp5"] Sep 11 16:27:09.239858 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:09.239825 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="340a9de4-01a7-4623-9e4b-0a6332bd8b35" path="/var/lib/kubelet/pods/340a9de4-01a7-4623-9e4b-0a6332bd8b35/volumes" Sep 11 16:27:16.831500 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:16.831462 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dx4xn"] Sep 11 16:27:21.158085 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:21.158049 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-59f108-openshift-default-6cc7dd95cd-c4pmd"] Sep 11 16:27:21.158485 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:21.158274 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-59f108-openshift-default-6cc7dd95cd-c4pmd" podUID="8716cbf6-b788-4cf2-adb4-e4b4e400a725" containerName="istio-proxy" containerID="cri-o://da04bd17b9a23ac0a08b84cf1311967b69dad3b929c7902a6d6f33a04ff7b355" gracePeriod=30 Sep 11 16:27:26.416809 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:26.416784 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-59f108-openshift-default-6cc7dd95cd-c4pmd" Sep 11 16:27:26.495905 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:26.495814 2566 generic.go:356] "Generic (PLEG): container finished" podID="8716cbf6-b788-4cf2-adb4-e4b4e400a725" containerID="da04bd17b9a23ac0a08b84cf1311967b69dad3b929c7902a6d6f33a04ff7b355" exitCode=0 Sep 11 16:27:26.495905 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:26.495859 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-59f108-openshift-default-6cc7dd95cd-c4pmd" Sep 11 16:27:26.496180 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:26.495897 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-59f108-openshift-default-6cc7dd95cd-c4pmd" event={"ID":"8716cbf6-b788-4cf2-adb4-e4b4e400a725","Type":"ContainerDied","Data":"da04bd17b9a23ac0a08b84cf1311967b69dad3b929c7902a6d6f33a04ff7b355"} Sep 11 16:27:26.496180 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:26.495962 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-59f108-openshift-default-6cc7dd95cd-c4pmd" event={"ID":"8716cbf6-b788-4cf2-adb4-e4b4e400a725","Type":"ContainerDied","Data":"4bdd7be9295cc68512c4d6a6cf54cb121245f99e01c83d7aa6dea403763ce283"} Sep 11 16:27:26.496180 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:26.496023 2566 scope.go:117] "RemoveContainer" containerID="da04bd17b9a23ac0a08b84cf1311967b69dad3b929c7902a6d6f33a04ff7b355" Sep 11 16:27:26.505453 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:26.505435 2566 scope.go:117] "RemoveContainer" containerID="5abd1863c90d2e699da52e79d00277555c75cdacb758af2ddf9613c6a7377d01" Sep 11 16:27:26.514225 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:26.514205 2566 scope.go:117] "RemoveContainer" containerID="da04bd17b9a23ac0a08b84cf1311967b69dad3b929c7902a6d6f33a04ff7b355" Sep 11 16:27:26.514499 ip-10-0-136-96 kubenswrapper[2566]: E0911 16:27:26.514481 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da04bd17b9a23ac0a08b84cf1311967b69dad3b929c7902a6d6f33a04ff7b355\": container with ID starting with da04bd17b9a23ac0a08b84cf1311967b69dad3b929c7902a6d6f33a04ff7b355 not found: ID does not exist" containerID="da04bd17b9a23ac0a08b84cf1311967b69dad3b929c7902a6d6f33a04ff7b355" Sep 11 16:27:26.514552 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:26.514510 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da04bd17b9a23ac0a08b84cf1311967b69dad3b929c7902a6d6f33a04ff7b355"} err="failed to get container status \"da04bd17b9a23ac0a08b84cf1311967b69dad3b929c7902a6d6f33a04ff7b355\": rpc error: code = NotFound desc = could not find container \"da04bd17b9a23ac0a08b84cf1311967b69dad3b929c7902a6d6f33a04ff7b355\": container with ID starting with da04bd17b9a23ac0a08b84cf1311967b69dad3b929c7902a6d6f33a04ff7b355 not found: ID does not exist" Sep 11 16:27:26.514552 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:26.514526 2566 scope.go:117] "RemoveContainer" containerID="5abd1863c90d2e699da52e79d00277555c75cdacb758af2ddf9613c6a7377d01" Sep 11 16:27:26.514770 ip-10-0-136-96 kubenswrapper[2566]: E0911 16:27:26.514752 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5abd1863c90d2e699da52e79d00277555c75cdacb758af2ddf9613c6a7377d01\": container with ID starting with 5abd1863c90d2e699da52e79d00277555c75cdacb758af2ddf9613c6a7377d01 not found: ID does not exist" containerID="5abd1863c90d2e699da52e79d00277555c75cdacb758af2ddf9613c6a7377d01" Sep 11 16:27:26.514812 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:26.514776 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5abd1863c90d2e699da52e79d00277555c75cdacb758af2ddf9613c6a7377d01"} err="failed to get container status \"5abd1863c90d2e699da52e79d00277555c75cdacb758af2ddf9613c6a7377d01\": rpc error: code = NotFound desc = could not find container \"5abd1863c90d2e699da52e79d00277555c75cdacb758af2ddf9613c6a7377d01\": container with ID starting with 5abd1863c90d2e699da52e79d00277555c75cdacb758af2ddf9613c6a7377d01 not found: ID does not exist" Sep 11 16:27:26.522598 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:26.522582 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8716cbf6-b788-4cf2-adb4-e4b4e400a725-istio-token\") pod \"8716cbf6-b788-4cf2-adb4-e4b4e400a725\" (UID: \"8716cbf6-b788-4cf2-adb4-e4b4e400a725\") " Sep 11 16:27:26.522656 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:26.522608 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8716cbf6-b788-4cf2-adb4-e4b4e400a725-workload-socket\") pod \"8716cbf6-b788-4cf2-adb4-e4b4e400a725\" (UID: \"8716cbf6-b788-4cf2-adb4-e4b4e400a725\") " Sep 11 16:27:26.522656 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:26.522639 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8716cbf6-b788-4cf2-adb4-e4b4e400a725-istio-envoy\") pod \"8716cbf6-b788-4cf2-adb4-e4b4e400a725\" (UID: \"8716cbf6-b788-4cf2-adb4-e4b4e400a725\") " Sep 11 16:27:26.522754 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:26.522662 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8716cbf6-b788-4cf2-adb4-e4b4e400a725-istio-data\") pod \"8716cbf6-b788-4cf2-adb4-e4b4e400a725\" (UID: \"8716cbf6-b788-4cf2-adb4-e4b4e400a725\") " Sep 11 16:27:26.522754 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:26.522714 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8716cbf6-b788-4cf2-adb4-e4b4e400a725-credential-socket\") pod \"8716cbf6-b788-4cf2-adb4-e4b4e400a725\" (UID: \"8716cbf6-b788-4cf2-adb4-e4b4e400a725\") " Sep 11 16:27:26.522831 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:26.522767 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8716cbf6-b788-4cf2-adb4-e4b4e400a725-workload-certs\") pod \"8716cbf6-b788-4cf2-adb4-e4b4e400a725\" (UID: \"8716cbf6-b788-4cf2-adb4-e4b4e400a725\") " Sep 11 16:27:26.522831 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:26.522795 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8716cbf6-b788-4cf2-adb4-e4b4e400a725-istio-podinfo\") pod \"8716cbf6-b788-4cf2-adb4-e4b4e400a725\" (UID: \"8716cbf6-b788-4cf2-adb4-e4b4e400a725\") " Sep 11 16:27:26.522908 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:26.522845 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8716cbf6-b788-4cf2-adb4-e4b4e400a725-istiod-ca-cert\") pod \"8716cbf6-b788-4cf2-adb4-e4b4e400a725\" (UID: \"8716cbf6-b788-4cf2-adb4-e4b4e400a725\") " Sep 11 16:27:26.522908 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:26.522873 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tx7ng\" (UniqueName: \"kubernetes.io/projected/8716cbf6-b788-4cf2-adb4-e4b4e400a725-kube-api-access-tx7ng\") pod \"8716cbf6-b788-4cf2-adb4-e4b4e400a725\" (UID: \"8716cbf6-b788-4cf2-adb4-e4b4e400a725\") " Sep 11 16:27:26.523027 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:26.522938 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8716cbf6-b788-4cf2-adb4-e4b4e400a725-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8716cbf6-b788-4cf2-adb4-e4b4e400a725" (UID: "8716cbf6-b788-4cf2-adb4-e4b4e400a725"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 16:27:26.523086 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:26.523054 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8716cbf6-b788-4cf2-adb4-e4b4e400a725-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8716cbf6-b788-4cf2-adb4-e4b4e400a725" (UID: "8716cbf6-b788-4cf2-adb4-e4b4e400a725"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 16:27:26.523219 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:26.523190 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8716cbf6-b788-4cf2-adb4-e4b4e400a725-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8716cbf6-b788-4cf2-adb4-e4b4e400a725" (UID: "8716cbf6-b788-4cf2-adb4-e4b4e400a725"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 16:27:26.523333 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:26.523277 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8716cbf6-b788-4cf2-adb4-e4b4e400a725-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8716cbf6-b788-4cf2-adb4-e4b4e400a725" (UID: "8716cbf6-b788-4cf2-adb4-e4b4e400a725"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 16:27:26.523400 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:26.523346 2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8716cbf6-b788-4cf2-adb4-e4b4e400a725-credential-socket\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:27:26.523400 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:26.523365 2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8716cbf6-b788-4cf2-adb4-e4b4e400a725-workload-certs\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:27:26.523400 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:26.523380 2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8716cbf6-b788-4cf2-adb4-e4b4e400a725-workload-socket\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:27:26.523558 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:26.523436 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8716cbf6-b788-4cf2-adb4-e4b4e400a725-istio-data" (OuterVolumeSpecName: "istio-data") pod "8716cbf6-b788-4cf2-adb4-e4b4e400a725" (UID: "8716cbf6-b788-4cf2-adb4-e4b4e400a725"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 16:27:26.524845 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:26.524821 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8716cbf6-b788-4cf2-adb4-e4b4e400a725-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8716cbf6-b788-4cf2-adb4-e4b4e400a725" (UID: "8716cbf6-b788-4cf2-adb4-e4b4e400a725"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 16:27:26.524951 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:26.524911 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8716cbf6-b788-4cf2-adb4-e4b4e400a725-istio-token" (OuterVolumeSpecName: "istio-token") pod "8716cbf6-b788-4cf2-adb4-e4b4e400a725" (UID: "8716cbf6-b788-4cf2-adb4-e4b4e400a725"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 16:27:26.525080 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:26.525060 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8716cbf6-b788-4cf2-adb4-e4b4e400a725-kube-api-access-tx7ng" (OuterVolumeSpecName: "kube-api-access-tx7ng") pod "8716cbf6-b788-4cf2-adb4-e4b4e400a725" (UID: "8716cbf6-b788-4cf2-adb4-e4b4e400a725"). InnerVolumeSpecName "kube-api-access-tx7ng". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 16:27:26.525129 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:26.525106 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8716cbf6-b788-4cf2-adb4-e4b4e400a725-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8716cbf6-b788-4cf2-adb4-e4b4e400a725" (UID: "8716cbf6-b788-4cf2-adb4-e4b4e400a725"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 16:27:26.624833 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:26.624791 2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8716cbf6-b788-4cf2-adb4-e4b4e400a725-istio-token\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:27:26.624833 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:26.624825 2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8716cbf6-b788-4cf2-adb4-e4b4e400a725-istio-envoy\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:27:26.624833 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:26.624834 2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8716cbf6-b788-4cf2-adb4-e4b4e400a725-istio-data\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:27:26.624833 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:26.624844 2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8716cbf6-b788-4cf2-adb4-e4b4e400a725-istio-podinfo\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:27:26.625122 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:26.624854 2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8716cbf6-b788-4cf2-adb4-e4b4e400a725-istiod-ca-cert\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:27:26.625122 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:26.624863 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tx7ng\" (UniqueName: \"kubernetes.io/projected/8716cbf6-b788-4cf2-adb4-e4b4e400a725-kube-api-access-tx7ng\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:27:26.825060 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:26.825020 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-59f108-openshift-default-6cc7dd95cd-c4pmd"] Sep 11 16:27:26.830330 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:26.830301 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-59f108-openshift-default-6cc7dd95cd-c4pmd"] Sep 11 16:27:27.238887 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:27.238808 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8716cbf6-b788-4cf2-adb4-e4b4e400a725" path="/var/lib/kubelet/pods/8716cbf6-b788-4cf2-adb4-e4b4e400a725/volumes" Sep 11 16:27:34.620285 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:34.620245 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dx4xn"] Sep 11 16:27:39.136431 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:39.136397 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-1e0e69-openshift-default-6f9dd57774-zczpr"] Sep 11 16:27:39.136851 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:39.136619 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-1e0e69-openshift-default-6f9dd57774-zczpr" podUID="50f58a6c-7361-4eb4-af63-e25efa7396ef" containerName="istio-proxy" containerID="cri-o://e69249c3f963a732066628d15b1c44cecaee22daed8a1e7a4b83854bba5466ba" gracePeriod=30 Sep 11 16:27:43.129164 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:43.129121 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dx4xn"] Sep 11 16:27:44.413594 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:44.413536 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-1e0e69-openshift-default-6f9dd57774-zczpr" Sep 11 16:27:44.597049 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:44.597007 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50f58a6c-7361-4eb4-af63-e25efa7396ef-istiod-ca-cert\") pod \"50f58a6c-7361-4eb4-af63-e25efa7396ef\" (UID: \"50f58a6c-7361-4eb4-af63-e25efa7396ef\") " Sep 11 16:27:44.597247 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:44.597069 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50f58a6c-7361-4eb4-af63-e25efa7396ef-workload-certs\") pod \"50f58a6c-7361-4eb4-af63-e25efa7396ef\" (UID: \"50f58a6c-7361-4eb4-af63-e25efa7396ef\") " Sep 11 16:27:44.597247 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:44.597104 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50f58a6c-7361-4eb4-af63-e25efa7396ef-workload-socket\") pod \"50f58a6c-7361-4eb4-af63-e25efa7396ef\" (UID: \"50f58a6c-7361-4eb4-af63-e25efa7396ef\") " Sep 11 16:27:44.597247 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:44.597126 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50f58a6c-7361-4eb4-af63-e25efa7396ef-credential-socket\") pod \"50f58a6c-7361-4eb4-af63-e25efa7396ef\" (UID: \"50f58a6c-7361-4eb4-af63-e25efa7396ef\") " Sep 11 16:27:44.597247 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:44.597149 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50f58a6c-7361-4eb4-af63-e25efa7396ef-istio-data\") pod \"50f58a6c-7361-4eb4-af63-e25efa7396ef\" (UID: \"50f58a6c-7361-4eb4-af63-e25efa7396ef\") " Sep 11 16:27:44.597247 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:44.597174 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wks9f\" (UniqueName: \"kubernetes.io/projected/50f58a6c-7361-4eb4-af63-e25efa7396ef-kube-api-access-wks9f\") pod \"50f58a6c-7361-4eb4-af63-e25efa7396ef\" (UID: \"50f58a6c-7361-4eb4-af63-e25efa7396ef\") " Sep 11 16:27:44.597247 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:44.597217 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50f58a6c-7361-4eb4-af63-e25efa7396ef-istio-envoy\") pod \"50f58a6c-7361-4eb4-af63-e25efa7396ef\" (UID: \"50f58a6c-7361-4eb4-af63-e25efa7396ef\") " Sep 11 16:27:44.597576 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:44.597251 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50f58a6c-7361-4eb4-af63-e25efa7396ef-istio-token\") pod \"50f58a6c-7361-4eb4-af63-e25efa7396ef\" (UID: \"50f58a6c-7361-4eb4-af63-e25efa7396ef\") " Sep 11 16:27:44.597576 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:44.597311 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50f58a6c-7361-4eb4-af63-e25efa7396ef-istio-podinfo\") pod \"50f58a6c-7361-4eb4-af63-e25efa7396ef\" (UID: \"50f58a6c-7361-4eb4-af63-e25efa7396ef\") " Sep 11 16:27:44.597576 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:44.597386 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50f58a6c-7361-4eb4-af63-e25efa7396ef-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "50f58a6c-7361-4eb4-af63-e25efa7396ef" (UID: "50f58a6c-7361-4eb4-af63-e25efa7396ef"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 16:27:44.597576 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:44.597423 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50f58a6c-7361-4eb4-af63-e25efa7396ef-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "50f58a6c-7361-4eb4-af63-e25efa7396ef" (UID: "50f58a6c-7361-4eb4-af63-e25efa7396ef"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 16:27:44.597576 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:44.597413 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50f58a6c-7361-4eb4-af63-e25efa7396ef-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "50f58a6c-7361-4eb4-af63-e25efa7396ef" (UID: "50f58a6c-7361-4eb4-af63-e25efa7396ef"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 16:27:44.597576 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:44.597498 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50f58a6c-7361-4eb4-af63-e25efa7396ef-istio-data" (OuterVolumeSpecName: "istio-data") pod "50f58a6c-7361-4eb4-af63-e25efa7396ef" (UID: "50f58a6c-7361-4eb4-af63-e25efa7396ef"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 16:27:44.597576 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:44.597506 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50f58a6c-7361-4eb4-af63-e25efa7396ef-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "50f58a6c-7361-4eb4-af63-e25efa7396ef" (UID: "50f58a6c-7361-4eb4-af63-e25efa7396ef"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 16:27:44.597883 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:44.597636 2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50f58a6c-7361-4eb4-af63-e25efa7396ef-workload-certs\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:27:44.597883 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:44.597653 2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50f58a6c-7361-4eb4-af63-e25efa7396ef-workload-socket\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:27:44.597883 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:44.597662 2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50f58a6c-7361-4eb4-af63-e25efa7396ef-credential-socket\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:27:44.597883 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:44.597671 2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50f58a6c-7361-4eb4-af63-e25efa7396ef-istio-data\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:27:44.597883 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:44.597680 2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50f58a6c-7361-4eb4-af63-e25efa7396ef-istiod-ca-cert\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:27:44.599585 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:44.599544 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/50f58a6c-7361-4eb4-af63-e25efa7396ef-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "50f58a6c-7361-4eb4-af63-e25efa7396ef" (UID: "50f58a6c-7361-4eb4-af63-e25efa7396ef"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 16:27:44.599711 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:44.599611 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50f58a6c-7361-4eb4-af63-e25efa7396ef-kube-api-access-wks9f" (OuterVolumeSpecName: "kube-api-access-wks9f") pod "50f58a6c-7361-4eb4-af63-e25efa7396ef" (UID: "50f58a6c-7361-4eb4-af63-e25efa7396ef"). InnerVolumeSpecName "kube-api-access-wks9f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 16:27:44.599903 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:44.599881 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50f58a6c-7361-4eb4-af63-e25efa7396ef-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "50f58a6c-7361-4eb4-af63-e25efa7396ef" (UID: "50f58a6c-7361-4eb4-af63-e25efa7396ef"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 16:27:44.600084 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:44.600065 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50f58a6c-7361-4eb4-af63-e25efa7396ef-istio-token" (OuterVolumeSpecName: "istio-token") pod "50f58a6c-7361-4eb4-af63-e25efa7396ef" (UID: "50f58a6c-7361-4eb4-af63-e25efa7396ef"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 16:27:44.607664 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:44.607593 2566 generic.go:356] "Generic (PLEG): container finished" podID="50f58a6c-7361-4eb4-af63-e25efa7396ef" containerID="e69249c3f963a732066628d15b1c44cecaee22daed8a1e7a4b83854bba5466ba" exitCode=0 Sep 11 16:27:44.607664 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:44.607634 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-1e0e69-openshift-default-6f9dd57774-zczpr" event={"ID":"50f58a6c-7361-4eb4-af63-e25efa7396ef","Type":"ContainerDied","Data":"e69249c3f963a732066628d15b1c44cecaee22daed8a1e7a4b83854bba5466ba"} Sep 11 16:27:44.607664 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:44.607659 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-1e0e69-openshift-default-6f9dd57774-zczpr" Sep 11 16:27:44.607875 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:44.607710 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-1e0e69-openshift-default-6f9dd57774-zczpr" event={"ID":"50f58a6c-7361-4eb4-af63-e25efa7396ef","Type":"ContainerDied","Data":"917fd2e0904ebfa073158ca5d0c15d626332968e8402caf68a43e144417d3b34"} Sep 11 16:27:44.607875 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:44.607749 2566 scope.go:117] "RemoveContainer" containerID="e69249c3f963a732066628d15b1c44cecaee22daed8a1e7a4b83854bba5466ba" Sep 11 16:27:44.621450 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:44.621425 2566 scope.go:117] "RemoveContainer" containerID="d5f1ad0b220f6acad7f87dcb7b1afa38c354a1409a50df0c822eb2608e61aa2e" Sep 11 16:27:44.630378 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:44.630354 2566 scope.go:117] "RemoveContainer" containerID="e69249c3f963a732066628d15b1c44cecaee22daed8a1e7a4b83854bba5466ba" Sep 11 16:27:44.630765 ip-10-0-136-96 kubenswrapper[2566]: E0911 16:27:44.630735 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e69249c3f963a732066628d15b1c44cecaee22daed8a1e7a4b83854bba5466ba\": container with ID starting with e69249c3f963a732066628d15b1c44cecaee22daed8a1e7a4b83854bba5466ba not found: ID does not exist" containerID="e69249c3f963a732066628d15b1c44cecaee22daed8a1e7a4b83854bba5466ba" Sep 11 16:27:44.630936 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:44.630778 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e69249c3f963a732066628d15b1c44cecaee22daed8a1e7a4b83854bba5466ba"} err="failed to get container status \"e69249c3f963a732066628d15b1c44cecaee22daed8a1e7a4b83854bba5466ba\": rpc error: code = NotFound desc = could not find container \"e69249c3f963a732066628d15b1c44cecaee22daed8a1e7a4b83854bba5466ba\": container with ID starting with e69249c3f963a732066628d15b1c44cecaee22daed8a1e7a4b83854bba5466ba not found: ID does not exist" Sep 11 16:27:44.630936 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:44.630806 2566 scope.go:117] "RemoveContainer" containerID="d5f1ad0b220f6acad7f87dcb7b1afa38c354a1409a50df0c822eb2608e61aa2e" Sep 11 16:27:44.632734 ip-10-0-136-96 kubenswrapper[2566]: E0911 16:27:44.631451 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5f1ad0b220f6acad7f87dcb7b1afa38c354a1409a50df0c822eb2608e61aa2e\": container with ID starting with d5f1ad0b220f6acad7f87dcb7b1afa38c354a1409a50df0c822eb2608e61aa2e not found: ID does not exist" containerID="d5f1ad0b220f6acad7f87dcb7b1afa38c354a1409a50df0c822eb2608e61aa2e" Sep 11 16:27:44.632734 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:44.631482 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5f1ad0b220f6acad7f87dcb7b1afa38c354a1409a50df0c822eb2608e61aa2e"} err="failed to get container status \"d5f1ad0b220f6acad7f87dcb7b1afa38c354a1409a50df0c822eb2608e61aa2e\": rpc error: code = NotFound desc = could not find container \"d5f1ad0b220f6acad7f87dcb7b1afa38c354a1409a50df0c822eb2608e61aa2e\": container with ID starting with d5f1ad0b220f6acad7f87dcb7b1afa38c354a1409a50df0c822eb2608e61aa2e not found: ID does not exist" Sep 11 16:27:44.634245 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:44.634222 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-1e0e69-openshift-default-6f9dd57774-zczpr"] Sep 11 16:27:44.639650 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:44.639629 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-1e0e69-openshift-default-6f9dd57774-zczpr"] Sep 11 16:27:44.698565 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:44.698487 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wks9f\" (UniqueName: \"kubernetes.io/projected/50f58a6c-7361-4eb4-af63-e25efa7396ef-kube-api-access-wks9f\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:27:44.698565 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:44.698517 2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50f58a6c-7361-4eb4-af63-e25efa7396ef-istio-envoy\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:27:44.698565 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:44.698528 2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50f58a6c-7361-4eb4-af63-e25efa7396ef-istio-token\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:27:44.698565 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:44.698537 2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50f58a6c-7361-4eb4-af63-e25efa7396ef-istio-podinfo\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:27:45.239471 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:45.239437 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50f58a6c-7361-4eb4-af63-e25efa7396ef" path="/var/lib/kubelet/pods/50f58a6c-7361-4eb4-af63-e25efa7396ef/volumes" Sep 11 16:27:53.821747 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:27:53.821668 2566 scope.go:117] "RemoveContainer" containerID="1aa50eae5b7519fd265328caad43d92f61bab5f397be4da48b1546a97cd25ed4" Sep 11 16:28:53.854860 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:28:53.854822 2566 scope.go:117] "RemoveContainer" containerID="97bf5cb3bec3e2492ad766d066f74c47d463213a447cbb094d90b4e707732bf3" Sep 11 16:29:07.315885 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:29:07.315835 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dx4xn"] Sep 11 16:29:33.914155 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:29:33.914117 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dx4xn"] Sep 11 16:29:41.421358 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:29:41.421323 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dx4xn"] Sep 11 16:29:50.622453 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:29:50.622417 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dx4xn"] Sep 11 16:29:53.517796 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:29:53.517766 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-xjvl5_7542ad77-c817-4370-91de-3b08db264753/console-operator/2.log" Sep 11 16:29:53.522785 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:29:53.522761 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-xjvl5_7542ad77-c817-4370-91de-3b08db264753/console-operator/2.log" Sep 11 16:29:53.524568 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:29:53.524549 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tgq5c_981d7d1e-f81f-48fc-8962-7831bc5c2b27/ovn-acl-logging/0.log" Sep 11 16:29:53.529196 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:29:53.529181 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tgq5c_981d7d1e-f81f-48fc-8962-7831bc5c2b27/ovn-acl-logging/0.log" Sep 11 16:30:07.826821 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:30:07.826781 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dx4xn"] Sep 11 16:31:36.421456 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:31:36.421421 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dx4xn"] Sep 11 16:31:44.128453 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:31:44.128416 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dx4xn"] Sep 11 16:31:50.218401 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:31:50.218366 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dx4xn"] Sep 11 16:31:53.903247 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:31:53.903212 2566 scope.go:117] "RemoveContainer" containerID="2aa31cd0227c47ac1f41868a97f7c4c021d74cb9ab292fa54555e5449510e53c" Sep 11 16:31:53.912406 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:31:53.912381 2566 scope.go:117] "RemoveContainer" containerID="2331522b9b2006a3412ee128959d745301dcdcee216008e944b19a25ba8054a6" Sep 11 16:31:58.270299 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:31:58.270258 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dx4xn"] Sep 11 16:32:08.201777 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:32:08.201727 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dx4xn"] Sep 11 16:32:26.576245 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:32:26.576207 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dx4xn"] Sep 11 16:32:33.663059 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:32:33.663014 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dx4xn"] Sep 11 16:32:42.915883 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:32:42.915835 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dx4xn"] Sep 11 16:32:43.156781 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:32:43.156747 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-585675b678-pqm8q"] Sep 11 16:32:43.157286 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:32:43.157255 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a25bef5-7a3e-4271-bc59-95ba530e19ad" containerName="revoke-keys" Sep 11 16:32:43.157286 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:32:43.157281 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a25bef5-7a3e-4271-bc59-95ba530e19ad" containerName="revoke-keys" Sep 11 16:32:43.157286 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:32:43.157295 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8716cbf6-b788-4cf2-adb4-e4b4e400a725" containerName="istio-proxy" Sep 11 16:32:43.157526 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:32:43.157301 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="8716cbf6-b788-4cf2-adb4-e4b4e400a725" containerName="istio-proxy" Sep 11 16:32:43.157526 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:32:43.157309 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8716cbf6-b788-4cf2-adb4-e4b4e400a725" containerName="istio-proxy" Sep 11 16:32:43.157526 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:32:43.157314 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="8716cbf6-b788-4cf2-adb4-e4b4e400a725" containerName="istio-proxy" Sep 11 16:32:43.157526 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:32:43.157333 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50f58a6c-7361-4eb4-af63-e25efa7396ef" containerName="istio-proxy" Sep 11 16:32:43.157526 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:32:43.157341 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="50f58a6c-7361-4eb4-af63-e25efa7396ef" containerName="istio-proxy" Sep 11 16:32:43.157526 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:32:43.157350 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="340a9de4-01a7-4623-9e4b-0a6332bd8b35" containerName="istio-proxy" Sep 11 16:32:43.157526 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:32:43.157357 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="340a9de4-01a7-4623-9e4b-0a6332bd8b35" containerName="istio-proxy" Sep 11 16:32:43.157526 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:32:43.157365 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0" containerName="istio-proxy" Sep 11 16:32:43.157526 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:32:43.157373 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0" containerName="istio-proxy" Sep 11 16:32:43.157526 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:32:43.157400 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63e70794-d8a3-4a93-af85-6060def3d813" containerName="istio-proxy" Sep 11 16:32:43.157526 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:32:43.157408 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="63e70794-d8a3-4a93-af85-6060def3d813" containerName="istio-proxy" Sep 11 16:32:43.157526 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:32:43.157422 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0" containerName="istio-proxy" Sep 11 16:32:43.157526 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:32:43.157429 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0" containerName="istio-proxy" Sep 11 16:32:43.157526 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:32:43.157439 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="340a9de4-01a7-4623-9e4b-0a6332bd8b35" containerName="istio-proxy" Sep 11 16:32:43.157526 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:32:43.157446 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="340a9de4-01a7-4623-9e4b-0a6332bd8b35" containerName="istio-proxy" Sep 11 16:32:43.157526 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:32:43.157457 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63e70794-d8a3-4a93-af85-6060def3d813" containerName="istio-proxy" Sep 11 16:32:43.157526 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:32:43.157464 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="63e70794-d8a3-4a93-af85-6060def3d813" containerName="istio-proxy" Sep 11 16:32:43.158304 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:32:43.157560 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="63e70794-d8a3-4a93-af85-6060def3d813" containerName="istio-proxy" Sep 11 16:32:43.158304 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:32:43.157570 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="50f58a6c-7361-4eb4-af63-e25efa7396ef" containerName="istio-proxy" Sep 11 16:32:43.158304 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:32:43.157576 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="8716cbf6-b788-4cf2-adb4-e4b4e400a725" containerName="istio-proxy" Sep 11 16:32:43.158304 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:32:43.157583 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0" containerName="istio-proxy" Sep 11 16:32:43.158304 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:32:43.157588 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="8716cbf6-b788-4cf2-adb4-e4b4e400a725" containerName="istio-proxy" Sep 11 16:32:43.158304 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:32:43.157594 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a25bef5-7a3e-4271-bc59-95ba530e19ad" containerName="revoke-keys" Sep 11 16:32:43.158304 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:32:43.157599 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ee2db09-9134-4ed6-a4ab-f72aeef8b7e0" containerName="istio-proxy" Sep 11 16:32:43.158304 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:32:43.157606 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="340a9de4-01a7-4623-9e4b-0a6332bd8b35" containerName="istio-proxy" Sep 11 16:32:43.158304 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:32:43.157616 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="50f58a6c-7361-4eb4-af63-e25efa7396ef" containerName="istio-proxy" Sep 11 16:32:43.161087 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:32:43.161063 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-585675b678-pqm8q" Sep 11 16:32:43.164010 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:32:43.163990 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 11 16:32:43.164120 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:32:43.164022 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Sep 11 16:32:43.164498 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:32:43.164477 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 16:32:43.164602 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:32:43.164524 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 11 16:32:43.165217 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:32:43.165200 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-nfbz8\"" Sep 11 16:32:43.181709 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:32:43.181685 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-585675b678-pqm8q"] Sep 11 16:32:43.265566 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:32:43.265535 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/567d970d-9d5f-4f35-96eb-5852145d5285-openshift-service-ca\") pod \"maas-api-585675b678-pqm8q\" (UID: \"567d970d-9d5f-4f35-96eb-5852145d5285\") " pod="odh-ai-gateway-infra/maas-api-585675b678-pqm8q" Sep 11 16:32:43.265742 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:32:43.265582 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6gzp\" (UniqueName: \"kubernetes.io/projected/567d970d-9d5f-4f35-96eb-5852145d5285-kube-api-access-s6gzp\") pod \"maas-api-585675b678-pqm8q\" (UID: \"567d970d-9d5f-4f35-96eb-5852145d5285\") " pod="odh-ai-gateway-infra/maas-api-585675b678-pqm8q" Sep 11 16:32:43.265742 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:32:43.265604 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/567d970d-9d5f-4f35-96eb-5852145d5285-metrics-certs\") pod \"maas-api-585675b678-pqm8q\" (UID: \"567d970d-9d5f-4f35-96eb-5852145d5285\") " pod="odh-ai-gateway-infra/maas-api-585675b678-pqm8q" Sep 11 16:32:43.265742 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:32:43.265684 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/567d970d-9d5f-4f35-96eb-5852145d5285-maas-api-tls\") pod \"maas-api-585675b678-pqm8q\" (UID: \"567d970d-9d5f-4f35-96eb-5852145d5285\") " pod="odh-ai-gateway-infra/maas-api-585675b678-pqm8q" Sep 11 16:32:43.367023 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:32:43.366956 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/567d970d-9d5f-4f35-96eb-5852145d5285-openshift-service-ca\") pod \"maas-api-585675b678-pqm8q\" (UID: \"567d970d-9d5f-4f35-96eb-5852145d5285\") " pod="odh-ai-gateway-infra/maas-api-585675b678-pqm8q" Sep 11 16:32:43.367225 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:32:43.367082 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6gzp\" (UniqueName: \"kubernetes.io/projected/567d970d-9d5f-4f35-96eb-5852145d5285-kube-api-access-s6gzp\") pod \"maas-api-585675b678-pqm8q\" (UID: \"567d970d-9d5f-4f35-96eb-5852145d5285\") " pod="odh-ai-gateway-infra/maas-api-585675b678-pqm8q" Sep 11 16:32:43.367225 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:32:43.367108 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/567d970d-9d5f-4f35-96eb-5852145d5285-metrics-certs\") pod \"maas-api-585675b678-pqm8q\" (UID: \"567d970d-9d5f-4f35-96eb-5852145d5285\") " pod="odh-ai-gateway-infra/maas-api-585675b678-pqm8q" Sep 11 16:32:43.367225 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:32:43.367144 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/567d970d-9d5f-4f35-96eb-5852145d5285-maas-api-tls\") pod \"maas-api-585675b678-pqm8q\" (UID: \"567d970d-9d5f-4f35-96eb-5852145d5285\") " pod="odh-ai-gateway-infra/maas-api-585675b678-pqm8q" Sep 11 16:32:43.367683 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:32:43.367658 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/567d970d-9d5f-4f35-96eb-5852145d5285-openshift-service-ca\") pod \"maas-api-585675b678-pqm8q\" (UID: \"567d970d-9d5f-4f35-96eb-5852145d5285\") " pod="odh-ai-gateway-infra/maas-api-585675b678-pqm8q" Sep 11 16:32:43.369698 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:32:43.369676 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/567d970d-9d5f-4f35-96eb-5852145d5285-maas-api-tls\") pod \"maas-api-585675b678-pqm8q\" (UID: \"567d970d-9d5f-4f35-96eb-5852145d5285\") " pod="odh-ai-gateway-infra/maas-api-585675b678-pqm8q" Sep 11 16:32:43.369698 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:32:43.369691 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/567d970d-9d5f-4f35-96eb-5852145d5285-metrics-certs\") pod \"maas-api-585675b678-pqm8q\" (UID: \"567d970d-9d5f-4f35-96eb-5852145d5285\") " pod="odh-ai-gateway-infra/maas-api-585675b678-pqm8q" Sep 11 16:32:43.378427 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:32:43.378405 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6gzp\" (UniqueName: \"kubernetes.io/projected/567d970d-9d5f-4f35-96eb-5852145d5285-kube-api-access-s6gzp\") pod \"maas-api-585675b678-pqm8q\" (UID: \"567d970d-9d5f-4f35-96eb-5852145d5285\") " pod="odh-ai-gateway-infra/maas-api-585675b678-pqm8q" Sep 11 16:32:43.473034 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:32:43.472942 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-585675b678-pqm8q" Sep 11 16:32:43.762161 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:32:43.762135 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-585675b678-pqm8q"] Sep 11 16:32:43.763351 ip-10-0-136-96 kubenswrapper[2566]: W0911 16:32:43.763328 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod567d970d_9d5f_4f35_96eb_5852145d5285.slice/crio-8fdcea8c86a027e4be52848aff5129c9ee9f92f07545fcd105ea7a0c85f72561 WatchSource:0}: Error finding container 8fdcea8c86a027e4be52848aff5129c9ee9f92f07545fcd105ea7a0c85f72561: Status 404 returned error can't find the container with id 8fdcea8c86a027e4be52848aff5129c9ee9f92f07545fcd105ea7a0c85f72561 Sep 11 16:32:43.764986 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:32:43.764954 2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 16:32:44.364513 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:32:44.364481 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-585675b678-pqm8q" event={"ID":"567d970d-9d5f-4f35-96eb-5852145d5285","Type":"ContainerStarted","Data":"8fdcea8c86a027e4be52848aff5129c9ee9f92f07545fcd105ea7a0c85f72561"} Sep 11 16:32:46.379065 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:32:46.379025 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-585675b678-pqm8q" event={"ID":"567d970d-9d5f-4f35-96eb-5852145d5285","Type":"ContainerStarted","Data":"9a699d9962d5f501c41b12f7eab4db5ac0fd1a76e5755556ca396aee3f7ebb50"} Sep 11 16:32:46.379451 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:32:46.379201 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-585675b678-pqm8q" Sep 11 16:32:46.398703 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:32:46.398645 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-585675b678-pqm8q" podStartSLOduration=0.962222339 podStartE2EDuration="3.398625554s" totalImagesPullingTime="2.436403215s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:32:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:32:46.395061522 +0000 UTC m=+2273.609895424" watchObservedRunningTime="2026-09-11 16:32:46.398625554 +0000 UTC m=+2273.613459460" Sep 11 16:32:50.839721 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:32:50.839685 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dx4xn"] Sep 11 16:32:51.066657 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:32:51.066615 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-585675b678-pqm8q"] Sep 11 16:32:51.066883 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:32:51.066799 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-585675b678-pqm8q" podUID="567d970d-9d5f-4f35-96eb-5852145d5285" containerName="maas-api" containerID="cri-o://9a699d9962d5f501c41b12f7eab4db5ac0fd1a76e5755556ca396aee3f7ebb50" gracePeriod=30 Sep 11 16:32:51.072183 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:32:51.072159 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-585675b678-pqm8q" Sep 11 16:32:51.313617 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:32:51.313591 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-585675b678-pqm8q" Sep 11 16:32:51.407388 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:32:51.407357 2566 generic.go:356] "Generic (PLEG): container finished" podID="567d970d-9d5f-4f35-96eb-5852145d5285" containerID="9a699d9962d5f501c41b12f7eab4db5ac0fd1a76e5755556ca396aee3f7ebb50" exitCode=0 Sep 11 16:32:51.407617 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:32:51.407406 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-585675b678-pqm8q" Sep 11 16:32:51.407617 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:32:51.407449 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-585675b678-pqm8q" event={"ID":"567d970d-9d5f-4f35-96eb-5852145d5285","Type":"ContainerDied","Data":"9a699d9962d5f501c41b12f7eab4db5ac0fd1a76e5755556ca396aee3f7ebb50"} Sep 11 16:32:51.407617 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:32:51.407506 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-585675b678-pqm8q" event={"ID":"567d970d-9d5f-4f35-96eb-5852145d5285","Type":"ContainerDied","Data":"8fdcea8c86a027e4be52848aff5129c9ee9f92f07545fcd105ea7a0c85f72561"} Sep 11 16:32:51.407617 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:32:51.407535 2566 scope.go:117] "RemoveContainer" containerID="9a699d9962d5f501c41b12f7eab4db5ac0fd1a76e5755556ca396aee3f7ebb50" Sep 11 16:32:51.417348 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:32:51.417330 2566 scope.go:117] "RemoveContainer" containerID="9a699d9962d5f501c41b12f7eab4db5ac0fd1a76e5755556ca396aee3f7ebb50" Sep 11 16:32:51.417645 ip-10-0-136-96 kubenswrapper[2566]: E0911 16:32:51.417625 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a699d9962d5f501c41b12f7eab4db5ac0fd1a76e5755556ca396aee3f7ebb50\": container with ID starting with 9a699d9962d5f501c41b12f7eab4db5ac0fd1a76e5755556ca396aee3f7ebb50 not found: ID does not exist" containerID="9a699d9962d5f501c41b12f7eab4db5ac0fd1a76e5755556ca396aee3f7ebb50" Sep 11 16:32:51.417729 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:32:51.417652 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a699d9962d5f501c41b12f7eab4db5ac0fd1a76e5755556ca396aee3f7ebb50"} err="failed to get container status \"9a699d9962d5f501c41b12f7eab4db5ac0fd1a76e5755556ca396aee3f7ebb50\": rpc error: code = NotFound desc = could not find container \"9a699d9962d5f501c41b12f7eab4db5ac0fd1a76e5755556ca396aee3f7ebb50\": container with ID starting with 9a699d9962d5f501c41b12f7eab4db5ac0fd1a76e5755556ca396aee3f7ebb50 not found: ID does not exist" Sep 11 16:32:51.449073 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:32:51.448984 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/567d970d-9d5f-4f35-96eb-5852145d5285-maas-api-tls\") pod \"567d970d-9d5f-4f35-96eb-5852145d5285\" (UID: \"567d970d-9d5f-4f35-96eb-5852145d5285\") " Sep 11 16:32:51.449073 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:32:51.449029 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s6gzp\" (UniqueName: \"kubernetes.io/projected/567d970d-9d5f-4f35-96eb-5852145d5285-kube-api-access-s6gzp\") pod \"567d970d-9d5f-4f35-96eb-5852145d5285\" (UID: \"567d970d-9d5f-4f35-96eb-5852145d5285\") " Sep 11 16:32:51.449073 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:32:51.449046 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/567d970d-9d5f-4f35-96eb-5852145d5285-metrics-certs\") pod \"567d970d-9d5f-4f35-96eb-5852145d5285\" (UID: \"567d970d-9d5f-4f35-96eb-5852145d5285\") " Sep 11 16:32:51.449339 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:32:51.449086 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/567d970d-9d5f-4f35-96eb-5852145d5285-openshift-service-ca\") pod \"567d970d-9d5f-4f35-96eb-5852145d5285\" (UID: \"567d970d-9d5f-4f35-96eb-5852145d5285\") " Sep 11 16:32:51.449533 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:32:51.449503 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/567d970d-9d5f-4f35-96eb-5852145d5285-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "567d970d-9d5f-4f35-96eb-5852145d5285" (UID: "567d970d-9d5f-4f35-96eb-5852145d5285"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 16:32:51.451445 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:32:51.451413 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/567d970d-9d5f-4f35-96eb-5852145d5285-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "567d970d-9d5f-4f35-96eb-5852145d5285" (UID: "567d970d-9d5f-4f35-96eb-5852145d5285"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 16:32:51.451445 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:32:51.451436 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/567d970d-9d5f-4f35-96eb-5852145d5285-kube-api-access-s6gzp" (OuterVolumeSpecName: "kube-api-access-s6gzp") pod "567d970d-9d5f-4f35-96eb-5852145d5285" (UID: "567d970d-9d5f-4f35-96eb-5852145d5285"). InnerVolumeSpecName "kube-api-access-s6gzp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 16:32:51.451593 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:32:51.451474 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/567d970d-9d5f-4f35-96eb-5852145d5285-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "567d970d-9d5f-4f35-96eb-5852145d5285" (UID: "567d970d-9d5f-4f35-96eb-5852145d5285"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 16:32:51.550549 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:32:51.550511 2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/567d970d-9d5f-4f35-96eb-5852145d5285-maas-api-tls\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:32:51.550549 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:32:51.550549 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6gzp\" (UniqueName: \"kubernetes.io/projected/567d970d-9d5f-4f35-96eb-5852145d5285-kube-api-access-s6gzp\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:32:51.550775 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:32:51.550563 2566 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/567d970d-9d5f-4f35-96eb-5852145d5285-metrics-certs\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:32:51.550775 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:32:51.550578 2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/567d970d-9d5f-4f35-96eb-5852145d5285-openshift-service-ca\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 11 16:32:51.731030 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:32:51.730996 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-585675b678-pqm8q"] Sep 11 16:32:51.735132 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:32:51.735102 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-585675b678-pqm8q"] Sep 11 16:32:53.240402 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:32:53.240369 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="567d970d-9d5f-4f35-96eb-5852145d5285" path="/var/lib/kubelet/pods/567d970d-9d5f-4f35-96eb-5852145d5285/volumes" Sep 11 16:32:57.522557 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:32:57.522518 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dx4xn"] Sep 11 16:33:03.245836 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:33:03.245773 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dx4xn"] Sep 11 16:33:10.187472 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:33:10.187435 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-dx4xn"] Sep 11 16:33:13.180134 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:33:13.180105 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-65696b975d-t9k9v_9face0b1-6f4b-4bff-a174-b928e8fe3c42/authorino/0.log" Sep 11 16:33:18.604054 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:33:18.604022 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-48gfx_bfd28efb-548a-4ed8-a6c5-2bd79df593ab/manager/0.log" Sep 11 16:33:18.847732 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:33:18.847707 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-rmzgg_25f4149c-81d0-4345-94cf-2d650d114a4e/server/0.log" Sep 11 16:33:19.224528 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:33:19.224494 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-58647b66f4-kj5l9_4a19c7c5-5d8e-48b1-977a-08b22dbfa94d/copy-manifests/0.log" Sep 11 16:33:19.231639 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:33:19.231614 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-58647b66f4-kj5l9_4a19c7c5-5d8e-48b1-977a-08b22dbfa94d/manager/0.log" Sep 11 16:33:19.943962 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:33:19.943924 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29819055-pck9z_2d419112-60b9-4f74-ac53-8cdb8565d6f4/cleanup/0.log" Sep 11 16:33:20.935283 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:33:20.935254 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-65696b975d-t9k9v_9face0b1-6f4b-4bff-a174-b928e8fe3c42/authorino/0.log" Sep 11 16:33:21.508523 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:33:21.508486 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-dx4xn_1842dece-04ed-460c-879c-1e62e6548a67/limitador/0.log" Sep 11 16:33:21.617358 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:33:21.617326 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-p55g5_4246efe8-35df-4aaf-8bb9-f01789b50185/manager/0.log" Sep 11 16:33:21.955705 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:33:21.955677 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-zdlss_1cd096b0-a02a-4cc7-accd-7986120b8a76/istio-proxy/0.log" Sep 11 16:33:22.443564 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:33:22.443509 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-2gpqh_b4fb754a-0ba5-47b6-8fca-ec6a846499f5/istio-proxy/1.log" Sep 11 16:33:22.808078 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:33:22.808053 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-844dfdb99b-766v5_313238e9-79d7-45f7-8d96-beb55feb415a/router/0.log" Sep 11 16:33:23.814076 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:33:23.814046 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-g54l9_a53fb426-3ed8-49f0-9c0e-041a23cd5e5a/main/0.log" Sep 11 16:33:28.823803 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:33:28.823765 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-bg9tr/must-gather-j6zl5"] Sep 11 16:33:28.824231 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:33:28.824195 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="567d970d-9d5f-4f35-96eb-5852145d5285" containerName="maas-api" Sep 11 16:33:28.824231 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:33:28.824207 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="567d970d-9d5f-4f35-96eb-5852145d5285" containerName="maas-api" Sep 11 16:33:28.824231 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:33:28.824230 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50f58a6c-7361-4eb4-af63-e25efa7396ef" containerName="istio-proxy" Sep 11 16:33:28.824339 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:33:28.824235 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="50f58a6c-7361-4eb4-af63-e25efa7396ef" containerName="istio-proxy" Sep 11 16:33:28.824339 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:33:28.824296 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="567d970d-9d5f-4f35-96eb-5852145d5285" containerName="maas-api" Sep 11 16:33:28.824339 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:33:28.824305 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="63e70794-d8a3-4a93-af85-6060def3d813" containerName="istio-proxy" Sep 11 16:33:28.824339 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:33:28.824315 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="340a9de4-01a7-4623-9e4b-0a6332bd8b35" containerName="istio-proxy" Sep 11 16:33:28.831418 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:33:28.831395 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bg9tr/must-gather-j6zl5" Sep 11 16:33:28.834439 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:33:28.834421 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-bg9tr\"/\"default-dockercfg-lx7vd\"" Sep 11 16:33:28.834569 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:33:28.834450 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bg9tr\"/\"openshift-service-ca.crt\"" Sep 11 16:33:28.834765 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:33:28.834748 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bg9tr\"/\"kube-root-ca.crt\"" Sep 11 16:33:28.839367 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:33:28.839308 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bg9tr/must-gather-j6zl5"] Sep 11 16:33:28.902452 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:33:28.902407 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/70fd06af-2d4b-4dc6-83fe-9de5d6c392c7-must-gather-output\") pod \"must-gather-j6zl5\" (UID: \"70fd06af-2d4b-4dc6-83fe-9de5d6c392c7\") " pod="openshift-must-gather-bg9tr/must-gather-j6zl5" Sep 11 16:33:28.902644 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:33:28.902491 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cxkz\" (UniqueName: \"kubernetes.io/projected/70fd06af-2d4b-4dc6-83fe-9de5d6c392c7-kube-api-access-5cxkz\") pod \"must-gather-j6zl5\" (UID: \"70fd06af-2d4b-4dc6-83fe-9de5d6c392c7\") " pod="openshift-must-gather-bg9tr/must-gather-j6zl5" Sep 11 16:33:29.003580 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:33:29.003536 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/70fd06af-2d4b-4dc6-83fe-9de5d6c392c7-must-gather-output\") pod \"must-gather-j6zl5\" (UID: \"70fd06af-2d4b-4dc6-83fe-9de5d6c392c7\") " pod="openshift-must-gather-bg9tr/must-gather-j6zl5" Sep 11 16:33:29.003788 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:33:29.003611 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cxkz\" (UniqueName: \"kubernetes.io/projected/70fd06af-2d4b-4dc6-83fe-9de5d6c392c7-kube-api-access-5cxkz\") pod \"must-gather-j6zl5\" (UID: \"70fd06af-2d4b-4dc6-83fe-9de5d6c392c7\") " pod="openshift-must-gather-bg9tr/must-gather-j6zl5" Sep 11 16:33:29.003988 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:33:29.003955 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/70fd06af-2d4b-4dc6-83fe-9de5d6c392c7-must-gather-output\") pod \"must-gather-j6zl5\" (UID: \"70fd06af-2d4b-4dc6-83fe-9de5d6c392c7\") " pod="openshift-must-gather-bg9tr/must-gather-j6zl5" Sep 11 16:33:29.013788 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:33:29.013757 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cxkz\" (UniqueName: \"kubernetes.io/projected/70fd06af-2d4b-4dc6-83fe-9de5d6c392c7-kube-api-access-5cxkz\") pod \"must-gather-j6zl5\" (UID: \"70fd06af-2d4b-4dc6-83fe-9de5d6c392c7\") " pod="openshift-must-gather-bg9tr/must-gather-j6zl5" Sep 11 16:33:29.142138 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:33:29.142038 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bg9tr/must-gather-j6zl5" Sep 11 16:33:29.422064 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:33:29.422033 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bg9tr/must-gather-j6zl5"] Sep 11 16:33:29.423905 ip-10-0-136-96 kubenswrapper[2566]: W0911 16:33:29.423875 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod70fd06af_2d4b_4dc6_83fe_9de5d6c392c7.slice/crio-53ef51c7782814911cb9b596fc39f9e5a4f61e8d4ded89e1339fb6ef07b10918 WatchSource:0}: Error finding container 53ef51c7782814911cb9b596fc39f9e5a4f61e8d4ded89e1339fb6ef07b10918: Status 404 returned error can't find the container with id 53ef51c7782814911cb9b596fc39f9e5a4f61e8d4ded89e1339fb6ef07b10918 Sep 11 16:33:29.629822 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:33:29.629782 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bg9tr/must-gather-j6zl5" event={"ID":"70fd06af-2d4b-4dc6-83fe-9de5d6c392c7","Type":"ContainerStarted","Data":"53ef51c7782814911cb9b596fc39f9e5a4f61e8d4ded89e1339fb6ef07b10918"} Sep 11 16:33:30.640420 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:33:30.640383 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bg9tr/must-gather-j6zl5" event={"ID":"70fd06af-2d4b-4dc6-83fe-9de5d6c392c7","Type":"ContainerStarted","Data":"50f0b9e498f31c7ea33d9275a8c972345027f17f6023ac5e08621bd1a0ca8361"} Sep 11 16:33:30.640895 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:33:30.640435 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bg9tr/must-gather-j6zl5" event={"ID":"70fd06af-2d4b-4dc6-83fe-9de5d6c392c7","Type":"ContainerStarted","Data":"531547eef1f8268a5c5b82fdbe447a226199b1def3690805ee21988810bbb652"} Sep 11 16:33:30.661880 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:33:30.661817 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-bg9tr/must-gather-j6zl5" podStartSLOduration=1.846049422 podStartE2EDuration="2.661795016s" totalImagesPullingTime="815.745594ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:33:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:33:30.658241669 +0000 UTC m=+2317.873075574" watchObservedRunningTime="2026-09-11 16:33:30.661795016 +0000 UTC m=+2317.876628921" Sep 11 16:33:31.903273 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:33:31.903243 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-95zvh_1c358fa6-a923-4589-953f-bd49ddd4235c/global-pull-secret-syncer/0.log" Sep 11 16:33:32.094711 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:33:32.094675 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-spxb6_94ab0396-7020-4ad3-b1b7-63cd05490e84/konnectivity-agent/0.log" Sep 11 16:33:32.176138 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:33:32.176062 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-96.ec2.internal_bd507eff9b3716da03d385309d794351/haproxy/0.log" Sep 11 16:33:36.270761 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:33:36.270693 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-65696b975d-t9k9v_9face0b1-6f4b-4bff-a174-b928e8fe3c42/authorino/0.log" Sep 11 16:33:36.502742 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:33:36.502643 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-dx4xn_1842dece-04ed-460c-879c-1e62e6548a67/limitador/0.log" Sep 11 16:33:36.584211 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:33:36.584107 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-p55g5_4246efe8-35df-4aaf-8bb9-f01789b50185/manager/0.log" Sep 11 16:33:38.272389 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:33:38.272358 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3722b1cf-6259-4249-91ca-50ef6fac0a5b/alertmanager/0.log" Sep 11 16:33:38.302511 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:33:38.302443 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3722b1cf-6259-4249-91ca-50ef6fac0a5b/config-reloader/0.log" Sep 11 16:33:38.330344 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:33:38.330307 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3722b1cf-6259-4249-91ca-50ef6fac0a5b/kube-rbac-proxy-web/0.log" Sep 11 16:33:38.353599 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:33:38.353574 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3722b1cf-6259-4249-91ca-50ef6fac0a5b/kube-rbac-proxy/0.log" Sep 11 16:33:38.378215 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:33:38.378186 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3722b1cf-6259-4249-91ca-50ef6fac0a5b/kube-rbac-proxy-metric/0.log" Sep 11 16:33:38.401868 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:33:38.401841 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3722b1cf-6259-4249-91ca-50ef6fac0a5b/prom-label-proxy/0.log" Sep 11 16:33:38.425411 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:33:38.425367 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3722b1cf-6259-4249-91ca-50ef6fac0a5b/init-config-reloader/0.log" Sep 11 16:33:38.472905 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:33:38.472866 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cd954b897-kqtwh_5429cde4-cb80-4579-9687-e71617b15b9f/cluster-monitoring-operator/0.log" Sep 11 16:33:38.501820 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:33:38.501783 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-wxn2r_562f8af8-8658-4994-a330-2bbfb42e8339/kube-state-metrics/0.log" Sep 11 16:33:38.523002 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:33:38.522907 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-wxn2r_562f8af8-8658-4994-a330-2bbfb42e8339/kube-rbac-proxy-main/0.log" Sep 11 16:33:38.548025 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:33:38.547964 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-wxn2r_562f8af8-8658-4994-a330-2bbfb42e8339/kube-rbac-proxy-self/0.log" Sep 11 16:33:38.683487 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:33:38.683438 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2thkm_8edb9d38-7d93-4bdd-bb6d-c0c566471af0/node-exporter/0.log" Sep 11 16:33:38.721439 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:33:38.721403 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2thkm_8edb9d38-7d93-4bdd-bb6d-c0c566471af0/kube-rbac-proxy/0.log" Sep 11 16:33:38.780672 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:33:38.780583 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2thkm_8edb9d38-7d93-4bdd-bb6d-c0c566471af0/init-textfile/0.log" Sep 11 16:33:39.497060 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:33:39.497024 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-77c6977cfd-hxwhg_de3f52af-8138-47b2-9096-74d6d4e421a2/telemeter-client/0.log" Sep 11 16:33:39.523240 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:33:39.523176 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-77c6977cfd-hxwhg_de3f52af-8138-47b2-9096-74d6d4e421a2/reload/0.log" Sep 11 16:33:39.550741 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:33:39.550644 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-77c6977cfd-hxwhg_de3f52af-8138-47b2-9096-74d6d4e421a2/kube-rbac-proxy/0.log" Sep 11 16:33:41.268735 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:33:41.268710 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-5dcbc7df6b-8q8dx_09f64aa8-4936-49d3-96f7-9895410e5dda/networking-console-plugin/0.log" Sep 11 16:33:41.880822 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:33:41.880791 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-xjvl5_7542ad77-c817-4370-91de-3b08db264753/console-operator/2.log" Sep 11 16:33:41.887668 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:33:41.887636 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-xjvl5_7542ad77-c817-4370-91de-3b08db264753/console-operator/3.log" Sep 11 16:33:42.038959 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:33:42.038913 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-bg9tr/perf-node-gather-daemonset-2dd5n"] Sep 11 16:33:42.045398 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:33:42.045358 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bg9tr/perf-node-gather-daemonset-2dd5n" Sep 11 16:33:42.055561 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:33:42.055530 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bg9tr/perf-node-gather-daemonset-2dd5n"] Sep 11 16:33:42.066690 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:33:42.066627 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6609e233-4f49-45f7-8cb0-5e31556df204-proc\") pod \"perf-node-gather-daemonset-2dd5n\" (UID: \"6609e233-4f49-45f7-8cb0-5e31556df204\") " pod="openshift-must-gather-bg9tr/perf-node-gather-daemonset-2dd5n" Sep 11 16:33:42.066866 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:33:42.066723 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6609e233-4f49-45f7-8cb0-5e31556df204-lib-modules\") pod \"perf-node-gather-daemonset-2dd5n\" (UID: \"6609e233-4f49-45f7-8cb0-5e31556df204\") " pod="openshift-must-gather-bg9tr/perf-node-gather-daemonset-2dd5n" Sep 11 16:33:42.066866 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:33:42.066766 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkrgb\" (UniqueName: \"kubernetes.io/projected/6609e233-4f49-45f7-8cb0-5e31556df204-kube-api-access-vkrgb\") pod \"perf-node-gather-daemonset-2dd5n\" (UID: \"6609e233-4f49-45f7-8cb0-5e31556df204\") " pod="openshift-must-gather-bg9tr/perf-node-gather-daemonset-2dd5n" Sep 11 16:33:42.067029 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:33:42.066868 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6609e233-4f49-45f7-8cb0-5e31556df204-podres\") pod \"perf-node-gather-daemonset-2dd5n\" (UID: \"6609e233-4f49-45f7-8cb0-5e31556df204\") " pod="openshift-must-gather-bg9tr/perf-node-gather-daemonset-2dd5n" Sep 11 16:33:42.067029 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:33:42.066917 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6609e233-4f49-45f7-8cb0-5e31556df204-sys\") pod \"perf-node-gather-daemonset-2dd5n\" (UID: \"6609e233-4f49-45f7-8cb0-5e31556df204\") " pod="openshift-must-gather-bg9tr/perf-node-gather-daemonset-2dd5n" Sep 11 16:33:42.168351 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:33:42.168258 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6609e233-4f49-45f7-8cb0-5e31556df204-proc\") pod \"perf-node-gather-daemonset-2dd5n\" (UID: \"6609e233-4f49-45f7-8cb0-5e31556df204\") " pod="openshift-must-gather-bg9tr/perf-node-gather-daemonset-2dd5n" Sep 11 16:33:42.168351 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:33:42.168312 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6609e233-4f49-45f7-8cb0-5e31556df204-lib-modules\") pod \"perf-node-gather-daemonset-2dd5n\" (UID: \"6609e233-4f49-45f7-8cb0-5e31556df204\") " pod="openshift-must-gather-bg9tr/perf-node-gather-daemonset-2dd5n" Sep 11 16:33:42.168351 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:33:42.168343 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkrgb\" (UniqueName: \"kubernetes.io/projected/6609e233-4f49-45f7-8cb0-5e31556df204-kube-api-access-vkrgb\") pod \"perf-node-gather-daemonset-2dd5n\" (UID: \"6609e233-4f49-45f7-8cb0-5e31556df204\") " pod="openshift-must-gather-bg9tr/perf-node-gather-daemonset-2dd5n" Sep 11 16:33:42.168655 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:33:42.168379 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6609e233-4f49-45f7-8cb0-5e31556df204-podres\") pod \"perf-node-gather-daemonset-2dd5n\" (UID: \"6609e233-4f49-45f7-8cb0-5e31556df204\") " pod="openshift-must-gather-bg9tr/perf-node-gather-daemonset-2dd5n" Sep 11 16:33:42.168655 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:33:42.168417 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6609e233-4f49-45f7-8cb0-5e31556df204-sys\") pod \"perf-node-gather-daemonset-2dd5n\" (UID: \"6609e233-4f49-45f7-8cb0-5e31556df204\") " pod="openshift-must-gather-bg9tr/perf-node-gather-daemonset-2dd5n" Sep 11 16:33:42.168655 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:33:42.168606 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6609e233-4f49-45f7-8cb0-5e31556df204-sys\") pod \"perf-node-gather-daemonset-2dd5n\" (UID: \"6609e233-4f49-45f7-8cb0-5e31556df204\") " pod="openshift-must-gather-bg9tr/perf-node-gather-daemonset-2dd5n" Sep 11 16:33:42.168826 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:33:42.168669 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6609e233-4f49-45f7-8cb0-5e31556df204-proc\") pod \"perf-node-gather-daemonset-2dd5n\" (UID: \"6609e233-4f49-45f7-8cb0-5e31556df204\") " pod="openshift-must-gather-bg9tr/perf-node-gather-daemonset-2dd5n" Sep 11 16:33:42.168826 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:33:42.168786 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6609e233-4f49-45f7-8cb0-5e31556df204-lib-modules\") pod \"perf-node-gather-daemonset-2dd5n\" (UID: \"6609e233-4f49-45f7-8cb0-5e31556df204\") " pod="openshift-must-gather-bg9tr/perf-node-gather-daemonset-2dd5n" Sep 11 16:33:42.169261 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:33:42.169233 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6609e233-4f49-45f7-8cb0-5e31556df204-podres\") pod \"perf-node-gather-daemonset-2dd5n\" (UID: \"6609e233-4f49-45f7-8cb0-5e31556df204\") " pod="openshift-must-gather-bg9tr/perf-node-gather-daemonset-2dd5n" Sep 11 16:33:42.178749 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:33:42.178721 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkrgb\" (UniqueName: \"kubernetes.io/projected/6609e233-4f49-45f7-8cb0-5e31556df204-kube-api-access-vkrgb\") pod \"perf-node-gather-daemonset-2dd5n\" (UID: \"6609e233-4f49-45f7-8cb0-5e31556df204\") " pod="openshift-must-gather-bg9tr/perf-node-gather-daemonset-2dd5n" Sep 11 16:33:42.364435 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:33:42.364396 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bg9tr/perf-node-gather-daemonset-2dd5n" Sep 11 16:33:42.472321 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:33:42.472266 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-84cd566d5d-njpzr_5ca9fbab-d9ba-4396-a734-ea2785cfc13d/console/0.log" Sep 11 16:33:42.511172 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:33:42.511140 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-76f99584f4-2f2j4_6e65377d-74a2-4a00-9752-ce734ca7d357/download-server/0.log" Sep 11 16:33:42.671934 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:33:42.671888 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bg9tr/perf-node-gather-daemonset-2dd5n"] Sep 11 16:33:42.673312 ip-10-0-136-96 kubenswrapper[2566]: W0911 16:33:42.673275 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod6609e233_4f49_45f7_8cb0_5e31556df204.slice/crio-2662fb1d365473abd924206f18e9a80ba2c3e2c81fd0e26e3627b559c60a9682 WatchSource:0}: Error finding container 2662fb1d365473abd924206f18e9a80ba2c3e2c81fd0e26e3627b559c60a9682: Status 404 returned error can't find the container with id 2662fb1d365473abd924206f18e9a80ba2c3e2c81fd0e26e3627b559c60a9682 Sep 11 16:33:42.771010 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:33:42.768642 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bg9tr/perf-node-gather-daemonset-2dd5n" event={"ID":"6609e233-4f49-45f7-8cb0-5e31556df204","Type":"ContainerStarted","Data":"2662fb1d365473abd924206f18e9a80ba2c3e2c81fd0e26e3627b559c60a9682"} Sep 11 16:33:42.771010 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:33:42.769228 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-bg9tr/perf-node-gather-daemonset-2dd5n" Sep 11 16:33:42.787135 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:33:42.786940 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-bg9tr/perf-node-gather-daemonset-2dd5n" podStartSLOduration=0.786917773 podStartE2EDuration="786.917773ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 16:33:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:33:42.784884828 +0000 UTC m=+2329.999718734" watchObservedRunningTime="2026-09-11 16:33:42.786917773 +0000 UTC m=+2330.001751680" Sep 11 16:33:43.207958 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:33:43.207922 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-86695b7cc7-ssz6b_ddc77228-8fb6-473e-9940-7eae0c5457aa/volume-data-source-validator/0.log" Sep 11 16:33:43.778271 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:33:43.778232 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bg9tr/perf-node-gather-daemonset-2dd5n" event={"ID":"6609e233-4f49-45f7-8cb0-5e31556df204","Type":"ContainerStarted","Data":"10654c38b750b280fedbf1132b23682f0919b0d82a0633e7a0ed969f917902a5"} Sep 11 16:33:44.432548 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:33:44.432513 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-gx9fm_a1e7a08b-8aa9-42e9-aa46-3225f61299e0/dns/0.log" Sep 11 16:33:44.457312 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:33:44.457282 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-gx9fm_a1e7a08b-8aa9-42e9-aa46-3225f61299e0/kube-rbac-proxy/0.log" Sep 11 16:33:44.636840 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:33:44.636804 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-sxqj2_ed90592a-177c-4128-b153-b51d18b8f619/dns-node-resolver/0.log" Sep 11 16:33:45.194513 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:33:45.194485 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bctd5_249a0e7e-1da6-422a-8492-1820f6437318/node-ca/0.log" Sep 11 16:33:46.216489 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:33:46.216385 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-zdlss_1cd096b0-a02a-4cc7-accd-7986120b8a76/istio-proxy/0.log" Sep 11 16:33:46.869513 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:33:46.869478 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-2gpqh_b4fb754a-0ba5-47b6-8fca-ec6a846499f5/istio-proxy/0.log" Sep 11 16:33:46.881872 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:33:46.881843 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-2gpqh_b4fb754a-0ba5-47b6-8fca-ec6a846499f5/istio-proxy/1.log" Sep 11 16:33:46.992526 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:33:46.992498 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-844dfdb99b-766v5_313238e9-79d7-45f7-8d96-beb55feb415a/router/0.log" Sep 11 16:33:47.910153 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:33:47.910122 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-4kl5b_4d14aeeb-9182-4334-956a-d0cc21ca6633/serve-healthcheck-canary/0.log" Sep 11 16:33:48.426990 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:33:48.426938 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-mjh4m_9f6c5948-c774-49b6-aaf5-026c7b5db718/insights-operator/0.log" Sep 11 16:33:48.427610 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:33:48.427589 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-mjh4m_9f6c5948-c774-49b6-aaf5-026c7b5db718/insights-operator/1.log" Sep 11 16:33:48.594433 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:33:48.594399 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-qdzw9_715b88ae-4c15-451d-acda-67b2ec875590/kube-rbac-proxy/0.log" Sep 11 16:33:48.615464 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:33:48.615432 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-qdzw9_715b88ae-4c15-451d-acda-67b2ec875590/exporter/0.log" Sep 11 16:33:48.637128 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:33:48.637104 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-qdzw9_715b88ae-4c15-451d-acda-67b2ec875590/extractor/0.log" Sep 11 16:33:49.796469 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:33:49.796435 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-bg9tr/perf-node-gather-daemonset-2dd5n" Sep 11 16:33:51.218405 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:33:51.218365 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-48gfx_bfd28efb-548a-4ed8-a6c5-2bd79df593ab/manager/0.log" Sep 11 16:33:51.304833 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:33:51.304790 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-rmzgg_25f4149c-81d0-4345-94cf-2d650d114a4e/server/0.log" Sep 11 16:33:51.521599 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:33:51.521533 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-58647b66f4-kj5l9_4a19c7c5-5d8e-48b1-977a-08b22dbfa94d/manager/0.log" Sep 11 16:33:51.544122 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:33:51.544096 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-58647b66f4-kj5l9_4a19c7c5-5d8e-48b1-977a-08b22dbfa94d/copy-manifests/0.log" Sep 11 16:33:52.799897 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:33:52.799856 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-zdlss_1cd096b0-a02a-4cc7-accd-7986120b8a76/istio-proxy/0.log" Sep 11 16:33:52.986299 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:33:52.986268 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-6848676c-8zbl6_4291750d-8a58-4dfe-b6a7-6d959f0f63cd/manager/0.log" Sep 11 16:33:53.444351 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:33:53.444308 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-2gpqh_b4fb754a-0ba5-47b6-8fca-ec6a846499f5/istio-proxy/0.log" Sep 11 16:33:53.457154 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:33:53.457080 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-2gpqh_b4fb754a-0ba5-47b6-8fca-ec6a846499f5/istio-proxy/1.log" Sep 11 16:33:53.547589 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:33:53.547560 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-844dfdb99b-766v5_313238e9-79d7-45f7-8d96-beb55feb415a/router/0.log" Sep 11 16:33:55.004966 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:33:55.004934 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-zdlss_1cd096b0-a02a-4cc7-accd-7986120b8a76/istio-proxy/0.log" Sep 11 16:33:55.616399 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:33:55.616360 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-2gpqh_b4fb754a-0ba5-47b6-8fca-ec6a846499f5/istio-proxy/0.log" Sep 11 16:33:55.627542 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:33:55.627497 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-2gpqh_b4fb754a-0ba5-47b6-8fca-ec6a846499f5/istio-proxy/1.log" Sep 11 16:33:55.722196 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:33:55.722166 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-844dfdb99b-766v5_313238e9-79d7-45f7-8d96-beb55feb415a/router/0.log" Sep 11 16:33:58.715420 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:33:58.715378 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-djghf_07d36306-7ec8-4c74-9b97-7545503a2783/kube-storage-version-migrator-operator/1.log" Sep 11 16:33:58.716276 ip-10-0-136-96 kubenswrapper[2566]: I0911 16:33:58.716258 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-djghf_07d36306-7ec8-4c74-9b97-7545503a2783/kube-storage-version-migrator-operator/0.log"