Sep 10 23:36:46.896836 ip-10-0-138-128 systemd[1]: Starting Kubernetes Kubelet... Sep 10 23:36:47.277363 ip-10-0-138-128 kubenswrapper[2574]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 10 23:36:47.277363 ip-10-0-138-128 kubenswrapper[2574]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 10 23:36:47.277363 ip-10-0-138-128 kubenswrapper[2574]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 10 23:36:47.277363 ip-10-0-138-128 kubenswrapper[2574]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 10 23:36:47.277363 ip-10-0-138-128 kubenswrapper[2574]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 10 23:36:47.279026 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.278943 2574 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 10 23:36:47.283092 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.283076 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 23:36:47.283092 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.283093 2574 feature_gate.go:328] unrecognized feature gate: Example Sep 10 23:36:47.283155 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.283096 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 23:36:47.283155 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.283099 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 23:36:47.283155 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.283103 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 23:36:47.283155 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.283106 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 23:36:47.283155 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.283108 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 23:36:47.283155 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.283111 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 23:36:47.283155 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.283113 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 23:36:47.283155 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.283116 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 23:36:47.283155 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.283118 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 23:36:47.283155 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.283121 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 23:36:47.283155 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.283123 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 23:36:47.283155 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.283126 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 23:36:47.283155 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.283128 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 23:36:47.283155 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.283131 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 23:36:47.283155 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.283133 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 23:36:47.283155 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.283139 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 23:36:47.283155 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.283142 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 23:36:47.283155 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.283145 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 23:36:47.283155 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.283147 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 23:36:47.283155 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.283150 2574 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 23:36:47.283155 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.283153 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 23:36:47.283664 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.283156 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 23:36:47.283664 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.283160 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 23:36:47.283664 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.283164 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 23:36:47.283664 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.283167 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 23:36:47.283664 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.283170 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 23:36:47.283664 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.283173 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 23:36:47.283664 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.283176 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 23:36:47.283664 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.283178 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 23:36:47.283664 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.283181 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 23:36:47.283664 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.283184 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 23:36:47.283664 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.283186 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 23:36:47.283664 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.283189 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 23:36:47.283664 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.283191 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 23:36:47.283664 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.283194 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 23:36:47.283664 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.283196 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 23:36:47.283664 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.283199 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 23:36:47.283664 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.283202 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 23:36:47.283664 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.283205 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 23:36:47.283664 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.283207 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 23:36:47.283664 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.283211 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 23:36:47.284194 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.283214 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 23:36:47.284194 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.283217 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 23:36:47.284194 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.283220 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 23:36:47.284194 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.283222 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 23:36:47.284194 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.283225 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 23:36:47.284194 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.283228 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 23:36:47.284194 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.283230 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 23:36:47.284194 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.283233 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 23:36:47.284194 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.283235 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 23:36:47.284194 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.283238 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 23:36:47.284194 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.283242 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 23:36:47.284194 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.283246 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 23:36:47.284194 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.283249 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 23:36:47.284194 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.283254 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 23:36:47.284194 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.283257 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 23:36:47.284194 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.283259 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 23:36:47.284194 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.283262 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 23:36:47.284194 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.283264 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 23:36:47.284194 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.283267 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 23:36:47.284194 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.283270 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 23:36:47.284707 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.283275 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 23:36:47.284707 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.283279 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 23:36:47.284707 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.283282 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 23:36:47.284707 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.283285 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 23:36:47.284707 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.283288 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 23:36:47.284707 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.283291 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 23:36:47.284707 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.283293 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 23:36:47.284707 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.283296 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 23:36:47.284707 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.283299 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 23:36:47.284707 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.283302 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 23:36:47.284707 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.283306 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 23:36:47.284707 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.283309 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 23:36:47.284707 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.283312 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 23:36:47.284707 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.283315 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 23:36:47.284707 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.283318 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 23:36:47.284707 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.283321 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 23:36:47.284707 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.283323 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 23:36:47.284707 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.283326 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 23:36:47.285144 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.283330 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 23:36:47.285144 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.283332 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 23:36:47.285144 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.283335 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 23:36:47.285144 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.283338 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 23:36:47.285144 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.283340 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 23:36:47.285144 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.283343 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 23:36:47.285144 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.283346 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 23:36:47.285144 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.283349 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 23:36:47.285144 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.283352 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 23:36:47.285144 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.283355 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 23:36:47.285144 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.283358 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 23:36:47.285144 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.283360 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 23:36:47.285144 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.283363 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 23:36:47.285144 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.283365 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 23:36:47.285144 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.283368 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 23:36:47.285144 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.283370 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 23:36:47.285144 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.283373 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 23:36:47.285144 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.283376 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 23:36:47.285144 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284101 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 23:36:47.285627 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284108 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 23:36:47.285627 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284111 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 23:36:47.285627 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284114 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 23:36:47.285627 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284118 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 23:36:47.285627 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284120 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 23:36:47.285627 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284123 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 23:36:47.285627 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284125 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 23:36:47.285627 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284128 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 23:36:47.285627 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284131 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 23:36:47.285627 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284134 2574 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 23:36:47.285627 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284138 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 23:36:47.285627 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284142 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 23:36:47.285627 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284145 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 23:36:47.285627 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284148 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 23:36:47.285627 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284151 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 23:36:47.285627 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284154 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 23:36:47.285627 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284156 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 23:36:47.285627 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284159 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 23:36:47.285627 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284161 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 23:36:47.286099 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284166 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 23:36:47.286099 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284170 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 23:36:47.286099 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284176 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 23:36:47.286099 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284180 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 23:36:47.286099 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284183 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 23:36:47.286099 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284186 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 23:36:47.286099 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284189 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 23:36:47.286099 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284192 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 23:36:47.286099 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284195 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 23:36:47.286099 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284197 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 23:36:47.286099 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284200 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 23:36:47.286099 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284204 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 23:36:47.286099 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284208 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 23:36:47.286099 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284211 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 23:36:47.286099 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284214 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 23:36:47.286099 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284217 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 23:36:47.286099 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284220 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 23:36:47.286099 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284223 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 23:36:47.286099 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284225 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 23:36:47.286616 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284228 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 23:36:47.286616 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284231 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 23:36:47.286616 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284234 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 23:36:47.286616 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284239 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 23:36:47.286616 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284241 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 23:36:47.286616 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284245 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 23:36:47.286616 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284248 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 23:36:47.286616 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284251 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 23:36:47.286616 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284253 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 23:36:47.286616 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284256 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 23:36:47.286616 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284258 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 23:36:47.286616 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284260 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 23:36:47.286616 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284263 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 23:36:47.286616 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284267 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 23:36:47.286616 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284269 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 23:36:47.286616 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284272 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 23:36:47.286616 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284274 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 23:36:47.286616 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284277 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 23:36:47.286616 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284279 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 23:36:47.286616 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284282 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 23:36:47.287113 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284284 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 23:36:47.287113 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284287 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 23:36:47.287113 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284289 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 23:36:47.287113 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284291 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 23:36:47.287113 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284294 2574 feature_gate.go:328] unrecognized feature gate: Example Sep 10 23:36:47.287113 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284296 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 23:36:47.287113 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284299 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 23:36:47.287113 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284301 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 23:36:47.287113 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284303 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 23:36:47.287113 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284306 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 23:36:47.287113 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284308 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 23:36:47.287113 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284311 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 23:36:47.287113 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284314 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 23:36:47.287113 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284316 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 23:36:47.287113 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284319 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 23:36:47.287113 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284322 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 23:36:47.287113 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284326 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 23:36:47.287113 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284328 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 23:36:47.287113 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284331 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 23:36:47.287113 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284333 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 23:36:47.287615 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284336 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 23:36:47.287615 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284338 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 23:36:47.287615 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284341 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 23:36:47.287615 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284343 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 23:36:47.287615 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284346 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 23:36:47.287615 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284349 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 23:36:47.287615 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284351 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 23:36:47.287615 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284354 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 23:36:47.287615 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284357 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 23:36:47.287615 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284359 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 23:36:47.287615 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284362 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 23:36:47.287615 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284364 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 23:36:47.287615 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284367 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 23:36:47.287615 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284370 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 23:36:47.287615 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284372 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 23:36:47.287615 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284374 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 23:36:47.287615 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284377 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 23:36:47.287615 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284379 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 23:36:47.287615 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284382 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 23:36:47.287615 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284385 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 23:36:47.288099 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284778 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 23:36:47.288099 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284784 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 23:36:47.288099 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284786 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 23:36:47.288099 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284789 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 23:36:47.288099 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284792 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 23:36:47.288099 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284794 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 23:36:47.288099 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284798 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 23:36:47.288099 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284802 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 23:36:47.288099 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284806 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 23:36:47.288099 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284809 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 23:36:47.288099 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284812 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 23:36:47.288099 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284815 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 23:36:47.288099 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284817 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 23:36:47.288099 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284820 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 23:36:47.288099 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284823 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 23:36:47.288099 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284825 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 23:36:47.288099 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284828 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 23:36:47.288099 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284830 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 23:36:47.288099 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284834 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 23:36:47.288567 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284838 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 23:36:47.288567 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284841 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 23:36:47.288567 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284844 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 23:36:47.288567 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284847 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 23:36:47.288567 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284850 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 23:36:47.288567 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284853 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 23:36:47.288567 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284856 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 23:36:47.288567 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284859 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 23:36:47.288567 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284862 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 23:36:47.288567 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284865 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 23:36:47.288567 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284868 2574 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 23:36:47.288567 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284870 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 23:36:47.288567 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284873 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 23:36:47.288567 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284876 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 23:36:47.288567 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284879 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 23:36:47.288567 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284882 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 23:36:47.288567 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284884 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 23:36:47.288567 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284887 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 23:36:47.288567 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284889 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 23:36:47.288567 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284892 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 23:36:47.289069 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284894 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 23:36:47.289069 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284897 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 23:36:47.289069 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284902 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 23:36:47.289069 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284905 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 23:36:47.289069 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284907 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 23:36:47.289069 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284910 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 23:36:47.289069 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284912 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 23:36:47.289069 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284915 2574 feature_gate.go:328] unrecognized feature gate: Example Sep 10 23:36:47.289069 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284917 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 23:36:47.289069 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284919 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 23:36:47.289069 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284922 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 23:36:47.289069 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284924 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 23:36:47.289069 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284927 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 23:36:47.289069 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284929 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 23:36:47.289069 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284932 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 23:36:47.289069 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284934 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 23:36:47.289069 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284937 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 23:36:47.289069 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284940 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 23:36:47.289069 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284942 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 23:36:47.289069 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284945 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 23:36:47.289069 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284947 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 23:36:47.289591 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284950 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 23:36:47.289591 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284952 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 23:36:47.289591 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284955 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 23:36:47.289591 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284957 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 23:36:47.289591 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284960 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 23:36:47.289591 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284962 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 23:36:47.289591 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284965 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 23:36:47.289591 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284968 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 23:36:47.289591 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284970 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 23:36:47.289591 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284972 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 23:36:47.289591 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284975 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 23:36:47.289591 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284977 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 23:36:47.289591 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284980 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 23:36:47.289591 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284983 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 23:36:47.289591 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284986 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 23:36:47.289591 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284988 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 23:36:47.289591 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284991 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 23:36:47.289591 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284994 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 23:36:47.289591 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284996 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 23:36:47.289591 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.284998 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 23:36:47.290077 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.285001 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 23:36:47.290077 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.285005 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 23:36:47.290077 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.285008 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 23:36:47.290077 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.285011 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 23:36:47.290077 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.285014 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 23:36:47.290077 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.285017 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 23:36:47.290077 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.285019 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 23:36:47.290077 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.285022 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 23:36:47.290077 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.285025 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 23:36:47.290077 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.285029 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 23:36:47.290077 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.285032 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 23:36:47.290077 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.285034 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 23:36:47.290077 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.285037 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 23:36:47.290077 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.285039 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 23:36:47.290077 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.285042 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 23:36:47.290077 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.285045 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 23:36:47.290077 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.285047 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 23:36:47.290077 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.285049 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 23:36:47.290077 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.285052 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 23:36:47.290077 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285127 2574 flags.go:64] FLAG: --address="0.0.0.0" Sep 10 23:36:47.290584 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285160 2574 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 10 23:36:47.290584 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285166 2574 flags.go:64] FLAG: --anonymous-auth="true" Sep 10 23:36:47.290584 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285170 2574 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 10 23:36:47.290584 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285175 2574 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 10 23:36:47.290584 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285179 2574 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 10 23:36:47.290584 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285183 2574 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 10 23:36:47.290584 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285187 2574 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 10 23:36:47.290584 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285191 2574 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 10 23:36:47.290584 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285195 2574 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 10 23:36:47.290584 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285198 2574 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 10 23:36:47.290584 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285201 2574 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 10 23:36:47.290584 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285204 2574 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 10 23:36:47.290584 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285207 2574 flags.go:64] FLAG: --cgroup-root="" Sep 10 23:36:47.290584 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285210 2574 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 10 23:36:47.290584 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285213 2574 flags.go:64] FLAG: --client-ca-file="" Sep 10 23:36:47.290584 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285216 2574 flags.go:64] FLAG: --cloud-provider="external" Sep 10 23:36:47.290584 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285218 2574 flags.go:64] FLAG: --cluster-dns="[]" Sep 10 23:36:47.290584 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285222 2574 flags.go:64] FLAG: --cluster-domain="" Sep 10 23:36:47.290584 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285225 2574 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 10 23:36:47.290584 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285228 2574 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 10 23:36:47.290584 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285231 2574 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 10 23:36:47.290584 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285235 2574 flags.go:64] FLAG: --container-log-max-files="5" Sep 10 23:36:47.290584 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285239 2574 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 10 23:36:47.290584 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285242 2574 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 10 23:36:47.291163 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285245 2574 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 10 23:36:47.291163 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285248 2574 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 10 23:36:47.291163 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285252 2574 flags.go:64] FLAG: --contention-profiling="false" Sep 10 23:36:47.291163 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285255 2574 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 10 23:36:47.291163 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285258 2574 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 10 23:36:47.291163 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285261 2574 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 10 23:36:47.291163 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285264 2574 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 10 23:36:47.291163 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285268 2574 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 10 23:36:47.291163 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285271 2574 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 10 23:36:47.291163 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285274 2574 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 10 23:36:47.291163 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285277 2574 flags.go:64] FLAG: --enable-load-reader="false" Sep 10 23:36:47.291163 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285280 2574 flags.go:64] FLAG: --enable-server="true" Sep 10 23:36:47.291163 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285283 2574 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 10 23:36:47.291163 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285286 2574 flags.go:64] FLAG: --event-burst="100" Sep 10 23:36:47.291163 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285289 2574 flags.go:64] FLAG: --event-qps="50" Sep 10 23:36:47.291163 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285293 2574 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 10 23:36:47.291163 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285296 2574 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 10 23:36:47.291163 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285299 2574 flags.go:64] FLAG: --eviction-hard="" Sep 10 23:36:47.291163 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285303 2574 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 10 23:36:47.291163 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285306 2574 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 10 23:36:47.291163 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285309 2574 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 10 23:36:47.291163 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285312 2574 flags.go:64] FLAG: --eviction-soft="" Sep 10 23:36:47.291163 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285315 2574 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 10 23:36:47.291163 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285318 2574 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 10 23:36:47.291163 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285321 2574 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 10 23:36:47.292165 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285323 2574 flags.go:64] FLAG: --experimental-mounter-path="" Sep 10 23:36:47.292165 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285326 2574 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 10 23:36:47.292165 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285329 2574 flags.go:64] FLAG: --fail-swap-on="true" Sep 10 23:36:47.292165 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285332 2574 flags.go:64] FLAG: --feature-gates="" Sep 10 23:36:47.292165 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285336 2574 flags.go:64] FLAG: --file-check-frequency="20s" Sep 10 23:36:47.292165 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285339 2574 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 10 23:36:47.292165 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285342 2574 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 10 23:36:47.292165 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285346 2574 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 10 23:36:47.292165 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285348 2574 flags.go:64] FLAG: --healthz-port="10248" Sep 10 23:36:47.292165 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285354 2574 flags.go:64] FLAG: --help="false" Sep 10 23:36:47.292165 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285356 2574 flags.go:64] FLAG: --hostname-override="ip-10-0-138-128.ec2.internal" Sep 10 23:36:47.292165 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285360 2574 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 10 23:36:47.292165 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285363 2574 flags.go:64] FLAG: --http-check-frequency="20s" Sep 10 23:36:47.292165 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285366 2574 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 10 23:36:47.292165 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285369 2574 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 10 23:36:47.292165 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285372 2574 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 10 23:36:47.292165 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285375 2574 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 10 23:36:47.292165 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285378 2574 flags.go:64] FLAG: --image-service-endpoint="" Sep 10 23:36:47.292165 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285381 2574 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 10 23:36:47.292165 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285384 2574 flags.go:64] FLAG: --kube-api-burst="100" Sep 10 23:36:47.292165 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285387 2574 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 10 23:36:47.292165 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285390 2574 flags.go:64] FLAG: --kube-api-qps="50" Sep 10 23:36:47.292165 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285393 2574 flags.go:64] FLAG: --kube-reserved="" Sep 10 23:36:47.292165 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285396 2574 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 10 23:36:47.293191 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285399 2574 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 10 23:36:47.293191 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285402 2574 flags.go:64] FLAG: --kubelet-cgroups="" Sep 10 23:36:47.293191 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285405 2574 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 10 23:36:47.293191 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285408 2574 flags.go:64] FLAG: --lock-file="" Sep 10 23:36:47.293191 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285411 2574 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 10 23:36:47.293191 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285414 2574 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 10 23:36:47.293191 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285417 2574 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 10 23:36:47.293191 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285430 2574 flags.go:64] FLAG: --log-json-split-stream="false" Sep 10 23:36:47.293191 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285433 2574 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 10 23:36:47.293191 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285436 2574 flags.go:64] FLAG: --log-text-split-stream="false" Sep 10 23:36:47.293191 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285439 2574 flags.go:64] FLAG: --logging-format="text" Sep 10 23:36:47.293191 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285442 2574 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 10 23:36:47.293191 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285445 2574 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 10 23:36:47.293191 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285448 2574 flags.go:64] FLAG: --manifest-url="" Sep 10 23:36:47.293191 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285451 2574 flags.go:64] FLAG: --manifest-url-header="" Sep 10 23:36:47.293191 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285455 2574 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 10 23:36:47.293191 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285458 2574 flags.go:64] FLAG: --max-open-files="1000000" Sep 10 23:36:47.293191 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285464 2574 flags.go:64] FLAG: --max-pods="110" Sep 10 23:36:47.293191 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285467 2574 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 10 23:36:47.293191 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285470 2574 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 10 23:36:47.293191 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285473 2574 flags.go:64] FLAG: --memory-manager-policy="None" Sep 10 23:36:47.293191 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285476 2574 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 10 23:36:47.293191 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285479 2574 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 10 23:36:47.293191 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285482 2574 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 10 23:36:47.293191 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285485 2574 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 10 23:36:47.294078 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285492 2574 flags.go:64] FLAG: --node-status-max-images="50" Sep 10 23:36:47.294078 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285494 2574 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 10 23:36:47.294078 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285497 2574 flags.go:64] FLAG: --oom-score-adj="-999" Sep 10 23:36:47.294078 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285501 2574 flags.go:64] FLAG: --pod-cidr="" Sep 10 23:36:47.294078 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285504 2574 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10" Sep 10 23:36:47.294078 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285509 2574 flags.go:64] FLAG: --pod-manifest-path="" Sep 10 23:36:47.294078 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285512 2574 flags.go:64] FLAG: --pod-max-pids="-1" Sep 10 23:36:47.294078 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285515 2574 flags.go:64] FLAG: --pods-per-core="0" Sep 10 23:36:47.294078 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285519 2574 flags.go:64] FLAG: --port="10250" Sep 10 23:36:47.294078 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285522 2574 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 10 23:36:47.294078 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285525 2574 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-00dcc016334f470b5" Sep 10 23:36:47.294078 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285528 2574 flags.go:64] FLAG: --qos-reserved="" Sep 10 23:36:47.294078 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285531 2574 flags.go:64] FLAG: --read-only-port="10255" Sep 10 23:36:47.294078 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285534 2574 flags.go:64] FLAG: --register-node="true" Sep 10 23:36:47.294078 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285537 2574 flags.go:64] FLAG: --register-with-taints="" Sep 10 23:36:47.294078 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285541 2574 flags.go:64] FLAG: --registry-burst="10" Sep 10 23:36:47.294078 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285543 2574 flags.go:64] FLAG: --registry-qps="5" Sep 10 23:36:47.294078 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285546 2574 flags.go:64] FLAG: --reserved-cpus="" Sep 10 23:36:47.294078 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285549 2574 flags.go:64] FLAG: --reserved-memory="" Sep 10 23:36:47.294078 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285566 2574 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 10 23:36:47.294078 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285571 2574 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 10 23:36:47.294078 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285576 2574 flags.go:64] FLAG: --rotate-certificates="false" Sep 10 23:36:47.294078 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285580 2574 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 10 23:36:47.294078 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285584 2574 flags.go:64] FLAG: --runonce="false" Sep 10 23:36:47.294078 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285587 2574 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 10 23:36:47.294717 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285591 2574 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 10 23:36:47.294717 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285594 2574 flags.go:64] FLAG: --seccomp-default="false" Sep 10 23:36:47.294717 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285597 2574 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 10 23:36:47.294717 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285600 2574 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 10 23:36:47.294717 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285604 2574 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 10 23:36:47.294717 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285607 2574 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 10 23:36:47.294717 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285611 2574 flags.go:64] FLAG: --storage-driver-password="root" Sep 10 23:36:47.294717 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285614 2574 flags.go:64] FLAG: --storage-driver-secure="false" Sep 10 23:36:47.294717 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285616 2574 flags.go:64] FLAG: --storage-driver-table="stats" Sep 10 23:36:47.294717 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285619 2574 flags.go:64] FLAG: --storage-driver-user="root" Sep 10 23:36:47.294717 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285622 2574 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 10 23:36:47.294717 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285626 2574 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 10 23:36:47.294717 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285629 2574 flags.go:64] FLAG: --system-cgroups="" Sep 10 23:36:47.294717 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285632 2574 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 10 23:36:47.294717 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285637 2574 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 10 23:36:47.294717 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285641 2574 flags.go:64] FLAG: --tls-cert-file="" Sep 10 23:36:47.294717 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285644 2574 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 10 23:36:47.294717 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285648 2574 flags.go:64] FLAG: --tls-min-version="" Sep 10 23:36:47.294717 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285651 2574 flags.go:64] FLAG: --tls-private-key-file="" Sep 10 23:36:47.294717 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285654 2574 flags.go:64] FLAG: --topology-manager-policy="none" Sep 10 23:36:47.294717 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285657 2574 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 10 23:36:47.294717 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285659 2574 flags.go:64] FLAG: --topology-manager-scope="container" Sep 10 23:36:47.294717 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285662 2574 flags.go:64] FLAG: --v="2" Sep 10 23:36:47.294717 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285668 2574 flags.go:64] FLAG: --version="false" Sep 10 23:36:47.294717 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285672 2574 flags.go:64] FLAG: --vmodule="" Sep 10 23:36:47.295312 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285676 2574 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 10 23:36:47.295312 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.285679 2574 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 10 23:36:47.295312 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.285791 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 23:36:47.295312 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.285796 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 23:36:47.295312 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.285807 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 23:36:47.295312 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.285810 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 23:36:47.295312 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.285812 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 23:36:47.295312 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.285816 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 23:36:47.295312 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.285819 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 23:36:47.295312 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.285822 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 23:36:47.295312 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.285824 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 23:36:47.295312 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.285827 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 23:36:47.295312 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.285830 2574 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 23:36:47.295312 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.285832 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 23:36:47.295312 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.285835 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 23:36:47.295312 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.285837 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 23:36:47.295312 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.285840 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 23:36:47.295312 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.285843 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 23:36:47.295312 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.285845 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 23:36:47.295312 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.285848 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 23:36:47.295818 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.285850 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 23:36:47.295818 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.285853 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 23:36:47.295818 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.285856 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 23:36:47.295818 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.285858 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 23:36:47.295818 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.285863 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 23:36:47.295818 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.285866 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 23:36:47.295818 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.285869 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 23:36:47.295818 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.285872 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 23:36:47.295818 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.285875 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 23:36:47.295818 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.285877 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 23:36:47.295818 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.285882 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 23:36:47.295818 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.285885 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 23:36:47.295818 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.285888 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 23:36:47.295818 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.285891 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 23:36:47.295818 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.285894 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 23:36:47.295818 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.285896 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 23:36:47.295818 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.285899 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 23:36:47.295818 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.285901 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 23:36:47.295818 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.285904 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 23:36:47.295818 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.285908 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 23:36:47.296385 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.285911 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 23:36:47.296385 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.285914 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 23:36:47.296385 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.285918 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 23:36:47.296385 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.285921 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 23:36:47.296385 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.285924 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 23:36:47.296385 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.285927 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 23:36:47.296385 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.285929 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 23:36:47.296385 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.285932 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 23:36:47.296385 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.285935 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 23:36:47.296385 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.285937 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 23:36:47.296385 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.285941 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 23:36:47.296385 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.285944 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 23:36:47.296385 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.285947 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 23:36:47.296385 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.285950 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 23:36:47.296385 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.285952 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 23:36:47.296385 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.285955 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 23:36:47.296385 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.285957 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 23:36:47.296385 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.285960 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 23:36:47.296385 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.285963 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 23:36:47.296916 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.285966 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 23:36:47.296916 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.285968 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 23:36:47.296916 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.285971 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 23:36:47.296916 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.285975 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 23:36:47.296916 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.285977 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 23:36:47.296916 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.285980 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 23:36:47.296916 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.285982 2574 feature_gate.go:328] unrecognized feature gate: Example Sep 10 23:36:47.296916 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.285985 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 23:36:47.296916 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.285987 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 23:36:47.296916 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.285990 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 23:36:47.296916 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.285993 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 23:36:47.296916 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.285995 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 23:36:47.296916 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.285999 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 23:36:47.296916 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.286002 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 23:36:47.296916 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.286005 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 23:36:47.296916 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.286007 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 23:36:47.296916 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.286010 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 23:36:47.296916 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.286013 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 23:36:47.296916 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.286015 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 23:36:47.296916 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.286018 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 23:36:47.297412 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.286020 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 23:36:47.297412 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.286023 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 23:36:47.297412 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.286026 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 23:36:47.297412 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.286029 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 23:36:47.297412 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.286031 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 23:36:47.297412 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.286034 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 23:36:47.297412 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.286036 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 23:36:47.297412 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.286039 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 23:36:47.297412 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.286041 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 23:36:47.297412 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.286044 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 23:36:47.297412 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.286047 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 23:36:47.297412 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.286050 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 23:36:47.297412 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.286053 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 23:36:47.297412 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.286055 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 23:36:47.297412 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.286058 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 23:36:47.297412 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.286061 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 23:36:47.297412 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.286064 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 23:36:47.297412 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.286066 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 23:36:47.297412 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.286069 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 23:36:47.297412 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.286071 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 23:36:47.297957 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.286074 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 23:36:47.297957 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.286076 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 23:36:47.297957 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.286081 2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 10 23:36:47.297957 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.294241 2574 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 10 23:36:47.297957 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.294256 2574 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 10 23:36:47.297957 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.294961 2574 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 10 23:36:47.297957 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.294974 2574 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 10 23:36:47.297957 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295035 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 23:36:47.297957 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295041 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 23:36:47.297957 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295045 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 23:36:47.297957 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295048 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 23:36:47.297957 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295051 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 23:36:47.297957 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295055 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 23:36:47.297957 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295059 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 23:36:47.298328 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295063 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 23:36:47.298328 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295066 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 23:36:47.298328 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295070 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 23:36:47.298328 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295073 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 23:36:47.298328 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295076 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 23:36:47.298328 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295079 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 23:36:47.298328 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295082 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 23:36:47.298328 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295084 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 23:36:47.298328 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295087 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 23:36:47.298328 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295091 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 23:36:47.298328 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295096 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 23:36:47.298328 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295099 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 23:36:47.298328 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295102 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 23:36:47.298328 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295105 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 23:36:47.298328 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295108 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 23:36:47.298328 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295110 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 23:36:47.298328 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295123 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 23:36:47.298328 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295126 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 23:36:47.298807 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295130 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 23:36:47.298807 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295133 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 23:36:47.298807 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295136 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 23:36:47.298807 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295140 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 23:36:47.298807 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295143 2574 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 23:36:47.298807 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295146 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 23:36:47.298807 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295149 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 23:36:47.298807 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295152 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 23:36:47.298807 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295155 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 23:36:47.298807 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295157 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 23:36:47.298807 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295160 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 23:36:47.298807 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295162 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 23:36:47.298807 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295165 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 23:36:47.298807 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295168 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 23:36:47.298807 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295170 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 23:36:47.298807 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295173 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 23:36:47.298807 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295176 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 23:36:47.298807 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295178 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 23:36:47.298807 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295181 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 23:36:47.299513 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295184 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 23:36:47.299513 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295187 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 23:36:47.299513 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295190 2574 feature_gate.go:328] unrecognized feature gate: Example Sep 10 23:36:47.299513 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295192 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 23:36:47.299513 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295195 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 23:36:47.299513 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295197 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 23:36:47.299513 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295200 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 23:36:47.299513 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295202 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 23:36:47.299513 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295205 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 23:36:47.299513 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295207 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 23:36:47.299513 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295210 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 23:36:47.299513 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295213 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 23:36:47.299513 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295215 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 23:36:47.299513 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295218 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 23:36:47.299513 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295221 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 23:36:47.299513 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295223 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 23:36:47.299513 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295226 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 23:36:47.299513 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295228 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 23:36:47.299513 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295231 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 23:36:47.299513 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295233 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 23:36:47.300263 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295236 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 23:36:47.300263 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295239 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 23:36:47.300263 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295241 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 23:36:47.300263 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295244 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 23:36:47.300263 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295246 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 23:36:47.300263 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295249 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 23:36:47.300263 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295251 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 23:36:47.300263 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295254 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 23:36:47.300263 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295256 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 23:36:47.300263 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295259 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 23:36:47.300263 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295261 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 23:36:47.300263 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295264 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 23:36:47.300263 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295267 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 23:36:47.300263 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295269 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 23:36:47.300263 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295272 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 23:36:47.300263 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295276 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 23:36:47.300263 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295279 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 23:36:47.300263 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295282 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 23:36:47.300263 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295284 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 23:36:47.300263 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295286 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 23:36:47.300882 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295289 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 23:36:47.300882 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295292 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 23:36:47.300882 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295294 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 23:36:47.300882 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295297 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 23:36:47.300882 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295299 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 23:36:47.300882 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295302 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 23:36:47.300882 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295304 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 23:36:47.300882 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295307 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 23:36:47.300882 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295310 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 23:36:47.300882 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295312 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 23:36:47.300882 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295315 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 23:36:47.300882 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295318 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 23:36:47.300882 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295321 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 23:36:47.300882 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295324 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 23:36:47.300882 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295327 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 23:36:47.301278 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.295332 2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 10 23:36:47.301278 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295453 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 23:36:47.301278 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295458 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 23:36:47.301278 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295461 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 23:36:47.301278 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295464 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 23:36:47.301278 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295467 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 23:36:47.301278 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295470 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 23:36:47.301278 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295473 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 23:36:47.301278 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295476 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 23:36:47.301278 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295479 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 23:36:47.301278 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295481 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 23:36:47.301278 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295484 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 23:36:47.301278 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295486 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 23:36:47.301278 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295490 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 23:36:47.301278 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295492 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 23:36:47.301727 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295495 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 23:36:47.301727 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295498 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 23:36:47.301727 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295501 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 23:36:47.301727 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295504 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 23:36:47.301727 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295506 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 23:36:47.301727 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295509 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 23:36:47.301727 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295511 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 23:36:47.301727 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295514 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 23:36:47.301727 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295517 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 23:36:47.301727 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295519 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 23:36:47.301727 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295522 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 23:36:47.301727 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295524 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 23:36:47.301727 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295527 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 23:36:47.301727 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295530 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 23:36:47.301727 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295532 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 23:36:47.301727 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295535 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 23:36:47.301727 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295537 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 23:36:47.301727 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295540 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 23:36:47.301727 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295542 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 23:36:47.301727 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295545 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 23:36:47.302250 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295548 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 23:36:47.302250 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295550 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 23:36:47.302250 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295568 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 23:36:47.302250 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295572 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 23:36:47.302250 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295577 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 23:36:47.302250 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295580 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 23:36:47.302250 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295583 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 23:36:47.302250 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295585 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 23:36:47.302250 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295588 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 23:36:47.302250 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295591 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 23:36:47.302250 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295594 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 23:36:47.302250 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295599 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 23:36:47.302250 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295603 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 23:36:47.302250 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295606 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 23:36:47.302250 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295609 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 23:36:47.302250 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295612 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 23:36:47.302250 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295615 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 23:36:47.302250 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295617 2574 feature_gate.go:328] unrecognized feature gate: Example Sep 10 23:36:47.302250 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295620 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 23:36:47.302250 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295622 2574 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 23:36:47.302766 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295626 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 23:36:47.302766 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295630 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 23:36:47.302766 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295634 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 23:36:47.302766 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295637 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 23:36:47.302766 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295640 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 23:36:47.302766 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295643 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 23:36:47.302766 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295646 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 23:36:47.302766 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295649 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 23:36:47.302766 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295652 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 23:36:47.302766 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295655 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 23:36:47.302766 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295658 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 23:36:47.302766 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295661 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 23:36:47.302766 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295664 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 23:36:47.302766 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295666 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 23:36:47.302766 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295669 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 23:36:47.302766 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295672 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 23:36:47.302766 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295674 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 23:36:47.302766 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295681 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 23:36:47.302766 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295683 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 23:36:47.303299 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295686 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 23:36:47.303299 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295689 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 23:36:47.303299 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295691 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 23:36:47.303299 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295694 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 23:36:47.303299 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295696 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 23:36:47.303299 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295700 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 23:36:47.303299 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295703 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 23:36:47.303299 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295705 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 23:36:47.303299 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295708 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 23:36:47.303299 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295711 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 23:36:47.303299 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295714 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 23:36:47.303299 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295716 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 23:36:47.303299 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295719 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 23:36:47.303299 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295722 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 23:36:47.303299 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295724 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 23:36:47.303299 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295727 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 23:36:47.303299 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295730 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 23:36:47.303299 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295732 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 23:36:47.303299 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295735 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 23:36:47.303299 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295737 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 23:36:47.303957 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295740 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 23:36:47.303957 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295742 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 23:36:47.303957 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295745 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 23:36:47.303957 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295747 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 23:36:47.303957 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295750 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 23:36:47.303957 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:47.295753 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 23:36:47.303957 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.295758 2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 10 23:36:47.303957 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.295864 2574 server.go:962] "Client rotation is on, will bootstrap in background" Sep 10 23:36:47.303957 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.297793 2574 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 10 23:36:47.303957 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.298854 2574 server.go:1019] "Starting client certificate rotation" Sep 10 23:36:47.303957 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.298966 2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 10 23:36:47.303957 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.299018 2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 10 23:36:47.303957 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.301198 2574 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 10 23:36:47.304349 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.302393 2574 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 10 23:36:47.312549 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.312530 2574 log.go:25] "Validated CRI v1 runtime API" Sep 10 23:36:47.316140 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.316127 2574 log.go:25] "Validated CRI v1 image API" Sep 10 23:36:47.317134 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.317119 2574 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 10 23:36:47.319655 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.319635 2574 fs.go:135] Filesystem UUIDs: map[4c8eec58-b878-4247-ad24-d04b60431db4:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 d8691948-cb88-4a53-b1ef-e1280fe18d49:/dev/nvme0n1p4] Sep 10 23:36:47.319713 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.319654 2574 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 10 23:36:47.325598 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.325477 2574 manager.go:216] Machine: {Timestamp:2026-09-10 23:36:47.324147918 +0000 UTC m=+0.316834718 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101918 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2ddec32f3ad24d016da5877f7549a8 SystemUUID:ec2ddec3-2f3a-d24d-016d-a5877f7549a8 BootID:0ebbd10a-0883-4cd6-b21b-d53ead7bf2d1 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:4d:ad:e1:b4:af Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:4d:ad:e1:b4:af Speed:0 Mtu:9001} {Name:ovs-system MacAddress:be:ec:29:00:ef:5f Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 10 23:36:47.325598 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.325594 2574 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 10 23:36:47.325726 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.325700 2574 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 10 23:36:47.326391 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.326368 2574 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 10 23:36:47.329276 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.329090 2574 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 10 23:36:47.329414 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.329281 2574 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-128.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 10 23:36:47.329956 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.329945 2574 topology_manager.go:138] "Creating topology manager with none policy" Sep 10 23:36:47.329989 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.329958 2574 container_manager_linux.go:309] "Creating device plugin manager" Sep 10 23:36:47.329989 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.329966 2574 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 10 23:36:47.330534 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.330524 2574 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 10 23:36:47.331289 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.331280 2574 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 10 23:36:47.334992 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.334981 2574 state_mem.go:36] "Initialized new in-memory state store" Sep 10 23:36:47.335112 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.335103 2574 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 10 23:36:47.335147 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.335133 2574 kubelet.go:486] "Attempting to sync node with API server" Sep 10 23:36:47.335147 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.335143 2574 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 10 23:36:47.335198 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.335157 2574 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 10 23:36:47.335198 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.335166 2574 kubelet.go:398] "Adding apiserver pod source" Sep 10 23:36:47.335198 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.335175 2574 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 10 23:36:47.335525 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.335514 2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 10 23:36:47.335591 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.335546 2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 10 23:36:47.337663 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.337647 2574 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 10 23:36:47.339727 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.339710 2574 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 10 23:36:47.339804 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.339748 2574 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 10 23:36:47.341314 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.341299 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 10 23:36:47.341386 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.341320 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 10 23:36:47.341386 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.341330 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 10 23:36:47.341386 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.341338 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 10 23:36:47.341386 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.341351 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 10 23:36:47.341386 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.341360 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 10 23:36:47.341386 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.341379 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 10 23:36:47.341386 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.341387 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 10 23:36:47.341641 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.341397 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 10 23:36:47.341641 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.341405 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 10 23:36:47.341641 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.341417 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 10 23:36:47.341641 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.341426 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 10 23:36:47.342076 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.342064 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 10 23:36:47.342130 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.342079 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 10 23:36:47.345782 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.345766 2574 server.go:1268] "Started kubelet" Sep 10 23:36:47.345849 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.345826 2574 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 10 23:36:47.346007 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.345911 2574 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 10 23:36:47.346095 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.346034 2574 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 10 23:36:47.346967 ip-10-0-138-128 systemd[1]: Started Kubernetes Kubelet. Sep 10 23:36:47.347314 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.347297 2574 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 10 23:36:47.347723 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.347688 2574 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-128.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 10 23:36:47.348133 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.348120 2574 server.go:310] "Adding debug handlers to kubelet server" Sep 10 23:36:47.348249 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:36:47.348229 2574 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-128.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 10 23:36:47.348302 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:36:47.348233 2574 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 10 23:36:47.351873 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.351854 2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 10 23:36:47.352474 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.352453 2574 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 10 23:36:47.352602 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:36:47.351655 2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-128.ec2.internal.18d419e148636b02 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-128.ec2.internal,UID:ip-10-0-138-128.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-128.ec2.internal,},FirstTimestamp:2026-09-10 23:36:47.345740546 +0000 UTC m=+0.338427345,LastTimestamp:2026-09-10 23:36:47.345740546 +0000 UTC m=+0.338427345,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-128.ec2.internal,}" Sep 10 23:36:47.352878 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:36:47.352856 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-128.ec2.internal\" not found" Sep 10 23:36:47.353209 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.353139 2574 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 10 23:36:47.353209 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.353161 2574 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 10 23:36:47.353328 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.353137 2574 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 10 23:36:47.354009 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.353990 2574 factory.go:55] Registering systemd factory Sep 10 23:36:47.354070 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.354014 2574 factory.go:223] Registration of the systemd container factory successfully Sep 10 23:36:47.354343 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.354329 2574 factory.go:153] Registering CRI-O factory Sep 10 23:36:47.354402 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.354376 2574 factory.go:223] Registration of the crio container factory successfully Sep 10 23:36:47.354509 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.354497 2574 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 10 23:36:47.354610 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.354519 2574 factory.go:103] Registering Raw factory Sep 10 23:36:47.354610 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.354533 2574 manager.go:1196] Started watching for new ooms in manager Sep 10 23:36:47.354932 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.354914 2574 manager.go:349] Starting recovery of all containers Sep 10 23:36:47.356592 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.354329 2574 reconstruct.go:97] "Volume reconstruction finished" Sep 10 23:36:47.356592 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.356242 2574 reconciler.go:29] "Reconciler: start to sync state" Sep 10 23:36:47.359497 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.359475 2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-fn2bj" Sep 10 23:36:47.361129 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:36:47.361099 2574 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 10 23:36:47.365934 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.365913 2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-fn2bj" Sep 10 23:36:47.366076 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:36:47.366056 2574 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-128.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 10 23:36:47.367192 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.367174 2574 manager.go:354] Recovery completed Sep 10 23:36:47.372042 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.372025 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 23:36:47.373549 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.373533 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-128.ec2.internal" event="NodeHasSufficientMemory" Sep 10 23:36:47.373653 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.373574 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-128.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 23:36:47.373653 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.373586 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-128.ec2.internal" event="NodeHasSufficientPID" Sep 10 23:36:47.374146 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.374132 2574 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 10 23:36:47.374146 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.374146 2574 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 10 23:36:47.374265 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.374149 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 23:36:47.374265 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.374161 2574 state_mem.go:36] "Initialized new in-memory state store" Sep 10 23:36:47.374793 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.374778 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-128.ec2.internal" event="NodeHasSufficientMemory" Sep 10 23:36:47.374854 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.374803 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-128.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 23:36:47.374854 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.374812 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-128.ec2.internal" event="NodeHasSufficientPID" Sep 10 23:36:47.375611 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:36:47.375506 2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-128.ec2.internal.18d419e14a0bb318 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-128.ec2.internal,UID:ip-10-0-138-128.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-138-128.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-138-128.ec2.internal,},FirstTimestamp:2026-09-10 23:36:47.373546264 +0000 UTC m=+0.366233063,LastTimestamp:2026-09-10 23:36:47.373546264 +0000 UTC m=+0.366233063,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-128.ec2.internal,}" Sep 10 23:36:47.375987 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.375974 2574 policy_none.go:49] "None policy: Start" Sep 10 23:36:47.376039 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.375994 2574 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 10 23:36:47.376039 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.376008 2574 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 10 23:36:47.377921 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.377908 2574 policy_none.go:47] "Start" Sep 10 23:36:47.434060 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.414197 2574 manager.go:337] "Starting Device Plugin manager" Sep 10 23:36:47.434060 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:36:47.414230 2574 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 10 23:36:47.434060 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.414244 2574 server.go:87] "Starting device plugin registration server" Sep 10 23:36:47.434060 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.414498 2574 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 10 23:36:47.434060 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.414508 2574 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 10 23:36:47.434060 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.414627 2574 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 10 23:36:47.434060 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.414695 2574 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 10 23:36:47.434060 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.414703 2574 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 10 23:36:47.434060 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:36:47.415139 2574 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 10 23:36:47.434060 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:36:47.415190 2574 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-128.ec2.internal\" not found" Sep 10 23:36:47.480416 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.480370 2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 10 23:36:47.481762 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.481746 2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 10 23:36:47.481859 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.481797 2574 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 10 23:36:47.481859 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.481828 2574 kubelet.go:2448] "Starting kubelet main sync loop" Sep 10 23:36:47.481951 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:36:47.481872 2574 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 10 23:36:47.484427 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.484408 2574 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 23:36:47.515410 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.515383 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 23:36:47.516285 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.516266 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-128.ec2.internal" event="NodeHasSufficientMemory" Sep 10 23:36:47.516376 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.516300 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-128.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 23:36:47.516376 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.516313 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-128.ec2.internal" event="NodeHasSufficientPID" Sep 10 23:36:47.516376 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.516341 2574 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-128.ec2.internal" Sep 10 23:36:47.525092 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.525075 2574 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-138-128.ec2.internal" Sep 10 23:36:47.525144 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:36:47.525102 2574 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-128.ec2.internal\": node \"ip-10-0-138-128.ec2.internal\" not found" Sep 10 23:36:47.541320 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:36:47.541246 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-128.ec2.internal\" not found" Sep 10 23:36:47.582352 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.582320 2574 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-128.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-128.ec2.internal"] Sep 10 23:36:47.582411 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.582392 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 23:36:47.583244 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.583229 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-128.ec2.internal" event="NodeHasSufficientMemory" Sep 10 23:36:47.583328 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.583261 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-128.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 23:36:47.583328 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.583275 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-128.ec2.internal" event="NodeHasSufficientPID" Sep 10 23:36:47.585500 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.585485 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 23:36:47.585643 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.585628 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-128.ec2.internal" Sep 10 23:36:47.585690 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.585660 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 23:36:47.586219 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.586204 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-128.ec2.internal" event="NodeHasSufficientMemory" Sep 10 23:36:47.586277 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.586228 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-128.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 23:36:47.586277 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.586234 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-128.ec2.internal" event="NodeHasSufficientMemory" Sep 10 23:36:47.586277 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.586256 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-128.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 23:36:47.586277 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.586270 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-128.ec2.internal" event="NodeHasSufficientPID" Sep 10 23:36:47.586393 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.586238 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-128.ec2.internal" event="NodeHasSufficientPID" Sep 10 23:36:47.588440 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.588424 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-128.ec2.internal" Sep 10 23:36:47.588498 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.588450 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 23:36:47.589106 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.589090 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-128.ec2.internal" event="NodeHasSufficientMemory" Sep 10 23:36:47.589185 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.589114 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-128.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 23:36:47.589185 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.589123 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-128.ec2.internal" event="NodeHasSufficientPID" Sep 10 23:36:47.612191 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:36:47.612172 2574 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-128.ec2.internal\" not found" node="ip-10-0-138-128.ec2.internal" Sep 10 23:36:47.616439 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:36:47.616424 2574 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-128.ec2.internal\" not found" node="ip-10-0-138-128.ec2.internal" Sep 10 23:36:47.641969 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:36:47.641951 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-128.ec2.internal\" not found" Sep 10 23:36:47.657734 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.657715 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/577bf0a7d48f3dd63acd7590896a721f-config\") pod \"kube-apiserver-proxy-ip-10-0-138-128.ec2.internal\" (UID: \"577bf0a7d48f3dd63acd7590896a721f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-128.ec2.internal" Sep 10 23:36:47.657794 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.657738 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ebf313d069610fccaa30accef842e07e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-128.ec2.internal\" (UID: \"ebf313d069610fccaa30accef842e07e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-128.ec2.internal" Sep 10 23:36:47.657794 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.657756 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ebf313d069610fccaa30accef842e07e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-128.ec2.internal\" (UID: \"ebf313d069610fccaa30accef842e07e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-128.ec2.internal" Sep 10 23:36:47.742993 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:36:47.742960 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-128.ec2.internal\" not found" Sep 10 23:36:47.758341 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.758320 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ebf313d069610fccaa30accef842e07e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-128.ec2.internal\" (UID: \"ebf313d069610fccaa30accef842e07e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-128.ec2.internal" Sep 10 23:36:47.758428 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.758354 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ebf313d069610fccaa30accef842e07e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-128.ec2.internal\" (UID: \"ebf313d069610fccaa30accef842e07e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-128.ec2.internal" Sep 10 23:36:47.758428 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.758379 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/577bf0a7d48f3dd63acd7590896a721f-config\") pod \"kube-apiserver-proxy-ip-10-0-138-128.ec2.internal\" (UID: \"577bf0a7d48f3dd63acd7590896a721f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-128.ec2.internal" Sep 10 23:36:47.758428 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.758405 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ebf313d069610fccaa30accef842e07e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-128.ec2.internal\" (UID: \"ebf313d069610fccaa30accef842e07e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-128.ec2.internal" Sep 10 23:36:47.758537 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.758432 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ebf313d069610fccaa30accef842e07e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-128.ec2.internal\" (UID: \"ebf313d069610fccaa30accef842e07e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-128.ec2.internal" Sep 10 23:36:47.758537 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.758433 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/577bf0a7d48f3dd63acd7590896a721f-config\") pod \"kube-apiserver-proxy-ip-10-0-138-128.ec2.internal\" (UID: \"577bf0a7d48f3dd63acd7590896a721f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-128.ec2.internal" Sep 10 23:36:47.843818 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:36:47.843741 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-128.ec2.internal\" not found" Sep 10 23:36:47.915218 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.915188 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-128.ec2.internal" Sep 10 23:36:47.919102 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:47.919085 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-128.ec2.internal" Sep 10 23:36:47.945659 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:36:47.944948 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-128.ec2.internal\" not found" Sep 10 23:36:48.045696 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:36:48.045663 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-128.ec2.internal\" not found" Sep 10 23:36:49.751305 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:36:49.751232 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-128.ec2.internal\" not found" Sep 10 23:36:49.851871 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:36:49.851833 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-128.ec2.internal\" not found" Sep 10 23:36:49.904303 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:49.904278 2574 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 10 23:36:49.904735 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:49.904423 2574 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 10 23:36:49.952829 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:36:49.952808 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-128.ec2.internal\" not found" Sep 10 23:36:49.976173 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:49.976131 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-09 23:31:48 +0000 UTC" deadline="2028-03-23 21:15:40.485191505 +0000 UTC" Sep 10 23:36:49.976173 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:49.976162 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13437h38m50.50903293s" Sep 10 23:36:50.052094 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:50.052020 2574 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 23:36:50.058761 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:50.058739 2574 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-128.ec2.internal" Sep 10 23:36:50.069183 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:50.069165 2574 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 10 23:36:50.070961 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:50.070947 2574 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-128.ec2.internal" Sep 10 23:36:50.080699 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:50.080682 2574 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 10 23:36:50.102888 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:50.102855 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebf313d069610fccaa30accef842e07e.slice/crio-97e9869442b10442d82c1a66a7b49e6bde93e3bd64e8d8f77e5538ff2b215d41 WatchSource:0}: Error finding container 97e9869442b10442d82c1a66a7b49e6bde93e3bd64e8d8f77e5538ff2b215d41: Status 404 returned error can't find the container with id 97e9869442b10442d82c1a66a7b49e6bde93e3bd64e8d8f77e5538ff2b215d41 Sep 10 23:36:50.103122 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:50.103100 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod577bf0a7d48f3dd63acd7590896a721f.slice/crio-0b6ba203dc441d90a506dbccfd3af9f39ce475e93b9cab01a2d01539d174c818 WatchSource:0}: Error finding container 0b6ba203dc441d90a506dbccfd3af9f39ce475e93b9cab01a2d01539d174c818: Status 404 returned error can't find the container with id 0b6ba203dc441d90a506dbccfd3af9f39ce475e93b9cab01a2d01539d174c818 Sep 10 23:36:50.109372 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:50.109356 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 23:36:50.240193 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:50.240164 2574 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 23:36:50.456150 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:50.456072 2574 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 23:36:50.941462 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:50.941383 2574 apiserver.go:52] "Watching apiserver" Sep 10 23:36:50.949031 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:50.949001 2574 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 10 23:36:50.949576 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:50.949549 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-v87ts","openshift-cluster-node-tuning-operator/tuned-4fp98","openshift-dns/node-resolver-6wz5g","openshift-image-registry/node-ca-fckpk","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-128.ec2.internal","openshift-multus/multus-hnjsh","openshift-network-diagnostics/network-check-target-zb6x4","openshift-network-operator/iptables-alerter-dds4w","openshift-ovn-kubernetes/ovnkube-node-zrjmz","kube-system/global-pull-secret-syncer-75vlp","kube-system/konnectivity-agent-dl8bc","kube-system/kube-apiserver-proxy-ip-10-0-138-128.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rsrtp","openshift-multus/multus-additional-cni-plugins-f47cx"] Sep 10 23:36:50.952120 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:50.952099 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zb6x4" Sep 10 23:36:50.952230 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:36:50.952198 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zb6x4" podUID="24dda81e-d88e-4f29-836b-44bb86bf461d" Sep 10 23:36:50.954301 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:50.954280 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-dl8bc" Sep 10 23:36:50.956281 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:50.956257 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 10 23:36:50.956281 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:50.956276 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 10 23:36:50.956430 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:50.956262 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-2hpfs\"" Sep 10 23:36:50.956549 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:50.956532 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6wz5g" Sep 10 23:36:50.957728 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:50.957501 2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 10 23:36:50.958743 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:50.958707 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-fckpk" Sep 10 23:36:50.959463 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:50.959443 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 10 23:36:50.959693 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:50.959676 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 10 23:36:50.960034 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:50.960016 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-mwmdk\"" Sep 10 23:36:50.960949 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:50.960934 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-f47cx" Sep 10 23:36:50.962480 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:50.962457 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 10 23:36:50.962927 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:50.962902 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 10 23:36:50.964248 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:50.963134 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-hnjsh" Sep 10 23:36:50.964248 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:50.963672 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 10 23:36:50.964248 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:50.963891 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-wdj8v\"" Sep 10 23:36:50.964248 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:50.964097 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 10 23:36:50.964248 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:50.964132 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 10 23:36:50.964248 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:50.964191 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 10 23:36:50.964584 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:50.964351 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 10 23:36:50.965515 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:50.964716 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-x9st2\"" Sep 10 23:36:50.965515 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:50.965287 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-cj4dq\"" Sep 10 23:36:50.965515 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:50.965501 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 10 23:36:50.965792 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:50.965768 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 10 23:36:50.966033 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:50.965877 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-zrjmz" Sep 10 23:36:50.968105 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:50.967964 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rsrtp" Sep 10 23:36:50.969739 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:50.969719 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 10 23:36:50.972678 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:50.971518 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 10 23:36:50.972678 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:50.971535 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-4fp98" Sep 10 23:36:50.972678 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:50.972158 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 10 23:36:50.972678 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:50.972233 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-vkhdv\"" Sep 10 23:36:50.972678 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:50.972305 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 10 23:36:50.972954 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:50.972715 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 10 23:36:50.972954 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:50.972804 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 10 23:36:50.973258 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:50.973243 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 10 23:36:50.973436 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:50.973419 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 10 23:36:50.974023 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:50.973675 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 10 23:36:50.974023 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:50.973709 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-t5ldc\"" Sep 10 23:36:50.974023 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:50.973757 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 10 23:36:50.974337 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:50.974249 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5d6d9866-cf11-4f49-877a-ecc951c01e42-hosts-file\") pod \"node-resolver-6wz5g\" (UID: \"5d6d9866-cf11-4f49-877a-ecc951c01e42\") " pod="openshift-dns/node-resolver-6wz5g" Sep 10 23:36:50.974337 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:50.974289 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d6d9866-cf11-4f49-877a-ecc951c01e42-tmp-dir\") pod \"node-resolver-6wz5g\" (UID: \"5d6d9866-cf11-4f49-877a-ecc951c01e42\") " pod="openshift-dns/node-resolver-6wz5g" Sep 10 23:36:50.974337 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:50.974333 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cbaa689c-2a82-4b1a-b61f-4e8f43efa7d5-serviceca\") pod \"node-ca-fckpk\" (UID: \"cbaa689c-2a82-4b1a-b61f-4e8f43efa7d5\") " pod="openshift-image-registry/node-ca-fckpk" Sep 10 23:36:50.974491 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:50.974398 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfh7k\" (UniqueName: \"kubernetes.io/projected/5d6d9866-cf11-4f49-877a-ecc951c01e42-kube-api-access-xfh7k\") pod \"node-resolver-6wz5g\" (UID: \"5d6d9866-cf11-4f49-877a-ecc951c01e42\") " pod="openshift-dns/node-resolver-6wz5g" Sep 10 23:36:50.975895 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:50.975878 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 10 23:36:50.976058 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:50.976017 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cbaa689c-2a82-4b1a-b61f-4e8f43efa7d5-host\") pod \"node-ca-fckpk\" (UID: \"cbaa689c-2a82-4b1a-b61f-4e8f43efa7d5\") " pod="openshift-image-registry/node-ca-fckpk" Sep 10 23:36:50.976132 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:50.976064 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-hb9kx\"" Sep 10 23:36:50.976132 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:50.976084 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dlhf\" (UniqueName: \"kubernetes.io/projected/cbaa689c-2a82-4b1a-b61f-4e8f43efa7d5-kube-api-access-5dlhf\") pod \"node-ca-fckpk\" (UID: \"cbaa689c-2a82-4b1a-b61f-4e8f43efa7d5\") " pod="openshift-image-registry/node-ca-fckpk" Sep 10 23:36:50.976132 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:50.976128 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7wk9\" (UniqueName: \"kubernetes.io/projected/24dda81e-d88e-4f29-836b-44bb86bf461d-kube-api-access-s7wk9\") pod \"network-check-target-zb6x4\" (UID: \"24dda81e-d88e-4f29-836b-44bb86bf461d\") " pod="openshift-network-diagnostics/network-check-target-zb6x4" Sep 10 23:36:50.976305 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:50.976194 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7ad8f728-76e5-47a2-8ab2-681ac07591f8-agent-certs\") pod \"konnectivity-agent-dl8bc\" (UID: \"7ad8f728-76e5-47a2-8ab2-681ac07591f8\") " pod="kube-system/konnectivity-agent-dl8bc" Sep 10 23:36:50.976305 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:50.976235 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7ad8f728-76e5-47a2-8ab2-681ac07591f8-konnectivity-ca\") pod \"konnectivity-agent-dl8bc\" (UID: \"7ad8f728-76e5-47a2-8ab2-681ac07591f8\") " pod="kube-system/konnectivity-agent-dl8bc" Sep 10 23:36:50.977019 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:50.976997 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-75vlp" Sep 10 23:36:50.977119 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:36:50.977080 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-75vlp" podUID="3a0ec520-b33d-49cf-9714-1662f5910e5b" Sep 10 23:36:50.980439 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:50.980367 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v87ts" Sep 10 23:36:50.980527 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:36:50.980508 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v87ts" podUID="6df7e606-319b-4b9a-b6da-a850ecc9d267" Sep 10 23:36:50.982941 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:50.982923 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-dds4w" Sep 10 23:36:50.988622 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:50.988555 2574 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 10 23:36:50.988931 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:50.988912 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 10 23:36:50.989003 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:50.988961 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 10 23:36:50.990179 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:50.989942 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 10 23:36:50.990179 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:50.990147 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-6n45x\"" Sep 10 23:36:51.012497 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.012473 2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-jkpdq" Sep 10 23:36:51.021142 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.021122 2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-jkpdq" Sep 10 23:36:51.059421 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.059401 2574 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 10 23:36:51.076792 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.076768 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gqgt\" (UniqueName: \"kubernetes.io/projected/c90ea9b8-efa1-423c-8d26-e4755af78f53-kube-api-access-8gqgt\") pod \"ovnkube-node-zrjmz\" (UID: \"c90ea9b8-efa1-423c-8d26-e4755af78f53\") " pod="openshift-ovn-kubernetes/ovnkube-node-zrjmz" Sep 10 23:36:51.076901 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.076805 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3ced5f14-cf96-419c-af9b-c218e93b0365-etc-modprobe-d\") pod \"tuned-4fp98\" (UID: \"3ced5f14-cf96-419c-af9b-c218e93b0365\") " pod="openshift-cluster-node-tuning-operator/tuned-4fp98" Sep 10 23:36:51.076901 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.076829 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3ced5f14-cf96-419c-af9b-c218e93b0365-sys\") pod \"tuned-4fp98\" (UID: \"3ced5f14-cf96-419c-af9b-c218e93b0365\") " pod="openshift-cluster-node-tuning-operator/tuned-4fp98" Sep 10 23:36:51.076901 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.076857 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ed113529-7712-4e4f-838a-835f3d6cf1b0-host-run-k8s-cni-cncf-io\") pod \"multus-hnjsh\" (UID: \"ed113529-7712-4e4f-838a-835f3d6cf1b0\") " pod="openshift-multus/multus-hnjsh" Sep 10 23:36:51.076901 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.076875 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/43e0f2f1-cd70-4612-828f-6cd98e1b77c2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-f47cx\" (UID: \"43e0f2f1-cd70-4612-828f-6cd98e1b77c2\") " pod="openshift-multus/multus-additional-cni-plugins-f47cx" Sep 10 23:36:51.076901 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.076898 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3ced5f14-cf96-419c-af9b-c218e93b0365-etc-kubernetes\") pod \"tuned-4fp98\" (UID: \"3ced5f14-cf96-419c-af9b-c218e93b0365\") " pod="openshift-cluster-node-tuning-operator/tuned-4fp98" Sep 10 23:36:51.077103 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.076941 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3ced5f14-cf96-419c-af9b-c218e93b0365-etc-tuned\") pod \"tuned-4fp98\" (UID: \"3ced5f14-cf96-419c-af9b-c218e93b0365\") " pod="openshift-cluster-node-tuning-operator/tuned-4fp98" Sep 10 23:36:51.077103 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.076968 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8zw6\" (UniqueName: \"kubernetes.io/projected/ed113529-7712-4e4f-838a-835f3d6cf1b0-kube-api-access-p8zw6\") pod \"multus-hnjsh\" (UID: \"ed113529-7712-4e4f-838a-835f3d6cf1b0\") " pod="openshift-multus/multus-hnjsh" Sep 10 23:36:51.077103 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.077028 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4d311f9d-1424-4282-9fd5-5b788bdc4cac-socket-dir\") pod \"aws-ebs-csi-driver-node-rsrtp\" (UID: \"4d311f9d-1424-4282-9fd5-5b788bdc4cac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rsrtp" Sep 10 23:36:51.077103 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.077064 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4d311f9d-1424-4282-9fd5-5b788bdc4cac-device-dir\") pod \"aws-ebs-csi-driver-node-rsrtp\" (UID: \"4d311f9d-1424-4282-9fd5-5b788bdc4cac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rsrtp" Sep 10 23:36:51.077103 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.077089 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4d311f9d-1424-4282-9fd5-5b788bdc4cac-etc-selinux\") pod \"aws-ebs-csi-driver-node-rsrtp\" (UID: \"4d311f9d-1424-4282-9fd5-5b788bdc4cac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rsrtp" Sep 10 23:36:51.077278 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.077115 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gww99\" (UniqueName: \"kubernetes.io/projected/6df7e606-319b-4b9a-b6da-a850ecc9d267-kube-api-access-gww99\") pod \"network-metrics-daemon-v87ts\" (UID: \"6df7e606-319b-4b9a-b6da-a850ecc9d267\") " pod="openshift-multus/network-metrics-daemon-v87ts" Sep 10 23:36:51.077278 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.077141 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3ced5f14-cf96-419c-af9b-c218e93b0365-var-lib-kubelet\") pod \"tuned-4fp98\" (UID: \"3ced5f14-cf96-419c-af9b-c218e93b0365\") " pod="openshift-cluster-node-tuning-operator/tuned-4fp98" Sep 10 23:36:51.077278 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.077167 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ed113529-7712-4e4f-838a-835f3d6cf1b0-multus-cni-dir\") pod \"multus-hnjsh\" (UID: \"ed113529-7712-4e4f-838a-835f3d6cf1b0\") " pod="openshift-multus/multus-hnjsh" Sep 10 23:36:51.077278 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.077236 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22b9m\" (UniqueName: \"kubernetes.io/projected/4d311f9d-1424-4282-9fd5-5b788bdc4cac-kube-api-access-22b9m\") pod \"aws-ebs-csi-driver-node-rsrtp\" (UID: \"4d311f9d-1424-4282-9fd5-5b788bdc4cac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rsrtp" Sep 10 23:36:51.077278 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.077270 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4hx5\" (UniqueName: \"kubernetes.io/projected/6c4cdd12-89f3-4dbf-be94-04c8c0cb2c4e-kube-api-access-f4hx5\") pod \"iptables-alerter-dds4w\" (UID: \"6c4cdd12-89f3-4dbf-be94-04c8c0cb2c4e\") " pod="openshift-network-operator/iptables-alerter-dds4w" Sep 10 23:36:51.077438 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.077293 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/43e0f2f1-cd70-4612-828f-6cd98e1b77c2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-f47cx\" (UID: \"43e0f2f1-cd70-4612-828f-6cd98e1b77c2\") " pod="openshift-multus/multus-additional-cni-plugins-f47cx" Sep 10 23:36:51.077438 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.077317 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ed113529-7712-4e4f-838a-835f3d6cf1b0-multus-daemon-config\") pod \"multus-hnjsh\" (UID: \"ed113529-7712-4e4f-838a-835f3d6cf1b0\") " pod="openshift-multus/multus-hnjsh" Sep 10 23:36:51.077438 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.077341 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3a0ec520-b33d-49cf-9714-1662f5910e5b-original-pull-secret\") pod \"global-pull-secret-syncer-75vlp\" (UID: \"3a0ec520-b33d-49cf-9714-1662f5910e5b\") " pod="kube-system/global-pull-secret-syncer-75vlp" Sep 10 23:36:51.077438 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.077369 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6df7e606-319b-4b9a-b6da-a850ecc9d267-metrics-certs\") pod \"network-metrics-daemon-v87ts\" (UID: \"6df7e606-319b-4b9a-b6da-a850ecc9d267\") " pod="openshift-multus/network-metrics-daemon-v87ts" Sep 10 23:36:51.077438 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.077394 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c90ea9b8-efa1-423c-8d26-e4755af78f53-log-socket\") pod \"ovnkube-node-zrjmz\" (UID: \"c90ea9b8-efa1-423c-8d26-e4755af78f53\") " pod="openshift-ovn-kubernetes/ovnkube-node-zrjmz" Sep 10 23:36:51.077660 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.077437 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c90ea9b8-efa1-423c-8d26-e4755af78f53-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zrjmz\" (UID: \"c90ea9b8-efa1-423c-8d26-e4755af78f53\") " pod="openshift-ovn-kubernetes/ovnkube-node-zrjmz" Sep 10 23:36:51.077660 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.077476 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rknnf\" (UniqueName: \"kubernetes.io/projected/3ced5f14-cf96-419c-af9b-c218e93b0365-kube-api-access-rknnf\") pod \"tuned-4fp98\" (UID: \"3ced5f14-cf96-419c-af9b-c218e93b0365\") " pod="openshift-cluster-node-tuning-operator/tuned-4fp98" Sep 10 23:36:51.077660 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.077505 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ed113529-7712-4e4f-838a-835f3d6cf1b0-host-run-multus-certs\") pod \"multus-hnjsh\" (UID: \"ed113529-7712-4e4f-838a-835f3d6cf1b0\") " pod="openshift-multus/multus-hnjsh" Sep 10 23:36:51.077660 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.077537 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ed113529-7712-4e4f-838a-835f3d6cf1b0-etc-kubernetes\") pod \"multus-hnjsh\" (UID: \"ed113529-7712-4e4f-838a-835f3d6cf1b0\") " pod="openshift-multus/multus-hnjsh" Sep 10 23:36:51.077660 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.077562 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c90ea9b8-efa1-423c-8d26-e4755af78f53-host-cni-bin\") pod \"ovnkube-node-zrjmz\" (UID: \"c90ea9b8-efa1-423c-8d26-e4755af78f53\") " pod="openshift-ovn-kubernetes/ovnkube-node-zrjmz" Sep 10 23:36:51.077660 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.077576 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c90ea9b8-efa1-423c-8d26-e4755af78f53-ovnkube-config\") pod \"ovnkube-node-zrjmz\" (UID: \"c90ea9b8-efa1-423c-8d26-e4755af78f53\") " pod="openshift-ovn-kubernetes/ovnkube-node-zrjmz" Sep 10 23:36:51.077660 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.077601 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ed113529-7712-4e4f-838a-835f3d6cf1b0-host-var-lib-kubelet\") pod \"multus-hnjsh\" (UID: \"ed113529-7712-4e4f-838a-835f3d6cf1b0\") " pod="openshift-multus/multus-hnjsh" Sep 10 23:36:51.077660 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.077633 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7wk9\" (UniqueName: \"kubernetes.io/projected/24dda81e-d88e-4f29-836b-44bb86bf461d-kube-api-access-s7wk9\") pod \"network-check-target-zb6x4\" (UID: \"24dda81e-d88e-4f29-836b-44bb86bf461d\") " pod="openshift-network-diagnostics/network-check-target-zb6x4" Sep 10 23:36:51.077977 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.077681 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5d6d9866-cf11-4f49-877a-ecc951c01e42-hosts-file\") pod \"node-resolver-6wz5g\" (UID: \"5d6d9866-cf11-4f49-877a-ecc951c01e42\") " pod="openshift-dns/node-resolver-6wz5g" Sep 10 23:36:51.077977 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.077767 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c90ea9b8-efa1-423c-8d26-e4755af78f53-host-slash\") pod \"ovnkube-node-zrjmz\" (UID: \"c90ea9b8-efa1-423c-8d26-e4755af78f53\") " pod="openshift-ovn-kubernetes/ovnkube-node-zrjmz" Sep 10 23:36:51.077977 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.077778 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5d6d9866-cf11-4f49-877a-ecc951c01e42-hosts-file\") pod \"node-resolver-6wz5g\" (UID: \"5d6d9866-cf11-4f49-877a-ecc951c01e42\") " pod="openshift-dns/node-resolver-6wz5g" Sep 10 23:36:51.077977 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.077801 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c90ea9b8-efa1-423c-8d26-e4755af78f53-etc-openvswitch\") pod \"ovnkube-node-zrjmz\" (UID: \"c90ea9b8-efa1-423c-8d26-e4755af78f53\") " pod="openshift-ovn-kubernetes/ovnkube-node-zrjmz" Sep 10 23:36:51.077977 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.077829 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ed113529-7712-4e4f-838a-835f3d6cf1b0-system-cni-dir\") pod \"multus-hnjsh\" (UID: \"ed113529-7712-4e4f-838a-835f3d6cf1b0\") " pod="openshift-multus/multus-hnjsh" Sep 10 23:36:51.077977 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.077859 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dlhf\" (UniqueName: \"kubernetes.io/projected/cbaa689c-2a82-4b1a-b61f-4e8f43efa7d5-kube-api-access-5dlhf\") pod \"node-ca-fckpk\" (UID: \"cbaa689c-2a82-4b1a-b61f-4e8f43efa7d5\") " pod="openshift-image-registry/node-ca-fckpk" Sep 10 23:36:51.077977 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.077882 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3ced5f14-cf96-419c-af9b-c218e93b0365-etc-systemd\") pod \"tuned-4fp98\" (UID: \"3ced5f14-cf96-419c-af9b-c218e93b0365\") " pod="openshift-cluster-node-tuning-operator/tuned-4fp98" Sep 10 23:36:51.077977 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.077913 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6c4cdd12-89f3-4dbf-be94-04c8c0cb2c4e-host-slash\") pod \"iptables-alerter-dds4w\" (UID: \"6c4cdd12-89f3-4dbf-be94-04c8c0cb2c4e\") " pod="openshift-network-operator/iptables-alerter-dds4w" Sep 10 23:36:51.077977 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.077937 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4d311f9d-1424-4282-9fd5-5b788bdc4cac-registration-dir\") pod \"aws-ebs-csi-driver-node-rsrtp\" (UID: \"4d311f9d-1424-4282-9fd5-5b788bdc4cac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rsrtp" Sep 10 23:36:51.078305 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.077995 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3a0ec520-b33d-49cf-9714-1662f5910e5b-kubelet-config\") pod \"global-pull-secret-syncer-75vlp\" (UID: \"3a0ec520-b33d-49cf-9714-1662f5910e5b\") " pod="kube-system/global-pull-secret-syncer-75vlp" Sep 10 23:36:51.078305 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.078024 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3a0ec520-b33d-49cf-9714-1662f5910e5b-dbus\") pod \"global-pull-secret-syncer-75vlp\" (UID: \"3a0ec520-b33d-49cf-9714-1662f5910e5b\") " pod="kube-system/global-pull-secret-syncer-75vlp" Sep 10 23:36:51.078305 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.078073 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/43e0f2f1-cd70-4612-828f-6cd98e1b77c2-system-cni-dir\") pod \"multus-additional-cni-plugins-f47cx\" (UID: \"43e0f2f1-cd70-4612-828f-6cd98e1b77c2\") " pod="openshift-multus/multus-additional-cni-plugins-f47cx" Sep 10 23:36:51.078305 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.078114 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7ad8f728-76e5-47a2-8ab2-681ac07591f8-konnectivity-ca\") pod \"konnectivity-agent-dl8bc\" (UID: \"7ad8f728-76e5-47a2-8ab2-681ac07591f8\") " pod="kube-system/konnectivity-agent-dl8bc" Sep 10 23:36:51.078305 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.078185 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cbaa689c-2a82-4b1a-b61f-4e8f43efa7d5-serviceca\") pod \"node-ca-fckpk\" (UID: \"cbaa689c-2a82-4b1a-b61f-4e8f43efa7d5\") " pod="openshift-image-registry/node-ca-fckpk" Sep 10 23:36:51.078305 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.078212 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/43e0f2f1-cd70-4612-828f-6cd98e1b77c2-cni-binary-copy\") pod \"multus-additional-cni-plugins-f47cx\" (UID: \"43e0f2f1-cd70-4612-828f-6cd98e1b77c2\") " pod="openshift-multus/multus-additional-cni-plugins-f47cx" Sep 10 23:36:51.078305 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.078235 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3ced5f14-cf96-419c-af9b-c218e93b0365-run\") pod \"tuned-4fp98\" (UID: \"3ced5f14-cf96-419c-af9b-c218e93b0365\") " pod="openshift-cluster-node-tuning-operator/tuned-4fp98" Sep 10 23:36:51.078572 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.078337 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ed113529-7712-4e4f-838a-835f3d6cf1b0-multus-socket-dir-parent\") pod \"multus-hnjsh\" (UID: \"ed113529-7712-4e4f-838a-835f3d6cf1b0\") " pod="openshift-multus/multus-hnjsh" Sep 10 23:36:51.078572 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.078368 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c90ea9b8-efa1-423c-8d26-e4755af78f53-host-cni-netd\") pod \"ovnkube-node-zrjmz\" (UID: \"c90ea9b8-efa1-423c-8d26-e4755af78f53\") " pod="openshift-ovn-kubernetes/ovnkube-node-zrjmz" Sep 10 23:36:51.078572 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.078391 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c90ea9b8-efa1-423c-8d26-e4755af78f53-ovn-node-metrics-cert\") pod \"ovnkube-node-zrjmz\" (UID: \"c90ea9b8-efa1-423c-8d26-e4755af78f53\") " pod="openshift-ovn-kubernetes/ovnkube-node-zrjmz" Sep 10 23:36:51.078572 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.078427 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3ced5f14-cf96-419c-af9b-c218e93b0365-etc-sysctl-conf\") pod \"tuned-4fp98\" (UID: \"3ced5f14-cf96-419c-af9b-c218e93b0365\") " pod="openshift-cluster-node-tuning-operator/tuned-4fp98" Sep 10 23:36:51.078572 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.078450 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c90ea9b8-efa1-423c-8d26-e4755af78f53-run-systemd\") pod \"ovnkube-node-zrjmz\" (UID: \"c90ea9b8-efa1-423c-8d26-e4755af78f53\") " pod="openshift-ovn-kubernetes/ovnkube-node-zrjmz" Sep 10 23:36:51.078572 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.078472 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c90ea9b8-efa1-423c-8d26-e4755af78f53-var-lib-openvswitch\") pod \"ovnkube-node-zrjmz\" (UID: \"c90ea9b8-efa1-423c-8d26-e4755af78f53\") " pod="openshift-ovn-kubernetes/ovnkube-node-zrjmz" Sep 10 23:36:51.078572 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.078494 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ed113529-7712-4e4f-838a-835f3d6cf1b0-host-var-lib-cni-bin\") pod \"multus-hnjsh\" (UID: \"ed113529-7712-4e4f-838a-835f3d6cf1b0\") " pod="openshift-multus/multus-hnjsh" Sep 10 23:36:51.078572 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.078542 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4d311f9d-1424-4282-9fd5-5b788bdc4cac-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rsrtp\" (UID: \"4d311f9d-1424-4282-9fd5-5b788bdc4cac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rsrtp" Sep 10 23:36:51.078572 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.078568 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c90ea9b8-efa1-423c-8d26-e4755af78f53-run-ovn\") pod \"ovnkube-node-zrjmz\" (UID: \"c90ea9b8-efa1-423c-8d26-e4755af78f53\") " pod="openshift-ovn-kubernetes/ovnkube-node-zrjmz" Sep 10 23:36:51.078983 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.078597 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/43e0f2f1-cd70-4612-828f-6cd98e1b77c2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-f47cx\" (UID: \"43e0f2f1-cd70-4612-828f-6cd98e1b77c2\") " pod="openshift-multus/multus-additional-cni-plugins-f47cx" Sep 10 23:36:51.078983 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.078621 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c90ea9b8-efa1-423c-8d26-e4755af78f53-host-run-netns\") pod \"ovnkube-node-zrjmz\" (UID: \"c90ea9b8-efa1-423c-8d26-e4755af78f53\") " pod="openshift-ovn-kubernetes/ovnkube-node-zrjmz" Sep 10 23:36:51.078983 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.078658 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3ced5f14-cf96-419c-af9b-c218e93b0365-etc-sysctl-d\") pod \"tuned-4fp98\" (UID: \"3ced5f14-cf96-419c-af9b-c218e93b0365\") " pod="openshift-cluster-node-tuning-operator/tuned-4fp98" Sep 10 23:36:51.078983 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.078681 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3ced5f14-cf96-419c-af9b-c218e93b0365-host\") pod \"tuned-4fp98\" (UID: \"3ced5f14-cf96-419c-af9b-c218e93b0365\") " pod="openshift-cluster-node-tuning-operator/tuned-4fp98" Sep 10 23:36:51.078983 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.078695 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cbaa689c-2a82-4b1a-b61f-4e8f43efa7d5-serviceca\") pod \"node-ca-fckpk\" (UID: \"cbaa689c-2a82-4b1a-b61f-4e8f43efa7d5\") " pod="openshift-image-registry/node-ca-fckpk" Sep 10 23:36:51.078983 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.078704 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxfkl\" (UniqueName: \"kubernetes.io/projected/43e0f2f1-cd70-4612-828f-6cd98e1b77c2-kube-api-access-zxfkl\") pod \"multus-additional-cni-plugins-f47cx\" (UID: \"43e0f2f1-cd70-4612-828f-6cd98e1b77c2\") " pod="openshift-multus/multus-additional-cni-plugins-f47cx" Sep 10 23:36:51.078983 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.078706 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7ad8f728-76e5-47a2-8ab2-681ac07591f8-konnectivity-ca\") pod \"konnectivity-agent-dl8bc\" (UID: \"7ad8f728-76e5-47a2-8ab2-681ac07591f8\") " pod="kube-system/konnectivity-agent-dl8bc" Sep 10 23:36:51.078983 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.078741 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7ad8f728-76e5-47a2-8ab2-681ac07591f8-agent-certs\") pod \"konnectivity-agent-dl8bc\" (UID: \"7ad8f728-76e5-47a2-8ab2-681ac07591f8\") " pod="kube-system/konnectivity-agent-dl8bc" Sep 10 23:36:51.078983 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.078766 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d6d9866-cf11-4f49-877a-ecc951c01e42-tmp-dir\") pod \"node-resolver-6wz5g\" (UID: \"5d6d9866-cf11-4f49-877a-ecc951c01e42\") " pod="openshift-dns/node-resolver-6wz5g" Sep 10 23:36:51.078983 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.078806 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3ced5f14-cf96-419c-af9b-c218e93b0365-etc-sysconfig\") pod \"tuned-4fp98\" (UID: \"3ced5f14-cf96-419c-af9b-c218e93b0365\") " pod="openshift-cluster-node-tuning-operator/tuned-4fp98" Sep 10 23:36:51.078983 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.078834 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ed113529-7712-4e4f-838a-835f3d6cf1b0-host-run-netns\") pod \"multus-hnjsh\" (UID: \"ed113529-7712-4e4f-838a-835f3d6cf1b0\") " pod="openshift-multus/multus-hnjsh" Sep 10 23:36:51.078983 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.078878 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4d311f9d-1424-4282-9fd5-5b788bdc4cac-sys-fs\") pod \"aws-ebs-csi-driver-node-rsrtp\" (UID: \"4d311f9d-1424-4282-9fd5-5b788bdc4cac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rsrtp" Sep 10 23:36:51.078983 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.078902 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cbaa689c-2a82-4b1a-b61f-4e8f43efa7d5-host\") pod \"node-ca-fckpk\" (UID: \"cbaa689c-2a82-4b1a-b61f-4e8f43efa7d5\") " pod="openshift-image-registry/node-ca-fckpk" Sep 10 23:36:51.078983 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.078960 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/43e0f2f1-cd70-4612-828f-6cd98e1b77c2-cnibin\") pod \"multus-additional-cni-plugins-f47cx\" (UID: \"43e0f2f1-cd70-4612-828f-6cd98e1b77c2\") " pod="openshift-multus/multus-additional-cni-plugins-f47cx" Sep 10 23:36:51.079622 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.078994 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c90ea9b8-efa1-423c-8d26-e4755af78f53-host-run-ovn-kubernetes\") pod \"ovnkube-node-zrjmz\" (UID: \"c90ea9b8-efa1-423c-8d26-e4755af78f53\") " pod="openshift-ovn-kubernetes/ovnkube-node-zrjmz" Sep 10 23:36:51.079622 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.079025 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c90ea9b8-efa1-423c-8d26-e4755af78f53-ovnkube-script-lib\") pod \"ovnkube-node-zrjmz\" (UID: \"c90ea9b8-efa1-423c-8d26-e4755af78f53\") " pod="openshift-ovn-kubernetes/ovnkube-node-zrjmz" Sep 10 23:36:51.079622 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.079051 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3ced5f14-cf96-419c-af9b-c218e93b0365-lib-modules\") pod \"tuned-4fp98\" (UID: \"3ced5f14-cf96-419c-af9b-c218e93b0365\") " pod="openshift-cluster-node-tuning-operator/tuned-4fp98" Sep 10 23:36:51.079622 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.079074 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3ced5f14-cf96-419c-af9b-c218e93b0365-tmp\") pod \"tuned-4fp98\" (UID: \"3ced5f14-cf96-419c-af9b-c218e93b0365\") " pod="openshift-cluster-node-tuning-operator/tuned-4fp98" Sep 10 23:36:51.079622 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.079079 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d6d9866-cf11-4f49-877a-ecc951c01e42-tmp-dir\") pod \"node-resolver-6wz5g\" (UID: \"5d6d9866-cf11-4f49-877a-ecc951c01e42\") " pod="openshift-dns/node-resolver-6wz5g" Sep 10 23:36:51.079622 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.079099 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6c4cdd12-89f3-4dbf-be94-04c8c0cb2c4e-iptables-alerter-script\") pod \"iptables-alerter-dds4w\" (UID: \"6c4cdd12-89f3-4dbf-be94-04c8c0cb2c4e\") " pod="openshift-network-operator/iptables-alerter-dds4w" Sep 10 23:36:51.079622 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.079151 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ed113529-7712-4e4f-838a-835f3d6cf1b0-cni-binary-copy\") pod \"multus-hnjsh\" (UID: \"ed113529-7712-4e4f-838a-835f3d6cf1b0\") " pod="openshift-multus/multus-hnjsh" Sep 10 23:36:51.079622 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.079159 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cbaa689c-2a82-4b1a-b61f-4e8f43efa7d5-host\") pod \"node-ca-fckpk\" (UID: \"cbaa689c-2a82-4b1a-b61f-4e8f43efa7d5\") " pod="openshift-image-registry/node-ca-fckpk" Sep 10 23:36:51.079622 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.079152 2574 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 10 23:36:51.079622 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.079183 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c90ea9b8-efa1-423c-8d26-e4755af78f53-systemd-units\") pod \"ovnkube-node-zrjmz\" (UID: \"c90ea9b8-efa1-423c-8d26-e4755af78f53\") " pod="openshift-ovn-kubernetes/ovnkube-node-zrjmz" Sep 10 23:36:51.079622 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.079211 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c90ea9b8-efa1-423c-8d26-e4755af78f53-node-log\") pod \"ovnkube-node-zrjmz\" (UID: \"c90ea9b8-efa1-423c-8d26-e4755af78f53\") " pod="openshift-ovn-kubernetes/ovnkube-node-zrjmz" Sep 10 23:36:51.079622 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.079239 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c90ea9b8-efa1-423c-8d26-e4755af78f53-env-overrides\") pod \"ovnkube-node-zrjmz\" (UID: \"c90ea9b8-efa1-423c-8d26-e4755af78f53\") " pod="openshift-ovn-kubernetes/ovnkube-node-zrjmz" Sep 10 23:36:51.079622 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.079283 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ed113529-7712-4e4f-838a-835f3d6cf1b0-cnibin\") pod \"multus-hnjsh\" (UID: \"ed113529-7712-4e4f-838a-835f3d6cf1b0\") " pod="openshift-multus/multus-hnjsh" Sep 10 23:36:51.079622 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.079323 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ed113529-7712-4e4f-838a-835f3d6cf1b0-host-var-lib-cni-multus\") pod \"multus-hnjsh\" (UID: \"ed113529-7712-4e4f-838a-835f3d6cf1b0\") " pod="openshift-multus/multus-hnjsh" Sep 10 23:36:51.079622 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.079345 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ed113529-7712-4e4f-838a-835f3d6cf1b0-hostroot\") pod \"multus-hnjsh\" (UID: \"ed113529-7712-4e4f-838a-835f3d6cf1b0\") " pod="openshift-multus/multus-hnjsh" Sep 10 23:36:51.079622 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.079377 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ed113529-7712-4e4f-838a-835f3d6cf1b0-multus-conf-dir\") pod \"multus-hnjsh\" (UID: \"ed113529-7712-4e4f-838a-835f3d6cf1b0\") " pod="openshift-multus/multus-hnjsh" Sep 10 23:36:51.079622 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.079418 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfh7k\" (UniqueName: \"kubernetes.io/projected/5d6d9866-cf11-4f49-877a-ecc951c01e42-kube-api-access-xfh7k\") pod \"node-resolver-6wz5g\" (UID: \"5d6d9866-cf11-4f49-877a-ecc951c01e42\") " pod="openshift-dns/node-resolver-6wz5g" Sep 10 23:36:51.080373 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.079464 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c90ea9b8-efa1-423c-8d26-e4755af78f53-host-kubelet\") pod \"ovnkube-node-zrjmz\" (UID: \"c90ea9b8-efa1-423c-8d26-e4755af78f53\") " pod="openshift-ovn-kubernetes/ovnkube-node-zrjmz" Sep 10 23:36:51.080373 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.079574 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c90ea9b8-efa1-423c-8d26-e4755af78f53-run-openvswitch\") pod \"ovnkube-node-zrjmz\" (UID: \"c90ea9b8-efa1-423c-8d26-e4755af78f53\") " pod="openshift-ovn-kubernetes/ovnkube-node-zrjmz" Sep 10 23:36:51.083232 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.083212 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7ad8f728-76e5-47a2-8ab2-681ac07591f8-agent-certs\") pod \"konnectivity-agent-dl8bc\" (UID: \"7ad8f728-76e5-47a2-8ab2-681ac07591f8\") " pod="kube-system/konnectivity-agent-dl8bc" Sep 10 23:36:51.084854 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:36:51.084833 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 23:36:51.084854 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:36:51.084856 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 23:36:51.085003 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:36:51.084873 2574 projected.go:196] Error preparing data for projected volume kube-api-access-s7wk9 for pod openshift-network-diagnostics/network-check-target-zb6x4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 23:36:51.085003 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:36:51.084937 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/24dda81e-d88e-4f29-836b-44bb86bf461d-kube-api-access-s7wk9 podName:24dda81e-d88e-4f29-836b-44bb86bf461d nodeName:}" failed. No retries permitted until 2026-09-10 23:36:51.584915892 +0000 UTC m=+2.972518664 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-s7wk9" (UniqueName: "kubernetes.io/projected/24dda81e-d88e-4f29-836b-44bb86bf461d-kube-api-access-s7wk9") pod "network-check-target-zb6x4" (UID: "24dda81e-d88e-4f29-836b-44bb86bf461d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 23:36:51.087080 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.087062 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dlhf\" (UniqueName: \"kubernetes.io/projected/cbaa689c-2a82-4b1a-b61f-4e8f43efa7d5-kube-api-access-5dlhf\") pod \"node-ca-fckpk\" (UID: \"cbaa689c-2a82-4b1a-b61f-4e8f43efa7d5\") " pod="openshift-image-registry/node-ca-fckpk" Sep 10 23:36:51.090395 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.090364 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfh7k\" (UniqueName: \"kubernetes.io/projected/5d6d9866-cf11-4f49-877a-ecc951c01e42-kube-api-access-xfh7k\") pod \"node-resolver-6wz5g\" (UID: \"5d6d9866-cf11-4f49-877a-ecc951c01e42\") " pod="openshift-dns/node-resolver-6wz5g" Sep 10 23:36:51.091718 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.091670 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-128.ec2.internal" event={"ID":"ebf313d069610fccaa30accef842e07e","Type":"ContainerStarted","Data":"97e9869442b10442d82c1a66a7b49e6bde93e3bd64e8d8f77e5538ff2b215d41"} Sep 10 23:36:51.092881 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.092859 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-128.ec2.internal" event={"ID":"577bf0a7d48f3dd63acd7590896a721f","Type":"ContainerStarted","Data":"0b6ba203dc441d90a506dbccfd3af9f39ce475e93b9cab01a2d01539d174c818"} Sep 10 23:36:51.180820 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.180788 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3a0ec520-b33d-49cf-9714-1662f5910e5b-original-pull-secret\") pod \"global-pull-secret-syncer-75vlp\" (UID: \"3a0ec520-b33d-49cf-9714-1662f5910e5b\") " pod="kube-system/global-pull-secret-syncer-75vlp" Sep 10 23:36:51.180820 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.180823 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6df7e606-319b-4b9a-b6da-a850ecc9d267-metrics-certs\") pod \"network-metrics-daemon-v87ts\" (UID: \"6df7e606-319b-4b9a-b6da-a850ecc9d267\") " pod="openshift-multus/network-metrics-daemon-v87ts" Sep 10 23:36:51.181058 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.180848 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c90ea9b8-efa1-423c-8d26-e4755af78f53-log-socket\") pod \"ovnkube-node-zrjmz\" (UID: \"c90ea9b8-efa1-423c-8d26-e4755af78f53\") " pod="openshift-ovn-kubernetes/ovnkube-node-zrjmz" Sep 10 23:36:51.181058 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.180914 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c90ea9b8-efa1-423c-8d26-e4755af78f53-log-socket\") pod \"ovnkube-node-zrjmz\" (UID: \"c90ea9b8-efa1-423c-8d26-e4755af78f53\") " pod="openshift-ovn-kubernetes/ovnkube-node-zrjmz" Sep 10 23:36:51.181058 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.180947 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c90ea9b8-efa1-423c-8d26-e4755af78f53-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zrjmz\" (UID: \"c90ea9b8-efa1-423c-8d26-e4755af78f53\") " pod="openshift-ovn-kubernetes/ovnkube-node-zrjmz" Sep 10 23:36:51.181058 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:36:51.180959 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 23:36:51.181058 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.180972 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rknnf\" (UniqueName: \"kubernetes.io/projected/3ced5f14-cf96-419c-af9b-c218e93b0365-kube-api-access-rknnf\") pod \"tuned-4fp98\" (UID: \"3ced5f14-cf96-419c-af9b-c218e93b0365\") " pod="openshift-cluster-node-tuning-operator/tuned-4fp98" Sep 10 23:36:51.181058 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.180996 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ed113529-7712-4e4f-838a-835f3d6cf1b0-host-run-multus-certs\") pod \"multus-hnjsh\" (UID: \"ed113529-7712-4e4f-838a-835f3d6cf1b0\") " pod="openshift-multus/multus-hnjsh" Sep 10 23:36:51.181058 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:36:51.181025 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6df7e606-319b-4b9a-b6da-a850ecc9d267-metrics-certs podName:6df7e606-319b-4b9a-b6da-a850ecc9d267 nodeName:}" failed. No retries permitted until 2026-09-10 23:36:51.681005686 +0000 UTC m=+3.068608438 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6df7e606-319b-4b9a-b6da-a850ecc9d267-metrics-certs") pod "network-metrics-daemon-v87ts" (UID: "6df7e606-319b-4b9a-b6da-a850ecc9d267") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 23:36:51.181058 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.181036 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c90ea9b8-efa1-423c-8d26-e4755af78f53-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zrjmz\" (UID: \"c90ea9b8-efa1-423c-8d26-e4755af78f53\") " pod="openshift-ovn-kubernetes/ovnkube-node-zrjmz" Sep 10 23:36:51.181058 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.181047 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ed113529-7712-4e4f-838a-835f3d6cf1b0-etc-kubernetes\") pod \"multus-hnjsh\" (UID: \"ed113529-7712-4e4f-838a-835f3d6cf1b0\") " pod="openshift-multus/multus-hnjsh" Sep 10 23:36:51.181058 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.181042 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ed113529-7712-4e4f-838a-835f3d6cf1b0-host-run-multus-certs\") pod \"multus-hnjsh\" (UID: \"ed113529-7712-4e4f-838a-835f3d6cf1b0\") " pod="openshift-multus/multus-hnjsh" Sep 10 23:36:51.181541 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.181079 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c90ea9b8-efa1-423c-8d26-e4755af78f53-host-cni-bin\") pod \"ovnkube-node-zrjmz\" (UID: \"c90ea9b8-efa1-423c-8d26-e4755af78f53\") " pod="openshift-ovn-kubernetes/ovnkube-node-zrjmz" Sep 10 23:36:51.181541 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.181103 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c90ea9b8-efa1-423c-8d26-e4755af78f53-ovnkube-config\") pod \"ovnkube-node-zrjmz\" (UID: \"c90ea9b8-efa1-423c-8d26-e4755af78f53\") " pod="openshift-ovn-kubernetes/ovnkube-node-zrjmz" Sep 10 23:36:51.181541 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:36:51.181110 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 23:36:51.181541 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.181128 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ed113529-7712-4e4f-838a-835f3d6cf1b0-host-var-lib-kubelet\") pod \"multus-hnjsh\" (UID: \"ed113529-7712-4e4f-838a-835f3d6cf1b0\") " pod="openshift-multus/multus-hnjsh" Sep 10 23:36:51.181541 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:36:51.181156 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a0ec520-b33d-49cf-9714-1662f5910e5b-original-pull-secret podName:3a0ec520-b33d-49cf-9714-1662f5910e5b nodeName:}" failed. No retries permitted until 2026-09-10 23:36:51.681141565 +0000 UTC m=+3.068744323 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3a0ec520-b33d-49cf-9714-1662f5910e5b-original-pull-secret") pod "global-pull-secret-syncer-75vlp" (UID: "3a0ec520-b33d-49cf-9714-1662f5910e5b") : object "kube-system"/"original-pull-secret" not registered Sep 10 23:36:51.181541 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.181173 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ed113529-7712-4e4f-838a-835f3d6cf1b0-host-var-lib-kubelet\") pod \"multus-hnjsh\" (UID: \"ed113529-7712-4e4f-838a-835f3d6cf1b0\") " pod="openshift-multus/multus-hnjsh" Sep 10 23:36:51.181541 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.181175 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c90ea9b8-efa1-423c-8d26-e4755af78f53-host-cni-bin\") pod \"ovnkube-node-zrjmz\" (UID: \"c90ea9b8-efa1-423c-8d26-e4755af78f53\") " pod="openshift-ovn-kubernetes/ovnkube-node-zrjmz" Sep 10 23:36:51.181541 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.181169 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ed113529-7712-4e4f-838a-835f3d6cf1b0-etc-kubernetes\") pod \"multus-hnjsh\" (UID: \"ed113529-7712-4e4f-838a-835f3d6cf1b0\") " pod="openshift-multus/multus-hnjsh" Sep 10 23:36:51.181541 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.181187 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c90ea9b8-efa1-423c-8d26-e4755af78f53-host-slash\") pod \"ovnkube-node-zrjmz\" (UID: \"c90ea9b8-efa1-423c-8d26-e4755af78f53\") " pod="openshift-ovn-kubernetes/ovnkube-node-zrjmz" Sep 10 23:36:51.181541 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.181219 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c90ea9b8-efa1-423c-8d26-e4755af78f53-etc-openvswitch\") pod \"ovnkube-node-zrjmz\" (UID: \"c90ea9b8-efa1-423c-8d26-e4755af78f53\") " pod="openshift-ovn-kubernetes/ovnkube-node-zrjmz" Sep 10 23:36:51.181541 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.181237 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ed113529-7712-4e4f-838a-835f3d6cf1b0-system-cni-dir\") pod \"multus-hnjsh\" (UID: \"ed113529-7712-4e4f-838a-835f3d6cf1b0\") " pod="openshift-multus/multus-hnjsh" Sep 10 23:36:51.181541 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.181253 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3ced5f14-cf96-419c-af9b-c218e93b0365-etc-systemd\") pod \"tuned-4fp98\" (UID: \"3ced5f14-cf96-419c-af9b-c218e93b0365\") " pod="openshift-cluster-node-tuning-operator/tuned-4fp98" Sep 10 23:36:51.181541 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.181264 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c90ea9b8-efa1-423c-8d26-e4755af78f53-host-slash\") pod \"ovnkube-node-zrjmz\" (UID: \"c90ea9b8-efa1-423c-8d26-e4755af78f53\") " pod="openshift-ovn-kubernetes/ovnkube-node-zrjmz" Sep 10 23:36:51.181541 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.181283 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c90ea9b8-efa1-423c-8d26-e4755af78f53-etc-openvswitch\") pod \"ovnkube-node-zrjmz\" (UID: \"c90ea9b8-efa1-423c-8d26-e4755af78f53\") " pod="openshift-ovn-kubernetes/ovnkube-node-zrjmz" Sep 10 23:36:51.181541 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.181289 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6c4cdd12-89f3-4dbf-be94-04c8c0cb2c4e-host-slash\") pod \"iptables-alerter-dds4w\" (UID: \"6c4cdd12-89f3-4dbf-be94-04c8c0cb2c4e\") " pod="openshift-network-operator/iptables-alerter-dds4w" Sep 10 23:36:51.181541 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.181307 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3ced5f14-cf96-419c-af9b-c218e93b0365-etc-systemd\") pod \"tuned-4fp98\" (UID: \"3ced5f14-cf96-419c-af9b-c218e93b0365\") " pod="openshift-cluster-node-tuning-operator/tuned-4fp98" Sep 10 23:36:51.181541 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.181319 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6c4cdd12-89f3-4dbf-be94-04c8c0cb2c4e-host-slash\") pod \"iptables-alerter-dds4w\" (UID: \"6c4cdd12-89f3-4dbf-be94-04c8c0cb2c4e\") " pod="openshift-network-operator/iptables-alerter-dds4w" Sep 10 23:36:51.182363 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.181326 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4d311f9d-1424-4282-9fd5-5b788bdc4cac-registration-dir\") pod \"aws-ebs-csi-driver-node-rsrtp\" (UID: \"4d311f9d-1424-4282-9fd5-5b788bdc4cac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rsrtp" Sep 10 23:36:51.182363 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.181344 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3a0ec520-b33d-49cf-9714-1662f5910e5b-kubelet-config\") pod \"global-pull-secret-syncer-75vlp\" (UID: \"3a0ec520-b33d-49cf-9714-1662f5910e5b\") " pod="kube-system/global-pull-secret-syncer-75vlp" Sep 10 23:36:51.182363 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.181361 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ed113529-7712-4e4f-838a-835f3d6cf1b0-system-cni-dir\") pod \"multus-hnjsh\" (UID: \"ed113529-7712-4e4f-838a-835f3d6cf1b0\") " pod="openshift-multus/multus-hnjsh" Sep 10 23:36:51.182363 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.181365 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3a0ec520-b33d-49cf-9714-1662f5910e5b-dbus\") pod \"global-pull-secret-syncer-75vlp\" (UID: \"3a0ec520-b33d-49cf-9714-1662f5910e5b\") " pod="kube-system/global-pull-secret-syncer-75vlp" Sep 10 23:36:51.182363 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.181404 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4d311f9d-1424-4282-9fd5-5b788bdc4cac-registration-dir\") pod \"aws-ebs-csi-driver-node-rsrtp\" (UID: \"4d311f9d-1424-4282-9fd5-5b788bdc4cac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rsrtp" Sep 10 23:36:51.182363 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.181404 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/43e0f2f1-cd70-4612-828f-6cd98e1b77c2-system-cni-dir\") pod \"multus-additional-cni-plugins-f47cx\" (UID: \"43e0f2f1-cd70-4612-828f-6cd98e1b77c2\") " pod="openshift-multus/multus-additional-cni-plugins-f47cx" Sep 10 23:36:51.182363 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.181422 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3a0ec520-b33d-49cf-9714-1662f5910e5b-kubelet-config\") pod \"global-pull-secret-syncer-75vlp\" (UID: \"3a0ec520-b33d-49cf-9714-1662f5910e5b\") " pod="kube-system/global-pull-secret-syncer-75vlp" Sep 10 23:36:51.182363 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.181444 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/43e0f2f1-cd70-4612-828f-6cd98e1b77c2-cni-binary-copy\") pod \"multus-additional-cni-plugins-f47cx\" (UID: \"43e0f2f1-cd70-4612-828f-6cd98e1b77c2\") " pod="openshift-multus/multus-additional-cni-plugins-f47cx" Sep 10 23:36:51.182363 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.181446 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/43e0f2f1-cd70-4612-828f-6cd98e1b77c2-system-cni-dir\") pod \"multus-additional-cni-plugins-f47cx\" (UID: \"43e0f2f1-cd70-4612-828f-6cd98e1b77c2\") " pod="openshift-multus/multus-additional-cni-plugins-f47cx" Sep 10 23:36:51.182363 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.181470 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3ced5f14-cf96-419c-af9b-c218e93b0365-run\") pod \"tuned-4fp98\" (UID: \"3ced5f14-cf96-419c-af9b-c218e93b0365\") " pod="openshift-cluster-node-tuning-operator/tuned-4fp98" Sep 10 23:36:51.182363 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.181489 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ed113529-7712-4e4f-838a-835f3d6cf1b0-multus-socket-dir-parent\") pod \"multus-hnjsh\" (UID: \"ed113529-7712-4e4f-838a-835f3d6cf1b0\") " pod="openshift-multus/multus-hnjsh" Sep 10 23:36:51.182363 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.181507 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c90ea9b8-efa1-423c-8d26-e4755af78f53-host-cni-netd\") pod \"ovnkube-node-zrjmz\" (UID: \"c90ea9b8-efa1-423c-8d26-e4755af78f53\") " pod="openshift-ovn-kubernetes/ovnkube-node-zrjmz" Sep 10 23:36:51.182363 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.181532 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c90ea9b8-efa1-423c-8d26-e4755af78f53-ovn-node-metrics-cert\") pod \"ovnkube-node-zrjmz\" (UID: \"c90ea9b8-efa1-423c-8d26-e4755af78f53\") " pod="openshift-ovn-kubernetes/ovnkube-node-zrjmz" Sep 10 23:36:51.182363 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.181552 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3ced5f14-cf96-419c-af9b-c218e93b0365-etc-sysctl-conf\") pod \"tuned-4fp98\" (UID: \"3ced5f14-cf96-419c-af9b-c218e93b0365\") " pod="openshift-cluster-node-tuning-operator/tuned-4fp98" Sep 10 23:36:51.182363 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.181573 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3a0ec520-b33d-49cf-9714-1662f5910e5b-dbus\") pod \"global-pull-secret-syncer-75vlp\" (UID: \"3a0ec520-b33d-49cf-9714-1662f5910e5b\") " pod="kube-system/global-pull-secret-syncer-75vlp" Sep 10 23:36:51.182363 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.181583 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c90ea9b8-efa1-423c-8d26-e4755af78f53-run-systemd\") pod \"ovnkube-node-zrjmz\" (UID: \"c90ea9b8-efa1-423c-8d26-e4755af78f53\") " pod="openshift-ovn-kubernetes/ovnkube-node-zrjmz" Sep 10 23:36:51.182363 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.181612 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c90ea9b8-efa1-423c-8d26-e4755af78f53-host-cni-netd\") pod \"ovnkube-node-zrjmz\" (UID: \"c90ea9b8-efa1-423c-8d26-e4755af78f53\") " pod="openshift-ovn-kubernetes/ovnkube-node-zrjmz" Sep 10 23:36:51.183217 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.181575 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3ced5f14-cf96-419c-af9b-c218e93b0365-run\") pod \"tuned-4fp98\" (UID: \"3ced5f14-cf96-419c-af9b-c218e93b0365\") " pod="openshift-cluster-node-tuning-operator/tuned-4fp98" Sep 10 23:36:51.183217 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.181613 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c90ea9b8-efa1-423c-8d26-e4755af78f53-run-systemd\") pod \"ovnkube-node-zrjmz\" (UID: \"c90ea9b8-efa1-423c-8d26-e4755af78f53\") " pod="openshift-ovn-kubernetes/ovnkube-node-zrjmz" Sep 10 23:36:51.183217 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.181612 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c90ea9b8-efa1-423c-8d26-e4755af78f53-var-lib-openvswitch\") pod \"ovnkube-node-zrjmz\" (UID: \"c90ea9b8-efa1-423c-8d26-e4755af78f53\") " pod="openshift-ovn-kubernetes/ovnkube-node-zrjmz" Sep 10 23:36:51.183217 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.181675 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ed113529-7712-4e4f-838a-835f3d6cf1b0-multus-socket-dir-parent\") pod \"multus-hnjsh\" (UID: \"ed113529-7712-4e4f-838a-835f3d6cf1b0\") " pod="openshift-multus/multus-hnjsh" Sep 10 23:36:51.183217 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.181677 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ed113529-7712-4e4f-838a-835f3d6cf1b0-host-var-lib-cni-bin\") pod \"multus-hnjsh\" (UID: \"ed113529-7712-4e4f-838a-835f3d6cf1b0\") " pod="openshift-multus/multus-hnjsh" Sep 10 23:36:51.183217 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.181690 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c90ea9b8-efa1-423c-8d26-e4755af78f53-ovnkube-config\") pod \"ovnkube-node-zrjmz\" (UID: \"c90ea9b8-efa1-423c-8d26-e4755af78f53\") " pod="openshift-ovn-kubernetes/ovnkube-node-zrjmz" Sep 10 23:36:51.183217 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.181709 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4d311f9d-1424-4282-9fd5-5b788bdc4cac-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rsrtp\" (UID: \"4d311f9d-1424-4282-9fd5-5b788bdc4cac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rsrtp" Sep 10 23:36:51.183217 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.181709 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ed113529-7712-4e4f-838a-835f3d6cf1b0-host-var-lib-cni-bin\") pod \"multus-hnjsh\" (UID: \"ed113529-7712-4e4f-838a-835f3d6cf1b0\") " pod="openshift-multus/multus-hnjsh" Sep 10 23:36:51.183217 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.181730 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3ced5f14-cf96-419c-af9b-c218e93b0365-etc-sysctl-conf\") pod \"tuned-4fp98\" (UID: \"3ced5f14-cf96-419c-af9b-c218e93b0365\") " pod="openshift-cluster-node-tuning-operator/tuned-4fp98" Sep 10 23:36:51.183217 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.181735 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c90ea9b8-efa1-423c-8d26-e4755af78f53-run-ovn\") pod \"ovnkube-node-zrjmz\" (UID: \"c90ea9b8-efa1-423c-8d26-e4755af78f53\") " pod="openshift-ovn-kubernetes/ovnkube-node-zrjmz" Sep 10 23:36:51.183217 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.181739 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c90ea9b8-efa1-423c-8d26-e4755af78f53-var-lib-openvswitch\") pod \"ovnkube-node-zrjmz\" (UID: \"c90ea9b8-efa1-423c-8d26-e4755af78f53\") " pod="openshift-ovn-kubernetes/ovnkube-node-zrjmz" Sep 10 23:36:51.183217 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.181759 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/43e0f2f1-cd70-4612-828f-6cd98e1b77c2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-f47cx\" (UID: \"43e0f2f1-cd70-4612-828f-6cd98e1b77c2\") " pod="openshift-multus/multus-additional-cni-plugins-f47cx" Sep 10 23:36:51.183217 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.181782 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4d311f9d-1424-4282-9fd5-5b788bdc4cac-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rsrtp\" (UID: \"4d311f9d-1424-4282-9fd5-5b788bdc4cac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rsrtp" Sep 10 23:36:51.183217 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.181792 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c90ea9b8-efa1-423c-8d26-e4755af78f53-host-run-netns\") pod \"ovnkube-node-zrjmz\" (UID: \"c90ea9b8-efa1-423c-8d26-e4755af78f53\") " pod="openshift-ovn-kubernetes/ovnkube-node-zrjmz" Sep 10 23:36:51.183217 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.181794 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c90ea9b8-efa1-423c-8d26-e4755af78f53-run-ovn\") pod \"ovnkube-node-zrjmz\" (UID: \"c90ea9b8-efa1-423c-8d26-e4755af78f53\") " pod="openshift-ovn-kubernetes/ovnkube-node-zrjmz" Sep 10 23:36:51.183217 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.181816 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3ced5f14-cf96-419c-af9b-c218e93b0365-etc-sysctl-d\") pod \"tuned-4fp98\" (UID: \"3ced5f14-cf96-419c-af9b-c218e93b0365\") " pod="openshift-cluster-node-tuning-operator/tuned-4fp98" Sep 10 23:36:51.183217 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.181840 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c90ea9b8-efa1-423c-8d26-e4755af78f53-host-run-netns\") pod \"ovnkube-node-zrjmz\" (UID: \"c90ea9b8-efa1-423c-8d26-e4755af78f53\") " pod="openshift-ovn-kubernetes/ovnkube-node-zrjmz" Sep 10 23:36:51.183217 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.181848 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3ced5f14-cf96-419c-af9b-c218e93b0365-host\") pod \"tuned-4fp98\" (UID: \"3ced5f14-cf96-419c-af9b-c218e93b0365\") " pod="openshift-cluster-node-tuning-operator/tuned-4fp98" Sep 10 23:36:51.184093 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.181892 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxfkl\" (UniqueName: \"kubernetes.io/projected/43e0f2f1-cd70-4612-828f-6cd98e1b77c2-kube-api-access-zxfkl\") pod \"multus-additional-cni-plugins-f47cx\" (UID: \"43e0f2f1-cd70-4612-828f-6cd98e1b77c2\") " pod="openshift-multus/multus-additional-cni-plugins-f47cx" Sep 10 23:36:51.184093 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.181910 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3ced5f14-cf96-419c-af9b-c218e93b0365-etc-sysctl-d\") pod \"tuned-4fp98\" (UID: \"3ced5f14-cf96-419c-af9b-c218e93b0365\") " pod="openshift-cluster-node-tuning-operator/tuned-4fp98" Sep 10 23:36:51.184093 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.181956 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3ced5f14-cf96-419c-af9b-c218e93b0365-etc-sysconfig\") pod \"tuned-4fp98\" (UID: \"3ced5f14-cf96-419c-af9b-c218e93b0365\") " pod="openshift-cluster-node-tuning-operator/tuned-4fp98" Sep 10 23:36:51.184093 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.181982 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ed113529-7712-4e4f-838a-835f3d6cf1b0-host-run-netns\") pod \"multus-hnjsh\" (UID: \"ed113529-7712-4e4f-838a-835f3d6cf1b0\") " pod="openshift-multus/multus-hnjsh" Sep 10 23:36:51.184093 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.181897 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3ced5f14-cf96-419c-af9b-c218e93b0365-host\") pod \"tuned-4fp98\" (UID: \"3ced5f14-cf96-419c-af9b-c218e93b0365\") " pod="openshift-cluster-node-tuning-operator/tuned-4fp98" Sep 10 23:36:51.184093 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.182004 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3ced5f14-cf96-419c-af9b-c218e93b0365-etc-sysconfig\") pod \"tuned-4fp98\" (UID: \"3ced5f14-cf96-419c-af9b-c218e93b0365\") " pod="openshift-cluster-node-tuning-operator/tuned-4fp98" Sep 10 23:36:51.184093 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.182008 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4d311f9d-1424-4282-9fd5-5b788bdc4cac-sys-fs\") pod \"aws-ebs-csi-driver-node-rsrtp\" (UID: \"4d311f9d-1424-4282-9fd5-5b788bdc4cac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rsrtp" Sep 10 23:36:51.184093 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.182037 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/43e0f2f1-cd70-4612-828f-6cd98e1b77c2-cni-binary-copy\") pod \"multus-additional-cni-plugins-f47cx\" (UID: \"43e0f2f1-cd70-4612-828f-6cd98e1b77c2\") " pod="openshift-multus/multus-additional-cni-plugins-f47cx" Sep 10 23:36:51.184093 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.182052 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/43e0f2f1-cd70-4612-828f-6cd98e1b77c2-cnibin\") pod \"multus-additional-cni-plugins-f47cx\" (UID: \"43e0f2f1-cd70-4612-828f-6cd98e1b77c2\") " pod="openshift-multus/multus-additional-cni-plugins-f47cx" Sep 10 23:36:51.184093 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.182075 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4d311f9d-1424-4282-9fd5-5b788bdc4cac-sys-fs\") pod \"aws-ebs-csi-driver-node-rsrtp\" (UID: \"4d311f9d-1424-4282-9fd5-5b788bdc4cac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rsrtp" Sep 10 23:36:51.184093 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.182087 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/43e0f2f1-cd70-4612-828f-6cd98e1b77c2-cnibin\") pod \"multus-additional-cni-plugins-f47cx\" (UID: \"43e0f2f1-cd70-4612-828f-6cd98e1b77c2\") " pod="openshift-multus/multus-additional-cni-plugins-f47cx" Sep 10 23:36:51.184093 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.182092 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c90ea9b8-efa1-423c-8d26-e4755af78f53-host-run-ovn-kubernetes\") pod \"ovnkube-node-zrjmz\" (UID: \"c90ea9b8-efa1-423c-8d26-e4755af78f53\") " pod="openshift-ovn-kubernetes/ovnkube-node-zrjmz" Sep 10 23:36:51.184093 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.182044 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ed113529-7712-4e4f-838a-835f3d6cf1b0-host-run-netns\") pod \"multus-hnjsh\" (UID: \"ed113529-7712-4e4f-838a-835f3d6cf1b0\") " pod="openshift-multus/multus-hnjsh" Sep 10 23:36:51.184093 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.182120 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c90ea9b8-efa1-423c-8d26-e4755af78f53-ovnkube-script-lib\") pod \"ovnkube-node-zrjmz\" (UID: \"c90ea9b8-efa1-423c-8d26-e4755af78f53\") " pod="openshift-ovn-kubernetes/ovnkube-node-zrjmz" Sep 10 23:36:51.184093 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.182144 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3ced5f14-cf96-419c-af9b-c218e93b0365-lib-modules\") pod \"tuned-4fp98\" (UID: \"3ced5f14-cf96-419c-af9b-c218e93b0365\") " pod="openshift-cluster-node-tuning-operator/tuned-4fp98" Sep 10 23:36:51.184093 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.182145 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c90ea9b8-efa1-423c-8d26-e4755af78f53-host-run-ovn-kubernetes\") pod \"ovnkube-node-zrjmz\" (UID: \"c90ea9b8-efa1-423c-8d26-e4755af78f53\") " pod="openshift-ovn-kubernetes/ovnkube-node-zrjmz" Sep 10 23:36:51.184093 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.182169 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3ced5f14-cf96-419c-af9b-c218e93b0365-tmp\") pod \"tuned-4fp98\" (UID: \"3ced5f14-cf96-419c-af9b-c218e93b0365\") " pod="openshift-cluster-node-tuning-operator/tuned-4fp98" Sep 10 23:36:51.184876 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.182193 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6c4cdd12-89f3-4dbf-be94-04c8c0cb2c4e-iptables-alerter-script\") pod \"iptables-alerter-dds4w\" (UID: \"6c4cdd12-89f3-4dbf-be94-04c8c0cb2c4e\") " pod="openshift-network-operator/iptables-alerter-dds4w" Sep 10 23:36:51.184876 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.182212 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/43e0f2f1-cd70-4612-828f-6cd98e1b77c2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-f47cx\" (UID: \"43e0f2f1-cd70-4612-828f-6cd98e1b77c2\") " pod="openshift-multus/multus-additional-cni-plugins-f47cx" Sep 10 23:36:51.184876 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.182224 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ed113529-7712-4e4f-838a-835f3d6cf1b0-cni-binary-copy\") pod \"multus-hnjsh\" (UID: \"ed113529-7712-4e4f-838a-835f3d6cf1b0\") " pod="openshift-multus/multus-hnjsh" Sep 10 23:36:51.184876 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.182247 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3ced5f14-cf96-419c-af9b-c218e93b0365-lib-modules\") pod \"tuned-4fp98\" (UID: \"3ced5f14-cf96-419c-af9b-c218e93b0365\") " pod="openshift-cluster-node-tuning-operator/tuned-4fp98" Sep 10 23:36:51.184876 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.182257 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c90ea9b8-efa1-423c-8d26-e4755af78f53-systemd-units\") pod \"ovnkube-node-zrjmz\" (UID: \"c90ea9b8-efa1-423c-8d26-e4755af78f53\") " pod="openshift-ovn-kubernetes/ovnkube-node-zrjmz" Sep 10 23:36:51.184876 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.182284 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c90ea9b8-efa1-423c-8d26-e4755af78f53-node-log\") pod \"ovnkube-node-zrjmz\" (UID: \"c90ea9b8-efa1-423c-8d26-e4755af78f53\") " pod="openshift-ovn-kubernetes/ovnkube-node-zrjmz" Sep 10 23:36:51.184876 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.182301 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c90ea9b8-efa1-423c-8d26-e4755af78f53-systemd-units\") pod \"ovnkube-node-zrjmz\" (UID: \"c90ea9b8-efa1-423c-8d26-e4755af78f53\") " pod="openshift-ovn-kubernetes/ovnkube-node-zrjmz" Sep 10 23:36:51.184876 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.182307 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c90ea9b8-efa1-423c-8d26-e4755af78f53-env-overrides\") pod \"ovnkube-node-zrjmz\" (UID: \"c90ea9b8-efa1-423c-8d26-e4755af78f53\") " pod="openshift-ovn-kubernetes/ovnkube-node-zrjmz" Sep 10 23:36:51.184876 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.182338 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ed113529-7712-4e4f-838a-835f3d6cf1b0-cnibin\") pod \"multus-hnjsh\" (UID: \"ed113529-7712-4e4f-838a-835f3d6cf1b0\") " pod="openshift-multus/multus-hnjsh" Sep 10 23:36:51.184876 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.182364 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ed113529-7712-4e4f-838a-835f3d6cf1b0-host-var-lib-cni-multus\") pod \"multus-hnjsh\" (UID: \"ed113529-7712-4e4f-838a-835f3d6cf1b0\") " pod="openshift-multus/multus-hnjsh" Sep 10 23:36:51.184876 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.182388 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ed113529-7712-4e4f-838a-835f3d6cf1b0-hostroot\") pod \"multus-hnjsh\" (UID: \"ed113529-7712-4e4f-838a-835f3d6cf1b0\") " pod="openshift-multus/multus-hnjsh" Sep 10 23:36:51.184876 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.182412 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ed113529-7712-4e4f-838a-835f3d6cf1b0-multus-conf-dir\") pod \"multus-hnjsh\" (UID: \"ed113529-7712-4e4f-838a-835f3d6cf1b0\") " pod="openshift-multus/multus-hnjsh" Sep 10 23:36:51.184876 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.182439 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c90ea9b8-efa1-423c-8d26-e4755af78f53-host-kubelet\") pod \"ovnkube-node-zrjmz\" (UID: \"c90ea9b8-efa1-423c-8d26-e4755af78f53\") " pod="openshift-ovn-kubernetes/ovnkube-node-zrjmz" Sep 10 23:36:51.184876 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.182466 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c90ea9b8-efa1-423c-8d26-e4755af78f53-run-openvswitch\") pod \"ovnkube-node-zrjmz\" (UID: \"c90ea9b8-efa1-423c-8d26-e4755af78f53\") " pod="openshift-ovn-kubernetes/ovnkube-node-zrjmz" Sep 10 23:36:51.184876 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.182490 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gqgt\" (UniqueName: \"kubernetes.io/projected/c90ea9b8-efa1-423c-8d26-e4755af78f53-kube-api-access-8gqgt\") pod \"ovnkube-node-zrjmz\" (UID: \"c90ea9b8-efa1-423c-8d26-e4755af78f53\") " pod="openshift-ovn-kubernetes/ovnkube-node-zrjmz" Sep 10 23:36:51.184876 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.182497 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ed113529-7712-4e4f-838a-835f3d6cf1b0-host-var-lib-cni-multus\") pod \"multus-hnjsh\" (UID: \"ed113529-7712-4e4f-838a-835f3d6cf1b0\") " pod="openshift-multus/multus-hnjsh" Sep 10 23:36:51.184876 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.182512 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3ced5f14-cf96-419c-af9b-c218e93b0365-etc-modprobe-d\") pod \"tuned-4fp98\" (UID: \"3ced5f14-cf96-419c-af9b-c218e93b0365\") " pod="openshift-cluster-node-tuning-operator/tuned-4fp98" Sep 10 23:36:51.184876 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.182538 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3ced5f14-cf96-419c-af9b-c218e93b0365-sys\") pod \"tuned-4fp98\" (UID: \"3ced5f14-cf96-419c-af9b-c218e93b0365\") " pod="openshift-cluster-node-tuning-operator/tuned-4fp98" Sep 10 23:36:51.185553 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.182559 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ed113529-7712-4e4f-838a-835f3d6cf1b0-cnibin\") pod \"multus-hnjsh\" (UID: \"ed113529-7712-4e4f-838a-835f3d6cf1b0\") " pod="openshift-multus/multus-hnjsh" Sep 10 23:36:51.185553 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.182564 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ed113529-7712-4e4f-838a-835f3d6cf1b0-host-run-k8s-cni-cncf-io\") pod \"multus-hnjsh\" (UID: \"ed113529-7712-4e4f-838a-835f3d6cf1b0\") " pod="openshift-multus/multus-hnjsh" Sep 10 23:36:51.185553 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.182592 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/43e0f2f1-cd70-4612-828f-6cd98e1b77c2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-f47cx\" (UID: \"43e0f2f1-cd70-4612-828f-6cd98e1b77c2\") " pod="openshift-multus/multus-additional-cni-plugins-f47cx" Sep 10 23:36:51.185553 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.182603 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c90ea9b8-efa1-423c-8d26-e4755af78f53-node-log\") pod \"ovnkube-node-zrjmz\" (UID: \"c90ea9b8-efa1-423c-8d26-e4755af78f53\") " pod="openshift-ovn-kubernetes/ovnkube-node-zrjmz" Sep 10 23:36:51.185553 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.182617 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3ced5f14-cf96-419c-af9b-c218e93b0365-etc-kubernetes\") pod \"tuned-4fp98\" (UID: \"3ced5f14-cf96-419c-af9b-c218e93b0365\") " pod="openshift-cluster-node-tuning-operator/tuned-4fp98" Sep 10 23:36:51.185553 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.182666 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3ced5f14-cf96-419c-af9b-c218e93b0365-etc-tuned\") pod \"tuned-4fp98\" (UID: \"3ced5f14-cf96-419c-af9b-c218e93b0365\") " pod="openshift-cluster-node-tuning-operator/tuned-4fp98" Sep 10 23:36:51.185553 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.182723 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8zw6\" (UniqueName: \"kubernetes.io/projected/ed113529-7712-4e4f-838a-835f3d6cf1b0-kube-api-access-p8zw6\") pod \"multus-hnjsh\" (UID: \"ed113529-7712-4e4f-838a-835f3d6cf1b0\") " pod="openshift-multus/multus-hnjsh" Sep 10 23:36:51.185553 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.182733 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c90ea9b8-efa1-423c-8d26-e4755af78f53-ovnkube-script-lib\") pod \"ovnkube-node-zrjmz\" (UID: \"c90ea9b8-efa1-423c-8d26-e4755af78f53\") " pod="openshift-ovn-kubernetes/ovnkube-node-zrjmz" Sep 10 23:36:51.185553 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.182748 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4d311f9d-1424-4282-9fd5-5b788bdc4cac-socket-dir\") pod \"aws-ebs-csi-driver-node-rsrtp\" (UID: \"4d311f9d-1424-4282-9fd5-5b788bdc4cac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rsrtp" Sep 10 23:36:51.185553 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.182780 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6c4cdd12-89f3-4dbf-be94-04c8c0cb2c4e-iptables-alerter-script\") pod \"iptables-alerter-dds4w\" (UID: \"6c4cdd12-89f3-4dbf-be94-04c8c0cb2c4e\") " pod="openshift-network-operator/iptables-alerter-dds4w" Sep 10 23:36:51.185553 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.182856 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ed113529-7712-4e4f-838a-835f3d6cf1b0-hostroot\") pod \"multus-hnjsh\" (UID: \"ed113529-7712-4e4f-838a-835f3d6cf1b0\") " pod="openshift-multus/multus-hnjsh" Sep 10 23:36:51.185553 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.182862 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4d311f9d-1424-4282-9fd5-5b788bdc4cac-socket-dir\") pod \"aws-ebs-csi-driver-node-rsrtp\" (UID: \"4d311f9d-1424-4282-9fd5-5b788bdc4cac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rsrtp" Sep 10 23:36:51.185553 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.182901 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c90ea9b8-efa1-423c-8d26-e4755af78f53-host-kubelet\") pod \"ovnkube-node-zrjmz\" (UID: \"c90ea9b8-efa1-423c-8d26-e4755af78f53\") " pod="openshift-ovn-kubernetes/ovnkube-node-zrjmz" Sep 10 23:36:51.185553 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.182912 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ed113529-7712-4e4f-838a-835f3d6cf1b0-multus-conf-dir\") pod \"multus-hnjsh\" (UID: \"ed113529-7712-4e4f-838a-835f3d6cf1b0\") " pod="openshift-multus/multus-hnjsh" Sep 10 23:36:51.185553 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.182939 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c90ea9b8-efa1-423c-8d26-e4755af78f53-run-openvswitch\") pod \"ovnkube-node-zrjmz\" (UID: \"c90ea9b8-efa1-423c-8d26-e4755af78f53\") " pod="openshift-ovn-kubernetes/ovnkube-node-zrjmz" Sep 10 23:36:51.185553 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.182723 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c90ea9b8-efa1-423c-8d26-e4755af78f53-env-overrides\") pod \"ovnkube-node-zrjmz\" (UID: \"c90ea9b8-efa1-423c-8d26-e4755af78f53\") " pod="openshift-ovn-kubernetes/ovnkube-node-zrjmz" Sep 10 23:36:51.185553 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.183009 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3ced5f14-cf96-419c-af9b-c218e93b0365-etc-kubernetes\") pod \"tuned-4fp98\" (UID: \"3ced5f14-cf96-419c-af9b-c218e93b0365\") " pod="openshift-cluster-node-tuning-operator/tuned-4fp98" Sep 10 23:36:51.185553 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.183051 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3ced5f14-cf96-419c-af9b-c218e93b0365-sys\") pod \"tuned-4fp98\" (UID: \"3ced5f14-cf96-419c-af9b-c218e93b0365\") " pod="openshift-cluster-node-tuning-operator/tuned-4fp98" Sep 10 23:36:51.186234 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.183131 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3ced5f14-cf96-419c-af9b-c218e93b0365-etc-modprobe-d\") pod \"tuned-4fp98\" (UID: \"3ced5f14-cf96-419c-af9b-c218e93b0365\") " pod="openshift-cluster-node-tuning-operator/tuned-4fp98" Sep 10 23:36:51.186234 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.183158 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4d311f9d-1424-4282-9fd5-5b788bdc4cac-device-dir\") pod \"aws-ebs-csi-driver-node-rsrtp\" (UID: \"4d311f9d-1424-4282-9fd5-5b788bdc4cac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rsrtp" Sep 10 23:36:51.186234 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.183165 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/43e0f2f1-cd70-4612-828f-6cd98e1b77c2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-f47cx\" (UID: \"43e0f2f1-cd70-4612-828f-6cd98e1b77c2\") " pod="openshift-multus/multus-additional-cni-plugins-f47cx" Sep 10 23:36:51.186234 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.183192 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4d311f9d-1424-4282-9fd5-5b788bdc4cac-etc-selinux\") pod \"aws-ebs-csi-driver-node-rsrtp\" (UID: \"4d311f9d-1424-4282-9fd5-5b788bdc4cac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rsrtp" Sep 10 23:36:51.186234 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.183191 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ed113529-7712-4e4f-838a-835f3d6cf1b0-host-run-k8s-cni-cncf-io\") pod \"multus-hnjsh\" (UID: \"ed113529-7712-4e4f-838a-835f3d6cf1b0\") " pod="openshift-multus/multus-hnjsh" Sep 10 23:36:51.186234 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.183216 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gww99\" (UniqueName: \"kubernetes.io/projected/6df7e606-319b-4b9a-b6da-a850ecc9d267-kube-api-access-gww99\") pod \"network-metrics-daemon-v87ts\" (UID: \"6df7e606-319b-4b9a-b6da-a850ecc9d267\") " pod="openshift-multus/network-metrics-daemon-v87ts" Sep 10 23:36:51.186234 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.183230 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4d311f9d-1424-4282-9fd5-5b788bdc4cac-device-dir\") pod \"aws-ebs-csi-driver-node-rsrtp\" (UID: \"4d311f9d-1424-4282-9fd5-5b788bdc4cac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rsrtp" Sep 10 23:36:51.186234 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.183240 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3ced5f14-cf96-419c-af9b-c218e93b0365-var-lib-kubelet\") pod \"tuned-4fp98\" (UID: \"3ced5f14-cf96-419c-af9b-c218e93b0365\") " pod="openshift-cluster-node-tuning-operator/tuned-4fp98" Sep 10 23:36:51.186234 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.183264 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ed113529-7712-4e4f-838a-835f3d6cf1b0-multus-cni-dir\") pod \"multus-hnjsh\" (UID: \"ed113529-7712-4e4f-838a-835f3d6cf1b0\") " pod="openshift-multus/multus-hnjsh" Sep 10 23:36:51.186234 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.183286 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4d311f9d-1424-4282-9fd5-5b788bdc4cac-etc-selinux\") pod \"aws-ebs-csi-driver-node-rsrtp\" (UID: \"4d311f9d-1424-4282-9fd5-5b788bdc4cac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rsrtp" Sep 10 23:36:51.186234 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.183296 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ed113529-7712-4e4f-838a-835f3d6cf1b0-cni-binary-copy\") pod \"multus-hnjsh\" (UID: \"ed113529-7712-4e4f-838a-835f3d6cf1b0\") " pod="openshift-multus/multus-hnjsh" Sep 10 23:36:51.186234 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.183332 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3ced5f14-cf96-419c-af9b-c218e93b0365-var-lib-kubelet\") pod \"tuned-4fp98\" (UID: \"3ced5f14-cf96-419c-af9b-c218e93b0365\") " pod="openshift-cluster-node-tuning-operator/tuned-4fp98" Sep 10 23:36:51.186234 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.183288 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22b9m\" (UniqueName: \"kubernetes.io/projected/4d311f9d-1424-4282-9fd5-5b788bdc4cac-kube-api-access-22b9m\") pod \"aws-ebs-csi-driver-node-rsrtp\" (UID: \"4d311f9d-1424-4282-9fd5-5b788bdc4cac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rsrtp" Sep 10 23:36:51.186234 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.183368 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4hx5\" (UniqueName: \"kubernetes.io/projected/6c4cdd12-89f3-4dbf-be94-04c8c0cb2c4e-kube-api-access-f4hx5\") pod \"iptables-alerter-dds4w\" (UID: \"6c4cdd12-89f3-4dbf-be94-04c8c0cb2c4e\") " pod="openshift-network-operator/iptables-alerter-dds4w" Sep 10 23:36:51.186234 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.183374 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ed113529-7712-4e4f-838a-835f3d6cf1b0-multus-cni-dir\") pod \"multus-hnjsh\" (UID: \"ed113529-7712-4e4f-838a-835f3d6cf1b0\") " pod="openshift-multus/multus-hnjsh" Sep 10 23:36:51.186234 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.183391 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/43e0f2f1-cd70-4612-828f-6cd98e1b77c2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-f47cx\" (UID: \"43e0f2f1-cd70-4612-828f-6cd98e1b77c2\") " pod="openshift-multus/multus-additional-cni-plugins-f47cx" Sep 10 23:36:51.186234 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.183416 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ed113529-7712-4e4f-838a-835f3d6cf1b0-multus-daemon-config\") pod \"multus-hnjsh\" (UID: \"ed113529-7712-4e4f-838a-835f3d6cf1b0\") " pod="openshift-multus/multus-hnjsh" Sep 10 23:36:51.186865 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.183545 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/43e0f2f1-cd70-4612-828f-6cd98e1b77c2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-f47cx\" (UID: \"43e0f2f1-cd70-4612-828f-6cd98e1b77c2\") " pod="openshift-multus/multus-additional-cni-plugins-f47cx" Sep 10 23:36:51.186865 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.183891 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ed113529-7712-4e4f-838a-835f3d6cf1b0-multus-daemon-config\") pod \"multus-hnjsh\" (UID: \"ed113529-7712-4e4f-838a-835f3d6cf1b0\") " pod="openshift-multus/multus-hnjsh" Sep 10 23:36:51.186865 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.184365 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c90ea9b8-efa1-423c-8d26-e4755af78f53-ovn-node-metrics-cert\") pod \"ovnkube-node-zrjmz\" (UID: \"c90ea9b8-efa1-423c-8d26-e4755af78f53\") " pod="openshift-ovn-kubernetes/ovnkube-node-zrjmz" Sep 10 23:36:51.186865 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.184875 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3ced5f14-cf96-419c-af9b-c218e93b0365-tmp\") pod \"tuned-4fp98\" (UID: \"3ced5f14-cf96-419c-af9b-c218e93b0365\") " pod="openshift-cluster-node-tuning-operator/tuned-4fp98" Sep 10 23:36:51.186865 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.185201 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3ced5f14-cf96-419c-af9b-c218e93b0365-etc-tuned\") pod \"tuned-4fp98\" (UID: \"3ced5f14-cf96-419c-af9b-c218e93b0365\") " pod="openshift-cluster-node-tuning-operator/tuned-4fp98" Sep 10 23:36:51.204895 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.204836 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rknnf\" (UniqueName: \"kubernetes.io/projected/3ced5f14-cf96-419c-af9b-c218e93b0365-kube-api-access-rknnf\") pod \"tuned-4fp98\" (UID: \"3ced5f14-cf96-419c-af9b-c218e93b0365\") " pod="openshift-cluster-node-tuning-operator/tuned-4fp98" Sep 10 23:36:51.213597 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.213576 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxfkl\" (UniqueName: \"kubernetes.io/projected/43e0f2f1-cd70-4612-828f-6cd98e1b77c2-kube-api-access-zxfkl\") pod \"multus-additional-cni-plugins-f47cx\" (UID: \"43e0f2f1-cd70-4612-828f-6cd98e1b77c2\") " pod="openshift-multus/multus-additional-cni-plugins-f47cx" Sep 10 23:36:51.216540 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.216512 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8zw6\" (UniqueName: \"kubernetes.io/projected/ed113529-7712-4e4f-838a-835f3d6cf1b0-kube-api-access-p8zw6\") pod \"multus-hnjsh\" (UID: \"ed113529-7712-4e4f-838a-835f3d6cf1b0\") " pod="openshift-multus/multus-hnjsh" Sep 10 23:36:51.216892 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.216867 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-22b9m\" (UniqueName: \"kubernetes.io/projected/4d311f9d-1424-4282-9fd5-5b788bdc4cac-kube-api-access-22b9m\") pod \"aws-ebs-csi-driver-node-rsrtp\" (UID: \"4d311f9d-1424-4282-9fd5-5b788bdc4cac\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rsrtp" Sep 10 23:36:51.217789 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.217763 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gww99\" (UniqueName: \"kubernetes.io/projected/6df7e606-319b-4b9a-b6da-a850ecc9d267-kube-api-access-gww99\") pod \"network-metrics-daemon-v87ts\" (UID: \"6df7e606-319b-4b9a-b6da-a850ecc9d267\") " pod="openshift-multus/network-metrics-daemon-v87ts" Sep 10 23:36:51.217947 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.217928 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gqgt\" (UniqueName: \"kubernetes.io/projected/c90ea9b8-efa1-423c-8d26-e4755af78f53-kube-api-access-8gqgt\") pod \"ovnkube-node-zrjmz\" (UID: \"c90ea9b8-efa1-423c-8d26-e4755af78f53\") " pod="openshift-ovn-kubernetes/ovnkube-node-zrjmz" Sep 10 23:36:51.218361 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.218346 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4hx5\" (UniqueName: \"kubernetes.io/projected/6c4cdd12-89f3-4dbf-be94-04c8c0cb2c4e-kube-api-access-f4hx5\") pod \"iptables-alerter-dds4w\" (UID: \"6c4cdd12-89f3-4dbf-be94-04c8c0cb2c4e\") " pod="openshift-network-operator/iptables-alerter-dds4w" Sep 10 23:36:51.257483 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.257459 2574 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 23:36:51.268980 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.268955 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-dl8bc" Sep 10 23:36:51.278418 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.278401 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6wz5g" Sep 10 23:36:51.287108 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.287088 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-fckpk" Sep 10 23:36:51.293687 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.293667 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-f47cx" Sep 10 23:36:51.301293 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.301272 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-hnjsh" Sep 10 23:36:51.306940 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.306921 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-zrjmz" Sep 10 23:36:51.315633 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.315615 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rsrtp" Sep 10 23:36:51.323190 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.323172 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-4fp98" Sep 10 23:36:51.329629 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.329611 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-dds4w" Sep 10 23:36:51.586134 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.586050 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7wk9\" (UniqueName: \"kubernetes.io/projected/24dda81e-d88e-4f29-836b-44bb86bf461d-kube-api-access-s7wk9\") pod \"network-check-target-zb6x4\" (UID: \"24dda81e-d88e-4f29-836b-44bb86bf461d\") " pod="openshift-network-diagnostics/network-check-target-zb6x4" Sep 10 23:36:51.586270 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:36:51.586233 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 23:36:51.586270 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:36:51.586258 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 23:36:51.586270 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:36:51.586270 2574 projected.go:196] Error preparing data for projected volume kube-api-access-s7wk9 for pod openshift-network-diagnostics/network-check-target-zb6x4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 23:36:51.586386 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:36:51.586328 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/24dda81e-d88e-4f29-836b-44bb86bf461d-kube-api-access-s7wk9 podName:24dda81e-d88e-4f29-836b-44bb86bf461d nodeName:}" failed. No retries permitted until 2026-09-10 23:36:52.58630978 +0000 UTC m=+3.973912536 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-s7wk9" (UniqueName: "kubernetes.io/projected/24dda81e-d88e-4f29-836b-44bb86bf461d-kube-api-access-s7wk9") pod "network-check-target-zb6x4" (UID: "24dda81e-d88e-4f29-836b-44bb86bf461d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 23:36:51.687254 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.687216 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3a0ec520-b33d-49cf-9714-1662f5910e5b-original-pull-secret\") pod \"global-pull-secret-syncer-75vlp\" (UID: \"3a0ec520-b33d-49cf-9714-1662f5910e5b\") " pod="kube-system/global-pull-secret-syncer-75vlp" Sep 10 23:36:51.687254 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:51.687249 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6df7e606-319b-4b9a-b6da-a850ecc9d267-metrics-certs\") pod \"network-metrics-daemon-v87ts\" (UID: \"6df7e606-319b-4b9a-b6da-a850ecc9d267\") " pod="openshift-multus/network-metrics-daemon-v87ts" Sep 10 23:36:51.687468 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:36:51.687336 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 23:36:51.687468 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:36:51.687344 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 23:36:51.687468 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:36:51.687391 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6df7e606-319b-4b9a-b6da-a850ecc9d267-metrics-certs podName:6df7e606-319b-4b9a-b6da-a850ecc9d267 nodeName:}" failed. No retries permitted until 2026-09-10 23:36:52.687373953 +0000 UTC m=+4.074976707 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6df7e606-319b-4b9a-b6da-a850ecc9d267-metrics-certs") pod "network-metrics-daemon-v87ts" (UID: "6df7e606-319b-4b9a-b6da-a850ecc9d267") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 23:36:51.687468 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:36:51.687462 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a0ec520-b33d-49cf-9714-1662f5910e5b-original-pull-secret podName:3a0ec520-b33d-49cf-9714-1662f5910e5b nodeName:}" failed. No retries permitted until 2026-09-10 23:36:52.68744139 +0000 UTC m=+4.075044135 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3a0ec520-b33d-49cf-9714-1662f5910e5b-original-pull-secret") pod "global-pull-secret-syncer-75vlp" (UID: "3a0ec520-b33d-49cf-9714-1662f5910e5b") : object "kube-system"/"original-pull-secret" not registered Sep 10 23:36:52.006781 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:52.006755 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc90ea9b8_efa1_423c_8d26_e4755af78f53.slice/crio-caf022f16c931d26f30897504abdf4fc69832eaf9855b3a04018a5a9610af3a4 WatchSource:0}: Error finding container caf022f16c931d26f30897504abdf4fc69832eaf9855b3a04018a5a9610af3a4: Status 404 returned error can't find the container with id caf022f16c931d26f30897504abdf4fc69832eaf9855b3a04018a5a9610af3a4 Sep 10 23:36:52.010064 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:52.010027 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded113529_7712_4e4f_838a_835f3d6cf1b0.slice/crio-0afbcc869c0900245bffafcbbb4955fb167139e76bc45fc4722d564a4f15370f WatchSource:0}: Error finding container 0afbcc869c0900245bffafcbbb4955fb167139e76bc45fc4722d564a4f15370f: Status 404 returned error can't find the container with id 0afbcc869c0900245bffafcbbb4955fb167139e76bc45fc4722d564a4f15370f Sep 10 23:36:52.022934 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:36:52.021912 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ad8f728_76e5_47a2_8ab2_681ac07591f8.slice/crio-6d6e90ba3e5d042b35aec3ba124efa93ca028306fafdbad056a64ffc28170db1 WatchSource:0}: Error finding container 6d6e90ba3e5d042b35aec3ba124efa93ca028306fafdbad056a64ffc28170db1: Status 404 returned error can't find the container with id 6d6e90ba3e5d042b35aec3ba124efa93ca028306fafdbad056a64ffc28170db1 Sep 10 23:36:52.025362 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:52.025338 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 23:31:51 +0000 UTC" deadline="2028-04-21 21:28:21.061807474 +0000 UTC" Sep 10 23:36:52.025495 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:52.025363 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14133h51m29.036449079s" Sep 10 23:36:52.097927 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:52.097893 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-128.ec2.internal" event={"ID":"577bf0a7d48f3dd63acd7590896a721f","Type":"ContainerStarted","Data":"e1579b208b72a09ac6abe12f8747639eff991327cbb53c5d6c275c83bb82e55d"} Sep 10 23:36:52.100452 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:52.100425 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6wz5g" event={"ID":"5d6d9866-cf11-4f49-877a-ecc951c01e42","Type":"ContainerStarted","Data":"e069d121e302ccffb2cd3d75df02635ba041ffee7fa4deb38f11740557b63588"} Sep 10 23:36:52.101829 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:52.101806 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-4fp98" event={"ID":"3ced5f14-cf96-419c-af9b-c218e93b0365","Type":"ContainerStarted","Data":"a52f1f6a99336900eda73acd5a918f078bf68b917ef242dd85d883461f5572a6"} Sep 10 23:36:52.103055 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:52.103033 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-f47cx" event={"ID":"43e0f2f1-cd70-4612-828f-6cd98e1b77c2","Type":"ContainerStarted","Data":"dc595d90e63814c24b60f753ecd0728a1238b5f9f7624525d6023aad897c9119"} Sep 10 23:36:52.104214 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:52.104190 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rsrtp" event={"ID":"4d311f9d-1424-4282-9fd5-5b788bdc4cac","Type":"ContainerStarted","Data":"46ae4305f300e2198d1d16c56a9efef14f871b65804dc1a2deeeddae174f0ba5"} Sep 10 23:36:52.105499 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:52.105478 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-fckpk" event={"ID":"cbaa689c-2a82-4b1a-b61f-4e8f43efa7d5","Type":"ContainerStarted","Data":"fb012126a7d910d73441bcc39d13aa5ea6e9f36a9dbb7d9c80458160f0108f74"} Sep 10 23:36:52.106883 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:52.106860 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-hnjsh" event={"ID":"ed113529-7712-4e4f-838a-835f3d6cf1b0","Type":"ContainerStarted","Data":"0afbcc869c0900245bffafcbbb4955fb167139e76bc45fc4722d564a4f15370f"} Sep 10 23:36:52.108060 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:52.108040 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-dds4w" event={"ID":"6c4cdd12-89f3-4dbf-be94-04c8c0cb2c4e","Type":"ContainerStarted","Data":"7fc8dbdf127e54017d9017e023478f0758fea77271c32c0d198112d102fa3667"} Sep 10 23:36:52.109306 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:52.109286 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-dl8bc" event={"ID":"7ad8f728-76e5-47a2-8ab2-681ac07591f8","Type":"ContainerStarted","Data":"6d6e90ba3e5d042b35aec3ba124efa93ca028306fafdbad056a64ffc28170db1"} Sep 10 23:36:52.110742 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:52.110720 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zrjmz" event={"ID":"c90ea9b8-efa1-423c-8d26-e4755af78f53","Type":"ContainerStarted","Data":"caf022f16c931d26f30897504abdf4fc69832eaf9855b3a04018a5a9610af3a4"} Sep 10 23:36:52.596304 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:52.595866 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7wk9\" (UniqueName: \"kubernetes.io/projected/24dda81e-d88e-4f29-836b-44bb86bf461d-kube-api-access-s7wk9\") pod \"network-check-target-zb6x4\" (UID: \"24dda81e-d88e-4f29-836b-44bb86bf461d\") " pod="openshift-network-diagnostics/network-check-target-zb6x4" Sep 10 23:36:52.596304 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:36:52.596113 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 23:36:52.596304 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:36:52.596135 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 23:36:52.596304 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:36:52.596148 2574 projected.go:196] Error preparing data for projected volume kube-api-access-s7wk9 for pod openshift-network-diagnostics/network-check-target-zb6x4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 23:36:52.596304 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:36:52.596223 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/24dda81e-d88e-4f29-836b-44bb86bf461d-kube-api-access-s7wk9 podName:24dda81e-d88e-4f29-836b-44bb86bf461d nodeName:}" failed. No retries permitted until 2026-09-10 23:36:54.59619688 +0000 UTC m=+5.983799631 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-s7wk9" (UniqueName: "kubernetes.io/projected/24dda81e-d88e-4f29-836b-44bb86bf461d-kube-api-access-s7wk9") pod "network-check-target-zb6x4" (UID: "24dda81e-d88e-4f29-836b-44bb86bf461d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 23:36:52.697838 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:52.697051 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3a0ec520-b33d-49cf-9714-1662f5910e5b-original-pull-secret\") pod \"global-pull-secret-syncer-75vlp\" (UID: \"3a0ec520-b33d-49cf-9714-1662f5910e5b\") " pod="kube-system/global-pull-secret-syncer-75vlp" Sep 10 23:36:52.697838 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:52.697101 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6df7e606-319b-4b9a-b6da-a850ecc9d267-metrics-certs\") pod \"network-metrics-daemon-v87ts\" (UID: \"6df7e606-319b-4b9a-b6da-a850ecc9d267\") " pod="openshift-multus/network-metrics-daemon-v87ts" Sep 10 23:36:52.697838 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:36:52.697234 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 23:36:52.697838 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:36:52.697296 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6df7e606-319b-4b9a-b6da-a850ecc9d267-metrics-certs podName:6df7e606-319b-4b9a-b6da-a850ecc9d267 nodeName:}" failed. No retries permitted until 2026-09-10 23:36:54.697277306 +0000 UTC m=+6.084880047 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6df7e606-319b-4b9a-b6da-a850ecc9d267-metrics-certs") pod "network-metrics-daemon-v87ts" (UID: "6df7e606-319b-4b9a-b6da-a850ecc9d267") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 23:36:52.697838 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:36:52.697734 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 23:36:52.697838 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:36:52.697802 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a0ec520-b33d-49cf-9714-1662f5910e5b-original-pull-secret podName:3a0ec520-b33d-49cf-9714-1662f5910e5b nodeName:}" failed. No retries permitted until 2026-09-10 23:36:54.697785728 +0000 UTC m=+6.085388468 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3a0ec520-b33d-49cf-9714-1662f5910e5b-original-pull-secret") pod "global-pull-secret-syncer-75vlp" (UID: "3a0ec520-b33d-49cf-9714-1662f5910e5b") : object "kube-system"/"original-pull-secret" not registered Sep 10 23:36:53.026631 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:53.026518 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 23:31:51 +0000 UTC" deadline="2028-04-04 19:50:30.1902476 +0000 UTC" Sep 10 23:36:53.026631 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:53.026581 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13724h13m37.163670969s" Sep 10 23:36:53.089793 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:53.089755 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-75vlp" Sep 10 23:36:53.089944 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:36:53.089889 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-75vlp" podUID="3a0ec520-b33d-49cf-9714-1662f5910e5b" Sep 10 23:36:53.090302 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:53.090282 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v87ts" Sep 10 23:36:53.090426 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:36:53.090389 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v87ts" podUID="6df7e606-319b-4b9a-b6da-a850ecc9d267" Sep 10 23:36:53.090500 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:53.090452 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zb6x4" Sep 10 23:36:53.090550 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:36:53.090521 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zb6x4" podUID="24dda81e-d88e-4f29-836b-44bb86bf461d" Sep 10 23:36:53.120860 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:53.120824 2574 generic.go:356] "Generic (PLEG): container finished" podID="ebf313d069610fccaa30accef842e07e" containerID="20b32f8379fdfc7034fa3b06275a8f1a166e04f84f5910ced092646d08356d42" exitCode=0 Sep 10 23:36:53.121907 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:53.121881 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-128.ec2.internal" event={"ID":"ebf313d069610fccaa30accef842e07e","Type":"ContainerDied","Data":"20b32f8379fdfc7034fa3b06275a8f1a166e04f84f5910ced092646d08356d42"} Sep 10 23:36:53.135184 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:53.135134 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-128.ec2.internal" podStartSLOduration=3.135113742 podStartE2EDuration="3.135113742s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 23:36:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:36:53.133939823 +0000 UTC m=+4.521542590" watchObservedRunningTime="2026-09-10 23:36:53.135113742 +0000 UTC m=+4.522716503" Sep 10 23:36:54.175995 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:54.175957 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-128.ec2.internal" event={"ID":"ebf313d069610fccaa30accef842e07e","Type":"ContainerStarted","Data":"377feae4fb1517c09d611c6d4e318bc34d6f1f725f8899ad6c1580cc0c328f24"} Sep 10 23:36:54.196127 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:54.196070 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-128.ec2.internal" podStartSLOduration=4.196047234 podStartE2EDuration="4.196047234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 23:36:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:36:54.194978164 +0000 UTC m=+5.582580924" watchObservedRunningTime="2026-09-10 23:36:54.196047234 +0000 UTC m=+5.583649995" Sep 10 23:36:54.616634 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:54.615958 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7wk9\" (UniqueName: \"kubernetes.io/projected/24dda81e-d88e-4f29-836b-44bb86bf461d-kube-api-access-s7wk9\") pod \"network-check-target-zb6x4\" (UID: \"24dda81e-d88e-4f29-836b-44bb86bf461d\") " pod="openshift-network-diagnostics/network-check-target-zb6x4" Sep 10 23:36:54.616634 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:36:54.616206 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 23:36:54.616634 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:36:54.616229 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 23:36:54.616634 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:36:54.616242 2574 projected.go:196] Error preparing data for projected volume kube-api-access-s7wk9 for pod openshift-network-diagnostics/network-check-target-zb6x4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 23:36:54.616634 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:36:54.616303 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/24dda81e-d88e-4f29-836b-44bb86bf461d-kube-api-access-s7wk9 podName:24dda81e-d88e-4f29-836b-44bb86bf461d nodeName:}" failed. No retries permitted until 2026-09-10 23:36:58.616284436 +0000 UTC m=+10.003887195 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-s7wk9" (UniqueName: "kubernetes.io/projected/24dda81e-d88e-4f29-836b-44bb86bf461d-kube-api-access-s7wk9") pod "network-check-target-zb6x4" (UID: "24dda81e-d88e-4f29-836b-44bb86bf461d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 23:36:54.717350 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:54.717066 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3a0ec520-b33d-49cf-9714-1662f5910e5b-original-pull-secret\") pod \"global-pull-secret-syncer-75vlp\" (UID: \"3a0ec520-b33d-49cf-9714-1662f5910e5b\") " pod="kube-system/global-pull-secret-syncer-75vlp" Sep 10 23:36:54.717350 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:54.717114 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6df7e606-319b-4b9a-b6da-a850ecc9d267-metrics-certs\") pod \"network-metrics-daemon-v87ts\" (UID: \"6df7e606-319b-4b9a-b6da-a850ecc9d267\") " pod="openshift-multus/network-metrics-daemon-v87ts" Sep 10 23:36:54.717350 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:36:54.717223 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 23:36:54.717350 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:36:54.717273 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 23:36:54.717350 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:36:54.717296 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a0ec520-b33d-49cf-9714-1662f5910e5b-original-pull-secret podName:3a0ec520-b33d-49cf-9714-1662f5910e5b nodeName:}" failed. No retries permitted until 2026-09-10 23:36:58.717276956 +0000 UTC m=+10.104879711 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3a0ec520-b33d-49cf-9714-1662f5910e5b-original-pull-secret") pod "global-pull-secret-syncer-75vlp" (UID: "3a0ec520-b33d-49cf-9714-1662f5910e5b") : object "kube-system"/"original-pull-secret" not registered Sep 10 23:36:54.717350 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:36:54.717324 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6df7e606-319b-4b9a-b6da-a850ecc9d267-metrics-certs podName:6df7e606-319b-4b9a-b6da-a850ecc9d267 nodeName:}" failed. No retries permitted until 2026-09-10 23:36:58.717308508 +0000 UTC m=+10.104911249 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6df7e606-319b-4b9a-b6da-a850ecc9d267-metrics-certs") pod "network-metrics-daemon-v87ts" (UID: "6df7e606-319b-4b9a-b6da-a850ecc9d267") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 23:36:55.088749 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:55.088167 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zb6x4" Sep 10 23:36:55.088749 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:55.088175 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-75vlp" Sep 10 23:36:55.088749 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:36:55.088301 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zb6x4" podUID="24dda81e-d88e-4f29-836b-44bb86bf461d" Sep 10 23:36:55.088749 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:55.088318 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v87ts" Sep 10 23:36:55.088749 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:36:55.088484 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-75vlp" podUID="3a0ec520-b33d-49cf-9714-1662f5910e5b" Sep 10 23:36:55.088749 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:36:55.088572 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v87ts" podUID="6df7e606-319b-4b9a-b6da-a850ecc9d267" Sep 10 23:36:57.088211 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:57.088120 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v87ts" Sep 10 23:36:57.088211 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:57.088163 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zb6x4" Sep 10 23:36:57.088717 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:36:57.088264 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v87ts" podUID="6df7e606-319b-4b9a-b6da-a850ecc9d267" Sep 10 23:36:57.088717 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:57.088120 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-75vlp" Sep 10 23:36:57.088830 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:36:57.088754 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-75vlp" podUID="3a0ec520-b33d-49cf-9714-1662f5910e5b" Sep 10 23:36:57.088881 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:36:57.088841 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zb6x4" podUID="24dda81e-d88e-4f29-836b-44bb86bf461d" Sep 10 23:36:58.653524 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:58.653486 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7wk9\" (UniqueName: \"kubernetes.io/projected/24dda81e-d88e-4f29-836b-44bb86bf461d-kube-api-access-s7wk9\") pod \"network-check-target-zb6x4\" (UID: \"24dda81e-d88e-4f29-836b-44bb86bf461d\") " pod="openshift-network-diagnostics/network-check-target-zb6x4" Sep 10 23:36:58.654020 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:36:58.653803 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 23:36:58.654020 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:36:58.653834 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 23:36:58.654020 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:36:58.653855 2574 projected.go:196] Error preparing data for projected volume kube-api-access-s7wk9 for pod openshift-network-diagnostics/network-check-target-zb6x4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 23:36:58.654020 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:36:58.653929 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/24dda81e-d88e-4f29-836b-44bb86bf461d-kube-api-access-s7wk9 podName:24dda81e-d88e-4f29-836b-44bb86bf461d nodeName:}" failed. No retries permitted until 2026-09-10 23:37:06.653909521 +0000 UTC m=+18.041512273 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-s7wk9" (UniqueName: "kubernetes.io/projected/24dda81e-d88e-4f29-836b-44bb86bf461d-kube-api-access-s7wk9") pod "network-check-target-zb6x4" (UID: "24dda81e-d88e-4f29-836b-44bb86bf461d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 23:36:58.755149 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:58.754474 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3a0ec520-b33d-49cf-9714-1662f5910e5b-original-pull-secret\") pod \"global-pull-secret-syncer-75vlp\" (UID: \"3a0ec520-b33d-49cf-9714-1662f5910e5b\") " pod="kube-system/global-pull-secret-syncer-75vlp" Sep 10 23:36:58.755149 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:58.754527 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6df7e606-319b-4b9a-b6da-a850ecc9d267-metrics-certs\") pod \"network-metrics-daemon-v87ts\" (UID: \"6df7e606-319b-4b9a-b6da-a850ecc9d267\") " pod="openshift-multus/network-metrics-daemon-v87ts" Sep 10 23:36:58.755149 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:36:58.754693 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 23:36:58.755149 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:36:58.754759 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6df7e606-319b-4b9a-b6da-a850ecc9d267-metrics-certs podName:6df7e606-319b-4b9a-b6da-a850ecc9d267 nodeName:}" failed. No retries permitted until 2026-09-10 23:37:06.75473897 +0000 UTC m=+18.142341711 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6df7e606-319b-4b9a-b6da-a850ecc9d267-metrics-certs") pod "network-metrics-daemon-v87ts" (UID: "6df7e606-319b-4b9a-b6da-a850ecc9d267") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 23:36:58.755149 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:36:58.755136 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 23:36:58.755531 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:36:58.755201 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a0ec520-b33d-49cf-9714-1662f5910e5b-original-pull-secret podName:3a0ec520-b33d-49cf-9714-1662f5910e5b nodeName:}" failed. No retries permitted until 2026-09-10 23:37:06.755183573 +0000 UTC m=+18.142786316 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3a0ec520-b33d-49cf-9714-1662f5910e5b-original-pull-secret") pod "global-pull-secret-syncer-75vlp" (UID: "3a0ec520-b33d-49cf-9714-1662f5910e5b") : object "kube-system"/"original-pull-secret" not registered Sep 10 23:36:59.091085 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:59.091013 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zb6x4" Sep 10 23:36:59.091085 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:59.091031 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-75vlp" Sep 10 23:36:59.091085 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:36:59.091050 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v87ts" Sep 10 23:36:59.091322 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:36:59.091136 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zb6x4" podUID="24dda81e-d88e-4f29-836b-44bb86bf461d" Sep 10 23:36:59.091564 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:36:59.091540 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-75vlp" podUID="3a0ec520-b33d-49cf-9714-1662f5910e5b" Sep 10 23:36:59.091813 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:36:59.091687 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v87ts" podUID="6df7e606-319b-4b9a-b6da-a850ecc9d267" Sep 10 23:37:01.090602 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:01.090573 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v87ts" Sep 10 23:37:01.090980 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:01.090687 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zb6x4" Sep 10 23:37:01.090980 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:37:01.090694 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v87ts" podUID="6df7e606-319b-4b9a-b6da-a850ecc9d267" Sep 10 23:37:01.090980 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:01.090713 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-75vlp" Sep 10 23:37:01.090980 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:37:01.090781 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zb6x4" podUID="24dda81e-d88e-4f29-836b-44bb86bf461d" Sep 10 23:37:01.090980 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:37:01.090865 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-75vlp" podUID="3a0ec520-b33d-49cf-9714-1662f5910e5b" Sep 10 23:37:03.087633 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:03.087598 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v87ts" Sep 10 23:37:03.088128 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:03.087598 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-75vlp" Sep 10 23:37:03.088128 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:03.087607 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zb6x4" Sep 10 23:37:03.088128 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:37:03.087833 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-75vlp" podUID="3a0ec520-b33d-49cf-9714-1662f5910e5b" Sep 10 23:37:03.088128 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:37:03.087903 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zb6x4" podUID="24dda81e-d88e-4f29-836b-44bb86bf461d" Sep 10 23:37:03.088128 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:37:03.087792 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v87ts" podUID="6df7e606-319b-4b9a-b6da-a850ecc9d267" Sep 10 23:37:05.088015 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:05.087977 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zb6x4" Sep 10 23:37:05.088490 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:05.087978 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-75vlp" Sep 10 23:37:05.088490 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:37:05.088108 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zb6x4" podUID="24dda81e-d88e-4f29-836b-44bb86bf461d" Sep 10 23:37:05.088490 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:05.088121 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v87ts" Sep 10 23:37:05.088490 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:37:05.088213 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-75vlp" podUID="3a0ec520-b33d-49cf-9714-1662f5910e5b" Sep 10 23:37:05.088490 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:37:05.088310 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v87ts" podUID="6df7e606-319b-4b9a-b6da-a850ecc9d267" Sep 10 23:37:06.710296 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:06.710259 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7wk9\" (UniqueName: \"kubernetes.io/projected/24dda81e-d88e-4f29-836b-44bb86bf461d-kube-api-access-s7wk9\") pod \"network-check-target-zb6x4\" (UID: \"24dda81e-d88e-4f29-836b-44bb86bf461d\") " pod="openshift-network-diagnostics/network-check-target-zb6x4" Sep 10 23:37:06.710799 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:37:06.710390 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 23:37:06.710799 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:37:06.710415 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 23:37:06.710799 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:37:06.710428 2574 projected.go:196] Error preparing data for projected volume kube-api-access-s7wk9 for pod openshift-network-diagnostics/network-check-target-zb6x4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 23:37:06.710799 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:37:06.710488 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/24dda81e-d88e-4f29-836b-44bb86bf461d-kube-api-access-s7wk9 podName:24dda81e-d88e-4f29-836b-44bb86bf461d nodeName:}" failed. No retries permitted until 2026-09-10 23:37:22.71047495 +0000 UTC m=+34.098077687 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-s7wk9" (UniqueName: "kubernetes.io/projected/24dda81e-d88e-4f29-836b-44bb86bf461d-kube-api-access-s7wk9") pod "network-check-target-zb6x4" (UID: "24dda81e-d88e-4f29-836b-44bb86bf461d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 23:37:06.810916 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:06.810883 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3a0ec520-b33d-49cf-9714-1662f5910e5b-original-pull-secret\") pod \"global-pull-secret-syncer-75vlp\" (UID: \"3a0ec520-b33d-49cf-9714-1662f5910e5b\") " pod="kube-system/global-pull-secret-syncer-75vlp" Sep 10 23:37:06.811117 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:06.810926 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6df7e606-319b-4b9a-b6da-a850ecc9d267-metrics-certs\") pod \"network-metrics-daemon-v87ts\" (UID: \"6df7e606-319b-4b9a-b6da-a850ecc9d267\") " pod="openshift-multus/network-metrics-daemon-v87ts" Sep 10 23:37:06.811117 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:37:06.811056 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 23:37:06.811117 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:37:06.811058 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 23:37:06.811267 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:37:06.811128 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6df7e606-319b-4b9a-b6da-a850ecc9d267-metrics-certs podName:6df7e606-319b-4b9a-b6da-a850ecc9d267 nodeName:}" failed. No retries permitted until 2026-09-10 23:37:22.811107035 +0000 UTC m=+34.198709789 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6df7e606-319b-4b9a-b6da-a850ecc9d267-metrics-certs") pod "network-metrics-daemon-v87ts" (UID: "6df7e606-319b-4b9a-b6da-a850ecc9d267") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 23:37:06.811267 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:37:06.811145 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a0ec520-b33d-49cf-9714-1662f5910e5b-original-pull-secret podName:3a0ec520-b33d-49cf-9714-1662f5910e5b nodeName:}" failed. No retries permitted until 2026-09-10 23:37:22.81113728 +0000 UTC m=+34.198740033 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3a0ec520-b33d-49cf-9714-1662f5910e5b-original-pull-secret") pod "global-pull-secret-syncer-75vlp" (UID: "3a0ec520-b33d-49cf-9714-1662f5910e5b") : object "kube-system"/"original-pull-secret" not registered Sep 10 23:37:07.087763 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:07.087625 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-75vlp" Sep 10 23:37:07.087763 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:07.087680 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zb6x4" Sep 10 23:37:07.087763 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:07.087695 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v87ts" Sep 10 23:37:07.088017 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:37:07.087785 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-75vlp" podUID="3a0ec520-b33d-49cf-9714-1662f5910e5b" Sep 10 23:37:07.088017 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:37:07.087866 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zb6x4" podUID="24dda81e-d88e-4f29-836b-44bb86bf461d" Sep 10 23:37:07.088017 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:37:07.087986 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v87ts" podUID="6df7e606-319b-4b9a-b6da-a850ecc9d267" Sep 10 23:37:09.087895 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:09.087850 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zb6x4" Sep 10 23:37:09.088325 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:37:09.087967 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zb6x4" podUID="24dda81e-d88e-4f29-836b-44bb86bf461d" Sep 10 23:37:09.088325 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:09.088013 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-75vlp" Sep 10 23:37:09.088325 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:09.088096 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v87ts" Sep 10 23:37:09.088325 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:37:09.088204 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v87ts" podUID="6df7e606-319b-4b9a-b6da-a850ecc9d267" Sep 10 23:37:09.088325 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:37:09.088291 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-75vlp" podUID="3a0ec520-b33d-49cf-9714-1662f5910e5b" Sep 10 23:37:10.207564 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:10.207197 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6wz5g" event={"ID":"5d6d9866-cf11-4f49-877a-ecc951c01e42","Type":"ContainerStarted","Data":"e49aec00d076dde1b9d5bd2ec6044854de36b4cd46ed7fd3803ff9f6ee5ad610"} Sep 10 23:37:10.209072 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:10.209043 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-4fp98" event={"ID":"3ced5f14-cf96-419c-af9b-c218e93b0365","Type":"ContainerStarted","Data":"96f67f084131bb73822c0bedecefeb844dcf0d88ec34e7b75b1bfbca77e91e74"} Sep 10 23:37:10.211274 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:10.211252 2574 generic.go:356] "Generic (PLEG): container finished" podID="43e0f2f1-cd70-4612-828f-6cd98e1b77c2" containerID="8c27396ac2dfdaaa17b3ec95a38d43434b082a410a67a9d6d84033a69cb32c2e" exitCode=0 Sep 10 23:37:10.211380 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:10.211315 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-f47cx" event={"ID":"43e0f2f1-cd70-4612-828f-6cd98e1b77c2","Type":"ContainerDied","Data":"8c27396ac2dfdaaa17b3ec95a38d43434b082a410a67a9d6d84033a69cb32c2e"} Sep 10 23:37:10.213051 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:10.213029 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rsrtp" event={"ID":"4d311f9d-1424-4282-9fd5-5b788bdc4cac","Type":"ContainerStarted","Data":"5e58d70e856702fc292f278ca986dced1aff529adedf924eb4cbfecbe78027b1"} Sep 10 23:37:10.214947 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:10.214902 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-fckpk" event={"ID":"cbaa689c-2a82-4b1a-b61f-4e8f43efa7d5","Type":"ContainerStarted","Data":"8f0ab912f57b8c5da7a627672e6b536c2ba0564ea6953fd02cb607430b1a2b28"} Sep 10 23:37:10.216575 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:10.216553 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-hnjsh" event={"ID":"ed113529-7712-4e4f-838a-835f3d6cf1b0","Type":"ContainerStarted","Data":"b890dab444b4206679c71492d1fafca6759b1d7f8765c7b2af27ba1a42d8b5b5"} Sep 10 23:37:10.218442 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:10.218417 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-dl8bc" event={"ID":"7ad8f728-76e5-47a2-8ab2-681ac07591f8","Type":"ContainerStarted","Data":"aaf233ff6d3e5dd1222b53ff69220e2f6061c85514e8e022312d6c5828b0f5b5"} Sep 10 23:37:10.220515 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:10.220497 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zrjmz" event={"ID":"c90ea9b8-efa1-423c-8d26-e4755af78f53","Type":"ContainerStarted","Data":"087ad814d0a2774207fa8a127c7d290f8489a71bda772e144cfc204c7f135db0"} Sep 10 23:37:10.220618 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:10.220548 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zrjmz" event={"ID":"c90ea9b8-efa1-423c-8d26-e4755af78f53","Type":"ContainerStarted","Data":"6878ec2977c8c5e547693cf10c35d4ac01be5bd5ccca2ab4c5f8a5edc3a117c5"} Sep 10 23:37:10.220848 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:10.220798 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-6wz5g" podStartSLOduration=3.380696294 podStartE2EDuration="21.220781102s" totalImagesPullingTime="17.840084808s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 23:36:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:37:10.220553475 +0000 UTC m=+21.608156235" watchObservedRunningTime="2026-09-10 23:37:10.220781102 +0000 UTC m=+21.608383887" Sep 10 23:37:10.251424 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:10.251316 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-fckpk" podStartSLOduration=3.4090133910000002 podStartE2EDuration="21.251300938s" totalImagesPullingTime="17.842287547s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 23:36:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:37:10.233309832 +0000 UTC m=+21.620912592" watchObservedRunningTime="2026-09-10 23:37:10.251300938 +0000 UTC m=+21.638903697" Sep 10 23:37:10.266063 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:10.266024 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-4fp98" podStartSLOduration=3.422830744 podStartE2EDuration="21.266009168s" totalImagesPullingTime="17.843178424s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 23:36:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:37:10.265411424 +0000 UTC m=+21.653014185" watchObservedRunningTime="2026-09-10 23:37:10.266009168 +0000 UTC m=+21.653611926" Sep 10 23:37:10.280683 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:10.280620 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-dl8bc" podStartSLOduration=10.886890442 podStartE2EDuration="21.280604917s" totalImagesPullingTime="10.393714475s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:36:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:37:10.280370961 +0000 UTC m=+21.667973719" watchObservedRunningTime="2026-09-10 23:37:10.280604917 +0000 UTC m=+21.668207676" Sep 10 23:37:10.302438 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:10.302399 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-hnjsh" podStartSLOduration=3.399277067 podStartE2EDuration="21.302385465s" totalImagesPullingTime="17.903108398s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 23:36:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:37:10.302093184 +0000 UTC m=+21.689695942" watchObservedRunningTime="2026-09-10 23:37:10.302385465 +0000 UTC m=+21.689988224" Sep 10 23:37:11.087557 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:11.087525 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-75vlp" Sep 10 23:37:11.087761 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:11.087666 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v87ts" Sep 10 23:37:11.087761 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:37:11.087671 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-75vlp" podUID="3a0ec520-b33d-49cf-9714-1662f5910e5b" Sep 10 23:37:11.087761 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:11.087705 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zb6x4" Sep 10 23:37:11.087924 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:37:11.087809 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v87ts" podUID="6df7e606-319b-4b9a-b6da-a850ecc9d267" Sep 10 23:37:11.087924 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:37:11.087890 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zb6x4" podUID="24dda81e-d88e-4f29-836b-44bb86bf461d" Sep 10 23:37:11.227897 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:11.227849 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zrjmz" event={"ID":"c90ea9b8-efa1-423c-8d26-e4755af78f53","Type":"ContainerStarted","Data":"29b2cb2a810dcee53e7454529c53c6080d22ed428f478808c1a716e65a77824d"} Sep 10 23:37:11.228822 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:11.228601 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zrjmz" event={"ID":"c90ea9b8-efa1-423c-8d26-e4755af78f53","Type":"ContainerStarted","Data":"4306422d879a54c6acecebf4cdce9111c42ac02db9ee6007864381efc2c7cf8f"} Sep 10 23:37:11.228962 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:11.228905 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zrjmz" event={"ID":"c90ea9b8-efa1-423c-8d26-e4755af78f53","Type":"ContainerStarted","Data":"5cbd16b22fb82a001b67731931ddea0cf979b8279b3a36c2944b511695ac85fa"} Sep 10 23:37:11.229090 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:11.229061 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zrjmz" event={"ID":"c90ea9b8-efa1-423c-8d26-e4755af78f53","Type":"ContainerStarted","Data":"ba5ae9d5455efb8ca4d2e0a148e1dcc940a85af9f3936a6d62955f416b81dcde"} Sep 10 23:37:11.285970 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:11.285942 2574 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 10 23:37:12.032973 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:12.032849 2574 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-10T23:37:11.28596454Z","UUID":"7e8f20a6-cdb4-4102-b63d-c9f94bf540ad","Handler":null,"Name":"","Endpoint":""} Sep 10 23:37:12.034523 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:12.034501 2574 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 10 23:37:12.034674 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:12.034537 2574 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 10 23:37:12.233056 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:12.233019 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rsrtp" event={"ID":"4d311f9d-1424-4282-9fd5-5b788bdc4cac","Type":"ContainerStarted","Data":"f86b6bba19fd6b53d2b844b2ff36b9a1e930f1f5ea129d6cd2a3dca9f1c9a8c2"} Sep 10 23:37:12.234805 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:12.234772 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-dds4w" event={"ID":"6c4cdd12-89f3-4dbf-be94-04c8c0cb2c4e","Type":"ContainerStarted","Data":"bac71e8ed4d5981165b0305e9a9f3fafc53eadf174c845e3bfd4f4517fee23cf"} Sep 10 23:37:13.088069 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:13.088036 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zb6x4" Sep 10 23:37:13.088260 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:13.088036 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-75vlp" Sep 10 23:37:13.088260 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:37:13.088168 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zb6x4" podUID="24dda81e-d88e-4f29-836b-44bb86bf461d" Sep 10 23:37:13.088260 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:37:13.088242 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-75vlp" podUID="3a0ec520-b33d-49cf-9714-1662f5910e5b" Sep 10 23:37:13.088424 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:13.088036 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v87ts" Sep 10 23:37:13.088424 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:37:13.088342 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v87ts" podUID="6df7e606-319b-4b9a-b6da-a850ecc9d267" Sep 10 23:37:13.240341 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:13.240305 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rsrtp" event={"ID":"4d311f9d-1424-4282-9fd5-5b788bdc4cac","Type":"ContainerStarted","Data":"c32d10d1727103eb82d6680d592548e244067993d8ebd65ab83edf919435b55e"} Sep 10 23:37:13.244505 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:13.244479 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zrjmz" event={"ID":"c90ea9b8-efa1-423c-8d26-e4755af78f53","Type":"ContainerStarted","Data":"b375bc053fcefef9b96fce1d34bbac788ea50acc5c80e0a9ab13148750ccb053"} Sep 10 23:37:13.257093 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:13.257052 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-dds4w" podStartSLOduration=6.414720234 podStartE2EDuration="24.257036997s" totalImagesPullingTime="17.842316763s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 23:36:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:37:12.248506147 +0000 UTC m=+23.636108905" watchObservedRunningTime="2026-09-10 23:37:13.257036997 +0000 UTC m=+24.644639747" Sep 10 23:37:13.257417 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:13.257391 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rsrtp" podStartSLOduration=4.302201265 podStartE2EDuration="24.257383435s" totalImagesPullingTime="19.95518217s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 23:36:49 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:37:13.257009828 +0000 UTC m=+24.644612587" watchObservedRunningTime="2026-09-10 23:37:13.257383435 +0000 UTC m=+24.644986206" Sep 10 23:37:15.087853 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:15.087675 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-75vlp" Sep 10 23:37:15.088523 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:15.087674 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v87ts" Sep 10 23:37:15.088523 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:37:15.087935 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-75vlp" podUID="3a0ec520-b33d-49cf-9714-1662f5910e5b" Sep 10 23:37:15.088523 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:15.087714 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zb6x4" Sep 10 23:37:15.088523 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:37:15.088004 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v87ts" podUID="6df7e606-319b-4b9a-b6da-a850ecc9d267" Sep 10 23:37:15.088523 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:37:15.088109 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zb6x4" podUID="24dda81e-d88e-4f29-836b-44bb86bf461d" Sep 10 23:37:15.135592 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:15.135563 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-dl8bc" Sep 10 23:37:15.136118 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:15.136101 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-dl8bc" Sep 10 23:37:15.251021 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:15.250988 2574 generic.go:356] "Generic (PLEG): container finished" podID="43e0f2f1-cd70-4612-828f-6cd98e1b77c2" containerID="c44f67f29097b4256538b78fd348c6b4d5abf41e5ff8944bd8a0af8492adfad5" exitCode=0 Sep 10 23:37:15.251166 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:15.251065 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-f47cx" event={"ID":"43e0f2f1-cd70-4612-828f-6cd98e1b77c2","Type":"ContainerDied","Data":"c44f67f29097b4256538b78fd348c6b4d5abf41e5ff8944bd8a0af8492adfad5"} Sep 10 23:37:15.256430 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:15.256404 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zrjmz" event={"ID":"c90ea9b8-efa1-423c-8d26-e4755af78f53","Type":"ContainerStarted","Data":"89fb9b285809ca804b371b3e74df3e2cc62dfc5f4791efe9be800929d7344836"} Sep 10 23:37:15.256772 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:15.256751 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-dl8bc" Sep 10 23:37:15.257116 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:15.257095 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-dl8bc" Sep 10 23:37:15.308060 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:15.308035 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6wz5g_5d6d9866-cf11-4f49-877a-ecc951c01e42/dns-node-resolver/0.log" Sep 10 23:37:15.353436 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:15.353393 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-zrjmz" podStartSLOduration=8.451953668 podStartE2EDuration="26.353376829s" totalImagesPullingTime="17.901423161s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 23:36:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:37:15.353269186 +0000 UTC m=+26.740871946" watchObservedRunningTime="2026-09-10 23:37:15.353376829 +0000 UTC m=+26.740979587" Sep 10 23:37:16.258821 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:16.258782 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zrjmz" Sep 10 23:37:16.259296 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:16.259230 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zrjmz" Sep 10 23:37:16.259375 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:16.259301 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zrjmz" Sep 10 23:37:16.273466 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:16.273321 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zrjmz" Sep 10 23:37:16.273614 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:16.273575 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zrjmz" Sep 10 23:37:16.668143 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:16.668114 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-fckpk_cbaa689c-2a82-4b1a-b61f-4e8f43efa7d5/node-ca/0.log" Sep 10 23:37:16.752847 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:16.752813 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-75vlp"] Sep 10 23:37:16.753016 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:16.752973 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-75vlp" Sep 10 23:37:16.753075 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:37:16.753056 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-75vlp" podUID="3a0ec520-b33d-49cf-9714-1662f5910e5b" Sep 10 23:37:16.756342 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:16.756258 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-v87ts"] Sep 10 23:37:16.756475 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:16.756373 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v87ts" Sep 10 23:37:16.756553 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:37:16.756481 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v87ts" podUID="6df7e606-319b-4b9a-b6da-a850ecc9d267" Sep 10 23:37:16.756848 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:16.756830 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-zb6x4"] Sep 10 23:37:16.756943 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:16.756908 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zb6x4" Sep 10 23:37:16.757011 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:37:16.756982 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zb6x4" podUID="24dda81e-d88e-4f29-836b-44bb86bf461d" Sep 10 23:37:17.262725 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:17.262692 2574 generic.go:356] "Generic (PLEG): container finished" podID="43e0f2f1-cd70-4612-828f-6cd98e1b77c2" containerID="720e16e31fca69daba41b33fcaafeef90e6256147637b16ea968f433943fcbf7" exitCode=0 Sep 10 23:37:17.263314 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:17.262771 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-f47cx" event={"ID":"43e0f2f1-cd70-4612-828f-6cd98e1b77c2","Type":"ContainerDied","Data":"720e16e31fca69daba41b33fcaafeef90e6256147637b16ea968f433943fcbf7"} Sep 10 23:37:18.088019 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:18.087988 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-75vlp" Sep 10 23:37:18.088168 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:18.087998 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v87ts" Sep 10 23:37:18.088168 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:37:18.088092 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-75vlp" podUID="3a0ec520-b33d-49cf-9714-1662f5910e5b" Sep 10 23:37:18.088253 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:37:18.088213 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v87ts" podUID="6df7e606-319b-4b9a-b6da-a850ecc9d267" Sep 10 23:37:19.088126 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:19.088093 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zb6x4" Sep 10 23:37:19.088593 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:37:19.088183 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zb6x4" podUID="24dda81e-d88e-4f29-836b-44bb86bf461d" Sep 10 23:37:19.270502 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:19.270350 2574 generic.go:356] "Generic (PLEG): container finished" podID="43e0f2f1-cd70-4612-828f-6cd98e1b77c2" containerID="865a334da69e32369659bcb3553d6975c70516a16dbb7254b344147d616d79f1" exitCode=0 Sep 10 23:37:19.270502 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:19.270403 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-f47cx" event={"ID":"43e0f2f1-cd70-4612-828f-6cd98e1b77c2","Type":"ContainerDied","Data":"865a334da69e32369659bcb3553d6975c70516a16dbb7254b344147d616d79f1"} Sep 10 23:37:20.087507 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:20.087465 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-75vlp" Sep 10 23:37:20.087706 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:20.087473 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v87ts" Sep 10 23:37:20.087706 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:37:20.087582 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-75vlp" podUID="3a0ec520-b33d-49cf-9714-1662f5910e5b" Sep 10 23:37:20.087706 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:37:20.087668 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v87ts" podUID="6df7e606-319b-4b9a-b6da-a850ecc9d267" Sep 10 23:37:21.087451 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:21.087418 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zb6x4" Sep 10 23:37:21.087952 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:37:21.087540 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zb6x4" podUID="24dda81e-d88e-4f29-836b-44bb86bf461d" Sep 10 23:37:22.087814 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:22.087777 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v87ts" Sep 10 23:37:22.088271 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:22.087822 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-75vlp" Sep 10 23:37:22.088271 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:37:22.087924 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v87ts" podUID="6df7e606-319b-4b9a-b6da-a850ecc9d267" Sep 10 23:37:22.088271 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:37:22.088057 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-75vlp" podUID="3a0ec520-b33d-49cf-9714-1662f5910e5b" Sep 10 23:37:22.730257 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:22.730218 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7wk9\" (UniqueName: \"kubernetes.io/projected/24dda81e-d88e-4f29-836b-44bb86bf461d-kube-api-access-s7wk9\") pod \"network-check-target-zb6x4\" (UID: \"24dda81e-d88e-4f29-836b-44bb86bf461d\") " pod="openshift-network-diagnostics/network-check-target-zb6x4" Sep 10 23:37:22.730551 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:37:22.730410 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 23:37:22.730551 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:37:22.730437 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 23:37:22.730551 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:37:22.730450 2574 projected.go:196] Error preparing data for projected volume kube-api-access-s7wk9 for pod openshift-network-diagnostics/network-check-target-zb6x4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 23:37:22.730551 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:37:22.730522 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/24dda81e-d88e-4f29-836b-44bb86bf461d-kube-api-access-s7wk9 podName:24dda81e-d88e-4f29-836b-44bb86bf461d nodeName:}" failed. No retries permitted until 2026-09-10 23:37:54.730500607 +0000 UTC m=+66.118103369 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-s7wk9" (UniqueName: "kubernetes.io/projected/24dda81e-d88e-4f29-836b-44bb86bf461d-kube-api-access-s7wk9") pod "network-check-target-zb6x4" (UID: "24dda81e-d88e-4f29-836b-44bb86bf461d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 23:37:22.831353 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:22.831315 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3a0ec520-b33d-49cf-9714-1662f5910e5b-original-pull-secret\") pod \"global-pull-secret-syncer-75vlp\" (UID: \"3a0ec520-b33d-49cf-9714-1662f5910e5b\") " pod="kube-system/global-pull-secret-syncer-75vlp" Sep 10 23:37:22.831543 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:22.831363 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6df7e606-319b-4b9a-b6da-a850ecc9d267-metrics-certs\") pod \"network-metrics-daemon-v87ts\" (UID: \"6df7e606-319b-4b9a-b6da-a850ecc9d267\") " pod="openshift-multus/network-metrics-daemon-v87ts" Sep 10 23:37:22.831543 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:37:22.831475 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 23:37:22.831543 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:37:22.831534 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6df7e606-319b-4b9a-b6da-a850ecc9d267-metrics-certs podName:6df7e606-319b-4b9a-b6da-a850ecc9d267 nodeName:}" failed. No retries permitted until 2026-09-10 23:37:54.831519932 +0000 UTC m=+66.219122669 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6df7e606-319b-4b9a-b6da-a850ecc9d267-metrics-certs") pod "network-metrics-daemon-v87ts" (UID: "6df7e606-319b-4b9a-b6da-a850ecc9d267") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 23:37:22.831746 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:37:22.831475 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 23:37:22.831746 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:37:22.831624 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a0ec520-b33d-49cf-9714-1662f5910e5b-original-pull-secret podName:3a0ec520-b33d-49cf-9714-1662f5910e5b nodeName:}" failed. No retries permitted until 2026-09-10 23:37:54.831610397 +0000 UTC m=+66.219213147 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3a0ec520-b33d-49cf-9714-1662f5910e5b-original-pull-secret") pod "global-pull-secret-syncer-75vlp" (UID: "3a0ec520-b33d-49cf-9714-1662f5910e5b") : object "kube-system"/"original-pull-secret" not registered Sep 10 23:37:23.087611 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:23.087532 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zb6x4" Sep 10 23:37:23.087783 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:37:23.087634 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zb6x4" podUID="24dda81e-d88e-4f29-836b-44bb86bf461d" Sep 10 23:37:23.283233 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:23.283166 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-f47cx" event={"ID":"43e0f2f1-cd70-4612-828f-6cd98e1b77c2","Type":"ContainerStarted","Data":"6b973f37b95456754f242bc387b15cd370c60dfe9234f5b0604c510b17c5b78d"} Sep 10 23:37:24.087653 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:24.087608 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v87ts" Sep 10 23:37:24.087807 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:37:24.087741 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v87ts" podUID="6df7e606-319b-4b9a-b6da-a850ecc9d267" Sep 10 23:37:24.087807 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:24.087773 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-75vlp" Sep 10 23:37:24.087882 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:37:24.087839 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-75vlp" podUID="3a0ec520-b33d-49cf-9714-1662f5910e5b" Sep 10 23:37:24.287652 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:24.287607 2574 generic.go:356] "Generic (PLEG): container finished" podID="43e0f2f1-cd70-4612-828f-6cd98e1b77c2" containerID="6b973f37b95456754f242bc387b15cd370c60dfe9234f5b0604c510b17c5b78d" exitCode=0 Sep 10 23:37:24.288011 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:24.287687 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-f47cx" event={"ID":"43e0f2f1-cd70-4612-828f-6cd98e1b77c2","Type":"ContainerDied","Data":"6b973f37b95456754f242bc387b15cd370c60dfe9234f5b0604c510b17c5b78d"} Sep 10 23:37:25.088166 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:25.088131 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zb6x4" Sep 10 23:37:25.088346 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:37:25.088264 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zb6x4" podUID="24dda81e-d88e-4f29-836b-44bb86bf461d" Sep 10 23:37:25.292959 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:25.292923 2574 generic.go:356] "Generic (PLEG): container finished" podID="43e0f2f1-cd70-4612-828f-6cd98e1b77c2" containerID="d77c991d6879df14d32486fd600dfa6d9dca1d101b4278cd7bac143343125697" exitCode=0 Sep 10 23:37:25.293443 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:25.292975 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-f47cx" event={"ID":"43e0f2f1-cd70-4612-828f-6cd98e1b77c2","Type":"ContainerDied","Data":"d77c991d6879df14d32486fd600dfa6d9dca1d101b4278cd7bac143343125697"} Sep 10 23:37:26.088042 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:26.087997 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-75vlp" Sep 10 23:37:26.088042 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:26.088021 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v87ts" Sep 10 23:37:26.088307 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:37:26.088133 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-75vlp" podUID="3a0ec520-b33d-49cf-9714-1662f5910e5b" Sep 10 23:37:26.088307 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:37:26.088240 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v87ts" podUID="6df7e606-319b-4b9a-b6da-a850ecc9d267" Sep 10 23:37:26.298117 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:26.298082 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-f47cx" event={"ID":"43e0f2f1-cd70-4612-828f-6cd98e1b77c2","Type":"ContainerStarted","Data":"a0cc3d67376e15e5fa6278dbe1cc82515b9fee45d3c1c6244f61adf8c7687be0"} Sep 10 23:37:26.321715 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:26.321670 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-f47cx" podStartSLOduration=8.9890403 podStartE2EDuration="37.321631088s" totalImagesPullingTime="28.332590788s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 23:36:49 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:37:26.320296857 +0000 UTC m=+37.707899616" watchObservedRunningTime="2026-09-10 23:37:26.321631088 +0000 UTC m=+37.709233847" Sep 10 23:37:27.087596 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:27.087560 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zb6x4" Sep 10 23:37:27.087809 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:37:27.087688 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zb6x4" podUID="24dda81e-d88e-4f29-836b-44bb86bf461d" Sep 10 23:37:28.087899 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:28.087866 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v87ts" Sep 10 23:37:28.088276 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:37:28.087978 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v87ts" podUID="6df7e606-319b-4b9a-b6da-a850ecc9d267" Sep 10 23:37:28.088276 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:28.088036 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-75vlp" Sep 10 23:37:28.088276 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:37:28.088143 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-75vlp" podUID="3a0ec520-b33d-49cf-9714-1662f5910e5b" Sep 10 23:37:29.088025 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:29.087995 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zb6x4" Sep 10 23:37:29.088370 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:37:29.088074 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zb6x4" podUID="24dda81e-d88e-4f29-836b-44bb86bf461d" Sep 10 23:37:30.087564 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:30.087529 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v87ts" Sep 10 23:37:30.087763 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:37:30.087656 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v87ts" podUID="6df7e606-319b-4b9a-b6da-a850ecc9d267" Sep 10 23:37:30.087763 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:30.087696 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-75vlp" Sep 10 23:37:30.087838 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:37:30.087785 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-75vlp" podUID="3a0ec520-b33d-49cf-9714-1662f5910e5b" Sep 10 23:37:31.087982 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:31.087948 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zb6x4" Sep 10 23:37:31.088361 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:37:31.088050 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zb6x4" podUID="24dda81e-d88e-4f29-836b-44bb86bf461d" Sep 10 23:37:32.087699 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:32.087656 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-75vlp" Sep 10 23:37:32.087871 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:32.087677 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v87ts" Sep 10 23:37:32.087871 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:37:32.087763 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-75vlp" podUID="3a0ec520-b33d-49cf-9714-1662f5910e5b" Sep 10 23:37:32.087871 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:37:32.087836 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v87ts" podUID="6df7e606-319b-4b9a-b6da-a850ecc9d267" Sep 10 23:37:33.087983 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:33.087943 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zb6x4" Sep 10 23:37:33.088372 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:37:33.088076 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zb6x4" podUID="24dda81e-d88e-4f29-836b-44bb86bf461d" Sep 10 23:37:34.088107 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:34.088074 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v87ts" Sep 10 23:37:34.088472 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:37:34.088196 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v87ts" podUID="6df7e606-319b-4b9a-b6da-a850ecc9d267" Sep 10 23:37:34.088472 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:34.088248 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-75vlp" Sep 10 23:37:34.088472 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:37:34.088339 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-75vlp" podUID="3a0ec520-b33d-49cf-9714-1662f5910e5b" Sep 10 23:37:35.087504 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:35.087463 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zb6x4" Sep 10 23:37:35.087721 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:37:35.087575 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zb6x4" podUID="24dda81e-d88e-4f29-836b-44bb86bf461d" Sep 10 23:37:36.087558 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:36.087521 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v87ts" Sep 10 23:37:36.087961 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:36.087566 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-75vlp" Sep 10 23:37:36.087961 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:37:36.087631 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v87ts" podUID="6df7e606-319b-4b9a-b6da-a850ecc9d267" Sep 10 23:37:36.087961 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:37:36.087741 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-75vlp" podUID="3a0ec520-b33d-49cf-9714-1662f5910e5b" Sep 10 23:37:36.867218 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:36.867190 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-128.ec2.internal" event="NodeReady" Sep 10 23:37:36.867367 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:36.867299 2574 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 10 23:37:36.930299 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:36.930271 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-6bbbz"] Sep 10 23:37:36.956580 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:36.956555 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-l25dt"] Sep 10 23:37:36.956704 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:36.956686 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6bbbz" Sep 10 23:37:36.960128 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:36.960101 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-f9sdk\"" Sep 10 23:37:36.960262 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:36.960201 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 10 23:37:36.961137 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:36.961118 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 10 23:37:36.976678 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:36.976660 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-6bbbz"] Sep 10 23:37:36.976762 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:36.976751 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-l25dt" Sep 10 23:37:36.980092 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:36.980075 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 10 23:37:36.980092 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:36.980087 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 10 23:37:36.980279 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:36.980114 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 10 23:37:36.980279 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:36.980230 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 10 23:37:36.980564 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:36.980548 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-bptlm\"" Sep 10 23:37:36.980813 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:36.980794 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-l25dt"] Sep 10 23:37:37.052971 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:37.052945 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-48dgb"] Sep 10 23:37:37.074086 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:37.074065 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-48dgb"] Sep 10 23:37:37.074204 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:37.074148 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-48dgb" Sep 10 23:37:37.077074 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:37.076877 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 10 23:37:37.077074 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:37.076981 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 10 23:37:37.077074 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:37.077013 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-qfssj\"" Sep 10 23:37:37.077074 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:37.077021 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 10 23:37:37.087616 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:37.087597 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zb6x4" Sep 10 23:37:37.090218 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:37.090201 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 10 23:37:37.090490 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:37.090472 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 10 23:37:37.090540 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:37.090517 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-ppbqc\"" Sep 10 23:37:37.127602 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:37.127549 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3ebfe403-2315-4056-8daa-58d73e056140-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l25dt\" (UID: \"3ebfe403-2315-4056-8daa-58d73e056140\") " pod="openshift-insights/insights-runtime-extractor-l25dt" Sep 10 23:37:37.127602 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:37.127578 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6baf5c56-fb47-42e5-bc9e-d41ff149c33d-metrics-tls\") pod \"dns-default-6bbbz\" (UID: \"6baf5c56-fb47-42e5-bc9e-d41ff149c33d\") " pod="openshift-dns/dns-default-6bbbz" Sep 10 23:37:37.127602 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:37.127596 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3ebfe403-2315-4056-8daa-58d73e056140-data-volume\") pod \"insights-runtime-extractor-l25dt\" (UID: \"3ebfe403-2315-4056-8daa-58d73e056140\") " pod="openshift-insights/insights-runtime-extractor-l25dt" Sep 10 23:37:37.127765 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:37.127628 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3ebfe403-2315-4056-8daa-58d73e056140-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l25dt\" (UID: \"3ebfe403-2315-4056-8daa-58d73e056140\") " pod="openshift-insights/insights-runtime-extractor-l25dt" Sep 10 23:37:37.127765 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:37.127717 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkqvd\" (UniqueName: \"kubernetes.io/projected/6baf5c56-fb47-42e5-bc9e-d41ff149c33d-kube-api-access-hkqvd\") pod \"dns-default-6bbbz\" (UID: \"6baf5c56-fb47-42e5-bc9e-d41ff149c33d\") " pod="openshift-dns/dns-default-6bbbz" Sep 10 23:37:37.127765 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:37.127739 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3ebfe403-2315-4056-8daa-58d73e056140-crio-socket\") pod \"insights-runtime-extractor-l25dt\" (UID: \"3ebfe403-2315-4056-8daa-58d73e056140\") " pod="openshift-insights/insights-runtime-extractor-l25dt" Sep 10 23:37:37.127858 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:37.127780 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6baf5c56-fb47-42e5-bc9e-d41ff149c33d-config-volume\") pod \"dns-default-6bbbz\" (UID: \"6baf5c56-fb47-42e5-bc9e-d41ff149c33d\") " pod="openshift-dns/dns-default-6bbbz" Sep 10 23:37:37.127858 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:37.127811 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qx8m\" (UniqueName: \"kubernetes.io/projected/3ebfe403-2315-4056-8daa-58d73e056140-kube-api-access-4qx8m\") pod \"insights-runtime-extractor-l25dt\" (UID: \"3ebfe403-2315-4056-8daa-58d73e056140\") " pod="openshift-insights/insights-runtime-extractor-l25dt" Sep 10 23:37:37.127858 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:37.127843 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6baf5c56-fb47-42e5-bc9e-d41ff149c33d-tmp-dir\") pod \"dns-default-6bbbz\" (UID: \"6baf5c56-fb47-42e5-bc9e-d41ff149c33d\") " pod="openshift-dns/dns-default-6bbbz" Sep 10 23:37:37.228971 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:37.228931 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkqvd\" (UniqueName: \"kubernetes.io/projected/6baf5c56-fb47-42e5-bc9e-d41ff149c33d-kube-api-access-hkqvd\") pod \"dns-default-6bbbz\" (UID: \"6baf5c56-fb47-42e5-bc9e-d41ff149c33d\") " pod="openshift-dns/dns-default-6bbbz" Sep 10 23:37:37.228971 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:37.228968 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3ebfe403-2315-4056-8daa-58d73e056140-crio-socket\") pod \"insights-runtime-extractor-l25dt\" (UID: \"3ebfe403-2315-4056-8daa-58d73e056140\") " pod="openshift-insights/insights-runtime-extractor-l25dt" Sep 10 23:37:37.229179 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:37.229155 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6baf5c56-fb47-42e5-bc9e-d41ff149c33d-config-volume\") pod \"dns-default-6bbbz\" (UID: \"6baf5c56-fb47-42e5-bc9e-d41ff149c33d\") " pod="openshift-dns/dns-default-6bbbz" Sep 10 23:37:37.229216 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:37.229199 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qx8m\" (UniqueName: \"kubernetes.io/projected/3ebfe403-2315-4056-8daa-58d73e056140-kube-api-access-4qx8m\") pod \"insights-runtime-extractor-l25dt\" (UID: \"3ebfe403-2315-4056-8daa-58d73e056140\") " pod="openshift-insights/insights-runtime-extractor-l25dt" Sep 10 23:37:37.229258 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:37.229227 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/47d30ad8-0b9d-4c37-942d-b31a444e8420-cert\") pod \"ingress-canary-48dgb\" (UID: \"47d30ad8-0b9d-4c37-942d-b31a444e8420\") " pod="openshift-ingress-canary/ingress-canary-48dgb" Sep 10 23:37:37.229354 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:37.229340 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6baf5c56-fb47-42e5-bc9e-d41ff149c33d-tmp-dir\") pod \"dns-default-6bbbz\" (UID: \"6baf5c56-fb47-42e5-bc9e-d41ff149c33d\") " pod="openshift-dns/dns-default-6bbbz" Sep 10 23:37:37.229401 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:37.229355 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3ebfe403-2315-4056-8daa-58d73e056140-crio-socket\") pod \"insights-runtime-extractor-l25dt\" (UID: \"3ebfe403-2315-4056-8daa-58d73e056140\") " pod="openshift-insights/insights-runtime-extractor-l25dt" Sep 10 23:37:37.229401 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:37.229387 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rj9x\" (UniqueName: \"kubernetes.io/projected/47d30ad8-0b9d-4c37-942d-b31a444e8420-kube-api-access-9rj9x\") pod \"ingress-canary-48dgb\" (UID: \"47d30ad8-0b9d-4c37-942d-b31a444e8420\") " pod="openshift-ingress-canary/ingress-canary-48dgb" Sep 10 23:37:37.229474 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:37.229409 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3ebfe403-2315-4056-8daa-58d73e056140-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l25dt\" (UID: \"3ebfe403-2315-4056-8daa-58d73e056140\") " pod="openshift-insights/insights-runtime-extractor-l25dt" Sep 10 23:37:37.229474 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:37.229434 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6baf5c56-fb47-42e5-bc9e-d41ff149c33d-metrics-tls\") pod \"dns-default-6bbbz\" (UID: \"6baf5c56-fb47-42e5-bc9e-d41ff149c33d\") " pod="openshift-dns/dns-default-6bbbz" Sep 10 23:37:37.229474 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:37.229458 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3ebfe403-2315-4056-8daa-58d73e056140-data-volume\") pod \"insights-runtime-extractor-l25dt\" (UID: \"3ebfe403-2315-4056-8daa-58d73e056140\") " pod="openshift-insights/insights-runtime-extractor-l25dt" Sep 10 23:37:37.229608 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:37.229494 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3ebfe403-2315-4056-8daa-58d73e056140-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l25dt\" (UID: \"3ebfe403-2315-4056-8daa-58d73e056140\") " pod="openshift-insights/insights-runtime-extractor-l25dt" Sep 10 23:37:37.229683 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:37.229602 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6baf5c56-fb47-42e5-bc9e-d41ff149c33d-tmp-dir\") pod \"dns-default-6bbbz\" (UID: \"6baf5c56-fb47-42e5-bc9e-d41ff149c33d\") " pod="openshift-dns/dns-default-6bbbz" Sep 10 23:37:37.229860 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:37.229839 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3ebfe403-2315-4056-8daa-58d73e056140-data-volume\") pod \"insights-runtime-extractor-l25dt\" (UID: \"3ebfe403-2315-4056-8daa-58d73e056140\") " pod="openshift-insights/insights-runtime-extractor-l25dt" Sep 10 23:37:37.229927 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:37.229859 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6baf5c56-fb47-42e5-bc9e-d41ff149c33d-config-volume\") pod \"dns-default-6bbbz\" (UID: \"6baf5c56-fb47-42e5-bc9e-d41ff149c33d\") " pod="openshift-dns/dns-default-6bbbz" Sep 10 23:37:37.230070 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:37.230044 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3ebfe403-2315-4056-8daa-58d73e056140-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l25dt\" (UID: \"3ebfe403-2315-4056-8daa-58d73e056140\") " pod="openshift-insights/insights-runtime-extractor-l25dt" Sep 10 23:37:37.233311 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:37.233283 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6baf5c56-fb47-42e5-bc9e-d41ff149c33d-metrics-tls\") pod \"dns-default-6bbbz\" (UID: \"6baf5c56-fb47-42e5-bc9e-d41ff149c33d\") " pod="openshift-dns/dns-default-6bbbz" Sep 10 23:37:37.233420 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:37.233350 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3ebfe403-2315-4056-8daa-58d73e056140-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l25dt\" (UID: \"3ebfe403-2315-4056-8daa-58d73e056140\") " pod="openshift-insights/insights-runtime-extractor-l25dt" Sep 10 23:37:37.236869 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:37.236846 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkqvd\" (UniqueName: \"kubernetes.io/projected/6baf5c56-fb47-42e5-bc9e-d41ff149c33d-kube-api-access-hkqvd\") pod \"dns-default-6bbbz\" (UID: \"6baf5c56-fb47-42e5-bc9e-d41ff149c33d\") " pod="openshift-dns/dns-default-6bbbz" Sep 10 23:37:37.237051 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:37.237030 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qx8m\" (UniqueName: \"kubernetes.io/projected/3ebfe403-2315-4056-8daa-58d73e056140-kube-api-access-4qx8m\") pod \"insights-runtime-extractor-l25dt\" (UID: \"3ebfe403-2315-4056-8daa-58d73e056140\") " pod="openshift-insights/insights-runtime-extractor-l25dt" Sep 10 23:37:37.266949 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:37.266917 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6bbbz" Sep 10 23:37:37.285055 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:37.285033 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-l25dt" Sep 10 23:37:37.330290 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:37.330255 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/47d30ad8-0b9d-4c37-942d-b31a444e8420-cert\") pod \"ingress-canary-48dgb\" (UID: \"47d30ad8-0b9d-4c37-942d-b31a444e8420\") " pod="openshift-ingress-canary/ingress-canary-48dgb" Sep 10 23:37:37.330410 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:37.330328 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rj9x\" (UniqueName: \"kubernetes.io/projected/47d30ad8-0b9d-4c37-942d-b31a444e8420-kube-api-access-9rj9x\") pod \"ingress-canary-48dgb\" (UID: \"47d30ad8-0b9d-4c37-942d-b31a444e8420\") " pod="openshift-ingress-canary/ingress-canary-48dgb" Sep 10 23:37:37.332489 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:37.332464 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/47d30ad8-0b9d-4c37-942d-b31a444e8420-cert\") pod \"ingress-canary-48dgb\" (UID: \"47d30ad8-0b9d-4c37-942d-b31a444e8420\") " pod="openshift-ingress-canary/ingress-canary-48dgb" Sep 10 23:37:37.338811 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:37.338786 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rj9x\" (UniqueName: \"kubernetes.io/projected/47d30ad8-0b9d-4c37-942d-b31a444e8420-kube-api-access-9rj9x\") pod \"ingress-canary-48dgb\" (UID: \"47d30ad8-0b9d-4c37-942d-b31a444e8420\") " pod="openshift-ingress-canary/ingress-canary-48dgb" Sep 10 23:37:37.383263 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:37.383192 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-48dgb" Sep 10 23:37:37.598584 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:37.598558 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-68cb69c474-tfm62"] Sep 10 23:37:37.609025 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:37:37.608998 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6baf5c56_fb47_42e5_bc9e_d41ff149c33d.slice/crio-a9eb3c9863421701afff5f57b758cc8d8306a53faed336e803ed47d5beb150c2 WatchSource:0}: Error finding container a9eb3c9863421701afff5f57b758cc8d8306a53faed336e803ed47d5beb150c2: Status 404 returned error can't find the container with id a9eb3c9863421701afff5f57b758cc8d8306a53faed336e803ed47d5beb150c2 Sep 10 23:37:37.609615 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:37:37.609596 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ebfe403_2315_4056_8daa_58d73e056140.slice/crio-ce9df65f6dac8b09a386e458b0075e25571b1faf49fa53d76995204f0c1add40 WatchSource:0}: Error finding container ce9df65f6dac8b09a386e458b0075e25571b1faf49fa53d76995204f0c1add40: Status 404 returned error can't find the container with id ce9df65f6dac8b09a386e458b0075e25571b1faf49fa53d76995204f0c1add40 Sep 10 23:37:37.627568 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:37.627547 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-6bbbz"] Sep 10 23:37:37.627688 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:37.627591 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-l25dt"] Sep 10 23:37:37.627688 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:37.627625 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-68cb69c474-tfm62"] Sep 10 23:37:37.627688 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:37.627669 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68cb69c474-tfm62" Sep 10 23:37:37.629663 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:37.629632 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 10 23:37:37.629911 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:37.629897 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 10 23:37:37.630365 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:37.630346 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 10 23:37:37.630447 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:37.630428 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 10 23:37:37.630504 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:37.630461 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 10 23:37:37.630504 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:37.630485 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 10 23:37:37.630752 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:37.630739 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 10 23:37:37.630812 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:37.630795 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-b54lm\"" Sep 10 23:37:37.678680 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:37.678659 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-48dgb"] Sep 10 23:37:37.682117 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:37:37.682095 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47d30ad8_0b9d_4c37_942d_b31a444e8420.slice/crio-e671095ae28f0cc5bb9d7804fe0521f854eb7edbb8769d604adb8a35eb50fd73 WatchSource:0}: Error finding container e671095ae28f0cc5bb9d7804fe0521f854eb7edbb8769d604adb8a35eb50fd73: Status 404 returned error can't find the container with id e671095ae28f0cc5bb9d7804fe0521f854eb7edbb8769d604adb8a35eb50fd73 Sep 10 23:37:37.735636 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:37.735609 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8b9ade2c-dd2a-4be4-9002-c9086d64e5b1-console-serving-cert\") pod \"console-68cb69c474-tfm62\" (UID: \"8b9ade2c-dd2a-4be4-9002-c9086d64e5b1\") " pod="openshift-console/console-68cb69c474-tfm62" Sep 10 23:37:37.735728 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:37.735678 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8b9ade2c-dd2a-4be4-9002-c9086d64e5b1-console-config\") pod \"console-68cb69c474-tfm62\" (UID: \"8b9ade2c-dd2a-4be4-9002-c9086d64e5b1\") " pod="openshift-console/console-68cb69c474-tfm62" Sep 10 23:37:37.735728 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:37.735706 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndzl8\" (UniqueName: \"kubernetes.io/projected/8b9ade2c-dd2a-4be4-9002-c9086d64e5b1-kube-api-access-ndzl8\") pod \"console-68cb69c474-tfm62\" (UID: \"8b9ade2c-dd2a-4be4-9002-c9086d64e5b1\") " pod="openshift-console/console-68cb69c474-tfm62" Sep 10 23:37:37.735809 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:37.735755 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8b9ade2c-dd2a-4be4-9002-c9086d64e5b1-service-ca\") pod \"console-68cb69c474-tfm62\" (UID: \"8b9ade2c-dd2a-4be4-9002-c9086d64e5b1\") " pod="openshift-console/console-68cb69c474-tfm62" Sep 10 23:37:37.735865 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:37.735817 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8b9ade2c-dd2a-4be4-9002-c9086d64e5b1-oauth-serving-cert\") pod \"console-68cb69c474-tfm62\" (UID: \"8b9ade2c-dd2a-4be4-9002-c9086d64e5b1\") " pod="openshift-console/console-68cb69c474-tfm62" Sep 10 23:37:37.735865 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:37.735848 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8b9ade2c-dd2a-4be4-9002-c9086d64e5b1-console-oauth-config\") pod \"console-68cb69c474-tfm62\" (UID: \"8b9ade2c-dd2a-4be4-9002-c9086d64e5b1\") " pod="openshift-console/console-68cb69c474-tfm62" Sep 10 23:37:37.837089 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:37.837063 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8b9ade2c-dd2a-4be4-9002-c9086d64e5b1-oauth-serving-cert\") pod \"console-68cb69c474-tfm62\" (UID: \"8b9ade2c-dd2a-4be4-9002-c9086d64e5b1\") " pod="openshift-console/console-68cb69c474-tfm62" Sep 10 23:37:37.837194 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:37.837114 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8b9ade2c-dd2a-4be4-9002-c9086d64e5b1-console-oauth-config\") pod \"console-68cb69c474-tfm62\" (UID: \"8b9ade2c-dd2a-4be4-9002-c9086d64e5b1\") " pod="openshift-console/console-68cb69c474-tfm62" Sep 10 23:37:37.837194 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:37.837155 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8b9ade2c-dd2a-4be4-9002-c9086d64e5b1-console-serving-cert\") pod \"console-68cb69c474-tfm62\" (UID: \"8b9ade2c-dd2a-4be4-9002-c9086d64e5b1\") " pod="openshift-console/console-68cb69c474-tfm62" Sep 10 23:37:37.837194 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:37.837174 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8b9ade2c-dd2a-4be4-9002-c9086d64e5b1-console-config\") pod \"console-68cb69c474-tfm62\" (UID: \"8b9ade2c-dd2a-4be4-9002-c9086d64e5b1\") " pod="openshift-console/console-68cb69c474-tfm62" Sep 10 23:37:37.837194 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:37.837193 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndzl8\" (UniqueName: \"kubernetes.io/projected/8b9ade2c-dd2a-4be4-9002-c9086d64e5b1-kube-api-access-ndzl8\") pod \"console-68cb69c474-tfm62\" (UID: \"8b9ade2c-dd2a-4be4-9002-c9086d64e5b1\") " pod="openshift-console/console-68cb69c474-tfm62" Sep 10 23:37:37.837385 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:37.837208 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8b9ade2c-dd2a-4be4-9002-c9086d64e5b1-service-ca\") pod \"console-68cb69c474-tfm62\" (UID: \"8b9ade2c-dd2a-4be4-9002-c9086d64e5b1\") " pod="openshift-console/console-68cb69c474-tfm62" Sep 10 23:37:37.837822 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:37.837796 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8b9ade2c-dd2a-4be4-9002-c9086d64e5b1-oauth-serving-cert\") pod \"console-68cb69c474-tfm62\" (UID: \"8b9ade2c-dd2a-4be4-9002-c9086d64e5b1\") " pod="openshift-console/console-68cb69c474-tfm62" Sep 10 23:37:37.837934 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:37.837915 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8b9ade2c-dd2a-4be4-9002-c9086d64e5b1-console-config\") pod \"console-68cb69c474-tfm62\" (UID: \"8b9ade2c-dd2a-4be4-9002-c9086d64e5b1\") " pod="openshift-console/console-68cb69c474-tfm62" Sep 10 23:37:37.837934 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:37.837928 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8b9ade2c-dd2a-4be4-9002-c9086d64e5b1-service-ca\") pod \"console-68cb69c474-tfm62\" (UID: \"8b9ade2c-dd2a-4be4-9002-c9086d64e5b1\") " pod="openshift-console/console-68cb69c474-tfm62" Sep 10 23:37:37.839428 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:37.839408 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8b9ade2c-dd2a-4be4-9002-c9086d64e5b1-console-oauth-config\") pod \"console-68cb69c474-tfm62\" (UID: \"8b9ade2c-dd2a-4be4-9002-c9086d64e5b1\") " pod="openshift-console/console-68cb69c474-tfm62" Sep 10 23:37:37.839505 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:37.839492 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8b9ade2c-dd2a-4be4-9002-c9086d64e5b1-console-serving-cert\") pod \"console-68cb69c474-tfm62\" (UID: \"8b9ade2c-dd2a-4be4-9002-c9086d64e5b1\") " pod="openshift-console/console-68cb69c474-tfm62" Sep 10 23:37:37.844211 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:37.844191 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndzl8\" (UniqueName: \"kubernetes.io/projected/8b9ade2c-dd2a-4be4-9002-c9086d64e5b1-kube-api-access-ndzl8\") pod \"console-68cb69c474-tfm62\" (UID: \"8b9ade2c-dd2a-4be4-9002-c9086d64e5b1\") " pod="openshift-console/console-68cb69c474-tfm62" Sep 10 23:37:37.936764 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:37.936718 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68cb69c474-tfm62" Sep 10 23:37:38.088007 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:38.087974 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v87ts" Sep 10 23:37:38.088451 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:38.087975 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-75vlp" Sep 10 23:37:38.090367 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:38.090346 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 10 23:37:38.090997 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:38.090974 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 10 23:37:38.091094 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:38.090974 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-n6qg4\"" Sep 10 23:37:38.245678 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:38.245583 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-68cb69c474-tfm62"] Sep 10 23:37:38.326791 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:38.326752 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68cb69c474-tfm62" event={"ID":"8b9ade2c-dd2a-4be4-9002-c9086d64e5b1","Type":"ContainerStarted","Data":"3e3c90429262aa455c9f949798427a997fa7c5ec0c1705a949426c341c8afad2"} Sep 10 23:37:38.329356 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:38.329320 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-l25dt" event={"ID":"3ebfe403-2315-4056-8daa-58d73e056140","Type":"ContainerStarted","Data":"7ae525eca052af0d6483ccb682126ce3102f6a228d2bfbe897d91a8aa85a7bd2"} Sep 10 23:37:38.329486 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:38.329389 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-l25dt" event={"ID":"3ebfe403-2315-4056-8daa-58d73e056140","Type":"ContainerStarted","Data":"ce9df65f6dac8b09a386e458b0075e25571b1faf49fa53d76995204f0c1add40"} Sep 10 23:37:38.332398 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:38.332370 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6bbbz" event={"ID":"6baf5c56-fb47-42e5-bc9e-d41ff149c33d","Type":"ContainerStarted","Data":"a9eb3c9863421701afff5f57b758cc8d8306a53faed336e803ed47d5beb150c2"} Sep 10 23:37:38.335057 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:38.335027 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-48dgb" event={"ID":"47d30ad8-0b9d-4c37-942d-b31a444e8420","Type":"ContainerStarted","Data":"e671095ae28f0cc5bb9d7804fe0521f854eb7edbb8769d604adb8a35eb50fd73"} Sep 10 23:37:39.342813 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:39.342754 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-l25dt" event={"ID":"3ebfe403-2315-4056-8daa-58d73e056140","Type":"ContainerStarted","Data":"c52ecae8bfff93863addcb840309195e84a1923ee68b9e729d1d03217ad0227a"} Sep 10 23:37:42.177441 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:42.177410 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-dkphk"] Sep 10 23:37:42.179530 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:42.179502 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-dkphk" Sep 10 23:37:42.182055 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:42.182029 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-pj5x7\"" Sep 10 23:37:42.182160 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:42.182036 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Sep 10 23:37:42.191370 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:42.191329 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-dkphk"] Sep 10 23:37:42.272555 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:42.272474 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/f8b518fd-885c-42e8-b59f-a1d271e750dc-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-dkphk\" (UID: \"f8b518fd-885c-42e8-b59f-a1d271e750dc\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-dkphk" Sep 10 23:37:42.352529 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:42.352498 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-48dgb" event={"ID":"47d30ad8-0b9d-4c37-942d-b31a444e8420","Type":"ContainerStarted","Data":"809378db8b727060e250861c1534eadcb4fdcb8d06f5c966d0cfe9b22a53c898"} Sep 10 23:37:42.354108 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:42.354085 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68cb69c474-tfm62" event={"ID":"8b9ade2c-dd2a-4be4-9002-c9086d64e5b1","Type":"ContainerStarted","Data":"a1e6fbc81f5d113ba74830ffa8979f84ff87b75deb04be4eaf5ccf273b4903e2"} Sep 10 23:37:42.356178 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:42.356154 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-l25dt" event={"ID":"3ebfe403-2315-4056-8daa-58d73e056140","Type":"ContainerStarted","Data":"60e9ace24caef7dc0e18bc84a630742eadfde7d2e42ece61173e934bd1a0a941"} Sep 10 23:37:42.357808 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:42.357788 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6bbbz" event={"ID":"6baf5c56-fb47-42e5-bc9e-d41ff149c33d","Type":"ContainerStarted","Data":"799db0f92f4688c9aa755376a46cce2bdb4483fe61566755d22c516f05686a9d"} Sep 10 23:37:42.357887 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:42.357822 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6bbbz" event={"ID":"6baf5c56-fb47-42e5-bc9e-d41ff149c33d","Type":"ContainerStarted","Data":"78d41dc13b875420bd9a7084aca8b151627b54e82eb8368035ad726cf90f375c"} Sep 10 23:37:42.357933 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:42.357912 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-6bbbz" Sep 10 23:37:42.366904 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:42.366866 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-48dgb" podStartSLOduration=1.088457253 podStartE2EDuration="5.366852943s" totalImagesPullingTime="4.27839569s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:37:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:37:42.36631491 +0000 UTC m=+53.753917671" watchObservedRunningTime="2026-09-10 23:37:42.366852943 +0000 UTC m=+53.754455696" Sep 10 23:37:42.373562 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:42.373530 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/f8b518fd-885c-42e8-b59f-a1d271e750dc-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-dkphk\" (UID: \"f8b518fd-885c-42e8-b59f-a1d271e750dc\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-dkphk" Sep 10 23:37:42.373710 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:37:42.373682 2574 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found Sep 10 23:37:42.373780 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:37:42.373771 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8b518fd-885c-42e8-b59f-a1d271e750dc-tls-certificates podName:f8b518fd-885c-42e8-b59f-a1d271e750dc nodeName:}" failed. No retries permitted until 2026-09-10 23:37:42.873749326 +0000 UTC m=+54.261352079 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/f8b518fd-885c-42e8-b59f-a1d271e750dc-tls-certificates") pod "prometheus-operator-admission-webhook-5864cd99d7-dkphk" (UID: "f8b518fd-885c-42e8-b59f-a1d271e750dc") : secret "prometheus-operator-admission-webhook-tls" not found Sep 10 23:37:42.386405 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:42.384232 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-6bbbz" podStartSLOduration=2.032670873 podStartE2EDuration="6.384211747s" totalImagesPullingTime="4.351540874s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:37:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:37:42.38309682 +0000 UTC m=+53.770699578" watchObservedRunningTime="2026-09-10 23:37:42.384211747 +0000 UTC m=+53.771814509" Sep 10 23:37:42.401133 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:42.401094 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-l25dt" podStartSLOduration=2.250603594 podStartE2EDuration="6.401081823s" totalImagesPullingTime="4.150478229s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 23:37:36 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:37:42.400266571 +0000 UTC m=+53.787869499" watchObservedRunningTime="2026-09-10 23:37:42.401081823 +0000 UTC m=+53.788684581" Sep 10 23:37:42.414943 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:42.414899 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-68cb69c474-tfm62" podStartSLOduration=1.70689818 podStartE2EDuration="5.414887659s" totalImagesPullingTime="3.707989479s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:37:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:37:42.414791171 +0000 UTC m=+53.802393954" watchObservedRunningTime="2026-09-10 23:37:42.414887659 +0000 UTC m=+53.802490409" Sep 10 23:37:42.877146 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:42.877094 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/f8b518fd-885c-42e8-b59f-a1d271e750dc-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-dkphk\" (UID: \"f8b518fd-885c-42e8-b59f-a1d271e750dc\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-dkphk" Sep 10 23:37:42.879605 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:42.879583 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/f8b518fd-885c-42e8-b59f-a1d271e750dc-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-dkphk\" (UID: \"f8b518fd-885c-42e8-b59f-a1d271e750dc\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-dkphk" Sep 10 23:37:43.088433 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:43.088386 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-dkphk" Sep 10 23:37:43.352845 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:43.352813 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-dkphk"] Sep 10 23:37:43.355729 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:37:43.355695 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8b518fd_885c_42e8_b59f_a1d271e750dc.slice/crio-754a332a6cfac56ce0051a17e1ede14d33888cde96f6845047d8b2c496a122b1 WatchSource:0}: Error finding container 754a332a6cfac56ce0051a17e1ede14d33888cde96f6845047d8b2c496a122b1: Status 404 returned error can't find the container with id 754a332a6cfac56ce0051a17e1ede14d33888cde96f6845047d8b2c496a122b1 Sep 10 23:37:43.362088 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:43.362052 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-dkphk" event={"ID":"f8b518fd-885c-42e8-b59f-a1d271e750dc","Type":"ContainerStarted","Data":"754a332a6cfac56ce0051a17e1ede14d33888cde96f6845047d8b2c496a122b1"} Sep 10 23:37:45.371084 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:45.371046 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-dkphk" event={"ID":"f8b518fd-885c-42e8-b59f-a1d271e750dc","Type":"ContainerStarted","Data":"71a9f866428823d5bb794a4ff7d9a44e091443b236843c5b0aeeb44f1ec3a715"} Sep 10 23:37:45.371438 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:45.371247 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-dkphk" Sep 10 23:37:45.375771 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:45.375749 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-dkphk" Sep 10 23:37:45.386351 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:45.386315 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-dkphk" podStartSLOduration=2.424136513 podStartE2EDuration="3.386301165s" totalImagesPullingTime="962.164652ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:37:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:37:45.385771814 +0000 UTC m=+56.773374581" watchObservedRunningTime="2026-09-10 23:37:45.386301165 +0000 UTC m=+56.773903923" Sep 10 23:37:46.231840 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:46.231802 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-kckq4"] Sep 10 23:37:46.233935 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:46.233919 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-kckq4" Sep 10 23:37:46.236403 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:46.236375 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Sep 10 23:37:46.236403 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:46.236395 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 10 23:37:46.236837 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:46.236820 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Sep 10 23:37:46.237457 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:46.237437 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 10 23:37:46.237545 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:46.237522 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 10 23:37:46.237610 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:46.237565 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-g7slm\"" Sep 10 23:37:46.245101 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:46.245081 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-kckq4"] Sep 10 23:37:46.302229 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:46.302202 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/061f75eb-fab8-4777-a086-0d6b2c9abd7c-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-kckq4\" (UID: \"061f75eb-fab8-4777-a086-0d6b2c9abd7c\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-kckq4" Sep 10 23:37:46.302388 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:46.302235 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2j9d\" (UniqueName: \"kubernetes.io/projected/061f75eb-fab8-4777-a086-0d6b2c9abd7c-kube-api-access-b2j9d\") pod \"prometheus-operator-68bcd7f987-kckq4\" (UID: \"061f75eb-fab8-4777-a086-0d6b2c9abd7c\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-kckq4" Sep 10 23:37:46.302388 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:46.302315 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/061f75eb-fab8-4777-a086-0d6b2c9abd7c-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-kckq4\" (UID: \"061f75eb-fab8-4777-a086-0d6b2c9abd7c\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-kckq4" Sep 10 23:37:46.302388 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:46.302354 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/061f75eb-fab8-4777-a086-0d6b2c9abd7c-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-kckq4\" (UID: \"061f75eb-fab8-4777-a086-0d6b2c9abd7c\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-kckq4" Sep 10 23:37:46.403404 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:46.403373 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/061f75eb-fab8-4777-a086-0d6b2c9abd7c-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-kckq4\" (UID: \"061f75eb-fab8-4777-a086-0d6b2c9abd7c\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-kckq4" Sep 10 23:37:46.403861 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:46.403414 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2j9d\" (UniqueName: \"kubernetes.io/projected/061f75eb-fab8-4777-a086-0d6b2c9abd7c-kube-api-access-b2j9d\") pod \"prometheus-operator-68bcd7f987-kckq4\" (UID: \"061f75eb-fab8-4777-a086-0d6b2c9abd7c\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-kckq4" Sep 10 23:37:46.403861 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:46.403487 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/061f75eb-fab8-4777-a086-0d6b2c9abd7c-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-kckq4\" (UID: \"061f75eb-fab8-4777-a086-0d6b2c9abd7c\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-kckq4" Sep 10 23:37:46.403861 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:37:46.403517 2574 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Sep 10 23:37:46.403861 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:46.403540 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/061f75eb-fab8-4777-a086-0d6b2c9abd7c-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-kckq4\" (UID: \"061f75eb-fab8-4777-a086-0d6b2c9abd7c\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-kckq4" Sep 10 23:37:46.403861 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:37:46.403602 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/061f75eb-fab8-4777-a086-0d6b2c9abd7c-prometheus-operator-tls podName:061f75eb-fab8-4777-a086-0d6b2c9abd7c nodeName:}" failed. No retries permitted until 2026-09-10 23:37:46.903584485 +0000 UTC m=+58.291187226 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/061f75eb-fab8-4777-a086-0d6b2c9abd7c-prometheus-operator-tls") pod "prometheus-operator-68bcd7f987-kckq4" (UID: "061f75eb-fab8-4777-a086-0d6b2c9abd7c") : secret "prometheus-operator-tls" not found Sep 10 23:37:46.404210 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:46.404191 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/061f75eb-fab8-4777-a086-0d6b2c9abd7c-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-kckq4\" (UID: \"061f75eb-fab8-4777-a086-0d6b2c9abd7c\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-kckq4" Sep 10 23:37:46.405884 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:46.405862 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/061f75eb-fab8-4777-a086-0d6b2c9abd7c-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-kckq4\" (UID: \"061f75eb-fab8-4777-a086-0d6b2c9abd7c\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-kckq4" Sep 10 23:37:46.411777 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:46.411758 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2j9d\" (UniqueName: \"kubernetes.io/projected/061f75eb-fab8-4777-a086-0d6b2c9abd7c-kube-api-access-b2j9d\") pod \"prometheus-operator-68bcd7f987-kckq4\" (UID: \"061f75eb-fab8-4777-a086-0d6b2c9abd7c\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-kckq4" Sep 10 23:37:46.546045 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:46.545980 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-b89f77d6-6zkxh"] Sep 10 23:37:46.547797 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:46.547781 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b89f77d6-6zkxh" Sep 10 23:37:46.555459 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:46.555441 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Sep 10 23:37:46.560728 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:46.560664 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-b89f77d6-6zkxh"] Sep 10 23:37:46.604759 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:46.604728 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8p6r5\" (UniqueName: \"kubernetes.io/projected/deaa3139-0842-40a9-907a-306517f2ede3-kube-api-access-8p6r5\") pod \"console-b89f77d6-6zkxh\" (UID: \"deaa3139-0842-40a9-907a-306517f2ede3\") " pod="openshift-console/console-b89f77d6-6zkxh" Sep 10 23:37:46.604932 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:46.604773 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/deaa3139-0842-40a9-907a-306517f2ede3-service-ca\") pod \"console-b89f77d6-6zkxh\" (UID: \"deaa3139-0842-40a9-907a-306517f2ede3\") " pod="openshift-console/console-b89f77d6-6zkxh" Sep 10 23:37:46.604932 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:46.604802 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/deaa3139-0842-40a9-907a-306517f2ede3-console-oauth-config\") pod \"console-b89f77d6-6zkxh\" (UID: \"deaa3139-0842-40a9-907a-306517f2ede3\") " pod="openshift-console/console-b89f77d6-6zkxh" Sep 10 23:37:46.604932 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:46.604862 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/deaa3139-0842-40a9-907a-306517f2ede3-console-config\") pod \"console-b89f77d6-6zkxh\" (UID: \"deaa3139-0842-40a9-907a-306517f2ede3\") " pod="openshift-console/console-b89f77d6-6zkxh" Sep 10 23:37:46.604932 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:46.604888 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/deaa3139-0842-40a9-907a-306517f2ede3-console-serving-cert\") pod \"console-b89f77d6-6zkxh\" (UID: \"deaa3139-0842-40a9-907a-306517f2ede3\") " pod="openshift-console/console-b89f77d6-6zkxh" Sep 10 23:37:46.604932 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:46.604904 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/deaa3139-0842-40a9-907a-306517f2ede3-trusted-ca-bundle\") pod \"console-b89f77d6-6zkxh\" (UID: \"deaa3139-0842-40a9-907a-306517f2ede3\") " pod="openshift-console/console-b89f77d6-6zkxh" Sep 10 23:37:46.604932 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:46.604927 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/deaa3139-0842-40a9-907a-306517f2ede3-oauth-serving-cert\") pod \"console-b89f77d6-6zkxh\" (UID: \"deaa3139-0842-40a9-907a-306517f2ede3\") " pod="openshift-console/console-b89f77d6-6zkxh" Sep 10 23:37:46.706144 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:46.706099 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/deaa3139-0842-40a9-907a-306517f2ede3-console-config\") pod \"console-b89f77d6-6zkxh\" (UID: \"deaa3139-0842-40a9-907a-306517f2ede3\") " pod="openshift-console/console-b89f77d6-6zkxh" Sep 10 23:37:46.706144 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:46.706139 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/deaa3139-0842-40a9-907a-306517f2ede3-console-serving-cert\") pod \"console-b89f77d6-6zkxh\" (UID: \"deaa3139-0842-40a9-907a-306517f2ede3\") " pod="openshift-console/console-b89f77d6-6zkxh" Sep 10 23:37:46.706347 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:46.706155 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/deaa3139-0842-40a9-907a-306517f2ede3-trusted-ca-bundle\") pod \"console-b89f77d6-6zkxh\" (UID: \"deaa3139-0842-40a9-907a-306517f2ede3\") " pod="openshift-console/console-b89f77d6-6zkxh" Sep 10 23:37:46.706347 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:46.706186 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/deaa3139-0842-40a9-907a-306517f2ede3-oauth-serving-cert\") pod \"console-b89f77d6-6zkxh\" (UID: \"deaa3139-0842-40a9-907a-306517f2ede3\") " pod="openshift-console/console-b89f77d6-6zkxh" Sep 10 23:37:46.706347 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:46.706246 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8p6r5\" (UniqueName: \"kubernetes.io/projected/deaa3139-0842-40a9-907a-306517f2ede3-kube-api-access-8p6r5\") pod \"console-b89f77d6-6zkxh\" (UID: \"deaa3139-0842-40a9-907a-306517f2ede3\") " pod="openshift-console/console-b89f77d6-6zkxh" Sep 10 23:37:46.706347 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:46.706282 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/deaa3139-0842-40a9-907a-306517f2ede3-service-ca\") pod \"console-b89f77d6-6zkxh\" (UID: \"deaa3139-0842-40a9-907a-306517f2ede3\") " pod="openshift-console/console-b89f77d6-6zkxh" Sep 10 23:37:46.706347 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:46.706310 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/deaa3139-0842-40a9-907a-306517f2ede3-console-oauth-config\") pod \"console-b89f77d6-6zkxh\" (UID: \"deaa3139-0842-40a9-907a-306517f2ede3\") " pod="openshift-console/console-b89f77d6-6zkxh" Sep 10 23:37:46.707060 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:46.707025 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/deaa3139-0842-40a9-907a-306517f2ede3-console-config\") pod \"console-b89f77d6-6zkxh\" (UID: \"deaa3139-0842-40a9-907a-306517f2ede3\") " pod="openshift-console/console-b89f77d6-6zkxh" Sep 10 23:37:46.707060 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:46.707048 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/deaa3139-0842-40a9-907a-306517f2ede3-service-ca\") pod \"console-b89f77d6-6zkxh\" (UID: \"deaa3139-0842-40a9-907a-306517f2ede3\") " pod="openshift-console/console-b89f77d6-6zkxh" Sep 10 23:37:46.707222 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:46.707033 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/deaa3139-0842-40a9-907a-306517f2ede3-oauth-serving-cert\") pod \"console-b89f77d6-6zkxh\" (UID: \"deaa3139-0842-40a9-907a-306517f2ede3\") " pod="openshift-console/console-b89f77d6-6zkxh" Sep 10 23:37:46.707222 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:46.707160 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/deaa3139-0842-40a9-907a-306517f2ede3-trusted-ca-bundle\") pod \"console-b89f77d6-6zkxh\" (UID: \"deaa3139-0842-40a9-907a-306517f2ede3\") " pod="openshift-console/console-b89f77d6-6zkxh" Sep 10 23:37:46.708582 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:46.708555 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/deaa3139-0842-40a9-907a-306517f2ede3-console-serving-cert\") pod \"console-b89f77d6-6zkxh\" (UID: \"deaa3139-0842-40a9-907a-306517f2ede3\") " pod="openshift-console/console-b89f77d6-6zkxh" Sep 10 23:37:46.708707 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:46.708661 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/deaa3139-0842-40a9-907a-306517f2ede3-console-oauth-config\") pod \"console-b89f77d6-6zkxh\" (UID: \"deaa3139-0842-40a9-907a-306517f2ede3\") " pod="openshift-console/console-b89f77d6-6zkxh" Sep 10 23:37:46.716162 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:46.716135 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8p6r5\" (UniqueName: \"kubernetes.io/projected/deaa3139-0842-40a9-907a-306517f2ede3-kube-api-access-8p6r5\") pod \"console-b89f77d6-6zkxh\" (UID: \"deaa3139-0842-40a9-907a-306517f2ede3\") " pod="openshift-console/console-b89f77d6-6zkxh" Sep 10 23:37:46.856440 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:46.856361 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b89f77d6-6zkxh" Sep 10 23:37:46.908435 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:46.908407 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/061f75eb-fab8-4777-a086-0d6b2c9abd7c-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-kckq4\" (UID: \"061f75eb-fab8-4777-a086-0d6b2c9abd7c\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-kckq4" Sep 10 23:37:46.910933 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:46.910913 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/061f75eb-fab8-4777-a086-0d6b2c9abd7c-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-kckq4\" (UID: \"061f75eb-fab8-4777-a086-0d6b2c9abd7c\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-kckq4" Sep 10 23:37:47.122819 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:47.122742 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-b89f77d6-6zkxh"] Sep 10 23:37:47.125928 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:37:47.125893 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddeaa3139_0842_40a9_907a_306517f2ede3.slice/crio-96efcc8e4d324c12332af84dbfd7700c5e983fcf106db1b2407ee616dd5e2af0 WatchSource:0}: Error finding container 96efcc8e4d324c12332af84dbfd7700c5e983fcf106db1b2407ee616dd5e2af0: Status 404 returned error can't find the container with id 96efcc8e4d324c12332af84dbfd7700c5e983fcf106db1b2407ee616dd5e2af0 Sep 10 23:37:47.142805 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:47.142781 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-kckq4" Sep 10 23:37:47.379338 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:47.379258 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b89f77d6-6zkxh" event={"ID":"deaa3139-0842-40a9-907a-306517f2ede3","Type":"ContainerStarted","Data":"24a5e4150b9a11d59298f3a7630e96d175482ce6714b82fc8e748b64cfdcfd17"} Sep 10 23:37:47.379338 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:47.379314 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b89f77d6-6zkxh" event={"ID":"deaa3139-0842-40a9-907a-306517f2ede3","Type":"ContainerStarted","Data":"96efcc8e4d324c12332af84dbfd7700c5e983fcf106db1b2407ee616dd5e2af0"} Sep 10 23:37:47.396722 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:47.396667 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-b89f77d6-6zkxh" podStartSLOduration=1.396631831 podStartE2EDuration="1.396631831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:37:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:37:47.396127658 +0000 UTC m=+58.783730417" watchObservedRunningTime="2026-09-10 23:37:47.396631831 +0000 UTC m=+58.784234589" Sep 10 23:37:47.440772 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:47.440742 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-kckq4"] Sep 10 23:37:47.443347 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:37:47.443321 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod061f75eb_fab8_4777_a086_0d6b2c9abd7c.slice/crio-9309379e802f88b9d69cb0aae3bc053308f1be1f38fd77a36cb403062ffd3fd6 WatchSource:0}: Error finding container 9309379e802f88b9d69cb0aae3bc053308f1be1f38fd77a36cb403062ffd3fd6: Status 404 returned error can't find the container with id 9309379e802f88b9d69cb0aae3bc053308f1be1f38fd77a36cb403062ffd3fd6 Sep 10 23:37:47.937848 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:47.937797 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-68cb69c474-tfm62" Sep 10 23:37:47.938025 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:47.937878 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-68cb69c474-tfm62" Sep 10 23:37:47.943357 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:47.943319 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-68cb69c474-tfm62" Sep 10 23:37:48.285395 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:48.285363 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zrjmz" Sep 10 23:37:48.383668 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:48.383614 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-kckq4" event={"ID":"061f75eb-fab8-4777-a086-0d6b2c9abd7c","Type":"ContainerStarted","Data":"9309379e802f88b9d69cb0aae3bc053308f1be1f38fd77a36cb403062ffd3fd6"} Sep 10 23:37:48.387976 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:48.387952 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-68cb69c474-tfm62" Sep 10 23:37:49.388656 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:49.388608 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-kckq4" event={"ID":"061f75eb-fab8-4777-a086-0d6b2c9abd7c","Type":"ContainerStarted","Data":"794d7688138a2521317114229b15e6d4fc7e20490b664ac096ddd380499a4fe4"} Sep 10 23:37:49.389003 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:49.388681 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-kckq4" event={"ID":"061f75eb-fab8-4777-a086-0d6b2c9abd7c","Type":"ContainerStarted","Data":"e9bf733e5fc696b36f5170685336ecf694016bce1a2f382030c814ce10ded99c"} Sep 10 23:37:49.411124 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:49.411073 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-68bcd7f987-kckq4" podStartSLOduration=1.9842545139999999 podStartE2EDuration="3.411056333s" totalImagesPullingTime="1.426801819s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:37:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:37:49.409728969 +0000 UTC m=+60.797331705" watchObservedRunningTime="2026-09-10 23:37:49.411056333 +0000 UTC m=+60.798659091" Sep 10 23:37:51.623719 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:51.623684 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-p7gpg"] Sep 10 23:37:51.626982 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:51.626959 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-p7gpg" Sep 10 23:37:51.629205 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:51.629187 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Sep 10 23:37:51.629265 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:51.629208 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Sep 10 23:37:51.629543 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:51.629526 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-wfl9n\"" Sep 10 23:37:51.637828 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:51.637806 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-qbts4"] Sep 10 23:37:51.640847 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:51.640826 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-qbts4" Sep 10 23:37:51.642758 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:51.642735 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-p7gpg"] Sep 10 23:37:51.643248 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:51.643225 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Sep 10 23:37:51.643349 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:51.643306 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Sep 10 23:37:51.643416 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:51.643359 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-9kjgt\"" Sep 10 23:37:51.643416 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:51.643397 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Sep 10 23:37:51.654371 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:51.654349 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-qbts4"] Sep 10 23:37:51.658951 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:51.658868 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-m2v2w"] Sep 10 23:37:51.663266 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:51.663248 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-m2v2w" Sep 10 23:37:51.666584 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:51.666562 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 10 23:37:51.666703 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:51.666623 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 10 23:37:51.666775 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:51.666758 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-k96nt\"" Sep 10 23:37:51.666935 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:51.666914 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 10 23:37:51.746275 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:51.746235 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/869ffc54-b1e8-4f0b-bd8e-2c571d1c4584-node-exporter-wtmp\") pod \"node-exporter-m2v2w\" (UID: \"869ffc54-b1e8-4f0b-bd8e-2c571d1c4584\") " pod="openshift-monitoring/node-exporter-m2v2w" Sep 10 23:37:51.746275 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:51.746278 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a4d6f4b4-86db-445b-8ed8-20e0c4d622c2-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-qbts4\" (UID: \"a4d6f4b4-86db-445b-8ed8-20e0c4d622c2\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-qbts4" Sep 10 23:37:51.746496 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:51.746297 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmtsr\" (UniqueName: \"kubernetes.io/projected/af3034eb-2870-46ab-add1-005b59c04308-kube-api-access-mmtsr\") pod \"openshift-state-metrics-67dcf8c979-p7gpg\" (UID: \"af3034eb-2870-46ab-add1-005b59c04308\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-p7gpg" Sep 10 23:37:51.746496 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:51.746353 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/869ffc54-b1e8-4f0b-bd8e-2c571d1c4584-root\") pod \"node-exporter-m2v2w\" (UID: \"869ffc54-b1e8-4f0b-bd8e-2c571d1c4584\") " pod="openshift-monitoring/node-exporter-m2v2w" Sep 10 23:37:51.746496 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:51.746380 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/869ffc54-b1e8-4f0b-bd8e-2c571d1c4584-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-m2v2w\" (UID: \"869ffc54-b1e8-4f0b-bd8e-2c571d1c4584\") " pod="openshift-monitoring/node-exporter-m2v2w" Sep 10 23:37:51.746496 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:51.746401 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a4d6f4b4-86db-445b-8ed8-20e0c4d622c2-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-qbts4\" (UID: \"a4d6f4b4-86db-445b-8ed8-20e0c4d622c2\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-qbts4" Sep 10 23:37:51.746496 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:51.746418 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/af3034eb-2870-46ab-add1-005b59c04308-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-p7gpg\" (UID: \"af3034eb-2870-46ab-add1-005b59c04308\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-p7gpg" Sep 10 23:37:51.746496 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:51.746479 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/869ffc54-b1e8-4f0b-bd8e-2c571d1c4584-node-exporter-accelerators-collector-config\") pod \"node-exporter-m2v2w\" (UID: \"869ffc54-b1e8-4f0b-bd8e-2c571d1c4584\") " pod="openshift-monitoring/node-exporter-m2v2w" Sep 10 23:37:51.746750 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:51.746504 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a4d6f4b4-86db-445b-8ed8-20e0c4d622c2-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-qbts4\" (UID: \"a4d6f4b4-86db-445b-8ed8-20e0c4d622c2\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-qbts4" Sep 10 23:37:51.746750 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:51.746522 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/a4d6f4b4-86db-445b-8ed8-20e0c4d622c2-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-qbts4\" (UID: \"a4d6f4b4-86db-445b-8ed8-20e0c4d622c2\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-qbts4" Sep 10 23:37:51.746750 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:51.746539 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/af3034eb-2870-46ab-add1-005b59c04308-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-p7gpg\" (UID: \"af3034eb-2870-46ab-add1-005b59c04308\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-p7gpg" Sep 10 23:37:51.746750 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:51.746558 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/869ffc54-b1e8-4f0b-bd8e-2c571d1c4584-node-exporter-tls\") pod \"node-exporter-m2v2w\" (UID: \"869ffc54-b1e8-4f0b-bd8e-2c571d1c4584\") " pod="openshift-monitoring/node-exporter-m2v2w" Sep 10 23:37:51.746750 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:51.746575 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2c2m5\" (UniqueName: \"kubernetes.io/projected/869ffc54-b1e8-4f0b-bd8e-2c571d1c4584-kube-api-access-2c2m5\") pod \"node-exporter-m2v2w\" (UID: \"869ffc54-b1e8-4f0b-bd8e-2c571d1c4584\") " pod="openshift-monitoring/node-exporter-m2v2w" Sep 10 23:37:51.746750 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:51.746591 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/af3034eb-2870-46ab-add1-005b59c04308-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-p7gpg\" (UID: \"af3034eb-2870-46ab-add1-005b59c04308\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-p7gpg" Sep 10 23:37:51.746750 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:51.746659 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s88qs\" (UniqueName: \"kubernetes.io/projected/a4d6f4b4-86db-445b-8ed8-20e0c4d622c2-kube-api-access-s88qs\") pod \"kube-state-metrics-59fbdd77d-qbts4\" (UID: \"a4d6f4b4-86db-445b-8ed8-20e0c4d622c2\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-qbts4" Sep 10 23:37:51.746750 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:51.746696 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/869ffc54-b1e8-4f0b-bd8e-2c571d1c4584-node-exporter-textfile\") pod \"node-exporter-m2v2w\" (UID: \"869ffc54-b1e8-4f0b-bd8e-2c571d1c4584\") " pod="openshift-monitoring/node-exporter-m2v2w" Sep 10 23:37:51.746750 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:51.746731 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/a4d6f4b4-86db-445b-8ed8-20e0c4d622c2-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-qbts4\" (UID: \"a4d6f4b4-86db-445b-8ed8-20e0c4d622c2\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-qbts4" Sep 10 23:37:51.746750 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:51.746747 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/869ffc54-b1e8-4f0b-bd8e-2c571d1c4584-sys\") pod \"node-exporter-m2v2w\" (UID: \"869ffc54-b1e8-4f0b-bd8e-2c571d1c4584\") " pod="openshift-monitoring/node-exporter-m2v2w" Sep 10 23:37:51.747075 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:51.746792 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/869ffc54-b1e8-4f0b-bd8e-2c571d1c4584-metrics-client-ca\") pod \"node-exporter-m2v2w\" (UID: \"869ffc54-b1e8-4f0b-bd8e-2c571d1c4584\") " pod="openshift-monitoring/node-exporter-m2v2w" Sep 10 23:37:51.848059 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:51.848020 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/869ffc54-b1e8-4f0b-bd8e-2c571d1c4584-node-exporter-wtmp\") pod \"node-exporter-m2v2w\" (UID: \"869ffc54-b1e8-4f0b-bd8e-2c571d1c4584\") " pod="openshift-monitoring/node-exporter-m2v2w" Sep 10 23:37:51.848059 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:51.848066 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a4d6f4b4-86db-445b-8ed8-20e0c4d622c2-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-qbts4\" (UID: \"a4d6f4b4-86db-445b-8ed8-20e0c4d622c2\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-qbts4" Sep 10 23:37:51.848269 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:51.848086 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmtsr\" (UniqueName: \"kubernetes.io/projected/af3034eb-2870-46ab-add1-005b59c04308-kube-api-access-mmtsr\") pod \"openshift-state-metrics-67dcf8c979-p7gpg\" (UID: \"af3034eb-2870-46ab-add1-005b59c04308\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-p7gpg" Sep 10 23:37:51.848269 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:51.848107 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/869ffc54-b1e8-4f0b-bd8e-2c571d1c4584-root\") pod \"node-exporter-m2v2w\" (UID: \"869ffc54-b1e8-4f0b-bd8e-2c571d1c4584\") " pod="openshift-monitoring/node-exporter-m2v2w" Sep 10 23:37:51.848269 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:51.848132 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/869ffc54-b1e8-4f0b-bd8e-2c571d1c4584-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-m2v2w\" (UID: \"869ffc54-b1e8-4f0b-bd8e-2c571d1c4584\") " pod="openshift-monitoring/node-exporter-m2v2w" Sep 10 23:37:51.848269 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:51.848188 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/869ffc54-b1e8-4f0b-bd8e-2c571d1c4584-root\") pod \"node-exporter-m2v2w\" (UID: \"869ffc54-b1e8-4f0b-bd8e-2c571d1c4584\") " pod="openshift-monitoring/node-exporter-m2v2w" Sep 10 23:37:51.848269 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:37:51.848206 2574 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Sep 10 23:37:51.848269 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:51.848221 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/869ffc54-b1e8-4f0b-bd8e-2c571d1c4584-node-exporter-wtmp\") pod \"node-exporter-m2v2w\" (UID: \"869ffc54-b1e8-4f0b-bd8e-2c571d1c4584\") " pod="openshift-monitoring/node-exporter-m2v2w" Sep 10 23:37:51.848269 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:51.848235 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a4d6f4b4-86db-445b-8ed8-20e0c4d622c2-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-qbts4\" (UID: \"a4d6f4b4-86db-445b-8ed8-20e0c4d622c2\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-qbts4" Sep 10 23:37:51.848269 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:51.848258 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/af3034eb-2870-46ab-add1-005b59c04308-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-p7gpg\" (UID: \"af3034eb-2870-46ab-add1-005b59c04308\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-p7gpg" Sep 10 23:37:51.848665 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:37:51.848314 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4d6f4b4-86db-445b-8ed8-20e0c4d622c2-kube-state-metrics-tls podName:a4d6f4b4-86db-445b-8ed8-20e0c4d622c2 nodeName:}" failed. No retries permitted until 2026-09-10 23:37:52.34829382 +0000 UTC m=+63.735896557 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/a4d6f4b4-86db-445b-8ed8-20e0c4d622c2-kube-state-metrics-tls") pod "kube-state-metrics-59fbdd77d-qbts4" (UID: "a4d6f4b4-86db-445b-8ed8-20e0c4d622c2") : secret "kube-state-metrics-tls" not found Sep 10 23:37:51.848665 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:51.848342 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/869ffc54-b1e8-4f0b-bd8e-2c571d1c4584-node-exporter-accelerators-collector-config\") pod \"node-exporter-m2v2w\" (UID: \"869ffc54-b1e8-4f0b-bd8e-2c571d1c4584\") " pod="openshift-monitoring/node-exporter-m2v2w" Sep 10 23:37:51.848665 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:51.848374 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a4d6f4b4-86db-445b-8ed8-20e0c4d622c2-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-qbts4\" (UID: \"a4d6f4b4-86db-445b-8ed8-20e0c4d622c2\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-qbts4" Sep 10 23:37:51.848665 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:51.848400 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/a4d6f4b4-86db-445b-8ed8-20e0c4d622c2-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-qbts4\" (UID: \"a4d6f4b4-86db-445b-8ed8-20e0c4d622c2\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-qbts4" Sep 10 23:37:51.848665 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:51.848562 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/af3034eb-2870-46ab-add1-005b59c04308-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-p7gpg\" (UID: \"af3034eb-2870-46ab-add1-005b59c04308\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-p7gpg" Sep 10 23:37:51.848665 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:51.848609 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/869ffc54-b1e8-4f0b-bd8e-2c571d1c4584-node-exporter-tls\") pod \"node-exporter-m2v2w\" (UID: \"869ffc54-b1e8-4f0b-bd8e-2c571d1c4584\") " pod="openshift-monitoring/node-exporter-m2v2w" Sep 10 23:37:51.848665 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:51.848659 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c2m5\" (UniqueName: \"kubernetes.io/projected/869ffc54-b1e8-4f0b-bd8e-2c571d1c4584-kube-api-access-2c2m5\") pod \"node-exporter-m2v2w\" (UID: \"869ffc54-b1e8-4f0b-bd8e-2c571d1c4584\") " pod="openshift-monitoring/node-exporter-m2v2w" Sep 10 23:37:51.849035 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:51.848697 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/af3034eb-2870-46ab-add1-005b59c04308-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-p7gpg\" (UID: \"af3034eb-2870-46ab-add1-005b59c04308\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-p7gpg" Sep 10 23:37:51.849035 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:51.848731 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s88qs\" (UniqueName: \"kubernetes.io/projected/a4d6f4b4-86db-445b-8ed8-20e0c4d622c2-kube-api-access-s88qs\") pod \"kube-state-metrics-59fbdd77d-qbts4\" (UID: \"a4d6f4b4-86db-445b-8ed8-20e0c4d622c2\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-qbts4" Sep 10 23:37:51.849035 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:51.848756 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/869ffc54-b1e8-4f0b-bd8e-2c571d1c4584-node-exporter-textfile\") pod \"node-exporter-m2v2w\" (UID: \"869ffc54-b1e8-4f0b-bd8e-2c571d1c4584\") " pod="openshift-monitoring/node-exporter-m2v2w" Sep 10 23:37:51.849035 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:51.848808 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/a4d6f4b4-86db-445b-8ed8-20e0c4d622c2-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-qbts4\" (UID: \"a4d6f4b4-86db-445b-8ed8-20e0c4d622c2\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-qbts4" Sep 10 23:37:51.849035 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:51.848829 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/869ffc54-b1e8-4f0b-bd8e-2c571d1c4584-sys\") pod \"node-exporter-m2v2w\" (UID: \"869ffc54-b1e8-4f0b-bd8e-2c571d1c4584\") " pod="openshift-monitoring/node-exporter-m2v2w" Sep 10 23:37:51.849035 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:51.848870 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/869ffc54-b1e8-4f0b-bd8e-2c571d1c4584-metrics-client-ca\") pod \"node-exporter-m2v2w\" (UID: \"869ffc54-b1e8-4f0b-bd8e-2c571d1c4584\") " pod="openshift-monitoring/node-exporter-m2v2w" Sep 10 23:37:51.849035 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:51.849016 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/869ffc54-b1e8-4f0b-bd8e-2c571d1c4584-node-exporter-accelerators-collector-config\") pod \"node-exporter-m2v2w\" (UID: \"869ffc54-b1e8-4f0b-bd8e-2c571d1c4584\") " pod="openshift-monitoring/node-exporter-m2v2w" Sep 10 23:37:51.849376 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:37:51.849112 2574 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Sep 10 23:37:51.849376 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:51.849144 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/a4d6f4b4-86db-445b-8ed8-20e0c4d622c2-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-qbts4\" (UID: \"a4d6f4b4-86db-445b-8ed8-20e0c4d622c2\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-qbts4" Sep 10 23:37:51.849376 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:37:51.849182 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af3034eb-2870-46ab-add1-005b59c04308-openshift-state-metrics-tls podName:af3034eb-2870-46ab-add1-005b59c04308 nodeName:}" failed. No retries permitted until 2026-09-10 23:37:52.349162772 +0000 UTC m=+63.736765513 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/af3034eb-2870-46ab-add1-005b59c04308-openshift-state-metrics-tls") pod "openshift-state-metrics-67dcf8c979-p7gpg" (UID: "af3034eb-2870-46ab-add1-005b59c04308") : secret "openshift-state-metrics-tls" not found Sep 10 23:37:51.849543 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:51.849452 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/869ffc54-b1e8-4f0b-bd8e-2c571d1c4584-metrics-client-ca\") pod \"node-exporter-m2v2w\" (UID: \"869ffc54-b1e8-4f0b-bd8e-2c571d1c4584\") " pod="openshift-monitoring/node-exporter-m2v2w" Sep 10 23:37:51.849543 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:51.849470 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/869ffc54-b1e8-4f0b-bd8e-2c571d1c4584-node-exporter-textfile\") pod \"node-exporter-m2v2w\" (UID: \"869ffc54-b1e8-4f0b-bd8e-2c571d1c4584\") " pod="openshift-monitoring/node-exporter-m2v2w" Sep 10 23:37:51.849804 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:51.849775 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/a4d6f4b4-86db-445b-8ed8-20e0c4d622c2-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-qbts4\" (UID: \"a4d6f4b4-86db-445b-8ed8-20e0c4d622c2\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-qbts4" Sep 10 23:37:51.849885 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:51.849785 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/869ffc54-b1e8-4f0b-bd8e-2c571d1c4584-sys\") pod \"node-exporter-m2v2w\" (UID: \"869ffc54-b1e8-4f0b-bd8e-2c571d1c4584\") " pod="openshift-monitoring/node-exporter-m2v2w" Sep 10 23:37:51.850092 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:51.850066 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/af3034eb-2870-46ab-add1-005b59c04308-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-p7gpg\" (UID: \"af3034eb-2870-46ab-add1-005b59c04308\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-p7gpg" Sep 10 23:37:51.850159 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:51.850118 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a4d6f4b4-86db-445b-8ed8-20e0c4d622c2-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-qbts4\" (UID: \"a4d6f4b4-86db-445b-8ed8-20e0c4d622c2\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-qbts4" Sep 10 23:37:51.850842 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:51.850822 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/869ffc54-b1e8-4f0b-bd8e-2c571d1c4584-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-m2v2w\" (UID: \"869ffc54-b1e8-4f0b-bd8e-2c571d1c4584\") " pod="openshift-monitoring/node-exporter-m2v2w" Sep 10 23:37:51.851034 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:51.851011 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/af3034eb-2870-46ab-add1-005b59c04308-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-p7gpg\" (UID: \"af3034eb-2870-46ab-add1-005b59c04308\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-p7gpg" Sep 10 23:37:51.851217 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:51.851199 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a4d6f4b4-86db-445b-8ed8-20e0c4d622c2-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-qbts4\" (UID: \"a4d6f4b4-86db-445b-8ed8-20e0c4d622c2\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-qbts4" Sep 10 23:37:51.851454 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:51.851436 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/869ffc54-b1e8-4f0b-bd8e-2c571d1c4584-node-exporter-tls\") pod \"node-exporter-m2v2w\" (UID: \"869ffc54-b1e8-4f0b-bd8e-2c571d1c4584\") " pod="openshift-monitoring/node-exporter-m2v2w" Sep 10 23:37:51.866684 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:51.866659 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2c2m5\" (UniqueName: \"kubernetes.io/projected/869ffc54-b1e8-4f0b-bd8e-2c571d1c4584-kube-api-access-2c2m5\") pod \"node-exporter-m2v2w\" (UID: \"869ffc54-b1e8-4f0b-bd8e-2c571d1c4584\") " pod="openshift-monitoring/node-exporter-m2v2w" Sep 10 23:37:51.869806 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:51.869777 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmtsr\" (UniqueName: \"kubernetes.io/projected/af3034eb-2870-46ab-add1-005b59c04308-kube-api-access-mmtsr\") pod \"openshift-state-metrics-67dcf8c979-p7gpg\" (UID: \"af3034eb-2870-46ab-add1-005b59c04308\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-p7gpg" Sep 10 23:37:51.870970 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:51.870942 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s88qs\" (UniqueName: \"kubernetes.io/projected/a4d6f4b4-86db-445b-8ed8-20e0c4d622c2-kube-api-access-s88qs\") pod \"kube-state-metrics-59fbdd77d-qbts4\" (UID: \"a4d6f4b4-86db-445b-8ed8-20e0c4d622c2\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-qbts4" Sep 10 23:37:51.972375 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:51.972338 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-m2v2w" Sep 10 23:37:51.983484 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:37:51.983451 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod869ffc54_b1e8_4f0b_bd8e_2c571d1c4584.slice/crio-31b4284da1d35f1e9c769b0f1e9b7555070d69a7208dbf3253c3fb11d57b4dad WatchSource:0}: Error finding container 31b4284da1d35f1e9c769b0f1e9b7555070d69a7208dbf3253c3fb11d57b4dad: Status 404 returned error can't find the container with id 31b4284da1d35f1e9c769b0f1e9b7555070d69a7208dbf3253c3fb11d57b4dad Sep 10 23:37:52.352557 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:52.352470 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a4d6f4b4-86db-445b-8ed8-20e0c4d622c2-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-qbts4\" (UID: \"a4d6f4b4-86db-445b-8ed8-20e0c4d622c2\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-qbts4" Sep 10 23:37:52.352557 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:52.352533 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/af3034eb-2870-46ab-add1-005b59c04308-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-p7gpg\" (UID: \"af3034eb-2870-46ab-add1-005b59c04308\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-p7gpg" Sep 10 23:37:52.352820 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:37:52.352661 2574 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Sep 10 23:37:52.352820 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:37:52.352740 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4d6f4b4-86db-445b-8ed8-20e0c4d622c2-kube-state-metrics-tls podName:a4d6f4b4-86db-445b-8ed8-20e0c4d622c2 nodeName:}" failed. No retries permitted until 2026-09-10 23:37:53.352721129 +0000 UTC m=+64.740323869 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/a4d6f4b4-86db-445b-8ed8-20e0c4d622c2-kube-state-metrics-tls") pod "kube-state-metrics-59fbdd77d-qbts4" (UID: "a4d6f4b4-86db-445b-8ed8-20e0c4d622c2") : secret "kube-state-metrics-tls" not found Sep 10 23:37:52.354852 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:52.354830 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/af3034eb-2870-46ab-add1-005b59c04308-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-p7gpg\" (UID: \"af3034eb-2870-46ab-add1-005b59c04308\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-p7gpg" Sep 10 23:37:52.364969 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:52.364943 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-6bbbz" Sep 10 23:37:52.401032 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:52.400997 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-m2v2w" event={"ID":"869ffc54-b1e8-4f0b-bd8e-2c571d1c4584","Type":"ContainerStarted","Data":"31b4284da1d35f1e9c769b0f1e9b7555070d69a7208dbf3253c3fb11d57b4dad"} Sep 10 23:37:52.535258 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:52.535222 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-p7gpg" Sep 10 23:37:52.692353 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:52.692267 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 10 23:37:52.696273 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:52.696251 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 10 23:37:52.701437 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:52.698464 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Sep 10 23:37:52.701437 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:52.698501 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Sep 10 23:37:52.701437 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:52.698691 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Sep 10 23:37:52.701437 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:52.698770 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-rcn9z\"" Sep 10 23:37:52.701437 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:52.698883 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Sep 10 23:37:52.701437 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:52.699110 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Sep 10 23:37:52.701437 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:52.699299 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Sep 10 23:37:52.701437 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:52.699494 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Sep 10 23:37:52.701437 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:52.699707 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Sep 10 23:37:52.701437 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:52.699901 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Sep 10 23:37:52.710228 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:52.710205 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 10 23:37:52.756486 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:52.756445 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b5e6b6fa-bb37-4770-93b3-26b897d01b78-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b5e6b6fa-bb37-4770-93b3-26b897d01b78\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 23:37:52.756699 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:52.756495 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22xgw\" (UniqueName: \"kubernetes.io/projected/b5e6b6fa-bb37-4770-93b3-26b897d01b78-kube-api-access-22xgw\") pod \"alertmanager-main-0\" (UID: \"b5e6b6fa-bb37-4770-93b3-26b897d01b78\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 23:37:52.756699 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:52.756530 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b5e6b6fa-bb37-4770-93b3-26b897d01b78-config-out\") pod \"alertmanager-main-0\" (UID: \"b5e6b6fa-bb37-4770-93b3-26b897d01b78\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 23:37:52.756699 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:52.756566 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b5e6b6fa-bb37-4770-93b3-26b897d01b78-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b5e6b6fa-bb37-4770-93b3-26b897d01b78\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 23:37:52.756699 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:52.756597 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b5e6b6fa-bb37-4770-93b3-26b897d01b78-web-config\") pod \"alertmanager-main-0\" (UID: \"b5e6b6fa-bb37-4770-93b3-26b897d01b78\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 23:37:52.756699 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:52.756621 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b5e6b6fa-bb37-4770-93b3-26b897d01b78-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b5e6b6fa-bb37-4770-93b3-26b897d01b78\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 23:37:52.757003 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:52.756706 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b5e6b6fa-bb37-4770-93b3-26b897d01b78-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b5e6b6fa-bb37-4770-93b3-26b897d01b78\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 23:37:52.757003 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:52.756740 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b5e6b6fa-bb37-4770-93b3-26b897d01b78-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b5e6b6fa-bb37-4770-93b3-26b897d01b78\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 23:37:52.757003 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:52.756799 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b5e6b6fa-bb37-4770-93b3-26b897d01b78-config-volume\") pod \"alertmanager-main-0\" (UID: \"b5e6b6fa-bb37-4770-93b3-26b897d01b78\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 23:37:52.757003 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:52.756847 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5e6b6fa-bb37-4770-93b3-26b897d01b78-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b5e6b6fa-bb37-4770-93b3-26b897d01b78\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 23:37:52.757003 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:52.756886 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b5e6b6fa-bb37-4770-93b3-26b897d01b78-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b5e6b6fa-bb37-4770-93b3-26b897d01b78\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 23:37:52.757003 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:52.756994 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b5e6b6fa-bb37-4770-93b3-26b897d01b78-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b5e6b6fa-bb37-4770-93b3-26b897d01b78\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 23:37:52.757288 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:52.757030 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b5e6b6fa-bb37-4770-93b3-26b897d01b78-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b5e6b6fa-bb37-4770-93b3-26b897d01b78\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 23:37:52.818495 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:52.818033 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-p7gpg"] Sep 10 23:37:52.838850 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:37:52.838813 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf3034eb_2870_46ab_add1_005b59c04308.slice/crio-f39140d242a39f938e100e103cccd9d8b35a6510f44abcddc34e50137c21d780 WatchSource:0}: Error finding container f39140d242a39f938e100e103cccd9d8b35a6510f44abcddc34e50137c21d780: Status 404 returned error can't find the container with id f39140d242a39f938e100e103cccd9d8b35a6510f44abcddc34e50137c21d780 Sep 10 23:37:52.857804 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:52.857774 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b5e6b6fa-bb37-4770-93b3-26b897d01b78-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b5e6b6fa-bb37-4770-93b3-26b897d01b78\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 23:37:52.857926 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:52.857816 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b5e6b6fa-bb37-4770-93b3-26b897d01b78-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b5e6b6fa-bb37-4770-93b3-26b897d01b78\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 23:37:52.857926 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:52.857836 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b5e6b6fa-bb37-4770-93b3-26b897d01b78-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b5e6b6fa-bb37-4770-93b3-26b897d01b78\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 23:37:52.857926 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:52.857857 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22xgw\" (UniqueName: \"kubernetes.io/projected/b5e6b6fa-bb37-4770-93b3-26b897d01b78-kube-api-access-22xgw\") pod \"alertmanager-main-0\" (UID: \"b5e6b6fa-bb37-4770-93b3-26b897d01b78\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 23:37:52.857926 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:52.857877 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b5e6b6fa-bb37-4770-93b3-26b897d01b78-config-out\") pod \"alertmanager-main-0\" (UID: \"b5e6b6fa-bb37-4770-93b3-26b897d01b78\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 23:37:52.857926 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:52.857895 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b5e6b6fa-bb37-4770-93b3-26b897d01b78-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b5e6b6fa-bb37-4770-93b3-26b897d01b78\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 23:37:52.857926 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:52.857912 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b5e6b6fa-bb37-4770-93b3-26b897d01b78-web-config\") pod \"alertmanager-main-0\" (UID: \"b5e6b6fa-bb37-4770-93b3-26b897d01b78\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 23:37:52.858234 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:52.857931 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b5e6b6fa-bb37-4770-93b3-26b897d01b78-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b5e6b6fa-bb37-4770-93b3-26b897d01b78\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 23:37:52.858234 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:37:52.857938 2574 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Sep 10 23:37:52.858234 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:52.857954 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b5e6b6fa-bb37-4770-93b3-26b897d01b78-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b5e6b6fa-bb37-4770-93b3-26b897d01b78\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 23:37:52.858234 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:52.857972 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b5e6b6fa-bb37-4770-93b3-26b897d01b78-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b5e6b6fa-bb37-4770-93b3-26b897d01b78\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 23:37:52.858234 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:37:52.858008 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5e6b6fa-bb37-4770-93b3-26b897d01b78-secret-alertmanager-main-tls podName:b5e6b6fa-bb37-4770-93b3-26b897d01b78 nodeName:}" failed. No retries permitted until 2026-09-10 23:37:53.357986998 +0000 UTC m=+64.745589747 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/b5e6b6fa-bb37-4770-93b3-26b897d01b78-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "b5e6b6fa-bb37-4770-93b3-26b897d01b78") : secret "alertmanager-main-tls" not found Sep 10 23:37:52.858234 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:52.858034 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b5e6b6fa-bb37-4770-93b3-26b897d01b78-config-volume\") pod \"alertmanager-main-0\" (UID: \"b5e6b6fa-bb37-4770-93b3-26b897d01b78\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 23:37:52.858234 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:52.858087 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5e6b6fa-bb37-4770-93b3-26b897d01b78-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b5e6b6fa-bb37-4770-93b3-26b897d01b78\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 23:37:52.858234 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:52.858129 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b5e6b6fa-bb37-4770-93b3-26b897d01b78-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b5e6b6fa-bb37-4770-93b3-26b897d01b78\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 23:37:52.858634 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:52.858257 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b5e6b6fa-bb37-4770-93b3-26b897d01b78-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b5e6b6fa-bb37-4770-93b3-26b897d01b78\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 23:37:52.859222 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:52.859161 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b5e6b6fa-bb37-4770-93b3-26b897d01b78-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b5e6b6fa-bb37-4770-93b3-26b897d01b78\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 23:37:52.859222 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:37:52.859190 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b5e6b6fa-bb37-4770-93b3-26b897d01b78-alertmanager-trusted-ca-bundle podName:b5e6b6fa-bb37-4770-93b3-26b897d01b78 nodeName:}" failed. No retries permitted until 2026-09-10 23:37:53.359170331 +0000 UTC m=+64.746773088 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/b5e6b6fa-bb37-4770-93b3-26b897d01b78-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "b5e6b6fa-bb37-4770-93b3-26b897d01b78") : configmap references non-existent config key: ca-bundle.crt Sep 10 23:37:52.860842 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:52.860816 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b5e6b6fa-bb37-4770-93b3-26b897d01b78-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b5e6b6fa-bb37-4770-93b3-26b897d01b78\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 23:37:52.861013 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:52.860955 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b5e6b6fa-bb37-4770-93b3-26b897d01b78-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b5e6b6fa-bb37-4770-93b3-26b897d01b78\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 23:37:52.862149 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:52.861863 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b5e6b6fa-bb37-4770-93b3-26b897d01b78-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b5e6b6fa-bb37-4770-93b3-26b897d01b78\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 23:37:52.862149 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:52.862035 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b5e6b6fa-bb37-4770-93b3-26b897d01b78-config-volume\") pod \"alertmanager-main-0\" (UID: \"b5e6b6fa-bb37-4770-93b3-26b897d01b78\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 23:37:52.862149 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:52.862065 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b5e6b6fa-bb37-4770-93b3-26b897d01b78-web-config\") pod \"alertmanager-main-0\" (UID: \"b5e6b6fa-bb37-4770-93b3-26b897d01b78\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 23:37:52.862354 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:52.862298 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b5e6b6fa-bb37-4770-93b3-26b897d01b78-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b5e6b6fa-bb37-4770-93b3-26b897d01b78\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 23:37:52.862550 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:52.862532 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b5e6b6fa-bb37-4770-93b3-26b897d01b78-config-out\") pod \"alertmanager-main-0\" (UID: \"b5e6b6fa-bb37-4770-93b3-26b897d01b78\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 23:37:52.863482 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:52.863462 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b5e6b6fa-bb37-4770-93b3-26b897d01b78-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b5e6b6fa-bb37-4770-93b3-26b897d01b78\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 23:37:52.867781 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:52.867758 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-22xgw\" (UniqueName: \"kubernetes.io/projected/b5e6b6fa-bb37-4770-93b3-26b897d01b78-kube-api-access-22xgw\") pod \"alertmanager-main-0\" (UID: \"b5e6b6fa-bb37-4770-93b3-26b897d01b78\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 23:37:53.363321 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:53.363269 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b5e6b6fa-bb37-4770-93b3-26b897d01b78-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b5e6b6fa-bb37-4770-93b3-26b897d01b78\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 23:37:53.363534 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:53.363475 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5e6b6fa-bb37-4770-93b3-26b897d01b78-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b5e6b6fa-bb37-4770-93b3-26b897d01b78\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 23:37:53.363604 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:53.363539 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a4d6f4b4-86db-445b-8ed8-20e0c4d622c2-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-qbts4\" (UID: \"a4d6f4b4-86db-445b-8ed8-20e0c4d622c2\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-qbts4" Sep 10 23:37:53.364216 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:53.364192 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5e6b6fa-bb37-4770-93b3-26b897d01b78-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b5e6b6fa-bb37-4770-93b3-26b897d01b78\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 23:37:53.365731 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:53.365706 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b5e6b6fa-bb37-4770-93b3-26b897d01b78-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b5e6b6fa-bb37-4770-93b3-26b897d01b78\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 23:37:53.365808 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:53.365761 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a4d6f4b4-86db-445b-8ed8-20e0c4d622c2-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-qbts4\" (UID: \"a4d6f4b4-86db-445b-8ed8-20e0c4d622c2\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-qbts4" Sep 10 23:37:53.408146 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:53.408115 2574 generic.go:356] "Generic (PLEG): container finished" podID="869ffc54-b1e8-4f0b-bd8e-2c571d1c4584" containerID="c41b3788f6fe8a49c5d73b5f3986d3452187ce55a7878a158cd510a6985e2421" exitCode=0 Sep 10 23:37:53.408241 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:53.408201 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-m2v2w" event={"ID":"869ffc54-b1e8-4f0b-bd8e-2c571d1c4584","Type":"ContainerDied","Data":"c41b3788f6fe8a49c5d73b5f3986d3452187ce55a7878a158cd510a6985e2421"} Sep 10 23:37:53.410348 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:53.410320 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-p7gpg" event={"ID":"af3034eb-2870-46ab-add1-005b59c04308","Type":"ContainerStarted","Data":"50b2103aac0c3601c7a1f5bdd9ed1ac2a54fe3cd042f550129268935520b1fa0"} Sep 10 23:37:53.410444 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:53.410383 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-p7gpg" event={"ID":"af3034eb-2870-46ab-add1-005b59c04308","Type":"ContainerStarted","Data":"1923050282c7f611cbee80613efed7484c94a765e6c0fcd009f7e63226e5aa95"} Sep 10 23:37:53.410444 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:53.410412 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-p7gpg" event={"ID":"af3034eb-2870-46ab-add1-005b59c04308","Type":"ContainerStarted","Data":"f39140d242a39f938e100e103cccd9d8b35a6510f44abcddc34e50137c21d780"} Sep 10 23:37:53.451694 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:53.451669 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-qbts4" Sep 10 23:37:53.608998 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:53.608945 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 10 23:37:53.668675 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:53.665830 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-7f945b84b4-9zvwn"] Sep 10 23:37:53.671436 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:53.670659 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7f945b84b4-9zvwn" Sep 10 23:37:53.675907 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:53.675633 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Sep 10 23:37:53.675907 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:53.675712 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Sep 10 23:37:53.676101 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:53.675928 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-wtvk9\"" Sep 10 23:37:53.676101 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:53.675982 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Sep 10 23:37:53.676212 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:53.676195 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Sep 10 23:37:53.676336 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:53.676318 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Sep 10 23:37:53.676421 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:53.676401 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-7siqo7di8t684\"" Sep 10 23:37:53.688447 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:53.688423 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7f945b84b4-9zvwn"] Sep 10 23:37:53.747940 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:53.747903 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-qbts4"] Sep 10 23:37:53.769042 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:53.769009 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfqqx\" (UniqueName: \"kubernetes.io/projected/008bdffc-746e-448b-9bd0-18155f16320e-kube-api-access-wfqqx\") pod \"thanos-querier-7f945b84b4-9zvwn\" (UID: \"008bdffc-746e-448b-9bd0-18155f16320e\") " pod="openshift-monitoring/thanos-querier-7f945b84b4-9zvwn" Sep 10 23:37:53.769245 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:53.769082 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/008bdffc-746e-448b-9bd0-18155f16320e-metrics-client-ca\") pod \"thanos-querier-7f945b84b4-9zvwn\" (UID: \"008bdffc-746e-448b-9bd0-18155f16320e\") " pod="openshift-monitoring/thanos-querier-7f945b84b4-9zvwn" Sep 10 23:37:53.769245 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:53.769115 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/008bdffc-746e-448b-9bd0-18155f16320e-secret-thanos-querier-tls\") pod \"thanos-querier-7f945b84b4-9zvwn\" (UID: \"008bdffc-746e-448b-9bd0-18155f16320e\") " pod="openshift-monitoring/thanos-querier-7f945b84b4-9zvwn" Sep 10 23:37:53.769245 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:53.769161 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/008bdffc-746e-448b-9bd0-18155f16320e-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7f945b84b4-9zvwn\" (UID: \"008bdffc-746e-448b-9bd0-18155f16320e\") " pod="openshift-monitoring/thanos-querier-7f945b84b4-9zvwn" Sep 10 23:37:53.769245 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:53.769224 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/008bdffc-746e-448b-9bd0-18155f16320e-secret-grpc-tls\") pod \"thanos-querier-7f945b84b4-9zvwn\" (UID: \"008bdffc-746e-448b-9bd0-18155f16320e\") " pod="openshift-monitoring/thanos-querier-7f945b84b4-9zvwn" Sep 10 23:37:53.769406 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:53.769294 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/008bdffc-746e-448b-9bd0-18155f16320e-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7f945b84b4-9zvwn\" (UID: \"008bdffc-746e-448b-9bd0-18155f16320e\") " pod="openshift-monitoring/thanos-querier-7f945b84b4-9zvwn" Sep 10 23:37:53.769406 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:53.769331 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/008bdffc-746e-448b-9bd0-18155f16320e-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7f945b84b4-9zvwn\" (UID: \"008bdffc-746e-448b-9bd0-18155f16320e\") " pod="openshift-monitoring/thanos-querier-7f945b84b4-9zvwn" Sep 10 23:37:53.769499 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:53.769421 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/008bdffc-746e-448b-9bd0-18155f16320e-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7f945b84b4-9zvwn\" (UID: \"008bdffc-746e-448b-9bd0-18155f16320e\") " pod="openshift-monitoring/thanos-querier-7f945b84b4-9zvwn" Sep 10 23:37:53.870426 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:53.870342 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/008bdffc-746e-448b-9bd0-18155f16320e-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7f945b84b4-9zvwn\" (UID: \"008bdffc-746e-448b-9bd0-18155f16320e\") " pod="openshift-monitoring/thanos-querier-7f945b84b4-9zvwn" Sep 10 23:37:53.870426 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:53.870404 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/008bdffc-746e-448b-9bd0-18155f16320e-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7f945b84b4-9zvwn\" (UID: \"008bdffc-746e-448b-9bd0-18155f16320e\") " pod="openshift-monitoring/thanos-querier-7f945b84b4-9zvwn" Sep 10 23:37:53.870667 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:53.870481 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/008bdffc-746e-448b-9bd0-18155f16320e-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7f945b84b4-9zvwn\" (UID: \"008bdffc-746e-448b-9bd0-18155f16320e\") " pod="openshift-monitoring/thanos-querier-7f945b84b4-9zvwn" Sep 10 23:37:53.870667 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:53.870515 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfqqx\" (UniqueName: \"kubernetes.io/projected/008bdffc-746e-448b-9bd0-18155f16320e-kube-api-access-wfqqx\") pod \"thanos-querier-7f945b84b4-9zvwn\" (UID: \"008bdffc-746e-448b-9bd0-18155f16320e\") " pod="openshift-monitoring/thanos-querier-7f945b84b4-9zvwn" Sep 10 23:37:53.870667 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:53.870561 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/008bdffc-746e-448b-9bd0-18155f16320e-metrics-client-ca\") pod \"thanos-querier-7f945b84b4-9zvwn\" (UID: \"008bdffc-746e-448b-9bd0-18155f16320e\") " pod="openshift-monitoring/thanos-querier-7f945b84b4-9zvwn" Sep 10 23:37:53.870836 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:53.870802 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/008bdffc-746e-448b-9bd0-18155f16320e-secret-thanos-querier-tls\") pod \"thanos-querier-7f945b84b4-9zvwn\" (UID: \"008bdffc-746e-448b-9bd0-18155f16320e\") " pod="openshift-monitoring/thanos-querier-7f945b84b4-9zvwn" Sep 10 23:37:53.870894 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:53.870864 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/008bdffc-746e-448b-9bd0-18155f16320e-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7f945b84b4-9zvwn\" (UID: \"008bdffc-746e-448b-9bd0-18155f16320e\") " pod="openshift-monitoring/thanos-querier-7f945b84b4-9zvwn" Sep 10 23:37:53.870948 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:53.870894 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/008bdffc-746e-448b-9bd0-18155f16320e-secret-grpc-tls\") pod \"thanos-querier-7f945b84b4-9zvwn\" (UID: \"008bdffc-746e-448b-9bd0-18155f16320e\") " pod="openshift-monitoring/thanos-querier-7f945b84b4-9zvwn" Sep 10 23:37:53.871346 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:53.871295 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/008bdffc-746e-448b-9bd0-18155f16320e-metrics-client-ca\") pod \"thanos-querier-7f945b84b4-9zvwn\" (UID: \"008bdffc-746e-448b-9bd0-18155f16320e\") " pod="openshift-monitoring/thanos-querier-7f945b84b4-9zvwn" Sep 10 23:37:53.873690 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:53.873588 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/008bdffc-746e-448b-9bd0-18155f16320e-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7f945b84b4-9zvwn\" (UID: \"008bdffc-746e-448b-9bd0-18155f16320e\") " pod="openshift-monitoring/thanos-querier-7f945b84b4-9zvwn" Sep 10 23:37:53.873850 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:53.873828 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/008bdffc-746e-448b-9bd0-18155f16320e-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7f945b84b4-9zvwn\" (UID: \"008bdffc-746e-448b-9bd0-18155f16320e\") " pod="openshift-monitoring/thanos-querier-7f945b84b4-9zvwn" Sep 10 23:37:53.874424 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:53.874008 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/008bdffc-746e-448b-9bd0-18155f16320e-secret-grpc-tls\") pod \"thanos-querier-7f945b84b4-9zvwn\" (UID: \"008bdffc-746e-448b-9bd0-18155f16320e\") " pod="openshift-monitoring/thanos-querier-7f945b84b4-9zvwn" Sep 10 23:37:53.874424 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:53.874380 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/008bdffc-746e-448b-9bd0-18155f16320e-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7f945b84b4-9zvwn\" (UID: \"008bdffc-746e-448b-9bd0-18155f16320e\") " pod="openshift-monitoring/thanos-querier-7f945b84b4-9zvwn" Sep 10 23:37:53.875204 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:53.875177 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/008bdffc-746e-448b-9bd0-18155f16320e-secret-thanos-querier-tls\") pod \"thanos-querier-7f945b84b4-9zvwn\" (UID: \"008bdffc-746e-448b-9bd0-18155f16320e\") " pod="openshift-monitoring/thanos-querier-7f945b84b4-9zvwn" Sep 10 23:37:53.875809 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:53.875780 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/008bdffc-746e-448b-9bd0-18155f16320e-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7f945b84b4-9zvwn\" (UID: \"008bdffc-746e-448b-9bd0-18155f16320e\") " pod="openshift-monitoring/thanos-querier-7f945b84b4-9zvwn" Sep 10 23:37:53.878946 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:53.878924 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfqqx\" (UniqueName: \"kubernetes.io/projected/008bdffc-746e-448b-9bd0-18155f16320e-kube-api-access-wfqqx\") pod \"thanos-querier-7f945b84b4-9zvwn\" (UID: \"008bdffc-746e-448b-9bd0-18155f16320e\") " pod="openshift-monitoring/thanos-querier-7f945b84b4-9zvwn" Sep 10 23:37:53.890464 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:37:53.890428 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4d6f4b4_86db_445b_8ed8_20e0c4d622c2.slice/crio-d3c734187a8a150c2b4579e1d221b05227064954db611f9f52f54fa886e9b98e WatchSource:0}: Error finding container d3c734187a8a150c2b4579e1d221b05227064954db611f9f52f54fa886e9b98e: Status 404 returned error can't find the container with id d3c734187a8a150c2b4579e1d221b05227064954db611f9f52f54fa886e9b98e Sep 10 23:37:53.912221 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:53.912186 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 10 23:37:53.930562 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:37:53.930532 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5e6b6fa_bb37_4770_93b3_26b897d01b78.slice/crio-36eda9193db3164da027b3d3ec3600606f9fe827a6bc2fae69b7a32b9f3f5a95 WatchSource:0}: Error finding container 36eda9193db3164da027b3d3ec3600606f9fe827a6bc2fae69b7a32b9f3f5a95: Status 404 returned error can't find the container with id 36eda9193db3164da027b3d3ec3600606f9fe827a6bc2fae69b7a32b9f3f5a95 Sep 10 23:37:53.983745 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:53.983719 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7f945b84b4-9zvwn" Sep 10 23:37:54.264169 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:54.264135 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7f945b84b4-9zvwn"] Sep 10 23:37:54.269163 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:37:54.269126 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod008bdffc_746e_448b_9bd0_18155f16320e.slice/crio-1404c3d0006de73ca9af6378539ecc41ad90466517781509bf3eab51d488a323 WatchSource:0}: Error finding container 1404c3d0006de73ca9af6378539ecc41ad90466517781509bf3eab51d488a323: Status 404 returned error can't find the container with id 1404c3d0006de73ca9af6378539ecc41ad90466517781509bf3eab51d488a323 Sep 10 23:37:54.418983 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:54.418875 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7f945b84b4-9zvwn" event={"ID":"008bdffc-746e-448b-9bd0-18155f16320e","Type":"ContainerStarted","Data":"1404c3d0006de73ca9af6378539ecc41ad90466517781509bf3eab51d488a323"} Sep 10 23:37:54.421923 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:54.421851 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-m2v2w" event={"ID":"869ffc54-b1e8-4f0b-bd8e-2c571d1c4584","Type":"ContainerStarted","Data":"24d8b01aea9de7d9135232453f106fc05426d089afde1dbe4a18849629dc3fe4"} Sep 10 23:37:54.421923 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:54.421917 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-m2v2w" event={"ID":"869ffc54-b1e8-4f0b-bd8e-2c571d1c4584","Type":"ContainerStarted","Data":"6bc6e1387b96b9fd5b38d1ffb075208ae98697392eac63c1a46647bdfd2bf138"} Sep 10 23:37:54.426719 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:54.426680 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-p7gpg" event={"ID":"af3034eb-2870-46ab-add1-005b59c04308","Type":"ContainerStarted","Data":"9a590a50cd4b10711188864bc3ef2ff80f91db2556ca8e52035a1b59ec609d19"} Sep 10 23:37:54.431852 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:54.431816 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b5e6b6fa-bb37-4770-93b3-26b897d01b78","Type":"ContainerStarted","Data":"36eda9193db3164da027b3d3ec3600606f9fe827a6bc2fae69b7a32b9f3f5a95"} Sep 10 23:37:54.433511 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:54.433478 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-qbts4" event={"ID":"a4d6f4b4-86db-445b-8ed8-20e0c4d622c2","Type":"ContainerStarted","Data":"d3c734187a8a150c2b4579e1d221b05227064954db611f9f52f54fa886e9b98e"} Sep 10 23:37:54.442276 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:54.442214 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-m2v2w" podStartSLOduration=2.472259388 podStartE2EDuration="3.442198154s" totalImagesPullingTime="899.752857ms" totalInitContainerRuntime="70.185909ms" isStatefulPod=true podCreationTimestamp="2026-09-10 23:37:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:37:54.441126541 +0000 UTC m=+65.828729293" watchObservedRunningTime="2026-09-10 23:37:54.442198154 +0000 UTC m=+65.829800923" Sep 10 23:37:54.458368 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:54.458303 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-p7gpg" podStartSLOduration=2.5311716090000003 podStartE2EDuration="3.458279842s" totalImagesPullingTime="927.108233ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:37:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:37:54.457243301 +0000 UTC m=+65.844846070" watchObservedRunningTime="2026-09-10 23:37:54.458279842 +0000 UTC m=+65.845882601" Sep 10 23:37:54.780276 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:54.780236 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7wk9\" (UniqueName: \"kubernetes.io/projected/24dda81e-d88e-4f29-836b-44bb86bf461d-kube-api-access-s7wk9\") pod \"network-check-target-zb6x4\" (UID: \"24dda81e-d88e-4f29-836b-44bb86bf461d\") " pod="openshift-network-diagnostics/network-check-target-zb6x4" Sep 10 23:37:54.782538 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:54.782513 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 10 23:37:54.793328 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:54.793301 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 10 23:37:54.805102 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:54.805069 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7wk9\" (UniqueName: \"kubernetes.io/projected/24dda81e-d88e-4f29-836b-44bb86bf461d-kube-api-access-s7wk9\") pod \"network-check-target-zb6x4\" (UID: \"24dda81e-d88e-4f29-836b-44bb86bf461d\") " pod="openshift-network-diagnostics/network-check-target-zb6x4" Sep 10 23:37:54.881170 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:54.881129 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6df7e606-319b-4b9a-b6da-a850ecc9d267-metrics-certs\") pod \"network-metrics-daemon-v87ts\" (UID: \"6df7e606-319b-4b9a-b6da-a850ecc9d267\") " pod="openshift-multus/network-metrics-daemon-v87ts" Sep 10 23:37:54.881358 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:54.881228 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3a0ec520-b33d-49cf-9714-1662f5910e5b-original-pull-secret\") pod \"global-pull-secret-syncer-75vlp\" (UID: \"3a0ec520-b33d-49cf-9714-1662f5910e5b\") " pod="kube-system/global-pull-secret-syncer-75vlp" Sep 10 23:37:54.883783 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:54.883751 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 10 23:37:54.883910 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:54.883814 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 10 23:37:54.894235 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:54.894201 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3a0ec520-b33d-49cf-9714-1662f5910e5b-original-pull-secret\") pod \"global-pull-secret-syncer-75vlp\" (UID: \"3a0ec520-b33d-49cf-9714-1662f5910e5b\") " pod="kube-system/global-pull-secret-syncer-75vlp" Sep 10 23:37:54.895052 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:54.894978 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6df7e606-319b-4b9a-b6da-a850ecc9d267-metrics-certs\") pod \"network-metrics-daemon-v87ts\" (UID: \"6df7e606-319b-4b9a-b6da-a850ecc9d267\") " pod="openshift-multus/network-metrics-daemon-v87ts" Sep 10 23:37:54.903077 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:54.902887 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-n6qg4\"" Sep 10 23:37:54.905010 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:54.904987 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-75vlp" Sep 10 23:37:54.910193 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:54.910139 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v87ts" Sep 10 23:37:55.097325 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:55.097243 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-ppbqc\"" Sep 10 23:37:55.106128 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:55.106101 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zb6x4" Sep 10 23:37:55.954680 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:55.954625 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-6fb78998c8-bh5np"] Sep 10 23:37:55.957973 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:55.957949 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6fb78998c8-bh5np" Sep 10 23:37:55.960080 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:55.960045 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Sep 10 23:37:55.960999 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:55.960967 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Sep 10 23:37:55.960999 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:55.960995 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-g6fzx\"" Sep 10 23:37:55.961119 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:55.960973 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-25ejfqm22tc68\"" Sep 10 23:37:55.961119 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:55.961040 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Sep 10 23:37:55.961119 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:55.960973 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Sep 10 23:37:55.967876 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:55.967831 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6fb78998c8-bh5np"] Sep 10 23:37:55.992440 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:55.992403 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/cf3a41cb-434f-4206-8bfa-be659892e8d6-metrics-server-audit-profiles\") pod \"metrics-server-6fb78998c8-bh5np\" (UID: \"cf3a41cb-434f-4206-8bfa-be659892e8d6\") " pod="openshift-monitoring/metrics-server-6fb78998c8-bh5np" Sep 10 23:37:55.992631 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:55.992453 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/cf3a41cb-434f-4206-8bfa-be659892e8d6-client-ca-bundle\") pod \"metrics-server-6fb78998c8-bh5np\" (UID: \"cf3a41cb-434f-4206-8bfa-be659892e8d6\") " pod="openshift-monitoring/metrics-server-6fb78998c8-bh5np" Sep 10 23:37:55.992631 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:55.992556 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/cf3a41cb-434f-4206-8bfa-be659892e8d6-secret-metrics-server-client-certs\") pod \"metrics-server-6fb78998c8-bh5np\" (UID: \"cf3a41cb-434f-4206-8bfa-be659892e8d6\") " pod="openshift-monitoring/metrics-server-6fb78998c8-bh5np" Sep 10 23:37:55.992631 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:55.992584 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9w7r9\" (UniqueName: \"kubernetes.io/projected/cf3a41cb-434f-4206-8bfa-be659892e8d6-kube-api-access-9w7r9\") pod \"metrics-server-6fb78998c8-bh5np\" (UID: \"cf3a41cb-434f-4206-8bfa-be659892e8d6\") " pod="openshift-monitoring/metrics-server-6fb78998c8-bh5np" Sep 10 23:37:55.992631 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:55.992615 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/cf3a41cb-434f-4206-8bfa-be659892e8d6-audit-log\") pod \"metrics-server-6fb78998c8-bh5np\" (UID: \"cf3a41cb-434f-4206-8bfa-be659892e8d6\") " pod="openshift-monitoring/metrics-server-6fb78998c8-bh5np" Sep 10 23:37:55.992881 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:55.992764 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cf3a41cb-434f-4206-8bfa-be659892e8d6-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6fb78998c8-bh5np\" (UID: \"cf3a41cb-434f-4206-8bfa-be659892e8d6\") " pod="openshift-monitoring/metrics-server-6fb78998c8-bh5np" Sep 10 23:37:55.992881 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:55.992801 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/cf3a41cb-434f-4206-8bfa-be659892e8d6-secret-metrics-server-tls\") pod \"metrics-server-6fb78998c8-bh5np\" (UID: \"cf3a41cb-434f-4206-8bfa-be659892e8d6\") " pod="openshift-monitoring/metrics-server-6fb78998c8-bh5np" Sep 10 23:37:56.093750 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:56.093665 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/cf3a41cb-434f-4206-8bfa-be659892e8d6-metrics-server-audit-profiles\") pod \"metrics-server-6fb78998c8-bh5np\" (UID: \"cf3a41cb-434f-4206-8bfa-be659892e8d6\") " pod="openshift-monitoring/metrics-server-6fb78998c8-bh5np" Sep 10 23:37:56.093750 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:56.093704 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/cf3a41cb-434f-4206-8bfa-be659892e8d6-client-ca-bundle\") pod \"metrics-server-6fb78998c8-bh5np\" (UID: \"cf3a41cb-434f-4206-8bfa-be659892e8d6\") " pod="openshift-monitoring/metrics-server-6fb78998c8-bh5np" Sep 10 23:37:56.093954 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:56.093769 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/cf3a41cb-434f-4206-8bfa-be659892e8d6-secret-metrics-server-client-certs\") pod \"metrics-server-6fb78998c8-bh5np\" (UID: \"cf3a41cb-434f-4206-8bfa-be659892e8d6\") " pod="openshift-monitoring/metrics-server-6fb78998c8-bh5np" Sep 10 23:37:56.093954 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:56.093799 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9w7r9\" (UniqueName: \"kubernetes.io/projected/cf3a41cb-434f-4206-8bfa-be659892e8d6-kube-api-access-9w7r9\") pod \"metrics-server-6fb78998c8-bh5np\" (UID: \"cf3a41cb-434f-4206-8bfa-be659892e8d6\") " pod="openshift-monitoring/metrics-server-6fb78998c8-bh5np" Sep 10 23:37:56.093954 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:56.093826 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/cf3a41cb-434f-4206-8bfa-be659892e8d6-audit-log\") pod \"metrics-server-6fb78998c8-bh5np\" (UID: \"cf3a41cb-434f-4206-8bfa-be659892e8d6\") " pod="openshift-monitoring/metrics-server-6fb78998c8-bh5np" Sep 10 23:37:56.094087 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:56.093960 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cf3a41cb-434f-4206-8bfa-be659892e8d6-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6fb78998c8-bh5np\" (UID: \"cf3a41cb-434f-4206-8bfa-be659892e8d6\") " pod="openshift-monitoring/metrics-server-6fb78998c8-bh5np" Sep 10 23:37:56.094087 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:56.094001 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/cf3a41cb-434f-4206-8bfa-be659892e8d6-secret-metrics-server-tls\") pod \"metrics-server-6fb78998c8-bh5np\" (UID: \"cf3a41cb-434f-4206-8bfa-be659892e8d6\") " pod="openshift-monitoring/metrics-server-6fb78998c8-bh5np" Sep 10 23:37:56.094204 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:56.094180 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/cf3a41cb-434f-4206-8bfa-be659892e8d6-audit-log\") pod \"metrics-server-6fb78998c8-bh5np\" (UID: \"cf3a41cb-434f-4206-8bfa-be659892e8d6\") " pod="openshift-monitoring/metrics-server-6fb78998c8-bh5np" Sep 10 23:37:56.094804 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:56.094730 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cf3a41cb-434f-4206-8bfa-be659892e8d6-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6fb78998c8-bh5np\" (UID: \"cf3a41cb-434f-4206-8bfa-be659892e8d6\") " pod="openshift-monitoring/metrics-server-6fb78998c8-bh5np" Sep 10 23:37:56.094927 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:56.094806 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/cf3a41cb-434f-4206-8bfa-be659892e8d6-metrics-server-audit-profiles\") pod \"metrics-server-6fb78998c8-bh5np\" (UID: \"cf3a41cb-434f-4206-8bfa-be659892e8d6\") " pod="openshift-monitoring/metrics-server-6fb78998c8-bh5np" Sep 10 23:37:56.096654 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:56.096620 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/cf3a41cb-434f-4206-8bfa-be659892e8d6-secret-metrics-server-tls\") pod \"metrics-server-6fb78998c8-bh5np\" (UID: \"cf3a41cb-434f-4206-8bfa-be659892e8d6\") " pod="openshift-monitoring/metrics-server-6fb78998c8-bh5np" Sep 10 23:37:56.096835 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:56.096815 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/cf3a41cb-434f-4206-8bfa-be659892e8d6-secret-metrics-server-client-certs\") pod \"metrics-server-6fb78998c8-bh5np\" (UID: \"cf3a41cb-434f-4206-8bfa-be659892e8d6\") " pod="openshift-monitoring/metrics-server-6fb78998c8-bh5np" Sep 10 23:37:56.096949 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:56.096889 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/cf3a41cb-434f-4206-8bfa-be659892e8d6-client-ca-bundle\") pod \"metrics-server-6fb78998c8-bh5np\" (UID: \"cf3a41cb-434f-4206-8bfa-be659892e8d6\") " pod="openshift-monitoring/metrics-server-6fb78998c8-bh5np" Sep 10 23:37:56.104098 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:56.104070 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9w7r9\" (UniqueName: \"kubernetes.io/projected/cf3a41cb-434f-4206-8bfa-be659892e8d6-kube-api-access-9w7r9\") pod \"metrics-server-6fb78998c8-bh5np\" (UID: \"cf3a41cb-434f-4206-8bfa-be659892e8d6\") " pod="openshift-monitoring/metrics-server-6fb78998c8-bh5np" Sep 10 23:37:56.270335 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:56.270300 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6fb78998c8-bh5np" Sep 10 23:37:56.412437 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:56.410802 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-8kc4t"] Sep 10 23:37:56.416151 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:56.414763 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-8kc4t" Sep 10 23:37:56.425698 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:56.422252 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-jmp7q\"" Sep 10 23:37:56.425698 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:56.422515 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Sep 10 23:37:56.425698 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:56.425192 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-8kc4t"] Sep 10 23:37:56.498042 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:56.497914 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tfc7\" (UniqueName: \"kubernetes.io/projected/82b63a87-37d1-40bc-973d-478d4a807db3-kube-api-access-5tfc7\") pod \"monitoring-plugin-5d85778b9d-8kc4t\" (UID: \"82b63a87-37d1-40bc-973d-478d4a807db3\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-8kc4t" Sep 10 23:37:56.498042 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:56.497967 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/82b63a87-37d1-40bc-973d-478d4a807db3-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-8kc4t\" (UID: \"82b63a87-37d1-40bc-973d-478d4a807db3\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-8kc4t" Sep 10 23:37:56.602682 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:56.602435 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tfc7\" (UniqueName: \"kubernetes.io/projected/82b63a87-37d1-40bc-973d-478d4a807db3-kube-api-access-5tfc7\") pod \"monitoring-plugin-5d85778b9d-8kc4t\" (UID: \"82b63a87-37d1-40bc-973d-478d4a807db3\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-8kc4t" Sep 10 23:37:56.602682 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:56.602537 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/82b63a87-37d1-40bc-973d-478d4a807db3-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-8kc4t\" (UID: \"82b63a87-37d1-40bc-973d-478d4a807db3\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-8kc4t" Sep 10 23:37:56.602885 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:37:56.602752 2574 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Sep 10 23:37:56.602885 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:37:56.602824 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82b63a87-37d1-40bc-973d-478d4a807db3-monitoring-plugin-cert podName:82b63a87-37d1-40bc-973d-478d4a807db3 nodeName:}" failed. No retries permitted until 2026-09-10 23:37:57.102803286 +0000 UTC m=+68.490406043 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/82b63a87-37d1-40bc-973d-478d4a807db3-monitoring-plugin-cert") pod "monitoring-plugin-5d85778b9d-8kc4t" (UID: "82b63a87-37d1-40bc-973d-478d4a807db3") : secret "monitoring-plugin-cert" not found Sep 10 23:37:56.623588 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:56.623511 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-v87ts"] Sep 10 23:37:56.626688 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:56.625300 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-75vlp"] Sep 10 23:37:56.627196 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:56.627109 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-zb6x4"] Sep 10 23:37:56.628002 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:56.627974 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tfc7\" (UniqueName: \"kubernetes.io/projected/82b63a87-37d1-40bc-973d-478d4a807db3-kube-api-access-5tfc7\") pod \"monitoring-plugin-5d85778b9d-8kc4t\" (UID: \"82b63a87-37d1-40bc-973d-478d4a807db3\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-8kc4t" Sep 10 23:37:56.629686 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:37:56.629603 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a0ec520_b33d_49cf_9714_1662f5910e5b.slice/crio-d5852df5f3770c0334bc5630fcdbd018601faefaa6b634972783020bf07fc7aa WatchSource:0}: Error finding container d5852df5f3770c0334bc5630fcdbd018601faefaa6b634972783020bf07fc7aa: Status 404 returned error can't find the container with id d5852df5f3770c0334bc5630fcdbd018601faefaa6b634972783020bf07fc7aa Sep 10 23:37:56.631548 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:37:56.631522 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6df7e606_319b_4b9a_b6da_a850ecc9d267.slice/crio-6a6c7ed40d192fdf9cd444320906e109bdbe87413bfa8f859ac6bb1116847f9d WatchSource:0}: Error finding container 6a6c7ed40d192fdf9cd444320906e109bdbe87413bfa8f859ac6bb1116847f9d: Status 404 returned error can't find the container with id 6a6c7ed40d192fdf9cd444320906e109bdbe87413bfa8f859ac6bb1116847f9d Sep 10 23:37:56.633229 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:37:56.632504 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24dda81e_d88e_4f29_836b_44bb86bf461d.slice/crio-8343444adfa8aba01f13cdd88cb45d369b14c5f7a49b340f52175543ca55687c WatchSource:0}: Error finding container 8343444adfa8aba01f13cdd88cb45d369b14c5f7a49b340f52175543ca55687c: Status 404 returned error can't find the container with id 8343444adfa8aba01f13cdd88cb45d369b14c5f7a49b340f52175543ca55687c Sep 10 23:37:56.643561 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:56.643535 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6fb78998c8-bh5np"] Sep 10 23:37:56.655246 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:37:56.655206 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf3a41cb_434f_4206_8bfa_be659892e8d6.slice/crio-f4533479de747cf798a6cff153b94805b866e00c70c4dcc9d6cc0c7a2f808194 WatchSource:0}: Error finding container f4533479de747cf798a6cff153b94805b866e00c70c4dcc9d6cc0c7a2f808194: Status 404 returned error can't find the container with id f4533479de747cf798a6cff153b94805b866e00c70c4dcc9d6cc0c7a2f808194 Sep 10 23:37:56.826264 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:56.826231 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-b89f77d6-6zkxh"] Sep 10 23:37:56.857498 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:56.857414 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-b89f77d6-6zkxh" Sep 10 23:37:56.878650 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:56.878610 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-c7bd69f7f-jgkpc"] Sep 10 23:37:56.880820 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:56.880800 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c7bd69f7f-jgkpc" Sep 10 23:37:56.896837 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:56.896808 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-c7bd69f7f-jgkpc"] Sep 10 23:37:57.006066 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:57.006032 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/570bf3c6-39b8-4d1a-8957-3fd162f49f96-oauth-serving-cert\") pod \"console-c7bd69f7f-jgkpc\" (UID: \"570bf3c6-39b8-4d1a-8957-3fd162f49f96\") " pod="openshift-console/console-c7bd69f7f-jgkpc" Sep 10 23:37:57.006526 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:57.006071 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/570bf3c6-39b8-4d1a-8957-3fd162f49f96-console-config\") pod \"console-c7bd69f7f-jgkpc\" (UID: \"570bf3c6-39b8-4d1a-8957-3fd162f49f96\") " pod="openshift-console/console-c7bd69f7f-jgkpc" Sep 10 23:37:57.006526 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:57.006105 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2t9w6\" (UniqueName: \"kubernetes.io/projected/570bf3c6-39b8-4d1a-8957-3fd162f49f96-kube-api-access-2t9w6\") pod \"console-c7bd69f7f-jgkpc\" (UID: \"570bf3c6-39b8-4d1a-8957-3fd162f49f96\") " pod="openshift-console/console-c7bd69f7f-jgkpc" Sep 10 23:37:57.006526 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:57.006124 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/570bf3c6-39b8-4d1a-8957-3fd162f49f96-trusted-ca-bundle\") pod \"console-c7bd69f7f-jgkpc\" (UID: \"570bf3c6-39b8-4d1a-8957-3fd162f49f96\") " pod="openshift-console/console-c7bd69f7f-jgkpc" Sep 10 23:37:57.006526 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:57.006180 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/570bf3c6-39b8-4d1a-8957-3fd162f49f96-service-ca\") pod \"console-c7bd69f7f-jgkpc\" (UID: \"570bf3c6-39b8-4d1a-8957-3fd162f49f96\") " pod="openshift-console/console-c7bd69f7f-jgkpc" Sep 10 23:37:57.006526 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:57.006283 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/570bf3c6-39b8-4d1a-8957-3fd162f49f96-console-serving-cert\") pod \"console-c7bd69f7f-jgkpc\" (UID: \"570bf3c6-39b8-4d1a-8957-3fd162f49f96\") " pod="openshift-console/console-c7bd69f7f-jgkpc" Sep 10 23:37:57.006526 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:57.006363 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/570bf3c6-39b8-4d1a-8957-3fd162f49f96-console-oauth-config\") pod \"console-c7bd69f7f-jgkpc\" (UID: \"570bf3c6-39b8-4d1a-8957-3fd162f49f96\") " pod="openshift-console/console-c7bd69f7f-jgkpc" Sep 10 23:37:57.107758 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:57.107670 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/82b63a87-37d1-40bc-973d-478d4a807db3-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-8kc4t\" (UID: \"82b63a87-37d1-40bc-973d-478d4a807db3\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-8kc4t" Sep 10 23:37:57.107758 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:57.107720 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/570bf3c6-39b8-4d1a-8957-3fd162f49f96-console-serving-cert\") pod \"console-c7bd69f7f-jgkpc\" (UID: \"570bf3c6-39b8-4d1a-8957-3fd162f49f96\") " pod="openshift-console/console-c7bd69f7f-jgkpc" Sep 10 23:37:57.107967 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:57.107792 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/570bf3c6-39b8-4d1a-8957-3fd162f49f96-console-oauth-config\") pod \"console-c7bd69f7f-jgkpc\" (UID: \"570bf3c6-39b8-4d1a-8957-3fd162f49f96\") " pod="openshift-console/console-c7bd69f7f-jgkpc" Sep 10 23:37:57.107967 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:57.107843 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/570bf3c6-39b8-4d1a-8957-3fd162f49f96-oauth-serving-cert\") pod \"console-c7bd69f7f-jgkpc\" (UID: \"570bf3c6-39b8-4d1a-8957-3fd162f49f96\") " pod="openshift-console/console-c7bd69f7f-jgkpc" Sep 10 23:37:57.107967 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:57.107869 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/570bf3c6-39b8-4d1a-8957-3fd162f49f96-console-config\") pod \"console-c7bd69f7f-jgkpc\" (UID: \"570bf3c6-39b8-4d1a-8957-3fd162f49f96\") " pod="openshift-console/console-c7bd69f7f-jgkpc" Sep 10 23:37:57.107967 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:57.107909 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2t9w6\" (UniqueName: \"kubernetes.io/projected/570bf3c6-39b8-4d1a-8957-3fd162f49f96-kube-api-access-2t9w6\") pod \"console-c7bd69f7f-jgkpc\" (UID: \"570bf3c6-39b8-4d1a-8957-3fd162f49f96\") " pod="openshift-console/console-c7bd69f7f-jgkpc" Sep 10 23:37:57.107967 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:57.107936 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/570bf3c6-39b8-4d1a-8957-3fd162f49f96-trusted-ca-bundle\") pod \"console-c7bd69f7f-jgkpc\" (UID: \"570bf3c6-39b8-4d1a-8957-3fd162f49f96\") " pod="openshift-console/console-c7bd69f7f-jgkpc" Sep 10 23:37:57.108210 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:57.107970 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/570bf3c6-39b8-4d1a-8957-3fd162f49f96-service-ca\") pod \"console-c7bd69f7f-jgkpc\" (UID: \"570bf3c6-39b8-4d1a-8957-3fd162f49f96\") " pod="openshift-console/console-c7bd69f7f-jgkpc" Sep 10 23:37:57.109537 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:57.108738 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/570bf3c6-39b8-4d1a-8957-3fd162f49f96-service-ca\") pod \"console-c7bd69f7f-jgkpc\" (UID: \"570bf3c6-39b8-4d1a-8957-3fd162f49f96\") " pod="openshift-console/console-c7bd69f7f-jgkpc" Sep 10 23:37:57.109744 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:57.109712 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/570bf3c6-39b8-4d1a-8957-3fd162f49f96-oauth-serving-cert\") pod \"console-c7bd69f7f-jgkpc\" (UID: \"570bf3c6-39b8-4d1a-8957-3fd162f49f96\") " pod="openshift-console/console-c7bd69f7f-jgkpc" Sep 10 23:37:57.110938 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:57.110912 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/570bf3c6-39b8-4d1a-8957-3fd162f49f96-console-config\") pod \"console-c7bd69f7f-jgkpc\" (UID: \"570bf3c6-39b8-4d1a-8957-3fd162f49f96\") " pod="openshift-console/console-c7bd69f7f-jgkpc" Sep 10 23:37:57.111196 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:57.111168 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/570bf3c6-39b8-4d1a-8957-3fd162f49f96-trusted-ca-bundle\") pod \"console-c7bd69f7f-jgkpc\" (UID: \"570bf3c6-39b8-4d1a-8957-3fd162f49f96\") " pod="openshift-console/console-c7bd69f7f-jgkpc" Sep 10 23:37:57.113750 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:57.113693 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/82b63a87-37d1-40bc-973d-478d4a807db3-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-8kc4t\" (UID: \"82b63a87-37d1-40bc-973d-478d4a807db3\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-8kc4t" Sep 10 23:37:57.115602 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:57.115557 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/570bf3c6-39b8-4d1a-8957-3fd162f49f96-console-oauth-config\") pod \"console-c7bd69f7f-jgkpc\" (UID: \"570bf3c6-39b8-4d1a-8957-3fd162f49f96\") " pod="openshift-console/console-c7bd69f7f-jgkpc" Sep 10 23:37:57.116953 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:57.116392 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/570bf3c6-39b8-4d1a-8957-3fd162f49f96-console-serving-cert\") pod \"console-c7bd69f7f-jgkpc\" (UID: \"570bf3c6-39b8-4d1a-8957-3fd162f49f96\") " pod="openshift-console/console-c7bd69f7f-jgkpc" Sep 10 23:37:57.120403 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:57.120361 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2t9w6\" (UniqueName: \"kubernetes.io/projected/570bf3c6-39b8-4d1a-8957-3fd162f49f96-kube-api-access-2t9w6\") pod \"console-c7bd69f7f-jgkpc\" (UID: \"570bf3c6-39b8-4d1a-8957-3fd162f49f96\") " pod="openshift-console/console-c7bd69f7f-jgkpc" Sep 10 23:37:57.189342 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:57.189303 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c7bd69f7f-jgkpc" Sep 10 23:37:57.380515 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:57.380134 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-8kc4t" Sep 10 23:37:57.468605 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:57.467889 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-75vlp" event={"ID":"3a0ec520-b33d-49cf-9714-1662f5910e5b","Type":"ContainerStarted","Data":"d5852df5f3770c0334bc5630fcdbd018601faefaa6b634972783020bf07fc7aa"} Sep 10 23:37:57.473431 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:57.473399 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-zb6x4" event={"ID":"24dda81e-d88e-4f29-836b-44bb86bf461d","Type":"ContainerStarted","Data":"8343444adfa8aba01f13cdd88cb45d369b14c5f7a49b340f52175543ca55687c"} Sep 10 23:37:57.476041 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:57.475964 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-v87ts" event={"ID":"6df7e606-319b-4b9a-b6da-a850ecc9d267","Type":"ContainerStarted","Data":"6a6c7ed40d192fdf9cd444320906e109bdbe87413bfa8f859ac6bb1116847f9d"} Sep 10 23:37:57.479501 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:57.479322 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7f945b84b4-9zvwn" event={"ID":"008bdffc-746e-448b-9bd0-18155f16320e","Type":"ContainerStarted","Data":"6ebe2baef08de39e609173b2b6b70b3257cb9f91bb54518cbd6a36a39cdd0ffc"} Sep 10 23:37:57.479501 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:57.479391 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7f945b84b4-9zvwn" event={"ID":"008bdffc-746e-448b-9bd0-18155f16320e","Type":"ContainerStarted","Data":"d801b0adcf9552fe604bff322a59f348ea6ee274608f720d69ab405db656f91c"} Sep 10 23:37:57.479501 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:57.479432 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7f945b84b4-9zvwn" event={"ID":"008bdffc-746e-448b-9bd0-18155f16320e","Type":"ContainerStarted","Data":"a632bc693180371c6269ccc02c6e8ed4813eab44c00221eab9fa0e6de46d8408"} Sep 10 23:37:57.482783 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:57.481795 2574 generic.go:356] "Generic (PLEG): container finished" podID="b5e6b6fa-bb37-4770-93b3-26b897d01b78" containerID="873ad69cbae5056f37af4b529fbbb4328645ba2fc40fccd3ad6f44226d73a09c" exitCode=0 Sep 10 23:37:57.482783 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:57.481871 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b5e6b6fa-bb37-4770-93b3-26b897d01b78","Type":"ContainerDied","Data":"873ad69cbae5056f37af4b529fbbb4328645ba2fc40fccd3ad6f44226d73a09c"} Sep 10 23:37:57.489841 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:57.489047 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-qbts4" event={"ID":"a4d6f4b4-86db-445b-8ed8-20e0c4d622c2","Type":"ContainerStarted","Data":"b3d81c42164dc1e24c6dc8e2f43a44cb626cb2538210723efbaebadba620d7bc"} Sep 10 23:37:57.489841 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:57.489109 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-qbts4" event={"ID":"a4d6f4b4-86db-445b-8ed8-20e0c4d622c2","Type":"ContainerStarted","Data":"6c136822b718f7ad68a641e941028389e4105ab6e2537e50c656e608fc95f8ae"} Sep 10 23:37:57.489841 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:57.489140 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-qbts4" event={"ID":"a4d6f4b4-86db-445b-8ed8-20e0c4d622c2","Type":"ContainerStarted","Data":"892ebc7fb280e27aa3de08ef40cd692f565badca39c05f15d5afe233855fe3ae"} Sep 10 23:37:57.492890 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:57.492730 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6fb78998c8-bh5np" event={"ID":"cf3a41cb-434f-4206-8bfa-be659892e8d6","Type":"ContainerStarted","Data":"f4533479de747cf798a6cff153b94805b866e00c70c4dcc9d6cc0c7a2f808194"} Sep 10 23:37:57.506984 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:57.506956 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-c7bd69f7f-jgkpc"] Sep 10 23:37:57.539984 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:57.539826 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-qbts4" podStartSLOduration=4.100939652 podStartE2EDuration="6.539804707s" totalImagesPullingTime="2.438865055s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:37:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:37:57.539413269 +0000 UTC m=+68.927016026" watchObservedRunningTime="2026-09-10 23:37:57.539804707 +0000 UTC m=+68.927407477" Sep 10 23:37:57.678781 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:57.678022 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-8kc4t"] Sep 10 23:37:57.839188 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:37:57.838511 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod570bf3c6_39b8_4d1a_8957_3fd162f49f96.slice/crio-782ca31ebce3c1d29744589e0c5374b9757a5b8e6284c8970560af4dc1ce17ab WatchSource:0}: Error finding container 782ca31ebce3c1d29744589e0c5374b9757a5b8e6284c8970560af4dc1ce17ab: Status 404 returned error can't find the container with id 782ca31ebce3c1d29744589e0c5374b9757a5b8e6284c8970560af4dc1ce17ab Sep 10 23:37:58.036378 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:37:58.036266 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82b63a87_37d1_40bc_973d_478d4a807db3.slice/crio-c0b40c04a4cf60be8f3df034f0db99123ee981ec87da6a6ee385abe1a89061d6 WatchSource:0}: Error finding container c0b40c04a4cf60be8f3df034f0db99123ee981ec87da6a6ee385abe1a89061d6: Status 404 returned error can't find the container with id c0b40c04a4cf60be8f3df034f0db99123ee981ec87da6a6ee385abe1a89061d6 Sep 10 23:37:58.507413 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:58.507370 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c7bd69f7f-jgkpc" event={"ID":"570bf3c6-39b8-4d1a-8957-3fd162f49f96","Type":"ContainerStarted","Data":"782ca31ebce3c1d29744589e0c5374b9757a5b8e6284c8970560af4dc1ce17ab"} Sep 10 23:37:58.509990 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:58.509925 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-8kc4t" event={"ID":"82b63a87-37d1-40bc-973d-478d4a807db3","Type":"ContainerStarted","Data":"c0b40c04a4cf60be8f3df034f0db99123ee981ec87da6a6ee385abe1a89061d6"} Sep 10 23:37:59.524286 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:59.523403 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6fb78998c8-bh5np" event={"ID":"cf3a41cb-434f-4206-8bfa-be659892e8d6","Type":"ContainerStarted","Data":"24daf06c12832930b7dbe0119323751166e4442236fa6e50d9d616a31b16b787"} Sep 10 23:37:59.527706 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:59.527616 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-v87ts" event={"ID":"6df7e606-319b-4b9a-b6da-a850ecc9d267","Type":"ContainerStarted","Data":"feba12a39e86ff6b2eb99dd975d98c5f51c3c8f6fb5ffd1d7d63c20d03796529"} Sep 10 23:37:59.531338 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:59.531238 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7f945b84b4-9zvwn" event={"ID":"008bdffc-746e-448b-9bd0-18155f16320e","Type":"ContainerStarted","Data":"a6447ccf7ee2c2cec3ff57b3b761719bf624780bf6fc6f6932091f3806b97763"} Sep 10 23:37:59.534218 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:59.534195 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c7bd69f7f-jgkpc" event={"ID":"570bf3c6-39b8-4d1a-8957-3fd162f49f96","Type":"ContainerStarted","Data":"56bcc6fdcf2630b4364fa78063357bdcc193d19f980878ae3b8379c62645ece5"} Sep 10 23:37:59.541662 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:59.541032 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-6fb78998c8-bh5np" podStartSLOduration=2.354758364 podStartE2EDuration="4.541014826s" totalImagesPullingTime="2.186256462s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:37:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:37:59.53917037 +0000 UTC m=+70.926773152" watchObservedRunningTime="2026-09-10 23:37:59.541014826 +0000 UTC m=+70.928617587" Sep 10 23:37:59.564167 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:37:59.563629 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-c7bd69f7f-jgkpc" podStartSLOduration=3.563604887 podStartE2EDuration="3.563604887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:37:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:37:59.554918305 +0000 UTC m=+70.942521060" watchObservedRunningTime="2026-09-10 23:37:59.563604887 +0000 UTC m=+70.951207647" Sep 10 23:38:02.555141 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:38:02.554958 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7f945b84b4-9zvwn" event={"ID":"008bdffc-746e-448b-9bd0-18155f16320e","Type":"ContainerStarted","Data":"034199885ee3395989be33cc69e4bef76458bba95bf19926e830b695c0879fbe"} Sep 10 23:38:03.560112 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:38:03.560073 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-75vlp" event={"ID":"3a0ec520-b33d-49cf-9714-1662f5910e5b","Type":"ContainerStarted","Data":"2426465c3df80c27ace398a5fe9891e6e920af0f3688ce10764c0509472e900b"} Sep 10 23:38:03.561675 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:38:03.561624 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-8kc4t" event={"ID":"82b63a87-37d1-40bc-973d-478d4a807db3","Type":"ContainerStarted","Data":"7297c980410b6aaa04ffedfa6fa9635f966e6360a6b9786add75cbb5be7cfb08"} Sep 10 23:38:03.561837 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:38:03.561815 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-8kc4t" Sep 10 23:38:03.563273 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:38:03.563242 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-zb6x4" event={"ID":"24dda81e-d88e-4f29-836b-44bb86bf461d","Type":"ContainerStarted","Data":"c0703133906734b7c1b4b984794de4afc78f3c21ef72767d14a61971efa44556"} Sep 10 23:38:03.563408 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:38:03.563341 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-zb6x4" Sep 10 23:38:03.565173 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:38:03.565151 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-v87ts" event={"ID":"6df7e606-319b-4b9a-b6da-a850ecc9d267","Type":"ContainerStarted","Data":"3b06e98e7a529a99662c8732451b9ce8f0affbc682ac300e0045d0ae4c56ea9e"} Sep 10 23:38:03.567486 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:38:03.567467 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-8kc4t" Sep 10 23:38:03.568734 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:38:03.568715 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7f945b84b4-9zvwn" event={"ID":"008bdffc-746e-448b-9bd0-18155f16320e","Type":"ContainerStarted","Data":"c9cbc1871a099aa90f63322fb3c89cf44a28102e5a0abf335410fffa21d8efe0"} Sep 10 23:38:03.568870 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:38:03.568855 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-7f945b84b4-9zvwn" Sep 10 23:38:03.572472 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:38:03.572449 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b5e6b6fa-bb37-4770-93b3-26b897d01b78","Type":"ContainerStarted","Data":"a070f244866732c93907ad8342ba38d6adcea4587ac7096b44d3be2ba9b4034a"} Sep 10 23:38:03.572550 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:38:03.572500 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b5e6b6fa-bb37-4770-93b3-26b897d01b78","Type":"ContainerStarted","Data":"f80348225efe8840385b4eedc6c838207b0ba0e338f49e1b59f0d5c17e87494e"} Sep 10 23:38:03.572550 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:38:03.572526 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b5e6b6fa-bb37-4770-93b3-26b897d01b78","Type":"ContainerStarted","Data":"a7acd574b637aa6439f9028ffd8ad743a66e06ed499f2b90ec4d4101b4604be6"} Sep 10 23:38:03.572550 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:38:03.572550 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b5e6b6fa-bb37-4770-93b3-26b897d01b78","Type":"ContainerStarted","Data":"7ecd181e0560ca7138d9af0523dc157ba2415b198139ee29ad04bded645043f5"} Sep 10 23:38:03.572678 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:38:03.572574 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b5e6b6fa-bb37-4770-93b3-26b897d01b78","Type":"ContainerStarted","Data":"41d95795f5a27472cbd1150c87190456fd9c0026aa9812485f6af080e9d21638"} Sep 10 23:38:03.572678 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:38:03.572598 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b5e6b6fa-bb37-4770-93b3-26b897d01b78","Type":"ContainerStarted","Data":"fda59be1565242a8da7bdc3e969f44bdbf8fd82e0b3cc388e2a78e11fd8ddabc"} Sep 10 23:38:03.574353 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:38:03.574325 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-7f945b84b4-9zvwn" Sep 10 23:38:03.575982 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:38:03.575944 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-75vlp" podStartSLOduration=68.682647462 podStartE2EDuration="1m14.575931385s" totalImagesPullingTime="5.893283923s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 23:36:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:38:03.574624592 +0000 UTC m=+74.962227353" watchObservedRunningTime="2026-09-10 23:38:03.575931385 +0000 UTC m=+74.963534143" Sep 10 23:38:03.590839 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:38:03.590796 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-8kc4t" podStartSLOduration=3.191172588 podStartE2EDuration="7.590783191s" totalImagesPullingTime="4.399610603s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:37:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:38:03.589110067 +0000 UTC m=+74.976712825" watchObservedRunningTime="2026-09-10 23:38:03.590783191 +0000 UTC m=+74.978385967" Sep 10 23:38:03.623131 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:38:03.623075 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=4.228903809 podStartE2EDuration="11.623051904s" totalImagesPullingTime="7.374274889s" totalInitContainerRuntime="19.873206ms" isStatefulPod=false podCreationTimestamp="2026-09-10 23:37:52 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:38:03.620142629 +0000 UTC m=+75.007745410" watchObservedRunningTime="2026-09-10 23:38:03.623051904 +0000 UTC m=+75.010654665" Sep 10 23:38:03.640286 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:38:03.640236 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-v87ts" podStartSLOduration=72.431793359 podStartE2EDuration="1m14.640219672s" totalImagesPullingTime="2.208426313s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:36:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:38:03.639097181 +0000 UTC m=+75.026699939" watchObservedRunningTime="2026-09-10 23:38:03.640219672 +0000 UTC m=+75.027822431" Sep 10 23:38:03.669017 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:38:03.668962 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-7f945b84b4-9zvwn" podStartSLOduration=6.432372452 podStartE2EDuration="10.668945534s" totalImagesPullingTime="4.236573082s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:37:53 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:38:03.667267721 +0000 UTC m=+75.054870481" watchObservedRunningTime="2026-09-10 23:38:03.668945534 +0000 UTC m=+75.056548330" Sep 10 23:38:03.687233 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:38:03.687186 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-zb6x4" podStartSLOduration=68.848232966 podStartE2EDuration="1m14.68716998s" totalImagesPullingTime="5.838937014s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:36:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:38:03.686006324 +0000 UTC m=+75.073609083" watchObservedRunningTime="2026-09-10 23:38:03.68716998 +0000 UTC m=+75.074772738" Sep 10 23:38:07.189982 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:38:07.189933 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-c7bd69f7f-jgkpc" Sep 10 23:38:07.190337 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:38:07.190003 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-c7bd69f7f-jgkpc" Sep 10 23:38:07.194739 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:38:07.194716 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-c7bd69f7f-jgkpc" Sep 10 23:38:07.596059 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:38:07.596031 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-c7bd69f7f-jgkpc" Sep 10 23:38:07.639247 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:38:07.639213 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-68cb69c474-tfm62"] Sep 10 23:38:16.270862 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:38:16.270745 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-6fb78998c8-bh5np" Sep 10 23:38:16.270862 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:38:16.270807 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-6fb78998c8-bh5np" Sep 10 23:38:21.846044 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:38:21.845982 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-b89f77d6-6zkxh" podUID="deaa3139-0842-40a9-907a-306517f2ede3" containerName="console" containerID="cri-o://24a5e4150b9a11d59298f3a7630e96d175482ce6714b82fc8e748b64cfdcfd17" gracePeriod=15 Sep 10 23:38:22.083367 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:38:22.083340 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b89f77d6-6zkxh" Sep 10 23:38:22.134023 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:38:22.133956 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/deaa3139-0842-40a9-907a-306517f2ede3-console-serving-cert\") pod \"deaa3139-0842-40a9-907a-306517f2ede3\" (UID: \"deaa3139-0842-40a9-907a-306517f2ede3\") " Sep 10 23:38:22.134023 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:38:22.133997 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/deaa3139-0842-40a9-907a-306517f2ede3-trusted-ca-bundle\") pod \"deaa3139-0842-40a9-907a-306517f2ede3\" (UID: \"deaa3139-0842-40a9-907a-306517f2ede3\") " Sep 10 23:38:22.134023 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:38:22.134020 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8p6r5\" (UniqueName: \"kubernetes.io/projected/deaa3139-0842-40a9-907a-306517f2ede3-kube-api-access-8p6r5\") pod \"deaa3139-0842-40a9-907a-306517f2ede3\" (UID: \"deaa3139-0842-40a9-907a-306517f2ede3\") " Sep 10 23:38:22.134226 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:38:22.134066 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/deaa3139-0842-40a9-907a-306517f2ede3-console-oauth-config\") pod \"deaa3139-0842-40a9-907a-306517f2ede3\" (UID: \"deaa3139-0842-40a9-907a-306517f2ede3\") " Sep 10 23:38:22.134226 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:38:22.134087 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/deaa3139-0842-40a9-907a-306517f2ede3-service-ca\") pod \"deaa3139-0842-40a9-907a-306517f2ede3\" (UID: \"deaa3139-0842-40a9-907a-306517f2ede3\") " Sep 10 23:38:22.134226 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:38:22.134107 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/deaa3139-0842-40a9-907a-306517f2ede3-console-config\") pod \"deaa3139-0842-40a9-907a-306517f2ede3\" (UID: \"deaa3139-0842-40a9-907a-306517f2ede3\") " Sep 10 23:38:22.134226 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:38:22.134167 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/deaa3139-0842-40a9-907a-306517f2ede3-oauth-serving-cert\") pod \"deaa3139-0842-40a9-907a-306517f2ede3\" (UID: \"deaa3139-0842-40a9-907a-306517f2ede3\") " Sep 10 23:38:22.134518 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:38:22.134491 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/deaa3139-0842-40a9-907a-306517f2ede3-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "deaa3139-0842-40a9-907a-306517f2ede3" (UID: "deaa3139-0842-40a9-907a-306517f2ede3"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 23:38:22.134711 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:38:22.134624 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/deaa3139-0842-40a9-907a-306517f2ede3-service-ca" (OuterVolumeSpecName: "service-ca") pod "deaa3139-0842-40a9-907a-306517f2ede3" (UID: "deaa3139-0842-40a9-907a-306517f2ede3"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 23:38:22.134711 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:38:22.134691 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/deaa3139-0842-40a9-907a-306517f2ede3-console-config" (OuterVolumeSpecName: "console-config") pod "deaa3139-0842-40a9-907a-306517f2ede3" (UID: "deaa3139-0842-40a9-907a-306517f2ede3"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 23:38:22.134889 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:38:22.134698 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/deaa3139-0842-40a9-907a-306517f2ede3-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "deaa3139-0842-40a9-907a-306517f2ede3" (UID: "deaa3139-0842-40a9-907a-306517f2ede3"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 23:38:22.136436 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:38:22.136406 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/deaa3139-0842-40a9-907a-306517f2ede3-kube-api-access-8p6r5" (OuterVolumeSpecName: "kube-api-access-8p6r5") pod "deaa3139-0842-40a9-907a-306517f2ede3" (UID: "deaa3139-0842-40a9-907a-306517f2ede3"). InnerVolumeSpecName "kube-api-access-8p6r5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 23:38:22.136746 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:38:22.136415 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/deaa3139-0842-40a9-907a-306517f2ede3-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "deaa3139-0842-40a9-907a-306517f2ede3" (UID: "deaa3139-0842-40a9-907a-306517f2ede3"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 23:38:22.136746 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:38:22.136466 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/deaa3139-0842-40a9-907a-306517f2ede3-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "deaa3139-0842-40a9-907a-306517f2ede3" (UID: "deaa3139-0842-40a9-907a-306517f2ede3"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 23:38:22.234758 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:38:22.234727 2574 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/deaa3139-0842-40a9-907a-306517f2ede3-trusted-ca-bundle\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:38:22.234758 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:38:22.234753 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8p6r5\" (UniqueName: \"kubernetes.io/projected/deaa3139-0842-40a9-907a-306517f2ede3-kube-api-access-8p6r5\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:38:22.234758 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:38:22.234763 2574 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/deaa3139-0842-40a9-907a-306517f2ede3-console-oauth-config\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:38:22.234949 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:38:22.234772 2574 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/deaa3139-0842-40a9-907a-306517f2ede3-service-ca\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:38:22.234949 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:38:22.234782 2574 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/deaa3139-0842-40a9-907a-306517f2ede3-console-config\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:38:22.234949 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:38:22.234789 2574 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/deaa3139-0842-40a9-907a-306517f2ede3-oauth-serving-cert\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:38:22.234949 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:38:22.234799 2574 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/deaa3139-0842-40a9-907a-306517f2ede3-console-serving-cert\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:38:22.654752 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:38:22.654712 2574 generic.go:356] "Generic (PLEG): container finished" podID="deaa3139-0842-40a9-907a-306517f2ede3" containerID="24a5e4150b9a11d59298f3a7630e96d175482ce6714b82fc8e748b64cfdcfd17" exitCode=0 Sep 10 23:38:22.654888 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:38:22.654761 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b89f77d6-6zkxh" Sep 10 23:38:22.654888 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:38:22.654773 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b89f77d6-6zkxh" event={"ID":"deaa3139-0842-40a9-907a-306517f2ede3","Type":"ContainerDied","Data":"24a5e4150b9a11d59298f3a7630e96d175482ce6714b82fc8e748b64cfdcfd17"} Sep 10 23:38:22.654888 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:38:22.654825 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b89f77d6-6zkxh" event={"ID":"deaa3139-0842-40a9-907a-306517f2ede3","Type":"ContainerDied","Data":"96efcc8e4d324c12332af84dbfd7700c5e983fcf106db1b2407ee616dd5e2af0"} Sep 10 23:38:22.654888 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:38:22.654853 2574 scope.go:117] "RemoveContainer" containerID="24a5e4150b9a11d59298f3a7630e96d175482ce6714b82fc8e748b64cfdcfd17" Sep 10 23:38:22.664090 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:38:22.664072 2574 scope.go:117] "RemoveContainer" containerID="24a5e4150b9a11d59298f3a7630e96d175482ce6714b82fc8e748b64cfdcfd17" Sep 10 23:38:22.664385 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:38:22.664366 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24a5e4150b9a11d59298f3a7630e96d175482ce6714b82fc8e748b64cfdcfd17\": container with ID starting with 24a5e4150b9a11d59298f3a7630e96d175482ce6714b82fc8e748b64cfdcfd17 not found: ID does not exist" containerID="24a5e4150b9a11d59298f3a7630e96d175482ce6714b82fc8e748b64cfdcfd17" Sep 10 23:38:22.664434 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:38:22.664395 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24a5e4150b9a11d59298f3a7630e96d175482ce6714b82fc8e748b64cfdcfd17"} err="failed to get container status \"24a5e4150b9a11d59298f3a7630e96d175482ce6714b82fc8e748b64cfdcfd17\": rpc error: code = NotFound desc = could not find container \"24a5e4150b9a11d59298f3a7630e96d175482ce6714b82fc8e748b64cfdcfd17\": container with ID starting with 24a5e4150b9a11d59298f3a7630e96d175482ce6714b82fc8e748b64cfdcfd17 not found: ID does not exist" Sep 10 23:38:22.674408 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:38:22.674384 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-b89f77d6-6zkxh"] Sep 10 23:38:22.679796 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:38:22.679772 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-b89f77d6-6zkxh"] Sep 10 23:38:23.093263 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:38:23.093234 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="deaa3139-0842-40a9-907a-306517f2ede3" path="/var/lib/kubelet/pods/deaa3139-0842-40a9-907a-306517f2ede3/volumes" Sep 10 23:38:32.658073 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:38:32.658008 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-68cb69c474-tfm62" podUID="8b9ade2c-dd2a-4be4-9002-c9086d64e5b1" containerName="console" containerID="cri-o://a1e6fbc81f5d113ba74830ffa8979f84ff87b75deb04be4eaf5ccf273b4903e2" gracePeriod=15 Sep 10 23:38:32.920329 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:38:32.920253 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68cb69c474-tfm62" Sep 10 23:38:33.015313 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:38:33.015277 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8b9ade2c-dd2a-4be4-9002-c9086d64e5b1-oauth-serving-cert\") pod \"8b9ade2c-dd2a-4be4-9002-c9086d64e5b1\" (UID: \"8b9ade2c-dd2a-4be4-9002-c9086d64e5b1\") " Sep 10 23:38:33.015313 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:38:33.015320 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8b9ade2c-dd2a-4be4-9002-c9086d64e5b1-service-ca\") pod \"8b9ade2c-dd2a-4be4-9002-c9086d64e5b1\" (UID: \"8b9ade2c-dd2a-4be4-9002-c9086d64e5b1\") " Sep 10 23:38:33.015563 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:38:33.015338 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ndzl8\" (UniqueName: \"kubernetes.io/projected/8b9ade2c-dd2a-4be4-9002-c9086d64e5b1-kube-api-access-ndzl8\") pod \"8b9ade2c-dd2a-4be4-9002-c9086d64e5b1\" (UID: \"8b9ade2c-dd2a-4be4-9002-c9086d64e5b1\") " Sep 10 23:38:33.015563 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:38:33.015363 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8b9ade2c-dd2a-4be4-9002-c9086d64e5b1-console-config\") pod \"8b9ade2c-dd2a-4be4-9002-c9086d64e5b1\" (UID: \"8b9ade2c-dd2a-4be4-9002-c9086d64e5b1\") " Sep 10 23:38:33.015700 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:38:33.015562 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8b9ade2c-dd2a-4be4-9002-c9086d64e5b1-console-serving-cert\") pod \"8b9ade2c-dd2a-4be4-9002-c9086d64e5b1\" (UID: \"8b9ade2c-dd2a-4be4-9002-c9086d64e5b1\") " Sep 10 23:38:33.015757 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:38:33.015704 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8b9ade2c-dd2a-4be4-9002-c9086d64e5b1-console-oauth-config\") pod \"8b9ade2c-dd2a-4be4-9002-c9086d64e5b1\" (UID: \"8b9ade2c-dd2a-4be4-9002-c9086d64e5b1\") " Sep 10 23:38:33.015814 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:38:33.015751 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b9ade2c-dd2a-4be4-9002-c9086d64e5b1-console-config" (OuterVolumeSpecName: "console-config") pod "8b9ade2c-dd2a-4be4-9002-c9086d64e5b1" (UID: "8b9ade2c-dd2a-4be4-9002-c9086d64e5b1"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 23:38:33.015814 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:38:33.015770 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b9ade2c-dd2a-4be4-9002-c9086d64e5b1-service-ca" (OuterVolumeSpecName: "service-ca") pod "8b9ade2c-dd2a-4be4-9002-c9086d64e5b1" (UID: "8b9ade2c-dd2a-4be4-9002-c9086d64e5b1"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 23:38:33.015814 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:38:33.015779 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b9ade2c-dd2a-4be4-9002-c9086d64e5b1-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "8b9ade2c-dd2a-4be4-9002-c9086d64e5b1" (UID: "8b9ade2c-dd2a-4be4-9002-c9086d64e5b1"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 23:38:33.015938 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:38:33.015920 2574 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8b9ade2c-dd2a-4be4-9002-c9086d64e5b1-oauth-serving-cert\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:38:33.015979 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:38:33.015938 2574 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8b9ade2c-dd2a-4be4-9002-c9086d64e5b1-service-ca\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:38:33.015979 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:38:33.015948 2574 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8b9ade2c-dd2a-4be4-9002-c9086d64e5b1-console-config\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:38:33.017709 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:38:33.017688 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b9ade2c-dd2a-4be4-9002-c9086d64e5b1-kube-api-access-ndzl8" (OuterVolumeSpecName: "kube-api-access-ndzl8") pod "8b9ade2c-dd2a-4be4-9002-c9086d64e5b1" (UID: "8b9ade2c-dd2a-4be4-9002-c9086d64e5b1"). InnerVolumeSpecName "kube-api-access-ndzl8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 23:38:33.017799 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:38:33.017726 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b9ade2c-dd2a-4be4-9002-c9086d64e5b1-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "8b9ade2c-dd2a-4be4-9002-c9086d64e5b1" (UID: "8b9ade2c-dd2a-4be4-9002-c9086d64e5b1"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 23:38:33.017845 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:38:33.017799 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b9ade2c-dd2a-4be4-9002-c9086d64e5b1-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "8b9ade2c-dd2a-4be4-9002-c9086d64e5b1" (UID: "8b9ade2c-dd2a-4be4-9002-c9086d64e5b1"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 23:38:33.116458 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:38:33.116434 2574 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8b9ade2c-dd2a-4be4-9002-c9086d64e5b1-console-oauth-config\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:38:33.116458 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:38:33.116456 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndzl8\" (UniqueName: \"kubernetes.io/projected/8b9ade2c-dd2a-4be4-9002-c9086d64e5b1-kube-api-access-ndzl8\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:38:33.116596 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:38:33.116466 2574 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8b9ade2c-dd2a-4be4-9002-c9086d64e5b1-console-serving-cert\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:38:33.705327 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:38:33.705284 2574 generic.go:356] "Generic (PLEG): container finished" podID="8b9ade2c-dd2a-4be4-9002-c9086d64e5b1" containerID="a1e6fbc81f5d113ba74830ffa8979f84ff87b75deb04be4eaf5ccf273b4903e2" exitCode=0 Sep 10 23:38:33.705855 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:38:33.705343 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68cb69c474-tfm62" Sep 10 23:38:33.705855 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:38:33.705344 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68cb69c474-tfm62" event={"ID":"8b9ade2c-dd2a-4be4-9002-c9086d64e5b1","Type":"ContainerDied","Data":"a1e6fbc81f5d113ba74830ffa8979f84ff87b75deb04be4eaf5ccf273b4903e2"} Sep 10 23:38:33.705855 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:38:33.705403 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68cb69c474-tfm62" event={"ID":"8b9ade2c-dd2a-4be4-9002-c9086d64e5b1","Type":"ContainerDied","Data":"3e3c90429262aa455c9f949798427a997fa7c5ec0c1705a949426c341c8afad2"} Sep 10 23:38:33.705855 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:38:33.705427 2574 scope.go:117] "RemoveContainer" containerID="a1e6fbc81f5d113ba74830ffa8979f84ff87b75deb04be4eaf5ccf273b4903e2" Sep 10 23:38:33.713616 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:38:33.713598 2574 scope.go:117] "RemoveContainer" containerID="a1e6fbc81f5d113ba74830ffa8979f84ff87b75deb04be4eaf5ccf273b4903e2" Sep 10 23:38:33.713950 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:38:33.713931 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1e6fbc81f5d113ba74830ffa8979f84ff87b75deb04be4eaf5ccf273b4903e2\": container with ID starting with a1e6fbc81f5d113ba74830ffa8979f84ff87b75deb04be4eaf5ccf273b4903e2 not found: ID does not exist" containerID="a1e6fbc81f5d113ba74830ffa8979f84ff87b75deb04be4eaf5ccf273b4903e2" Sep 10 23:38:33.714039 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:38:33.713958 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1e6fbc81f5d113ba74830ffa8979f84ff87b75deb04be4eaf5ccf273b4903e2"} err="failed to get container status \"a1e6fbc81f5d113ba74830ffa8979f84ff87b75deb04be4eaf5ccf273b4903e2\": rpc error: code = NotFound desc = could not find container \"a1e6fbc81f5d113ba74830ffa8979f84ff87b75deb04be4eaf5ccf273b4903e2\": container with ID starting with a1e6fbc81f5d113ba74830ffa8979f84ff87b75deb04be4eaf5ccf273b4903e2 not found: ID does not exist" Sep 10 23:38:33.719902 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:38:33.719879 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-68cb69c474-tfm62"] Sep 10 23:38:33.723896 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:38:33.723877 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-68cb69c474-tfm62"] Sep 10 23:38:34.582380 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:38:34.582340 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-zb6x4" Sep 10 23:38:35.092845 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:38:35.092814 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b9ade2c-dd2a-4be4-9002-c9086d64e5b1" path="/var/lib/kubelet/pods/8b9ade2c-dd2a-4be4-9002-c9086d64e5b1/volumes" Sep 10 23:38:36.276896 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:38:36.276870 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-6fb78998c8-bh5np" Sep 10 23:38:36.280922 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:38:36.280884 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-6fb78998c8-bh5np" Sep 10 23:39:15.979033 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:15.978996 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-54dcfc4455-v9d4r"] Sep 10 23:39:15.979492 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:15.979301 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="deaa3139-0842-40a9-907a-306517f2ede3" containerName="console" Sep 10 23:39:15.979492 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:15.979313 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="deaa3139-0842-40a9-907a-306517f2ede3" containerName="console" Sep 10 23:39:15.979492 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:15.979329 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b9ade2c-dd2a-4be4-9002-c9086d64e5b1" containerName="console" Sep 10 23:39:15.979492 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:15.979335 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b9ade2c-dd2a-4be4-9002-c9086d64e5b1" containerName="console" Sep 10 23:39:15.979492 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:15.979394 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="deaa3139-0842-40a9-907a-306517f2ede3" containerName="console" Sep 10 23:39:15.979492 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:15.979407 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b9ade2c-dd2a-4be4-9002-c9086d64e5b1" containerName="console" Sep 10 23:39:15.981313 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:15.981297 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-54dcfc4455-v9d4r" Sep 10 23:39:15.983599 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:15.983575 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Sep 10 23:39:15.983806 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:15.983651 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-tqp9x\"" Sep 10 23:39:15.983806 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:15.983576 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Sep 10 23:39:15.983942 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:15.983921 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Sep 10 23:39:15.984004 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:15.983984 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/6db03de9-be49-4832-9ce9-5b131047b19b-secret-telemeter-client\") pod \"telemeter-client-54dcfc4455-v9d4r\" (UID: \"6db03de9-be49-4832-9ce9-5b131047b19b\") " pod="openshift-monitoring/telemeter-client-54dcfc4455-v9d4r" Sep 10 23:39:15.984049 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:15.984036 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6db03de9-be49-4832-9ce9-5b131047b19b-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-54dcfc4455-v9d4r\" (UID: \"6db03de9-be49-4832-9ce9-5b131047b19b\") " pod="openshift-monitoring/telemeter-client-54dcfc4455-v9d4r" Sep 10 23:39:15.984090 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:15.984059 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svhh4\" (UniqueName: \"kubernetes.io/projected/6db03de9-be49-4832-9ce9-5b131047b19b-kube-api-access-svhh4\") pod \"telemeter-client-54dcfc4455-v9d4r\" (UID: \"6db03de9-be49-4832-9ce9-5b131047b19b\") " pod="openshift-monitoring/telemeter-client-54dcfc4455-v9d4r" Sep 10 23:39:15.984196 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:15.984174 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/6db03de9-be49-4832-9ce9-5b131047b19b-federate-client-tls\") pod \"telemeter-client-54dcfc4455-v9d4r\" (UID: \"6db03de9-be49-4832-9ce9-5b131047b19b\") " pod="openshift-monitoring/telemeter-client-54dcfc4455-v9d4r" Sep 10 23:39:15.984260 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:15.984225 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6db03de9-be49-4832-9ce9-5b131047b19b-metrics-client-ca\") pod \"telemeter-client-54dcfc4455-v9d4r\" (UID: \"6db03de9-be49-4832-9ce9-5b131047b19b\") " pod="openshift-monitoring/telemeter-client-54dcfc4455-v9d4r" Sep 10 23:39:15.984260 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:15.984251 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/6db03de9-be49-4832-9ce9-5b131047b19b-telemeter-client-tls\") pod \"telemeter-client-54dcfc4455-v9d4r\" (UID: \"6db03de9-be49-4832-9ce9-5b131047b19b\") " pod="openshift-monitoring/telemeter-client-54dcfc4455-v9d4r" Sep 10 23:39:15.984357 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:15.984279 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6db03de9-be49-4832-9ce9-5b131047b19b-serving-certs-ca-bundle\") pod \"telemeter-client-54dcfc4455-v9d4r\" (UID: \"6db03de9-be49-4832-9ce9-5b131047b19b\") " pod="openshift-monitoring/telemeter-client-54dcfc4455-v9d4r" Sep 10 23:39:15.984357 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:15.984294 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Sep 10 23:39:15.984357 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:15.984317 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6db03de9-be49-4832-9ce9-5b131047b19b-telemeter-trusted-ca-bundle\") pod \"telemeter-client-54dcfc4455-v9d4r\" (UID: \"6db03de9-be49-4832-9ce9-5b131047b19b\") " pod="openshift-monitoring/telemeter-client-54dcfc4455-v9d4r" Sep 10 23:39:15.984501 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:15.984373 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Sep 10 23:39:15.990040 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:15.990017 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Sep 10 23:39:15.995520 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:15.995490 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-54dcfc4455-v9d4r"] Sep 10 23:39:16.085133 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:16.085095 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6db03de9-be49-4832-9ce9-5b131047b19b-metrics-client-ca\") pod \"telemeter-client-54dcfc4455-v9d4r\" (UID: \"6db03de9-be49-4832-9ce9-5b131047b19b\") " pod="openshift-monitoring/telemeter-client-54dcfc4455-v9d4r" Sep 10 23:39:16.085133 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:16.085130 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/6db03de9-be49-4832-9ce9-5b131047b19b-telemeter-client-tls\") pod \"telemeter-client-54dcfc4455-v9d4r\" (UID: \"6db03de9-be49-4832-9ce9-5b131047b19b\") " pod="openshift-monitoring/telemeter-client-54dcfc4455-v9d4r" Sep 10 23:39:16.085376 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:16.085147 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6db03de9-be49-4832-9ce9-5b131047b19b-serving-certs-ca-bundle\") pod \"telemeter-client-54dcfc4455-v9d4r\" (UID: \"6db03de9-be49-4832-9ce9-5b131047b19b\") " pod="openshift-monitoring/telemeter-client-54dcfc4455-v9d4r" Sep 10 23:39:16.085376 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:16.085171 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6db03de9-be49-4832-9ce9-5b131047b19b-telemeter-trusted-ca-bundle\") pod \"telemeter-client-54dcfc4455-v9d4r\" (UID: \"6db03de9-be49-4832-9ce9-5b131047b19b\") " pod="openshift-monitoring/telemeter-client-54dcfc4455-v9d4r" Sep 10 23:39:16.085376 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:16.085207 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/6db03de9-be49-4832-9ce9-5b131047b19b-secret-telemeter-client\") pod \"telemeter-client-54dcfc4455-v9d4r\" (UID: \"6db03de9-be49-4832-9ce9-5b131047b19b\") " pod="openshift-monitoring/telemeter-client-54dcfc4455-v9d4r" Sep 10 23:39:16.085376 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:16.085235 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6db03de9-be49-4832-9ce9-5b131047b19b-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-54dcfc4455-v9d4r\" (UID: \"6db03de9-be49-4832-9ce9-5b131047b19b\") " pod="openshift-monitoring/telemeter-client-54dcfc4455-v9d4r" Sep 10 23:39:16.085376 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:16.085260 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svhh4\" (UniqueName: \"kubernetes.io/projected/6db03de9-be49-4832-9ce9-5b131047b19b-kube-api-access-svhh4\") pod \"telemeter-client-54dcfc4455-v9d4r\" (UID: \"6db03de9-be49-4832-9ce9-5b131047b19b\") " pod="openshift-monitoring/telemeter-client-54dcfc4455-v9d4r" Sep 10 23:39:16.085376 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:16.085315 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/6db03de9-be49-4832-9ce9-5b131047b19b-federate-client-tls\") pod \"telemeter-client-54dcfc4455-v9d4r\" (UID: \"6db03de9-be49-4832-9ce9-5b131047b19b\") " pod="openshift-monitoring/telemeter-client-54dcfc4455-v9d4r" Sep 10 23:39:16.085984 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:16.085958 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6db03de9-be49-4832-9ce9-5b131047b19b-metrics-client-ca\") pod \"telemeter-client-54dcfc4455-v9d4r\" (UID: \"6db03de9-be49-4832-9ce9-5b131047b19b\") " pod="openshift-monitoring/telemeter-client-54dcfc4455-v9d4r" Sep 10 23:39:16.086159 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:16.086126 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6db03de9-be49-4832-9ce9-5b131047b19b-telemeter-trusted-ca-bundle\") pod \"telemeter-client-54dcfc4455-v9d4r\" (UID: \"6db03de9-be49-4832-9ce9-5b131047b19b\") " pod="openshift-monitoring/telemeter-client-54dcfc4455-v9d4r" Sep 10 23:39:16.086284 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:16.085976 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6db03de9-be49-4832-9ce9-5b131047b19b-serving-certs-ca-bundle\") pod \"telemeter-client-54dcfc4455-v9d4r\" (UID: \"6db03de9-be49-4832-9ce9-5b131047b19b\") " pod="openshift-monitoring/telemeter-client-54dcfc4455-v9d4r" Sep 10 23:39:16.087847 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:16.087826 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/6db03de9-be49-4832-9ce9-5b131047b19b-secret-telemeter-client\") pod \"telemeter-client-54dcfc4455-v9d4r\" (UID: \"6db03de9-be49-4832-9ce9-5b131047b19b\") " pod="openshift-monitoring/telemeter-client-54dcfc4455-v9d4r" Sep 10 23:39:16.087977 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:16.087958 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/6db03de9-be49-4832-9ce9-5b131047b19b-telemeter-client-tls\") pod \"telemeter-client-54dcfc4455-v9d4r\" (UID: \"6db03de9-be49-4832-9ce9-5b131047b19b\") " pod="openshift-monitoring/telemeter-client-54dcfc4455-v9d4r" Sep 10 23:39:16.088022 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:16.087987 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/6db03de9-be49-4832-9ce9-5b131047b19b-federate-client-tls\") pod \"telemeter-client-54dcfc4455-v9d4r\" (UID: \"6db03de9-be49-4832-9ce9-5b131047b19b\") " pod="openshift-monitoring/telemeter-client-54dcfc4455-v9d4r" Sep 10 23:39:16.088055 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:16.088029 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6db03de9-be49-4832-9ce9-5b131047b19b-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-54dcfc4455-v9d4r\" (UID: \"6db03de9-be49-4832-9ce9-5b131047b19b\") " pod="openshift-monitoring/telemeter-client-54dcfc4455-v9d4r" Sep 10 23:39:16.092238 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:16.092218 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-svhh4\" (UniqueName: \"kubernetes.io/projected/6db03de9-be49-4832-9ce9-5b131047b19b-kube-api-access-svhh4\") pod \"telemeter-client-54dcfc4455-v9d4r\" (UID: \"6db03de9-be49-4832-9ce9-5b131047b19b\") " pod="openshift-monitoring/telemeter-client-54dcfc4455-v9d4r" Sep 10 23:39:16.294296 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:16.294229 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-54dcfc4455-v9d4r" Sep 10 23:39:16.566699 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:16.566675 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-54dcfc4455-v9d4r"] Sep 10 23:39:16.568995 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:39:16.568968 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6db03de9_be49_4832_9ce9_5b131047b19b.slice/crio-db64762f3229c951cc10f5950b154f64c2b955d707eaa328fc9b43b3235b0603 WatchSource:0}: Error finding container db64762f3229c951cc10f5950b154f64c2b955d707eaa328fc9b43b3235b0603: Status 404 returned error can't find the container with id db64762f3229c951cc10f5950b154f64c2b955d707eaa328fc9b43b3235b0603 Sep 10 23:39:16.876496 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:16.876407 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-54dcfc4455-v9d4r" event={"ID":"6db03de9-be49-4832-9ce9-5b131047b19b","Type":"ContainerStarted","Data":"db64762f3229c951cc10f5950b154f64c2b955d707eaa328fc9b43b3235b0603"} Sep 10 23:39:18.886653 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:18.886610 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-54dcfc4455-v9d4r" event={"ID":"6db03de9-be49-4832-9ce9-5b131047b19b","Type":"ContainerStarted","Data":"944a08a4a60b11daefdc5b47e03c872217facd6200d235f0b819fc2b6a5fe5f9"} Sep 10 23:39:18.887038 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:18.886703 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-54dcfc4455-v9d4r" event={"ID":"6db03de9-be49-4832-9ce9-5b131047b19b","Type":"ContainerStarted","Data":"04c0600a700f604b367b75963abf6084162f437cdbb59628e8e19674a91716f7"} Sep 10 23:39:18.887038 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:18.886732 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-54dcfc4455-v9d4r" event={"ID":"6db03de9-be49-4832-9ce9-5b131047b19b","Type":"ContainerStarted","Data":"08176f5853913d1c88e456bcb95d165cf38b47d5fc685e960d7aef033f91c7e2"} Sep 10 23:39:18.906617 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:18.906563 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-54dcfc4455-v9d4r" podStartSLOduration=2.510574573 podStartE2EDuration="3.906546367s" totalImagesPullingTime="1.395971794s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:39:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:39:18.905005769 +0000 UTC m=+150.292608528" watchObservedRunningTime="2026-09-10 23:39:18.906546367 +0000 UTC m=+150.294149126" Sep 10 23:39:18.994155 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:18.994118 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7d68776c7f-xx5qf"] Sep 10 23:39:18.996451 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:18.996428 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d68776c7f-xx5qf" Sep 10 23:39:19.007946 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:19.007912 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bc902d48-8e0d-40dd-ba2a-3a43749c1a55-console-oauth-config\") pod \"console-7d68776c7f-xx5qf\" (UID: \"bc902d48-8e0d-40dd-ba2a-3a43749c1a55\") " pod="openshift-console/console-7d68776c7f-xx5qf" Sep 10 23:39:19.007946 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:19.007956 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7xf6\" (UniqueName: \"kubernetes.io/projected/bc902d48-8e0d-40dd-ba2a-3a43749c1a55-kube-api-access-m7xf6\") pod \"console-7d68776c7f-xx5qf\" (UID: \"bc902d48-8e0d-40dd-ba2a-3a43749c1a55\") " pod="openshift-console/console-7d68776c7f-xx5qf" Sep 10 23:39:19.008132 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:19.008039 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bc902d48-8e0d-40dd-ba2a-3a43749c1a55-service-ca\") pod \"console-7d68776c7f-xx5qf\" (UID: \"bc902d48-8e0d-40dd-ba2a-3a43749c1a55\") " pod="openshift-console/console-7d68776c7f-xx5qf" Sep 10 23:39:19.008132 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:19.008080 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc902d48-8e0d-40dd-ba2a-3a43749c1a55-trusted-ca-bundle\") pod \"console-7d68776c7f-xx5qf\" (UID: \"bc902d48-8e0d-40dd-ba2a-3a43749c1a55\") " pod="openshift-console/console-7d68776c7f-xx5qf" Sep 10 23:39:19.008132 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:19.008106 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bc902d48-8e0d-40dd-ba2a-3a43749c1a55-console-serving-cert\") pod \"console-7d68776c7f-xx5qf\" (UID: \"bc902d48-8e0d-40dd-ba2a-3a43749c1a55\") " pod="openshift-console/console-7d68776c7f-xx5qf" Sep 10 23:39:19.008132 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:19.008128 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bc902d48-8e0d-40dd-ba2a-3a43749c1a55-oauth-serving-cert\") pod \"console-7d68776c7f-xx5qf\" (UID: \"bc902d48-8e0d-40dd-ba2a-3a43749c1a55\") " pod="openshift-console/console-7d68776c7f-xx5qf" Sep 10 23:39:19.008332 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:19.008158 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bc902d48-8e0d-40dd-ba2a-3a43749c1a55-console-config\") pod \"console-7d68776c7f-xx5qf\" (UID: \"bc902d48-8e0d-40dd-ba2a-3a43749c1a55\") " pod="openshift-console/console-7d68776c7f-xx5qf" Sep 10 23:39:19.008744 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:19.008722 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7d68776c7f-xx5qf"] Sep 10 23:39:19.109550 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:19.109491 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bc902d48-8e0d-40dd-ba2a-3a43749c1a55-service-ca\") pod \"console-7d68776c7f-xx5qf\" (UID: \"bc902d48-8e0d-40dd-ba2a-3a43749c1a55\") " pod="openshift-console/console-7d68776c7f-xx5qf" Sep 10 23:39:19.109550 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:19.109546 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc902d48-8e0d-40dd-ba2a-3a43749c1a55-trusted-ca-bundle\") pod \"console-7d68776c7f-xx5qf\" (UID: \"bc902d48-8e0d-40dd-ba2a-3a43749c1a55\") " pod="openshift-console/console-7d68776c7f-xx5qf" Sep 10 23:39:19.109819 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:19.109575 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bc902d48-8e0d-40dd-ba2a-3a43749c1a55-console-serving-cert\") pod \"console-7d68776c7f-xx5qf\" (UID: \"bc902d48-8e0d-40dd-ba2a-3a43749c1a55\") " pod="openshift-console/console-7d68776c7f-xx5qf" Sep 10 23:39:19.109819 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:19.109599 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bc902d48-8e0d-40dd-ba2a-3a43749c1a55-oauth-serving-cert\") pod \"console-7d68776c7f-xx5qf\" (UID: \"bc902d48-8e0d-40dd-ba2a-3a43749c1a55\") " pod="openshift-console/console-7d68776c7f-xx5qf" Sep 10 23:39:19.109819 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:19.109631 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bc902d48-8e0d-40dd-ba2a-3a43749c1a55-console-config\") pod \"console-7d68776c7f-xx5qf\" (UID: \"bc902d48-8e0d-40dd-ba2a-3a43749c1a55\") " pod="openshift-console/console-7d68776c7f-xx5qf" Sep 10 23:39:19.109819 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:19.109726 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bc902d48-8e0d-40dd-ba2a-3a43749c1a55-console-oauth-config\") pod \"console-7d68776c7f-xx5qf\" (UID: \"bc902d48-8e0d-40dd-ba2a-3a43749c1a55\") " pod="openshift-console/console-7d68776c7f-xx5qf" Sep 10 23:39:19.109819 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:19.109752 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7xf6\" (UniqueName: \"kubernetes.io/projected/bc902d48-8e0d-40dd-ba2a-3a43749c1a55-kube-api-access-m7xf6\") pod \"console-7d68776c7f-xx5qf\" (UID: \"bc902d48-8e0d-40dd-ba2a-3a43749c1a55\") " pod="openshift-console/console-7d68776c7f-xx5qf" Sep 10 23:39:19.110450 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:19.110364 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bc902d48-8e0d-40dd-ba2a-3a43749c1a55-service-ca\") pod \"console-7d68776c7f-xx5qf\" (UID: \"bc902d48-8e0d-40dd-ba2a-3a43749c1a55\") " pod="openshift-console/console-7d68776c7f-xx5qf" Sep 10 23:39:19.110450 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:19.110414 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bc902d48-8e0d-40dd-ba2a-3a43749c1a55-oauth-serving-cert\") pod \"console-7d68776c7f-xx5qf\" (UID: \"bc902d48-8e0d-40dd-ba2a-3a43749c1a55\") " pod="openshift-console/console-7d68776c7f-xx5qf" Sep 10 23:39:19.110593 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:19.110448 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bc902d48-8e0d-40dd-ba2a-3a43749c1a55-console-config\") pod \"console-7d68776c7f-xx5qf\" (UID: \"bc902d48-8e0d-40dd-ba2a-3a43749c1a55\") " pod="openshift-console/console-7d68776c7f-xx5qf" Sep 10 23:39:19.110795 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:19.110770 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc902d48-8e0d-40dd-ba2a-3a43749c1a55-trusted-ca-bundle\") pod \"console-7d68776c7f-xx5qf\" (UID: \"bc902d48-8e0d-40dd-ba2a-3a43749c1a55\") " pod="openshift-console/console-7d68776c7f-xx5qf" Sep 10 23:39:19.112279 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:19.112251 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bc902d48-8e0d-40dd-ba2a-3a43749c1a55-console-serving-cert\") pod \"console-7d68776c7f-xx5qf\" (UID: \"bc902d48-8e0d-40dd-ba2a-3a43749c1a55\") " pod="openshift-console/console-7d68776c7f-xx5qf" Sep 10 23:39:19.112365 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:19.112303 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bc902d48-8e0d-40dd-ba2a-3a43749c1a55-console-oauth-config\") pod \"console-7d68776c7f-xx5qf\" (UID: \"bc902d48-8e0d-40dd-ba2a-3a43749c1a55\") " pod="openshift-console/console-7d68776c7f-xx5qf" Sep 10 23:39:19.117489 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:19.117472 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7xf6\" (UniqueName: \"kubernetes.io/projected/bc902d48-8e0d-40dd-ba2a-3a43749c1a55-kube-api-access-m7xf6\") pod \"console-7d68776c7f-xx5qf\" (UID: \"bc902d48-8e0d-40dd-ba2a-3a43749c1a55\") " pod="openshift-console/console-7d68776c7f-xx5qf" Sep 10 23:39:19.307921 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:19.307879 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d68776c7f-xx5qf" Sep 10 23:39:19.580440 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:19.580262 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7d68776c7f-xx5qf"] Sep 10 23:39:19.582968 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:39:19.582940 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc902d48_8e0d_40dd_ba2a_3a43749c1a55.slice/crio-169d6d0e81be7a5c6370f103e887c1ebccb06171933e7f84e9e8050892f57a3a WatchSource:0}: Error finding container 169d6d0e81be7a5c6370f103e887c1ebccb06171933e7f84e9e8050892f57a3a: Status 404 returned error can't find the container with id 169d6d0e81be7a5c6370f103e887c1ebccb06171933e7f84e9e8050892f57a3a Sep 10 23:39:19.896610 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:19.896481 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d68776c7f-xx5qf" event={"ID":"bc902d48-8e0d-40dd-ba2a-3a43749c1a55","Type":"ContainerStarted","Data":"b3ec4eb57b8d899aeafedcee23afa86fd552d681997a058ff58aab2f17131806"} Sep 10 23:39:19.897117 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:19.896665 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d68776c7f-xx5qf" event={"ID":"bc902d48-8e0d-40dd-ba2a-3a43749c1a55","Type":"ContainerStarted","Data":"169d6d0e81be7a5c6370f103e887c1ebccb06171933e7f84e9e8050892f57a3a"} Sep 10 23:39:19.913239 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:19.913194 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7d68776c7f-xx5qf" podStartSLOduration=1.9131695579999999 podStartE2EDuration="1.913169558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:39:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:39:19.911377944 +0000 UTC m=+151.298980703" watchObservedRunningTime="2026-09-10 23:39:19.913169558 +0000 UTC m=+151.300772316" Sep 10 23:39:29.309032 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:29.308985 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7d68776c7f-xx5qf" Sep 10 23:39:29.309470 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:29.309049 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7d68776c7f-xx5qf" Sep 10 23:39:29.313859 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:29.313837 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7d68776c7f-xx5qf" Sep 10 23:39:29.938457 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:29.938427 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7d68776c7f-xx5qf" Sep 10 23:39:29.980202 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:29.980158 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-c7bd69f7f-jgkpc"] Sep 10 23:39:30.092175 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:30.092144 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-778d864cdc-v6xfr"] Sep 10 23:39:30.098211 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:30.098186 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-778d864cdc-v6xfr" Sep 10 23:39:30.105866 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:30.105845 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-778d864cdc-v6xfr"] Sep 10 23:39:30.199528 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:30.199501 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/402e5517-4211-49e2-95a2-d9ea715f4a6c-oauth-serving-cert\") pod \"console-778d864cdc-v6xfr\" (UID: \"402e5517-4211-49e2-95a2-d9ea715f4a6c\") " pod="openshift-console/console-778d864cdc-v6xfr" Sep 10 23:39:30.199691 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:30.199539 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/402e5517-4211-49e2-95a2-d9ea715f4a6c-trusted-ca-bundle\") pod \"console-778d864cdc-v6xfr\" (UID: \"402e5517-4211-49e2-95a2-d9ea715f4a6c\") " pod="openshift-console/console-778d864cdc-v6xfr" Sep 10 23:39:30.199691 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:30.199565 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/402e5517-4211-49e2-95a2-d9ea715f4a6c-console-serving-cert\") pod \"console-778d864cdc-v6xfr\" (UID: \"402e5517-4211-49e2-95a2-d9ea715f4a6c\") " pod="openshift-console/console-778d864cdc-v6xfr" Sep 10 23:39:30.199691 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:30.199657 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/402e5517-4211-49e2-95a2-d9ea715f4a6c-service-ca\") pod \"console-778d864cdc-v6xfr\" (UID: \"402e5517-4211-49e2-95a2-d9ea715f4a6c\") " pod="openshift-console/console-778d864cdc-v6xfr" Sep 10 23:39:30.199803 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:30.199710 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/402e5517-4211-49e2-95a2-d9ea715f4a6c-console-config\") pod \"console-778d864cdc-v6xfr\" (UID: \"402e5517-4211-49e2-95a2-d9ea715f4a6c\") " pod="openshift-console/console-778d864cdc-v6xfr" Sep 10 23:39:30.199803 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:30.199729 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzzmh\" (UniqueName: \"kubernetes.io/projected/402e5517-4211-49e2-95a2-d9ea715f4a6c-kube-api-access-vzzmh\") pod \"console-778d864cdc-v6xfr\" (UID: \"402e5517-4211-49e2-95a2-d9ea715f4a6c\") " pod="openshift-console/console-778d864cdc-v6xfr" Sep 10 23:39:30.199803 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:30.199764 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/402e5517-4211-49e2-95a2-d9ea715f4a6c-console-oauth-config\") pod \"console-778d864cdc-v6xfr\" (UID: \"402e5517-4211-49e2-95a2-d9ea715f4a6c\") " pod="openshift-console/console-778d864cdc-v6xfr" Sep 10 23:39:30.300269 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:30.300235 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/402e5517-4211-49e2-95a2-d9ea715f4a6c-console-oauth-config\") pod \"console-778d864cdc-v6xfr\" (UID: \"402e5517-4211-49e2-95a2-d9ea715f4a6c\") " pod="openshift-console/console-778d864cdc-v6xfr" Sep 10 23:39:30.300269 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:30.300273 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/402e5517-4211-49e2-95a2-d9ea715f4a6c-oauth-serving-cert\") pod \"console-778d864cdc-v6xfr\" (UID: \"402e5517-4211-49e2-95a2-d9ea715f4a6c\") " pod="openshift-console/console-778d864cdc-v6xfr" Sep 10 23:39:30.300467 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:30.300394 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/402e5517-4211-49e2-95a2-d9ea715f4a6c-trusted-ca-bundle\") pod \"console-778d864cdc-v6xfr\" (UID: \"402e5517-4211-49e2-95a2-d9ea715f4a6c\") " pod="openshift-console/console-778d864cdc-v6xfr" Sep 10 23:39:30.300467 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:30.300440 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/402e5517-4211-49e2-95a2-d9ea715f4a6c-console-serving-cert\") pod \"console-778d864cdc-v6xfr\" (UID: \"402e5517-4211-49e2-95a2-d9ea715f4a6c\") " pod="openshift-console/console-778d864cdc-v6xfr" Sep 10 23:39:30.300553 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:30.300507 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/402e5517-4211-49e2-95a2-d9ea715f4a6c-service-ca\") pod \"console-778d864cdc-v6xfr\" (UID: \"402e5517-4211-49e2-95a2-d9ea715f4a6c\") " pod="openshift-console/console-778d864cdc-v6xfr" Sep 10 23:39:30.300606 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:30.300572 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/402e5517-4211-49e2-95a2-d9ea715f4a6c-console-config\") pod \"console-778d864cdc-v6xfr\" (UID: \"402e5517-4211-49e2-95a2-d9ea715f4a6c\") " pod="openshift-console/console-778d864cdc-v6xfr" Sep 10 23:39:30.300606 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:30.300598 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzzmh\" (UniqueName: \"kubernetes.io/projected/402e5517-4211-49e2-95a2-d9ea715f4a6c-kube-api-access-vzzmh\") pod \"console-778d864cdc-v6xfr\" (UID: \"402e5517-4211-49e2-95a2-d9ea715f4a6c\") " pod="openshift-console/console-778d864cdc-v6xfr" Sep 10 23:39:30.301038 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:30.300979 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/402e5517-4211-49e2-95a2-d9ea715f4a6c-oauth-serving-cert\") pod \"console-778d864cdc-v6xfr\" (UID: \"402e5517-4211-49e2-95a2-d9ea715f4a6c\") " pod="openshift-console/console-778d864cdc-v6xfr" Sep 10 23:39:30.301251 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:30.301227 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/402e5517-4211-49e2-95a2-d9ea715f4a6c-trusted-ca-bundle\") pod \"console-778d864cdc-v6xfr\" (UID: \"402e5517-4211-49e2-95a2-d9ea715f4a6c\") " pod="openshift-console/console-778d864cdc-v6xfr" Sep 10 23:39:30.301361 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:30.301248 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/402e5517-4211-49e2-95a2-d9ea715f4a6c-service-ca\") pod \"console-778d864cdc-v6xfr\" (UID: \"402e5517-4211-49e2-95a2-d9ea715f4a6c\") " pod="openshift-console/console-778d864cdc-v6xfr" Sep 10 23:39:30.301361 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:30.301294 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/402e5517-4211-49e2-95a2-d9ea715f4a6c-console-config\") pod \"console-778d864cdc-v6xfr\" (UID: \"402e5517-4211-49e2-95a2-d9ea715f4a6c\") " pod="openshift-console/console-778d864cdc-v6xfr" Sep 10 23:39:30.302719 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:30.302700 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/402e5517-4211-49e2-95a2-d9ea715f4a6c-console-oauth-config\") pod \"console-778d864cdc-v6xfr\" (UID: \"402e5517-4211-49e2-95a2-d9ea715f4a6c\") " pod="openshift-console/console-778d864cdc-v6xfr" Sep 10 23:39:30.302948 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:30.302929 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/402e5517-4211-49e2-95a2-d9ea715f4a6c-console-serving-cert\") pod \"console-778d864cdc-v6xfr\" (UID: \"402e5517-4211-49e2-95a2-d9ea715f4a6c\") " pod="openshift-console/console-778d864cdc-v6xfr" Sep 10 23:39:30.307535 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:30.307515 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzzmh\" (UniqueName: \"kubernetes.io/projected/402e5517-4211-49e2-95a2-d9ea715f4a6c-kube-api-access-vzzmh\") pod \"console-778d864cdc-v6xfr\" (UID: \"402e5517-4211-49e2-95a2-d9ea715f4a6c\") " pod="openshift-console/console-778d864cdc-v6xfr" Sep 10 23:39:30.408310 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:30.408250 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-778d864cdc-v6xfr" Sep 10 23:39:30.679473 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:30.679388 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-778d864cdc-v6xfr"] Sep 10 23:39:30.682257 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:39:30.682214 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod402e5517_4211_49e2_95a2_d9ea715f4a6c.slice/crio-8d071a0e4669d44924ff9d47614ac663a0c8655ce4fb6137b1b7797fdabae3fc WatchSource:0}: Error finding container 8d071a0e4669d44924ff9d47614ac663a0c8655ce4fb6137b1b7797fdabae3fc: Status 404 returned error can't find the container with id 8d071a0e4669d44924ff9d47614ac663a0c8655ce4fb6137b1b7797fdabae3fc Sep 10 23:39:30.940119 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:30.940084 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-778d864cdc-v6xfr" event={"ID":"402e5517-4211-49e2-95a2-d9ea715f4a6c","Type":"ContainerStarted","Data":"6c4f1d40f39a74469b215ff17af4efd2b6e5cfcb758ac41ae3bef268b495922a"} Sep 10 23:39:30.940281 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:30.940138 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-778d864cdc-v6xfr" event={"ID":"402e5517-4211-49e2-95a2-d9ea715f4a6c","Type":"ContainerStarted","Data":"8d071a0e4669d44924ff9d47614ac663a0c8655ce4fb6137b1b7797fdabae3fc"} Sep 10 23:39:30.959032 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:30.958985 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-778d864cdc-v6xfr" podStartSLOduration=0.958968186 podStartE2EDuration="958.968186ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:39:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:39:30.957699898 +0000 UTC m=+162.345302659" watchObservedRunningTime="2026-09-10 23:39:30.958968186 +0000 UTC m=+162.346570993" Sep 10 23:39:40.408517 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:40.408480 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-778d864cdc-v6xfr" Sep 10 23:39:40.409047 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:40.408561 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-778d864cdc-v6xfr" Sep 10 23:39:40.412884 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:40.412861 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-778d864cdc-v6xfr" Sep 10 23:39:40.982975 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:40.982942 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-778d864cdc-v6xfr" Sep 10 23:39:41.017326 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:41.017292 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7d68776c7f-xx5qf"] Sep 10 23:39:55.001728 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:55.001596 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-c7bd69f7f-jgkpc" podUID="570bf3c6-39b8-4d1a-8957-3fd162f49f96" containerName="console" containerID="cri-o://56bcc6fdcf2630b4364fa78063357bdcc193d19f980878ae3b8379c62645ece5" gracePeriod=15 Sep 10 23:39:55.238200 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:55.238173 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c7bd69f7f-jgkpc" Sep 10 23:39:55.419808 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:55.419732 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/570bf3c6-39b8-4d1a-8957-3fd162f49f96-service-ca\") pod \"570bf3c6-39b8-4d1a-8957-3fd162f49f96\" (UID: \"570bf3c6-39b8-4d1a-8957-3fd162f49f96\") " Sep 10 23:39:55.419808 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:55.419774 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/570bf3c6-39b8-4d1a-8957-3fd162f49f96-trusted-ca-bundle\") pod \"570bf3c6-39b8-4d1a-8957-3fd162f49f96\" (UID: \"570bf3c6-39b8-4d1a-8957-3fd162f49f96\") " Sep 10 23:39:55.420019 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:55.419811 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/570bf3c6-39b8-4d1a-8957-3fd162f49f96-console-serving-cert\") pod \"570bf3c6-39b8-4d1a-8957-3fd162f49f96\" (UID: \"570bf3c6-39b8-4d1a-8957-3fd162f49f96\") " Sep 10 23:39:55.420019 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:55.419914 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/570bf3c6-39b8-4d1a-8957-3fd162f49f96-console-config\") pod \"570bf3c6-39b8-4d1a-8957-3fd162f49f96\" (UID: \"570bf3c6-39b8-4d1a-8957-3fd162f49f96\") " Sep 10 23:39:55.420019 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:55.419949 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/570bf3c6-39b8-4d1a-8957-3fd162f49f96-oauth-serving-cert\") pod \"570bf3c6-39b8-4d1a-8957-3fd162f49f96\" (UID: \"570bf3c6-39b8-4d1a-8957-3fd162f49f96\") " Sep 10 23:39:55.420019 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:55.419994 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2t9w6\" (UniqueName: \"kubernetes.io/projected/570bf3c6-39b8-4d1a-8957-3fd162f49f96-kube-api-access-2t9w6\") pod \"570bf3c6-39b8-4d1a-8957-3fd162f49f96\" (UID: \"570bf3c6-39b8-4d1a-8957-3fd162f49f96\") " Sep 10 23:39:55.420224 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:55.420027 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/570bf3c6-39b8-4d1a-8957-3fd162f49f96-console-oauth-config\") pod \"570bf3c6-39b8-4d1a-8957-3fd162f49f96\" (UID: \"570bf3c6-39b8-4d1a-8957-3fd162f49f96\") " Sep 10 23:39:55.420295 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:55.420259 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/570bf3c6-39b8-4d1a-8957-3fd162f49f96-service-ca" (OuterVolumeSpecName: "service-ca") pod "570bf3c6-39b8-4d1a-8957-3fd162f49f96" (UID: "570bf3c6-39b8-4d1a-8957-3fd162f49f96"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 23:39:55.420344 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:55.420283 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/570bf3c6-39b8-4d1a-8957-3fd162f49f96-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "570bf3c6-39b8-4d1a-8957-3fd162f49f96" (UID: "570bf3c6-39b8-4d1a-8957-3fd162f49f96"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 23:39:55.420344 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:55.420310 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/570bf3c6-39b8-4d1a-8957-3fd162f49f96-console-config" (OuterVolumeSpecName: "console-config") pod "570bf3c6-39b8-4d1a-8957-3fd162f49f96" (UID: "570bf3c6-39b8-4d1a-8957-3fd162f49f96"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 23:39:55.420412 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:55.420366 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/570bf3c6-39b8-4d1a-8957-3fd162f49f96-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "570bf3c6-39b8-4d1a-8957-3fd162f49f96" (UID: "570bf3c6-39b8-4d1a-8957-3fd162f49f96"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 23:39:55.422075 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:55.422038 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/570bf3c6-39b8-4d1a-8957-3fd162f49f96-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "570bf3c6-39b8-4d1a-8957-3fd162f49f96" (UID: "570bf3c6-39b8-4d1a-8957-3fd162f49f96"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 23:39:55.422188 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:55.422082 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/570bf3c6-39b8-4d1a-8957-3fd162f49f96-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "570bf3c6-39b8-4d1a-8957-3fd162f49f96" (UID: "570bf3c6-39b8-4d1a-8957-3fd162f49f96"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 23:39:55.422243 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:55.422199 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/570bf3c6-39b8-4d1a-8957-3fd162f49f96-kube-api-access-2t9w6" (OuterVolumeSpecName: "kube-api-access-2t9w6") pod "570bf3c6-39b8-4d1a-8957-3fd162f49f96" (UID: "570bf3c6-39b8-4d1a-8957-3fd162f49f96"). InnerVolumeSpecName "kube-api-access-2t9w6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 23:39:55.521362 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:55.521336 2574 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/570bf3c6-39b8-4d1a-8957-3fd162f49f96-console-serving-cert\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:39:55.521362 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:55.521358 2574 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/570bf3c6-39b8-4d1a-8957-3fd162f49f96-console-config\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:39:55.521502 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:55.521367 2574 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/570bf3c6-39b8-4d1a-8957-3fd162f49f96-oauth-serving-cert\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:39:55.521502 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:55.521378 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2t9w6\" (UniqueName: \"kubernetes.io/projected/570bf3c6-39b8-4d1a-8957-3fd162f49f96-kube-api-access-2t9w6\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:39:55.521502 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:55.521387 2574 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/570bf3c6-39b8-4d1a-8957-3fd162f49f96-console-oauth-config\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:39:55.521502 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:55.521396 2574 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/570bf3c6-39b8-4d1a-8957-3fd162f49f96-service-ca\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:39:55.521502 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:55.521404 2574 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/570bf3c6-39b8-4d1a-8957-3fd162f49f96-trusted-ca-bundle\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:39:56.038040 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:56.038001 2574 generic.go:356] "Generic (PLEG): container finished" podID="570bf3c6-39b8-4d1a-8957-3fd162f49f96" containerID="56bcc6fdcf2630b4364fa78063357bdcc193d19f980878ae3b8379c62645ece5" exitCode=0 Sep 10 23:39:56.038447 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:56.038051 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c7bd69f7f-jgkpc" Sep 10 23:39:56.038447 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:56.038063 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c7bd69f7f-jgkpc" event={"ID":"570bf3c6-39b8-4d1a-8957-3fd162f49f96","Type":"ContainerDied","Data":"56bcc6fdcf2630b4364fa78063357bdcc193d19f980878ae3b8379c62645ece5"} Sep 10 23:39:56.038447 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:56.038111 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c7bd69f7f-jgkpc" event={"ID":"570bf3c6-39b8-4d1a-8957-3fd162f49f96","Type":"ContainerDied","Data":"782ca31ebce3c1d29744589e0c5374b9757a5b8e6284c8970560af4dc1ce17ab"} Sep 10 23:39:56.038447 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:56.038134 2574 scope.go:117] "RemoveContainer" containerID="56bcc6fdcf2630b4364fa78063357bdcc193d19f980878ae3b8379c62645ece5" Sep 10 23:39:56.047871 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:56.047839 2574 scope.go:117] "RemoveContainer" containerID="56bcc6fdcf2630b4364fa78063357bdcc193d19f980878ae3b8379c62645ece5" Sep 10 23:39:56.050026 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:39:56.049992 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56bcc6fdcf2630b4364fa78063357bdcc193d19f980878ae3b8379c62645ece5\": container with ID starting with 56bcc6fdcf2630b4364fa78063357bdcc193d19f980878ae3b8379c62645ece5 not found: ID does not exist" containerID="56bcc6fdcf2630b4364fa78063357bdcc193d19f980878ae3b8379c62645ece5" Sep 10 23:39:56.050182 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:56.050039 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56bcc6fdcf2630b4364fa78063357bdcc193d19f980878ae3b8379c62645ece5"} err="failed to get container status \"56bcc6fdcf2630b4364fa78063357bdcc193d19f980878ae3b8379c62645ece5\": rpc error: code = NotFound desc = could not find container \"56bcc6fdcf2630b4364fa78063357bdcc193d19f980878ae3b8379c62645ece5\": container with ID starting with 56bcc6fdcf2630b4364fa78063357bdcc193d19f980878ae3b8379c62645ece5 not found: ID does not exist" Sep 10 23:39:56.063339 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:56.063314 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-c7bd69f7f-jgkpc"] Sep 10 23:39:56.067243 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:56.067220 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-c7bd69f7f-jgkpc"] Sep 10 23:39:57.093423 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:39:57.093380 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="570bf3c6-39b8-4d1a-8957-3fd162f49f96" path="/var/lib/kubelet/pods/570bf3c6-39b8-4d1a-8957-3fd162f49f96/volumes" Sep 10 23:40:06.037564 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:06.037488 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7d68776c7f-xx5qf" podUID="bc902d48-8e0d-40dd-ba2a-3a43749c1a55" containerName="console" containerID="cri-o://b3ec4eb57b8d899aeafedcee23afa86fd552d681997a058ff58aab2f17131806" gracePeriod=15 Sep 10 23:40:06.285361 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:06.285335 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d68776c7f-xx5qf" Sep 10 23:40:06.320144 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:06.320074 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bc902d48-8e0d-40dd-ba2a-3a43749c1a55-service-ca\") pod \"bc902d48-8e0d-40dd-ba2a-3a43749c1a55\" (UID: \"bc902d48-8e0d-40dd-ba2a-3a43749c1a55\") " Sep 10 23:40:06.320144 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:06.320120 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bc902d48-8e0d-40dd-ba2a-3a43749c1a55-console-config\") pod \"bc902d48-8e0d-40dd-ba2a-3a43749c1a55\" (UID: \"bc902d48-8e0d-40dd-ba2a-3a43749c1a55\") " Sep 10 23:40:06.320357 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:06.320155 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc902d48-8e0d-40dd-ba2a-3a43749c1a55-trusted-ca-bundle\") pod \"bc902d48-8e0d-40dd-ba2a-3a43749c1a55\" (UID: \"bc902d48-8e0d-40dd-ba2a-3a43749c1a55\") " Sep 10 23:40:06.320357 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:06.320177 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bc902d48-8e0d-40dd-ba2a-3a43749c1a55-console-serving-cert\") pod \"bc902d48-8e0d-40dd-ba2a-3a43749c1a55\" (UID: \"bc902d48-8e0d-40dd-ba2a-3a43749c1a55\") " Sep 10 23:40:06.320357 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:06.320228 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bc902d48-8e0d-40dd-ba2a-3a43749c1a55-oauth-serving-cert\") pod \"bc902d48-8e0d-40dd-ba2a-3a43749c1a55\" (UID: \"bc902d48-8e0d-40dd-ba2a-3a43749c1a55\") " Sep 10 23:40:06.320357 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:06.320267 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bc902d48-8e0d-40dd-ba2a-3a43749c1a55-console-oauth-config\") pod \"bc902d48-8e0d-40dd-ba2a-3a43749c1a55\" (UID: \"bc902d48-8e0d-40dd-ba2a-3a43749c1a55\") " Sep 10 23:40:06.320357 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:06.320293 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m7xf6\" (UniqueName: \"kubernetes.io/projected/bc902d48-8e0d-40dd-ba2a-3a43749c1a55-kube-api-access-m7xf6\") pod \"bc902d48-8e0d-40dd-ba2a-3a43749c1a55\" (UID: \"bc902d48-8e0d-40dd-ba2a-3a43749c1a55\") " Sep 10 23:40:06.320607 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:06.320578 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc902d48-8e0d-40dd-ba2a-3a43749c1a55-console-config" (OuterVolumeSpecName: "console-config") pod "bc902d48-8e0d-40dd-ba2a-3a43749c1a55" (UID: "bc902d48-8e0d-40dd-ba2a-3a43749c1a55"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 23:40:06.320700 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:06.320511 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc902d48-8e0d-40dd-ba2a-3a43749c1a55-service-ca" (OuterVolumeSpecName: "service-ca") pod "bc902d48-8e0d-40dd-ba2a-3a43749c1a55" (UID: "bc902d48-8e0d-40dd-ba2a-3a43749c1a55"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 23:40:06.320790 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:06.320758 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc902d48-8e0d-40dd-ba2a-3a43749c1a55-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "bc902d48-8e0d-40dd-ba2a-3a43749c1a55" (UID: "bc902d48-8e0d-40dd-ba2a-3a43749c1a55"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 23:40:06.321023 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:06.320986 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc902d48-8e0d-40dd-ba2a-3a43749c1a55-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "bc902d48-8e0d-40dd-ba2a-3a43749c1a55" (UID: "bc902d48-8e0d-40dd-ba2a-3a43749c1a55"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 23:40:06.322424 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:06.322401 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc902d48-8e0d-40dd-ba2a-3a43749c1a55-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "bc902d48-8e0d-40dd-ba2a-3a43749c1a55" (UID: "bc902d48-8e0d-40dd-ba2a-3a43749c1a55"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 23:40:06.322720 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:06.322564 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc902d48-8e0d-40dd-ba2a-3a43749c1a55-kube-api-access-m7xf6" (OuterVolumeSpecName: "kube-api-access-m7xf6") pod "bc902d48-8e0d-40dd-ba2a-3a43749c1a55" (UID: "bc902d48-8e0d-40dd-ba2a-3a43749c1a55"). InnerVolumeSpecName "kube-api-access-m7xf6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 23:40:06.323684 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:06.322727 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc902d48-8e0d-40dd-ba2a-3a43749c1a55-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "bc902d48-8e0d-40dd-ba2a-3a43749c1a55" (UID: "bc902d48-8e0d-40dd-ba2a-3a43749c1a55"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 23:40:06.421313 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:06.421274 2574 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bc902d48-8e0d-40dd-ba2a-3a43749c1a55-oauth-serving-cert\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:40:06.421313 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:06.421308 2574 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bc902d48-8e0d-40dd-ba2a-3a43749c1a55-console-oauth-config\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:40:06.421313 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:06.421317 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7xf6\" (UniqueName: \"kubernetes.io/projected/bc902d48-8e0d-40dd-ba2a-3a43749c1a55-kube-api-access-m7xf6\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:40:06.421313 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:06.421327 2574 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bc902d48-8e0d-40dd-ba2a-3a43749c1a55-service-ca\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:40:06.421575 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:06.421337 2574 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bc902d48-8e0d-40dd-ba2a-3a43749c1a55-console-config\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:40:06.421575 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:06.421345 2574 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc902d48-8e0d-40dd-ba2a-3a43749c1a55-trusted-ca-bundle\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:40:06.421575 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:06.421354 2574 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bc902d48-8e0d-40dd-ba2a-3a43749c1a55-console-serving-cert\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:40:07.081952 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:07.081909 2574 generic.go:356] "Generic (PLEG): container finished" podID="bc902d48-8e0d-40dd-ba2a-3a43749c1a55" containerID="b3ec4eb57b8d899aeafedcee23afa86fd552d681997a058ff58aab2f17131806" exitCode=0 Sep 10 23:40:07.082357 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:07.081976 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d68776c7f-xx5qf" Sep 10 23:40:07.082357 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:07.081994 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d68776c7f-xx5qf" event={"ID":"bc902d48-8e0d-40dd-ba2a-3a43749c1a55","Type":"ContainerDied","Data":"b3ec4eb57b8d899aeafedcee23afa86fd552d681997a058ff58aab2f17131806"} Sep 10 23:40:07.082357 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:07.082079 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d68776c7f-xx5qf" event={"ID":"bc902d48-8e0d-40dd-ba2a-3a43749c1a55","Type":"ContainerDied","Data":"169d6d0e81be7a5c6370f103e887c1ebccb06171933e7f84e9e8050892f57a3a"} Sep 10 23:40:07.082357 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:07.082101 2574 scope.go:117] "RemoveContainer" containerID="b3ec4eb57b8d899aeafedcee23afa86fd552d681997a058ff58aab2f17131806" Sep 10 23:40:07.090780 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:07.090765 2574 scope.go:117] "RemoveContainer" containerID="b3ec4eb57b8d899aeafedcee23afa86fd552d681997a058ff58aab2f17131806" Sep 10 23:40:07.091046 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:40:07.091027 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3ec4eb57b8d899aeafedcee23afa86fd552d681997a058ff58aab2f17131806\": container with ID starting with b3ec4eb57b8d899aeafedcee23afa86fd552d681997a058ff58aab2f17131806 not found: ID does not exist" containerID="b3ec4eb57b8d899aeafedcee23afa86fd552d681997a058ff58aab2f17131806" Sep 10 23:40:07.091093 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:07.091053 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3ec4eb57b8d899aeafedcee23afa86fd552d681997a058ff58aab2f17131806"} err="failed to get container status \"b3ec4eb57b8d899aeafedcee23afa86fd552d681997a058ff58aab2f17131806\": rpc error: code = NotFound desc = could not find container \"b3ec4eb57b8d899aeafedcee23afa86fd552d681997a058ff58aab2f17131806\": container with ID starting with b3ec4eb57b8d899aeafedcee23afa86fd552d681997a058ff58aab2f17131806 not found: ID does not exist" Sep 10 23:40:07.101629 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:07.101607 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7d68776c7f-xx5qf"] Sep 10 23:40:07.105002 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:07.104978 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7d68776c7f-xx5qf"] Sep 10 23:40:09.093010 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:09.092976 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc902d48-8e0d-40dd-ba2a-3a43749c1a55" path="/var/lib/kubelet/pods/bc902d48-8e0d-40dd-ba2a-3a43749c1a55/volumes" Sep 10 23:40:28.077701 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:28.077664 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406tbsx"] Sep 10 23:40:28.078120 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:28.077992 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="570bf3c6-39b8-4d1a-8957-3fd162f49f96" containerName="console" Sep 10 23:40:28.078120 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:28.078004 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="570bf3c6-39b8-4d1a-8957-3fd162f49f96" containerName="console" Sep 10 23:40:28.078120 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:28.078034 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc902d48-8e0d-40dd-ba2a-3a43749c1a55" containerName="console" Sep 10 23:40:28.078120 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:28.078039 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc902d48-8e0d-40dd-ba2a-3a43749c1a55" containerName="console" Sep 10 23:40:28.078120 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:28.078098 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc902d48-8e0d-40dd-ba2a-3a43749c1a55" containerName="console" Sep 10 23:40:28.078120 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:28.078109 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="570bf3c6-39b8-4d1a-8957-3fd162f49f96" containerName="console" Sep 10 23:40:28.081285 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:28.081269 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406tbsx" Sep 10 23:40:28.083417 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:28.083397 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 10 23:40:28.083537 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:28.083399 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-hdmcv\"" Sep 10 23:40:28.083537 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:28.083399 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 10 23:40:28.089842 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:28.089822 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406tbsx"] Sep 10 23:40:28.198237 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:28.198207 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9lkm\" (UniqueName: \"kubernetes.io/projected/40aac6a6-c96f-4f6a-acb6-92156e05df83-kube-api-access-t9lkm\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406tbsx\" (UID: \"40aac6a6-c96f-4f6a-acb6-92156e05df83\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406tbsx" Sep 10 23:40:28.198413 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:28.198340 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/40aac6a6-c96f-4f6a-acb6-92156e05df83-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406tbsx\" (UID: \"40aac6a6-c96f-4f6a-acb6-92156e05df83\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406tbsx" Sep 10 23:40:28.198535 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:28.198485 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/40aac6a6-c96f-4f6a-acb6-92156e05df83-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406tbsx\" (UID: \"40aac6a6-c96f-4f6a-acb6-92156e05df83\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406tbsx" Sep 10 23:40:28.298814 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:28.298786 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/40aac6a6-c96f-4f6a-acb6-92156e05df83-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406tbsx\" (UID: \"40aac6a6-c96f-4f6a-acb6-92156e05df83\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406tbsx" Sep 10 23:40:28.298910 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:28.298836 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/40aac6a6-c96f-4f6a-acb6-92156e05df83-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406tbsx\" (UID: \"40aac6a6-c96f-4f6a-acb6-92156e05df83\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406tbsx" Sep 10 23:40:28.298910 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:28.298859 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9lkm\" (UniqueName: \"kubernetes.io/projected/40aac6a6-c96f-4f6a-acb6-92156e05df83-kube-api-access-t9lkm\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406tbsx\" (UID: \"40aac6a6-c96f-4f6a-acb6-92156e05df83\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406tbsx" Sep 10 23:40:28.299232 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:28.299207 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/40aac6a6-c96f-4f6a-acb6-92156e05df83-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406tbsx\" (UID: \"40aac6a6-c96f-4f6a-acb6-92156e05df83\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406tbsx" Sep 10 23:40:28.299274 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:28.299241 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/40aac6a6-c96f-4f6a-acb6-92156e05df83-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406tbsx\" (UID: \"40aac6a6-c96f-4f6a-acb6-92156e05df83\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406tbsx" Sep 10 23:40:28.306075 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:28.306056 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9lkm\" (UniqueName: \"kubernetes.io/projected/40aac6a6-c96f-4f6a-acb6-92156e05df83-kube-api-access-t9lkm\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406tbsx\" (UID: \"40aac6a6-c96f-4f6a-acb6-92156e05df83\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406tbsx" Sep 10 23:40:28.391234 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:28.391178 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406tbsx" Sep 10 23:40:28.659758 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:28.659679 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406tbsx"] Sep 10 23:40:28.662674 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:40:28.662630 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40aac6a6_c96f_4f6a_acb6_92156e05df83.slice/crio-ab9d6dffc8a099b3aeffb23be44b90aa8bed1f84df8dd426d4c216c45758cf48 WatchSource:0}: Error finding container ab9d6dffc8a099b3aeffb23be44b90aa8bed1f84df8dd426d4c216c45758cf48: Status 404 returned error can't find the container with id ab9d6dffc8a099b3aeffb23be44b90aa8bed1f84df8dd426d4c216c45758cf48 Sep 10 23:40:29.165312 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:29.165280 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406tbsx" event={"ID":"40aac6a6-c96f-4f6a-acb6-92156e05df83","Type":"ContainerStarted","Data":"ab9d6dffc8a099b3aeffb23be44b90aa8bed1f84df8dd426d4c216c45758cf48"} Sep 10 23:40:35.193725 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:35.193624 2574 generic.go:356] "Generic (PLEG): container finished" podID="40aac6a6-c96f-4f6a-acb6-92156e05df83" containerID="d7ee2c77fbb24884e26e97a6760aba6a89a0b70c647bd7853ad769e870acabc9" exitCode=0 Sep 10 23:40:35.193725 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:35.193679 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406tbsx" event={"ID":"40aac6a6-c96f-4f6a-acb6-92156e05df83","Type":"ContainerDied","Data":"d7ee2c77fbb24884e26e97a6760aba6a89a0b70c647bd7853ad769e870acabc9"} Sep 10 23:40:38.207867 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:38.207826 2574 generic.go:356] "Generic (PLEG): container finished" podID="40aac6a6-c96f-4f6a-acb6-92156e05df83" containerID="3b0f0f1df0924f35a64c57c81b9540c25518ee9c962b53b458dd3a9cbc60d76c" exitCode=0 Sep 10 23:40:38.208267 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:38.207916 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406tbsx" event={"ID":"40aac6a6-c96f-4f6a-acb6-92156e05df83","Type":"ContainerDied","Data":"3b0f0f1df0924f35a64c57c81b9540c25518ee9c962b53b458dd3a9cbc60d76c"} Sep 10 23:40:45.242123 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:45.242075 2574 generic.go:356] "Generic (PLEG): container finished" podID="40aac6a6-c96f-4f6a-acb6-92156e05df83" containerID="0f06416c39be9d93e53d222217017d2203c7027f8ce149623b4c26e83e596bcb" exitCode=0 Sep 10 23:40:45.242537 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:45.242155 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406tbsx" event={"ID":"40aac6a6-c96f-4f6a-acb6-92156e05df83","Type":"ContainerDied","Data":"0f06416c39be9d93e53d222217017d2203c7027f8ce149623b4c26e83e596bcb"} Sep 10 23:40:46.377199 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:46.377173 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406tbsx" Sep 10 23:40:46.562392 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:46.562300 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/40aac6a6-c96f-4f6a-acb6-92156e05df83-util\") pod \"40aac6a6-c96f-4f6a-acb6-92156e05df83\" (UID: \"40aac6a6-c96f-4f6a-acb6-92156e05df83\") " Sep 10 23:40:46.562392 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:46.562350 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/40aac6a6-c96f-4f6a-acb6-92156e05df83-bundle\") pod \"40aac6a6-c96f-4f6a-acb6-92156e05df83\" (UID: \"40aac6a6-c96f-4f6a-acb6-92156e05df83\") " Sep 10 23:40:46.562392 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:46.562378 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t9lkm\" (UniqueName: \"kubernetes.io/projected/40aac6a6-c96f-4f6a-acb6-92156e05df83-kube-api-access-t9lkm\") pod \"40aac6a6-c96f-4f6a-acb6-92156e05df83\" (UID: \"40aac6a6-c96f-4f6a-acb6-92156e05df83\") " Sep 10 23:40:46.563111 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:46.563083 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40aac6a6-c96f-4f6a-acb6-92156e05df83-bundle" (OuterVolumeSpecName: "bundle") pod "40aac6a6-c96f-4f6a-acb6-92156e05df83" (UID: "40aac6a6-c96f-4f6a-acb6-92156e05df83"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:40:46.564610 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:46.564588 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40aac6a6-c96f-4f6a-acb6-92156e05df83-kube-api-access-t9lkm" (OuterVolumeSpecName: "kube-api-access-t9lkm") pod "40aac6a6-c96f-4f6a-acb6-92156e05df83" (UID: "40aac6a6-c96f-4f6a-acb6-92156e05df83"). InnerVolumeSpecName "kube-api-access-t9lkm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 23:40:46.566706 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:46.566685 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40aac6a6-c96f-4f6a-acb6-92156e05df83-util" (OuterVolumeSpecName: "util") pod "40aac6a6-c96f-4f6a-acb6-92156e05df83" (UID: "40aac6a6-c96f-4f6a-acb6-92156e05df83"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:40:46.663169 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:46.663123 2574 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/40aac6a6-c96f-4f6a-acb6-92156e05df83-util\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:40:46.663169 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:46.663165 2574 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/40aac6a6-c96f-4f6a-acb6-92156e05df83-bundle\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:40:46.663169 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:46.663176 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9lkm\" (UniqueName: \"kubernetes.io/projected/40aac6a6-c96f-4f6a-acb6-92156e05df83-kube-api-access-t9lkm\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:40:47.251939 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:47.251902 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406tbsx" event={"ID":"40aac6a6-c96f-4f6a-acb6-92156e05df83","Type":"ContainerDied","Data":"ab9d6dffc8a099b3aeffb23be44b90aa8bed1f84df8dd426d4c216c45758cf48"} Sep 10 23:40:47.252142 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:47.251954 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ab9d6dffc8a099b3aeffb23be44b90aa8bed1f84df8dd426d4c216c45758cf48" Sep 10 23:40:47.252142 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:47.251953 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406tbsx" Sep 10 23:40:51.085943 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:51.085907 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bh8q5"] Sep 10 23:40:51.086331 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:51.086233 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40aac6a6-c96f-4f6a-acb6-92156e05df83" containerName="pull" Sep 10 23:40:51.086331 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:51.086244 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="40aac6a6-c96f-4f6a-acb6-92156e05df83" containerName="pull" Sep 10 23:40:51.086331 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:51.086259 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40aac6a6-c96f-4f6a-acb6-92156e05df83" containerName="util" Sep 10 23:40:51.086331 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:51.086265 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="40aac6a6-c96f-4f6a-acb6-92156e05df83" containerName="util" Sep 10 23:40:51.086331 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:51.086273 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40aac6a6-c96f-4f6a-acb6-92156e05df83" containerName="extract" Sep 10 23:40:51.086331 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:51.086279 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="40aac6a6-c96f-4f6a-acb6-92156e05df83" containerName="extract" Sep 10 23:40:51.086331 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:51.086325 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="40aac6a6-c96f-4f6a-acb6-92156e05df83" containerName="extract" Sep 10 23:40:51.090053 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:51.090019 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bh8q5" Sep 10 23:40:51.092572 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:51.092546 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Sep 10 23:40:51.092726 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:51.092713 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Sep 10 23:40:51.093020 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:51.092990 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-5thqv\"" Sep 10 23:40:51.102631 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:51.102610 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bh8q5"] Sep 10 23:40:51.201234 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:51.201185 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7bbc529f-0c71-4293-92bb-baf06166a7b7-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-bh8q5\" (UID: \"7bbc529f-0c71-4293-92bb-baf06166a7b7\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bh8q5" Sep 10 23:40:51.201234 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:51.201240 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j79mk\" (UniqueName: \"kubernetes.io/projected/7bbc529f-0c71-4293-92bb-baf06166a7b7-kube-api-access-j79mk\") pod \"cert-manager-operator-controller-manager-55c444bf86-bh8q5\" (UID: \"7bbc529f-0c71-4293-92bb-baf06166a7b7\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bh8q5" Sep 10 23:40:51.301891 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:51.301858 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j79mk\" (UniqueName: \"kubernetes.io/projected/7bbc529f-0c71-4293-92bb-baf06166a7b7-kube-api-access-j79mk\") pod \"cert-manager-operator-controller-manager-55c444bf86-bh8q5\" (UID: \"7bbc529f-0c71-4293-92bb-baf06166a7b7\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bh8q5" Sep 10 23:40:51.302071 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:51.301956 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7bbc529f-0c71-4293-92bb-baf06166a7b7-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-bh8q5\" (UID: \"7bbc529f-0c71-4293-92bb-baf06166a7b7\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bh8q5" Sep 10 23:40:51.302293 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:51.302274 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7bbc529f-0c71-4293-92bb-baf06166a7b7-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-bh8q5\" (UID: \"7bbc529f-0c71-4293-92bb-baf06166a7b7\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bh8q5" Sep 10 23:40:51.314655 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:51.314624 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j79mk\" (UniqueName: \"kubernetes.io/projected/7bbc529f-0c71-4293-92bb-baf06166a7b7-kube-api-access-j79mk\") pod \"cert-manager-operator-controller-manager-55c444bf86-bh8q5\" (UID: \"7bbc529f-0c71-4293-92bb-baf06166a7b7\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bh8q5" Sep 10 23:40:51.399551 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:51.399484 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bh8q5" Sep 10 23:40:51.673974 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:51.673945 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bh8q5"] Sep 10 23:40:51.676909 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:40:51.676879 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bbc529f_0c71_4293_92bb_baf06166a7b7.slice/crio-6aa39aab2c3f7409df1d6c5c189bedd9994ef36d3e586037cb35dc5a72f9f9a3 WatchSource:0}: Error finding container 6aa39aab2c3f7409df1d6c5c189bedd9994ef36d3e586037cb35dc5a72f9f9a3: Status 404 returned error can't find the container with id 6aa39aab2c3f7409df1d6c5c189bedd9994ef36d3e586037cb35dc5a72f9f9a3 Sep 10 23:40:52.273754 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:52.273713 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bh8q5" event={"ID":"7bbc529f-0c71-4293-92bb-baf06166a7b7","Type":"ContainerStarted","Data":"6aa39aab2c3f7409df1d6c5c189bedd9994ef36d3e586037cb35dc5a72f9f9a3"} Sep 10 23:40:54.283897 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:54.283855 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bh8q5" event={"ID":"7bbc529f-0c71-4293-92bb-baf06166a7b7","Type":"ContainerStarted","Data":"a73b9d88dcf98e0121356dd5709e7e53ad1abf9595d3a0c6b6b4140a3ab356a0"} Sep 10 23:40:54.304014 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:54.303907 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-bh8q5" podStartSLOduration=1.648084731 podStartE2EDuration="3.303887308s" totalImagesPullingTime="1.655802577s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:40:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:40:54.302887742 +0000 UTC m=+245.690490512" watchObservedRunningTime="2026-09-10 23:40:54.303887308 +0000 UTC m=+245.691490084" Sep 10 23:40:54.893226 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:54.893185 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnw4mq"] Sep 10 23:40:54.895921 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:54.895892 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnw4mq" Sep 10 23:40:54.898789 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:54.898764 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-hdmcv\"" Sep 10 23:40:54.898939 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:54.898795 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 10 23:40:54.898939 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:54.898798 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 10 23:40:54.913682 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:54.913635 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnw4mq"] Sep 10 23:40:54.927734 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:54.927697 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwkrr\" (UniqueName: \"kubernetes.io/projected/fab5af29-5bd5-4189-bd79-9f3fc1eafa71-kube-api-access-qwkrr\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnw4mq\" (UID: \"fab5af29-5bd5-4189-bd79-9f3fc1eafa71\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnw4mq" Sep 10 23:40:54.927856 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:54.927788 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fab5af29-5bd5-4189-bd79-9f3fc1eafa71-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnw4mq\" (UID: \"fab5af29-5bd5-4189-bd79-9f3fc1eafa71\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnw4mq" Sep 10 23:40:54.927856 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:54.927819 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fab5af29-5bd5-4189-bd79-9f3fc1eafa71-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnw4mq\" (UID: \"fab5af29-5bd5-4189-bd79-9f3fc1eafa71\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnw4mq" Sep 10 23:40:55.028526 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:55.028494 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwkrr\" (UniqueName: \"kubernetes.io/projected/fab5af29-5bd5-4189-bd79-9f3fc1eafa71-kube-api-access-qwkrr\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnw4mq\" (UID: \"fab5af29-5bd5-4189-bd79-9f3fc1eafa71\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnw4mq" Sep 10 23:40:55.028822 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:55.028546 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fab5af29-5bd5-4189-bd79-9f3fc1eafa71-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnw4mq\" (UID: \"fab5af29-5bd5-4189-bd79-9f3fc1eafa71\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnw4mq" Sep 10 23:40:55.028822 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:55.028575 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fab5af29-5bd5-4189-bd79-9f3fc1eafa71-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnw4mq\" (UID: \"fab5af29-5bd5-4189-bd79-9f3fc1eafa71\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnw4mq" Sep 10 23:40:55.028980 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:55.028961 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fab5af29-5bd5-4189-bd79-9f3fc1eafa71-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnw4mq\" (UID: \"fab5af29-5bd5-4189-bd79-9f3fc1eafa71\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnw4mq" Sep 10 23:40:55.029044 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:55.029022 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fab5af29-5bd5-4189-bd79-9f3fc1eafa71-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnw4mq\" (UID: \"fab5af29-5bd5-4189-bd79-9f3fc1eafa71\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnw4mq" Sep 10 23:40:55.039957 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:55.039922 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwkrr\" (UniqueName: \"kubernetes.io/projected/fab5af29-5bd5-4189-bd79-9f3fc1eafa71-kube-api-access-qwkrr\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnw4mq\" (UID: \"fab5af29-5bd5-4189-bd79-9f3fc1eafa71\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnw4mq" Sep 10 23:40:55.149587 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:55.149497 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-qk7d2"] Sep 10 23:40:55.151734 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:55.151711 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-qk7d2" Sep 10 23:40:55.153949 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:55.153928 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Sep 10 23:40:55.154082 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:55.154040 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-h2dw2\"" Sep 10 23:40:55.154255 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:55.154240 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Sep 10 23:40:55.163565 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:55.163537 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-qk7d2"] Sep 10 23:40:55.204954 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:55.204921 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnw4mq" Sep 10 23:40:55.229607 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:55.229573 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsmwt\" (UniqueName: \"kubernetes.io/projected/b1f86993-c17b-4269-80fe-9ae2b386badb-kube-api-access-vsmwt\") pod \"cert-manager-webhook-6486ffd997-qk7d2\" (UID: \"b1f86993-c17b-4269-80fe-9ae2b386badb\") " pod="cert-manager/cert-manager-webhook-6486ffd997-qk7d2" Sep 10 23:40:55.229745 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:55.229686 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b1f86993-c17b-4269-80fe-9ae2b386badb-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-qk7d2\" (UID: \"b1f86993-c17b-4269-80fe-9ae2b386badb\") " pod="cert-manager/cert-manager-webhook-6486ffd997-qk7d2" Sep 10 23:40:55.330184 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:55.330142 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b1f86993-c17b-4269-80fe-9ae2b386badb-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-qk7d2\" (UID: \"b1f86993-c17b-4269-80fe-9ae2b386badb\") " pod="cert-manager/cert-manager-webhook-6486ffd997-qk7d2" Sep 10 23:40:55.330545 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:55.330212 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsmwt\" (UniqueName: \"kubernetes.io/projected/b1f86993-c17b-4269-80fe-9ae2b386badb-kube-api-access-vsmwt\") pod \"cert-manager-webhook-6486ffd997-qk7d2\" (UID: \"b1f86993-c17b-4269-80fe-9ae2b386badb\") " pod="cert-manager/cert-manager-webhook-6486ffd997-qk7d2" Sep 10 23:40:55.340737 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:55.340708 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b1f86993-c17b-4269-80fe-9ae2b386badb-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-qk7d2\" (UID: \"b1f86993-c17b-4269-80fe-9ae2b386badb\") " pod="cert-manager/cert-manager-webhook-6486ffd997-qk7d2" Sep 10 23:40:55.340864 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:55.340785 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsmwt\" (UniqueName: \"kubernetes.io/projected/b1f86993-c17b-4269-80fe-9ae2b386badb-kube-api-access-vsmwt\") pod \"cert-manager-webhook-6486ffd997-qk7d2\" (UID: \"b1f86993-c17b-4269-80fe-9ae2b386badb\") " pod="cert-manager/cert-manager-webhook-6486ffd997-qk7d2" Sep 10 23:40:55.473081 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:55.473033 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-qk7d2" Sep 10 23:40:55.475576 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:55.475551 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnw4mq"] Sep 10 23:40:55.478291 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:40:55.478267 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfab5af29_5bd5_4189_bd79_9f3fc1eafa71.slice/crio-4b0ed39c02bca9c49aff98dfa39b72c87b1659f589a9aa16537792084ffc64c8 WatchSource:0}: Error finding container 4b0ed39c02bca9c49aff98dfa39b72c87b1659f589a9aa16537792084ffc64c8: Status 404 returned error can't find the container with id 4b0ed39c02bca9c49aff98dfa39b72c87b1659f589a9aa16537792084ffc64c8 Sep 10 23:40:55.746362 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:55.746244 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-qk7d2"] Sep 10 23:40:55.748986 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:40:55.748965 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb1f86993_c17b_4269_80fe_9ae2b386badb.slice/crio-ae79545a407ab5038e45a9847927a7b6fa5b71910afe2307a9d43936989c5651 WatchSource:0}: Error finding container ae79545a407ab5038e45a9847927a7b6fa5b71910afe2307a9d43936989c5651: Status 404 returned error can't find the container with id ae79545a407ab5038e45a9847927a7b6fa5b71910afe2307a9d43936989c5651 Sep 10 23:40:56.294002 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:56.293967 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-qk7d2" event={"ID":"b1f86993-c17b-4269-80fe-9ae2b386badb","Type":"ContainerStarted","Data":"ae79545a407ab5038e45a9847927a7b6fa5b71910afe2307a9d43936989c5651"} Sep 10 23:40:56.295440 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:56.295414 2574 generic.go:356] "Generic (PLEG): container finished" podID="fab5af29-5bd5-4189-bd79-9f3fc1eafa71" containerID="d50b81cfb2619aed84becf3942e82fc6541341808427f4e02a4697ff65f9a575" exitCode=0 Sep 10 23:40:56.295568 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:56.295460 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnw4mq" event={"ID":"fab5af29-5bd5-4189-bd79-9f3fc1eafa71","Type":"ContainerDied","Data":"d50b81cfb2619aed84becf3942e82fc6541341808427f4e02a4697ff65f9a575"} Sep 10 23:40:56.295568 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:56.295490 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnw4mq" event={"ID":"fab5af29-5bd5-4189-bd79-9f3fc1eafa71","Type":"ContainerStarted","Data":"4b0ed39c02bca9c49aff98dfa39b72c87b1659f589a9aa16537792084ffc64c8"} Sep 10 23:40:58.610035 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:58.609992 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-m5mj5"] Sep 10 23:40:58.612735 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:58.612711 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-m5mj5" Sep 10 23:40:58.614794 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:58.614761 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-74h4f\"" Sep 10 23:40:58.622720 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:58.622697 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-m5mj5"] Sep 10 23:40:58.651608 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:58.651569 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/210f6ba7-9801-45d9-9b8a-bcd20fc9b76f-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-m5mj5\" (UID: \"210f6ba7-9801-45d9-9b8a-bcd20fc9b76f\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-m5mj5" Sep 10 23:40:58.651815 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:58.651705 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zrrn\" (UniqueName: \"kubernetes.io/projected/210f6ba7-9801-45d9-9b8a-bcd20fc9b76f-kube-api-access-2zrrn\") pod \"cert-manager-cainjector-559cdd8c87-m5mj5\" (UID: \"210f6ba7-9801-45d9-9b8a-bcd20fc9b76f\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-m5mj5" Sep 10 23:40:58.752238 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:58.752172 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zrrn\" (UniqueName: \"kubernetes.io/projected/210f6ba7-9801-45d9-9b8a-bcd20fc9b76f-kube-api-access-2zrrn\") pod \"cert-manager-cainjector-559cdd8c87-m5mj5\" (UID: \"210f6ba7-9801-45d9-9b8a-bcd20fc9b76f\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-m5mj5" Sep 10 23:40:58.752415 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:58.752301 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/210f6ba7-9801-45d9-9b8a-bcd20fc9b76f-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-m5mj5\" (UID: \"210f6ba7-9801-45d9-9b8a-bcd20fc9b76f\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-m5mj5" Sep 10 23:40:58.769081 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:58.769049 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zrrn\" (UniqueName: \"kubernetes.io/projected/210f6ba7-9801-45d9-9b8a-bcd20fc9b76f-kube-api-access-2zrrn\") pod \"cert-manager-cainjector-559cdd8c87-m5mj5\" (UID: \"210f6ba7-9801-45d9-9b8a-bcd20fc9b76f\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-m5mj5" Sep 10 23:40:58.794408 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:58.794371 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/210f6ba7-9801-45d9-9b8a-bcd20fc9b76f-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-m5mj5\" (UID: \"210f6ba7-9801-45d9-9b8a-bcd20fc9b76f\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-m5mj5" Sep 10 23:40:58.924863 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:58.924782 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-m5mj5" Sep 10 23:40:59.313629 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:59.313599 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnw4mq" event={"ID":"fab5af29-5bd5-4189-bd79-9f3fc1eafa71","Type":"ContainerStarted","Data":"c7d786395a5e9272fe3026a044cc84e546675e0d3aec26afb4fd9ce9f8415df1"} Sep 10 23:40:59.315229 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:59.315204 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-qk7d2" event={"ID":"b1f86993-c17b-4269-80fe-9ae2b386badb","Type":"ContainerStarted","Data":"d5c1c220e604514cc6df3fe26c0149c7d3d49228558297d03be703841c2924c8"} Sep 10 23:40:59.315363 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:59.315330 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-qk7d2" Sep 10 23:40:59.343957 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:59.343909 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-qk7d2" podStartSLOduration=0.906205996 podStartE2EDuration="4.34389298s" totalImagesPullingTime="3.437686984s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:40:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:40:59.342836192 +0000 UTC m=+250.730438952" watchObservedRunningTime="2026-09-10 23:40:59.34389298 +0000 UTC m=+250.731495738" Sep 10 23:40:59.442910 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:40:59.442883 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-m5mj5"] Sep 10 23:40:59.444729 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:40:59.444704 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod210f6ba7_9801_45d9_9b8a_bcd20fc9b76f.slice/crio-9ff7c842fff1c653bafc83b0d29e6f6b18eaa143b04370b8cbab622f6dede344 WatchSource:0}: Error finding container 9ff7c842fff1c653bafc83b0d29e6f6b18eaa143b04370b8cbab622f6dede344: Status 404 returned error can't find the container with id 9ff7c842fff1c653bafc83b0d29e6f6b18eaa143b04370b8cbab622f6dede344 Sep 10 23:41:00.326433 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:00.326399 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-m5mj5" event={"ID":"210f6ba7-9801-45d9-9b8a-bcd20fc9b76f","Type":"ContainerStarted","Data":"c55a4cafed9eb4c7635e752b84fdb3f35d9d0402afb7de98c925a27ac83613c0"} Sep 10 23:41:00.326893 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:00.326444 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-m5mj5" event={"ID":"210f6ba7-9801-45d9-9b8a-bcd20fc9b76f","Type":"ContainerStarted","Data":"9ff7c842fff1c653bafc83b0d29e6f6b18eaa143b04370b8cbab622f6dede344"} Sep 10 23:41:00.328273 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:00.328246 2574 generic.go:356] "Generic (PLEG): container finished" podID="fab5af29-5bd5-4189-bd79-9f3fc1eafa71" containerID="c7d786395a5e9272fe3026a044cc84e546675e0d3aec26afb4fd9ce9f8415df1" exitCode=0 Sep 10 23:41:00.328358 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:00.328338 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnw4mq" event={"ID":"fab5af29-5bd5-4189-bd79-9f3fc1eafa71","Type":"ContainerDied","Data":"c7d786395a5e9272fe3026a044cc84e546675e0d3aec26afb4fd9ce9f8415df1"} Sep 10 23:41:00.344459 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:00.344407 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-m5mj5" podStartSLOduration=2.344386149 podStartE2EDuration="2.344386149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:40:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:41:00.343214579 +0000 UTC m=+251.730817340" watchObservedRunningTime="2026-09-10 23:41:00.344386149 +0000 UTC m=+251.731988912" Sep 10 23:41:01.334349 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:01.334314 2574 generic.go:356] "Generic (PLEG): container finished" podID="fab5af29-5bd5-4189-bd79-9f3fc1eafa71" containerID="ad93846de4e47dfeb2c490b8c2ffc48e74c16a26d923fff629b1655ba96552ba" exitCode=0 Sep 10 23:41:01.334729 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:01.334430 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnw4mq" event={"ID":"fab5af29-5bd5-4189-bd79-9f3fc1eafa71","Type":"ContainerDied","Data":"ad93846de4e47dfeb2c490b8c2ffc48e74c16a26d923fff629b1655ba96552ba"} Sep 10 23:41:02.471130 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:02.471104 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnw4mq" Sep 10 23:41:02.479616 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:02.479596 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fab5af29-5bd5-4189-bd79-9f3fc1eafa71-util\") pod \"fab5af29-5bd5-4189-bd79-9f3fc1eafa71\" (UID: \"fab5af29-5bd5-4189-bd79-9f3fc1eafa71\") " Sep 10 23:41:02.479733 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:02.479633 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fab5af29-5bd5-4189-bd79-9f3fc1eafa71-bundle\") pod \"fab5af29-5bd5-4189-bd79-9f3fc1eafa71\" (UID: \"fab5af29-5bd5-4189-bd79-9f3fc1eafa71\") " Sep 10 23:41:02.479733 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:02.479682 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qwkrr\" (UniqueName: \"kubernetes.io/projected/fab5af29-5bd5-4189-bd79-9f3fc1eafa71-kube-api-access-qwkrr\") pod \"fab5af29-5bd5-4189-bd79-9f3fc1eafa71\" (UID: \"fab5af29-5bd5-4189-bd79-9f3fc1eafa71\") " Sep 10 23:41:02.480019 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:02.479992 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fab5af29-5bd5-4189-bd79-9f3fc1eafa71-bundle" (OuterVolumeSpecName: "bundle") pod "fab5af29-5bd5-4189-bd79-9f3fc1eafa71" (UID: "fab5af29-5bd5-4189-bd79-9f3fc1eafa71"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:41:02.481653 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:02.481618 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fab5af29-5bd5-4189-bd79-9f3fc1eafa71-kube-api-access-qwkrr" (OuterVolumeSpecName: "kube-api-access-qwkrr") pod "fab5af29-5bd5-4189-bd79-9f3fc1eafa71" (UID: "fab5af29-5bd5-4189-bd79-9f3fc1eafa71"). InnerVolumeSpecName "kube-api-access-qwkrr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 23:41:02.483807 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:02.483788 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fab5af29-5bd5-4189-bd79-9f3fc1eafa71-util" (OuterVolumeSpecName: "util") pod "fab5af29-5bd5-4189-bd79-9f3fc1eafa71" (UID: "fab5af29-5bd5-4189-bd79-9f3fc1eafa71"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:41:02.580304 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:02.580271 2574 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fab5af29-5bd5-4189-bd79-9f3fc1eafa71-bundle\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:41:02.580304 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:02.580298 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwkrr\" (UniqueName: \"kubernetes.io/projected/fab5af29-5bd5-4189-bd79-9f3fc1eafa71-kube-api-access-qwkrr\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:41:02.580475 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:02.580312 2574 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fab5af29-5bd5-4189-bd79-9f3fc1eafa71-util\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:41:03.345949 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:03.345921 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnw4mq" Sep 10 23:41:03.346139 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:03.345915 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fnw4mq" event={"ID":"fab5af29-5bd5-4189-bd79-9f3fc1eafa71","Type":"ContainerDied","Data":"4b0ed39c02bca9c49aff98dfa39b72c87b1659f589a9aa16537792084ffc64c8"} Sep 10 23:41:03.346139 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:03.346045 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b0ed39c02bca9c49aff98dfa39b72c87b1659f589a9aa16537792084ffc64c8" Sep 10 23:41:05.331259 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:05.331223 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-qk7d2" Sep 10 23:41:05.830434 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:05.830398 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-rq9k9"] Sep 10 23:41:05.830737 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:05.830725 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fab5af29-5bd5-4189-bd79-9f3fc1eafa71" containerName="extract" Sep 10 23:41:05.830789 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:05.830739 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="fab5af29-5bd5-4189-bd79-9f3fc1eafa71" containerName="extract" Sep 10 23:41:05.830789 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:05.830748 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fab5af29-5bd5-4189-bd79-9f3fc1eafa71" containerName="pull" Sep 10 23:41:05.830789 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:05.830754 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="fab5af29-5bd5-4189-bd79-9f3fc1eafa71" containerName="pull" Sep 10 23:41:05.830789 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:05.830763 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fab5af29-5bd5-4189-bd79-9f3fc1eafa71" containerName="util" Sep 10 23:41:05.830789 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:05.830769 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="fab5af29-5bd5-4189-bd79-9f3fc1eafa71" containerName="util" Sep 10 23:41:05.830941 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:05.830835 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="fab5af29-5bd5-4189-bd79-9f3fc1eafa71" containerName="extract" Sep 10 23:41:05.834454 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:05.834435 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-rq9k9" Sep 10 23:41:05.836809 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:05.836785 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-wvdbw\"" Sep 10 23:41:05.847507 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:05.847480 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-rq9k9"] Sep 10 23:41:05.906166 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:05.906127 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4b1e8133-7bf5-48ff-a112-4588026dfc1c-bound-sa-token\") pod \"cert-manager-55b4d954d5-rq9k9\" (UID: \"4b1e8133-7bf5-48ff-a112-4588026dfc1c\") " pod="cert-manager/cert-manager-55b4d954d5-rq9k9" Sep 10 23:41:05.906345 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:05.906202 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrnbl\" (UniqueName: \"kubernetes.io/projected/4b1e8133-7bf5-48ff-a112-4588026dfc1c-kube-api-access-wrnbl\") pod \"cert-manager-55b4d954d5-rq9k9\" (UID: \"4b1e8133-7bf5-48ff-a112-4588026dfc1c\") " pod="cert-manager/cert-manager-55b4d954d5-rq9k9" Sep 10 23:41:06.007459 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:06.007424 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4b1e8133-7bf5-48ff-a112-4588026dfc1c-bound-sa-token\") pod \"cert-manager-55b4d954d5-rq9k9\" (UID: \"4b1e8133-7bf5-48ff-a112-4588026dfc1c\") " pod="cert-manager/cert-manager-55b4d954d5-rq9k9" Sep 10 23:41:06.007709 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:06.007495 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrnbl\" (UniqueName: \"kubernetes.io/projected/4b1e8133-7bf5-48ff-a112-4588026dfc1c-kube-api-access-wrnbl\") pod \"cert-manager-55b4d954d5-rq9k9\" (UID: \"4b1e8133-7bf5-48ff-a112-4588026dfc1c\") " pod="cert-manager/cert-manager-55b4d954d5-rq9k9" Sep 10 23:41:06.016101 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:06.016061 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4b1e8133-7bf5-48ff-a112-4588026dfc1c-bound-sa-token\") pod \"cert-manager-55b4d954d5-rq9k9\" (UID: \"4b1e8133-7bf5-48ff-a112-4588026dfc1c\") " pod="cert-manager/cert-manager-55b4d954d5-rq9k9" Sep 10 23:41:06.016192 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:06.016069 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrnbl\" (UniqueName: \"kubernetes.io/projected/4b1e8133-7bf5-48ff-a112-4588026dfc1c-kube-api-access-wrnbl\") pod \"cert-manager-55b4d954d5-rq9k9\" (UID: \"4b1e8133-7bf5-48ff-a112-4588026dfc1c\") " pod="cert-manager/cert-manager-55b4d954d5-rq9k9" Sep 10 23:41:06.143901 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:06.143811 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-rq9k9" Sep 10 23:41:06.414048 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:06.414024 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-rq9k9"] Sep 10 23:41:06.416450 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:41:06.416422 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4b1e8133_7bf5_48ff_a112_4588026dfc1c.slice/crio-f70da1970502051d535bb00267dbd324eabd3bd91e528261966bf8c859780db0 WatchSource:0}: Error finding container f70da1970502051d535bb00267dbd324eabd3bd91e528261966bf8c859780db0: Status 404 returned error can't find the container with id f70da1970502051d535bb00267dbd324eabd3bd91e528261966bf8c859780db0 Sep 10 23:41:07.365787 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:07.365754 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-rq9k9" event={"ID":"4b1e8133-7bf5-48ff-a112-4588026dfc1c","Type":"ContainerStarted","Data":"4ca46d27bb3673db899335b02054229393ac762ffc05557fe41ad7898f67e2a5"} Sep 10 23:41:07.365969 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:07.365799 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-rq9k9" event={"ID":"4b1e8133-7bf5-48ff-a112-4588026dfc1c","Type":"ContainerStarted","Data":"f70da1970502051d535bb00267dbd324eabd3bd91e528261966bf8c859780db0"} Sep 10 23:41:07.385111 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:07.385061 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-rq9k9" podStartSLOduration=2.385044887 podStartE2EDuration="2.385044887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:41:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:41:07.38443957 +0000 UTC m=+258.772042331" watchObservedRunningTime="2026-09-10 23:41:07.385044887 +0000 UTC m=+258.772647659" Sep 10 23:41:14.921378 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:14.921301 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhvnnf"] Sep 10 23:41:14.924813 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:14.924789 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhvnnf" Sep 10 23:41:14.926877 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:14.926852 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 10 23:41:14.927421 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:14.927404 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 10 23:41:14.927488 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:14.927411 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-hdmcv\"" Sep 10 23:41:14.932882 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:14.932861 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhvnnf"] Sep 10 23:41:14.983548 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:14.983518 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dfa0c8fd-bdef-4c28-b02a-9bff602e2d3f-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhvnnf\" (UID: \"dfa0c8fd-bdef-4c28-b02a-9bff602e2d3f\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhvnnf" Sep 10 23:41:14.983730 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:14.983560 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dfa0c8fd-bdef-4c28-b02a-9bff602e2d3f-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhvnnf\" (UID: \"dfa0c8fd-bdef-4c28-b02a-9bff602e2d3f\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhvnnf" Sep 10 23:41:14.983730 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:14.983613 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvmmf\" (UniqueName: \"kubernetes.io/projected/dfa0c8fd-bdef-4c28-b02a-9bff602e2d3f-kube-api-access-gvmmf\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhvnnf\" (UID: \"dfa0c8fd-bdef-4c28-b02a-9bff602e2d3f\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhvnnf" Sep 10 23:41:15.084596 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:15.084562 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dfa0c8fd-bdef-4c28-b02a-9bff602e2d3f-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhvnnf\" (UID: \"dfa0c8fd-bdef-4c28-b02a-9bff602e2d3f\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhvnnf" Sep 10 23:41:15.084780 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:15.084605 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dfa0c8fd-bdef-4c28-b02a-9bff602e2d3f-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhvnnf\" (UID: \"dfa0c8fd-bdef-4c28-b02a-9bff602e2d3f\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhvnnf" Sep 10 23:41:15.084780 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:15.084670 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvmmf\" (UniqueName: \"kubernetes.io/projected/dfa0c8fd-bdef-4c28-b02a-9bff602e2d3f-kube-api-access-gvmmf\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhvnnf\" (UID: \"dfa0c8fd-bdef-4c28-b02a-9bff602e2d3f\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhvnnf" Sep 10 23:41:15.085011 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:15.084992 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dfa0c8fd-bdef-4c28-b02a-9bff602e2d3f-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhvnnf\" (UID: \"dfa0c8fd-bdef-4c28-b02a-9bff602e2d3f\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhvnnf" Sep 10 23:41:15.085053 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:15.085010 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dfa0c8fd-bdef-4c28-b02a-9bff602e2d3f-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhvnnf\" (UID: \"dfa0c8fd-bdef-4c28-b02a-9bff602e2d3f\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhvnnf" Sep 10 23:41:15.092348 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:15.092315 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvmmf\" (UniqueName: \"kubernetes.io/projected/dfa0c8fd-bdef-4c28-b02a-9bff602e2d3f-kube-api-access-gvmmf\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhvnnf\" (UID: \"dfa0c8fd-bdef-4c28-b02a-9bff602e2d3f\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhvnnf" Sep 10 23:41:15.235443 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:15.235411 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhvnnf" Sep 10 23:41:15.507248 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:15.507034 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhvnnf"] Sep 10 23:41:15.509730 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:41:15.509699 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfa0c8fd_bdef_4c28_b02a_9bff602e2d3f.slice/crio-da311e685913b63248c8c98c5a3ef567fe96ff832bcb2eafeb9064d973d944d0 WatchSource:0}: Error finding container da311e685913b63248c8c98c5a3ef567fe96ff832bcb2eafeb9064d973d944d0: Status 404 returned error can't find the container with id da311e685913b63248c8c98c5a3ef567fe96ff832bcb2eafeb9064d973d944d0 Sep 10 23:41:16.405811 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:16.405732 2574 generic.go:356] "Generic (PLEG): container finished" podID="dfa0c8fd-bdef-4c28-b02a-9bff602e2d3f" containerID="13e47319043262c481f64cea432b5163d1e7ad171420bc04303fb888a7477aed" exitCode=0 Sep 10 23:41:16.406173 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:16.405822 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhvnnf" event={"ID":"dfa0c8fd-bdef-4c28-b02a-9bff602e2d3f","Type":"ContainerDied","Data":"13e47319043262c481f64cea432b5163d1e7ad171420bc04303fb888a7477aed"} Sep 10 23:41:16.406173 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:16.405893 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhvnnf" event={"ID":"dfa0c8fd-bdef-4c28-b02a-9bff602e2d3f","Type":"ContainerStarted","Data":"da311e685913b63248c8c98c5a3ef567fe96ff832bcb2eafeb9064d973d944d0"} Sep 10 23:41:18.417412 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:18.417378 2574 generic.go:356] "Generic (PLEG): container finished" podID="dfa0c8fd-bdef-4c28-b02a-9bff602e2d3f" containerID="a8a6d951c021b43ddb0d14b7dd18a3a87b6c13be990437973b4cbda319c72445" exitCode=0 Sep 10 23:41:18.417785 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:18.417442 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhvnnf" event={"ID":"dfa0c8fd-bdef-4c28-b02a-9bff602e2d3f","Type":"ContainerDied","Data":"a8a6d951c021b43ddb0d14b7dd18a3a87b6c13be990437973b4cbda319c72445"} Sep 10 23:41:19.424167 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:19.424134 2574 generic.go:356] "Generic (PLEG): container finished" podID="dfa0c8fd-bdef-4c28-b02a-9bff602e2d3f" containerID="88fbbeff30498d2c6538ed9621c43d462a6748f58be7098ae9b2389555021408" exitCode=0 Sep 10 23:41:19.424513 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:19.424214 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhvnnf" event={"ID":"dfa0c8fd-bdef-4c28-b02a-9bff602e2d3f","Type":"ContainerDied","Data":"88fbbeff30498d2c6538ed9621c43d462a6748f58be7098ae9b2389555021408"} Sep 10 23:41:20.553441 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:20.553412 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhvnnf" Sep 10 23:41:20.631314 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:20.631280 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dfa0c8fd-bdef-4c28-b02a-9bff602e2d3f-util\") pod \"dfa0c8fd-bdef-4c28-b02a-9bff602e2d3f\" (UID: \"dfa0c8fd-bdef-4c28-b02a-9bff602e2d3f\") " Sep 10 23:41:20.631482 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:20.631347 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gvmmf\" (UniqueName: \"kubernetes.io/projected/dfa0c8fd-bdef-4c28-b02a-9bff602e2d3f-kube-api-access-gvmmf\") pod \"dfa0c8fd-bdef-4c28-b02a-9bff602e2d3f\" (UID: \"dfa0c8fd-bdef-4c28-b02a-9bff602e2d3f\") " Sep 10 23:41:20.631482 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:20.631418 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dfa0c8fd-bdef-4c28-b02a-9bff602e2d3f-bundle\") pod \"dfa0c8fd-bdef-4c28-b02a-9bff602e2d3f\" (UID: \"dfa0c8fd-bdef-4c28-b02a-9bff602e2d3f\") " Sep 10 23:41:20.632273 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:20.632243 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfa0c8fd-bdef-4c28-b02a-9bff602e2d3f-bundle" (OuterVolumeSpecName: "bundle") pod "dfa0c8fd-bdef-4c28-b02a-9bff602e2d3f" (UID: "dfa0c8fd-bdef-4c28-b02a-9bff602e2d3f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:41:20.633386 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:20.633360 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfa0c8fd-bdef-4c28-b02a-9bff602e2d3f-kube-api-access-gvmmf" (OuterVolumeSpecName: "kube-api-access-gvmmf") pod "dfa0c8fd-bdef-4c28-b02a-9bff602e2d3f" (UID: "dfa0c8fd-bdef-4c28-b02a-9bff602e2d3f"). InnerVolumeSpecName "kube-api-access-gvmmf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 23:41:20.637438 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:20.637412 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfa0c8fd-bdef-4c28-b02a-9bff602e2d3f-util" (OuterVolumeSpecName: "util") pod "dfa0c8fd-bdef-4c28-b02a-9bff602e2d3f" (UID: "dfa0c8fd-bdef-4c28-b02a-9bff602e2d3f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:41:20.732907 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:20.732865 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvmmf\" (UniqueName: \"kubernetes.io/projected/dfa0c8fd-bdef-4c28-b02a-9bff602e2d3f-kube-api-access-gvmmf\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:41:20.732907 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:20.732906 2574 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dfa0c8fd-bdef-4c28-b02a-9bff602e2d3f-bundle\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:41:20.732907 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:20.732916 2574 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dfa0c8fd-bdef-4c28-b02a-9bff602e2d3f-util\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:41:21.435043 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:21.434972 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhvnnf" Sep 10 23:41:21.435182 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:21.434977 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhvnnf" event={"ID":"dfa0c8fd-bdef-4c28-b02a-9bff602e2d3f","Type":"ContainerDied","Data":"da311e685913b63248c8c98c5a3ef567fe96ff832bcb2eafeb9064d973d944d0"} Sep 10 23:41:21.435182 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:21.435109 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="da311e685913b63248c8c98c5a3ef567fe96ff832bcb2eafeb9064d973d944d0" Sep 10 23:41:31.339422 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:31.339389 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0c5rb8"] Sep 10 23:41:31.339818 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:31.339748 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dfa0c8fd-bdef-4c28-b02a-9bff602e2d3f" containerName="pull" Sep 10 23:41:31.339818 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:31.339763 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="dfa0c8fd-bdef-4c28-b02a-9bff602e2d3f" containerName="pull" Sep 10 23:41:31.339818 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:31.339772 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dfa0c8fd-bdef-4c28-b02a-9bff602e2d3f" containerName="extract" Sep 10 23:41:31.339818 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:31.339777 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="dfa0c8fd-bdef-4c28-b02a-9bff602e2d3f" containerName="extract" Sep 10 23:41:31.339818 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:31.339787 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dfa0c8fd-bdef-4c28-b02a-9bff602e2d3f" containerName="util" Sep 10 23:41:31.339818 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:31.339793 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="dfa0c8fd-bdef-4c28-b02a-9bff602e2d3f" containerName="util" Sep 10 23:41:31.339999 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:31.339851 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="dfa0c8fd-bdef-4c28-b02a-9bff602e2d3f" containerName="extract" Sep 10 23:41:31.387686 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:31.387655 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0c5rb8"] Sep 10 23:41:31.387830 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:31.387749 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0c5rb8" Sep 10 23:41:31.389993 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:31.389976 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 10 23:41:31.389993 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:31.389985 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 10 23:41:31.390124 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:31.390027 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-hdmcv\"" Sep 10 23:41:31.527469 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:31.527442 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/539900e1-0b7d-4ae5-822e-f2ba3407f8c3-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0c5rb8\" (UID: \"539900e1-0b7d-4ae5-822e-f2ba3407f8c3\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0c5rb8" Sep 10 23:41:31.527653 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:31.527481 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzn9z\" (UniqueName: \"kubernetes.io/projected/539900e1-0b7d-4ae5-822e-f2ba3407f8c3-kube-api-access-pzn9z\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0c5rb8\" (UID: \"539900e1-0b7d-4ae5-822e-f2ba3407f8c3\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0c5rb8" Sep 10 23:41:31.527653 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:31.527519 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/539900e1-0b7d-4ae5-822e-f2ba3407f8c3-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0c5rb8\" (UID: \"539900e1-0b7d-4ae5-822e-f2ba3407f8c3\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0c5rb8" Sep 10 23:41:31.628403 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:31.628321 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/539900e1-0b7d-4ae5-822e-f2ba3407f8c3-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0c5rb8\" (UID: \"539900e1-0b7d-4ae5-822e-f2ba3407f8c3\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0c5rb8" Sep 10 23:41:31.628403 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:31.628373 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzn9z\" (UniqueName: \"kubernetes.io/projected/539900e1-0b7d-4ae5-822e-f2ba3407f8c3-kube-api-access-pzn9z\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0c5rb8\" (UID: \"539900e1-0b7d-4ae5-822e-f2ba3407f8c3\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0c5rb8" Sep 10 23:41:31.628601 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:31.628425 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/539900e1-0b7d-4ae5-822e-f2ba3407f8c3-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0c5rb8\" (UID: \"539900e1-0b7d-4ae5-822e-f2ba3407f8c3\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0c5rb8" Sep 10 23:41:31.628835 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:31.628812 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/539900e1-0b7d-4ae5-822e-f2ba3407f8c3-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0c5rb8\" (UID: \"539900e1-0b7d-4ae5-822e-f2ba3407f8c3\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0c5rb8" Sep 10 23:41:31.628907 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:31.628822 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/539900e1-0b7d-4ae5-822e-f2ba3407f8c3-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0c5rb8\" (UID: \"539900e1-0b7d-4ae5-822e-f2ba3407f8c3\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0c5rb8" Sep 10 23:41:31.637159 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:31.637136 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzn9z\" (UniqueName: \"kubernetes.io/projected/539900e1-0b7d-4ae5-822e-f2ba3407f8c3-kube-api-access-pzn9z\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0c5rb8\" (UID: \"539900e1-0b7d-4ae5-822e-f2ba3407f8c3\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0c5rb8" Sep 10 23:41:31.696107 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:31.696071 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0c5rb8" Sep 10 23:41:32.001946 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:32.001865 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0c5rb8"] Sep 10 23:41:32.004725 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:41:32.004691 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod539900e1_0b7d_4ae5_822e_f2ba3407f8c3.slice/crio-e14a0e002831bb4ef1ff2290985e73e26516f7a1eb4ea68c329426ddb75a6626 WatchSource:0}: Error finding container e14a0e002831bb4ef1ff2290985e73e26516f7a1eb4ea68c329426ddb75a6626: Status 404 returned error can't find the container with id e14a0e002831bb4ef1ff2290985e73e26516f7a1eb4ea68c329426ddb75a6626 Sep 10 23:41:32.081820 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:32.081782 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-55d55f8464-vjvhz"] Sep 10 23:41:32.084097 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:32.084080 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-55d55f8464-vjvhz" Sep 10 23:41:32.086216 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:32.086176 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 10 23:41:32.086517 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:32.086489 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-b879w\"" Sep 10 23:41:32.086670 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:32.086604 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 10 23:41:32.086917 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:32.086894 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 10 23:41:32.086995 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:32.086955 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 10 23:41:32.109617 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:32.109583 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-55d55f8464-vjvhz"] Sep 10 23:41:32.234102 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:32.234055 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0ae1eec4-b5f9-457c-854c-316bda56d48a-manifests\") pod \"opendatahub-operator-controller-manager-55d55f8464-vjvhz\" (UID: \"0ae1eec4-b5f9-457c-854c-316bda56d48a\") " pod="opendatahub/opendatahub-operator-controller-manager-55d55f8464-vjvhz" Sep 10 23:41:32.234318 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:32.234180 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0ae1eec4-b5f9-457c-854c-316bda56d48a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-55d55f8464-vjvhz\" (UID: \"0ae1eec4-b5f9-457c-854c-316bda56d48a\") " pod="opendatahub/opendatahub-operator-controller-manager-55d55f8464-vjvhz" Sep 10 23:41:32.234318 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:32.234206 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0ae1eec4-b5f9-457c-854c-316bda56d48a-webhook-cert\") pod \"opendatahub-operator-controller-manager-55d55f8464-vjvhz\" (UID: \"0ae1eec4-b5f9-457c-854c-316bda56d48a\") " pod="opendatahub/opendatahub-operator-controller-manager-55d55f8464-vjvhz" Sep 10 23:41:32.234318 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:32.234240 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0ae1eec4-b5f9-457c-854c-316bda56d48a-tmp\") pod \"opendatahub-operator-controller-manager-55d55f8464-vjvhz\" (UID: \"0ae1eec4-b5f9-457c-854c-316bda56d48a\") " pod="opendatahub/opendatahub-operator-controller-manager-55d55f8464-vjvhz" Sep 10 23:41:32.234481 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:32.234353 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwgm9\" (UniqueName: \"kubernetes.io/projected/0ae1eec4-b5f9-457c-854c-316bda56d48a-kube-api-access-zwgm9\") pod \"opendatahub-operator-controller-manager-55d55f8464-vjvhz\" (UID: \"0ae1eec4-b5f9-457c-854c-316bda56d48a\") " pod="opendatahub/opendatahub-operator-controller-manager-55d55f8464-vjvhz" Sep 10 23:41:32.335174 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:32.335074 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0ae1eec4-b5f9-457c-854c-316bda56d48a-tmp\") pod \"opendatahub-operator-controller-manager-55d55f8464-vjvhz\" (UID: \"0ae1eec4-b5f9-457c-854c-316bda56d48a\") " pod="opendatahub/opendatahub-operator-controller-manager-55d55f8464-vjvhz" Sep 10 23:41:32.335174 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:32.335132 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwgm9\" (UniqueName: \"kubernetes.io/projected/0ae1eec4-b5f9-457c-854c-316bda56d48a-kube-api-access-zwgm9\") pod \"opendatahub-operator-controller-manager-55d55f8464-vjvhz\" (UID: \"0ae1eec4-b5f9-457c-854c-316bda56d48a\") " pod="opendatahub/opendatahub-operator-controller-manager-55d55f8464-vjvhz" Sep 10 23:41:32.335409 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:32.335184 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0ae1eec4-b5f9-457c-854c-316bda56d48a-manifests\") pod \"opendatahub-operator-controller-manager-55d55f8464-vjvhz\" (UID: \"0ae1eec4-b5f9-457c-854c-316bda56d48a\") " pod="opendatahub/opendatahub-operator-controller-manager-55d55f8464-vjvhz" Sep 10 23:41:32.335409 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:32.335201 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0ae1eec4-b5f9-457c-854c-316bda56d48a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-55d55f8464-vjvhz\" (UID: \"0ae1eec4-b5f9-457c-854c-316bda56d48a\") " pod="opendatahub/opendatahub-operator-controller-manager-55d55f8464-vjvhz" Sep 10 23:41:32.335409 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:32.335217 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0ae1eec4-b5f9-457c-854c-316bda56d48a-webhook-cert\") pod \"opendatahub-operator-controller-manager-55d55f8464-vjvhz\" (UID: \"0ae1eec4-b5f9-457c-854c-316bda56d48a\") " pod="opendatahub/opendatahub-operator-controller-manager-55d55f8464-vjvhz" Sep 10 23:41:32.335627 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:32.335604 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0ae1eec4-b5f9-457c-854c-316bda56d48a-manifests\") pod \"opendatahub-operator-controller-manager-55d55f8464-vjvhz\" (UID: \"0ae1eec4-b5f9-457c-854c-316bda56d48a\") " pod="opendatahub/opendatahub-operator-controller-manager-55d55f8464-vjvhz" Sep 10 23:41:32.337508 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:32.337476 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0ae1eec4-b5f9-457c-854c-316bda56d48a-tmp\") pod \"opendatahub-operator-controller-manager-55d55f8464-vjvhz\" (UID: \"0ae1eec4-b5f9-457c-854c-316bda56d48a\") " pod="opendatahub/opendatahub-operator-controller-manager-55d55f8464-vjvhz" Sep 10 23:41:32.337745 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:32.337724 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0ae1eec4-b5f9-457c-854c-316bda56d48a-webhook-cert\") pod \"opendatahub-operator-controller-manager-55d55f8464-vjvhz\" (UID: \"0ae1eec4-b5f9-457c-854c-316bda56d48a\") " pod="opendatahub/opendatahub-operator-controller-manager-55d55f8464-vjvhz" Sep 10 23:41:32.337793 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:32.337764 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0ae1eec4-b5f9-457c-854c-316bda56d48a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-55d55f8464-vjvhz\" (UID: \"0ae1eec4-b5f9-457c-854c-316bda56d48a\") " pod="opendatahub/opendatahub-operator-controller-manager-55d55f8464-vjvhz" Sep 10 23:41:32.344005 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:32.343981 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwgm9\" (UniqueName: \"kubernetes.io/projected/0ae1eec4-b5f9-457c-854c-316bda56d48a-kube-api-access-zwgm9\") pod \"opendatahub-operator-controller-manager-55d55f8464-vjvhz\" (UID: \"0ae1eec4-b5f9-457c-854c-316bda56d48a\") " pod="opendatahub/opendatahub-operator-controller-manager-55d55f8464-vjvhz" Sep 10 23:41:32.395708 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:32.395667 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-55d55f8464-vjvhz" Sep 10 23:41:32.486801 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:32.486769 2574 generic.go:356] "Generic (PLEG): container finished" podID="539900e1-0b7d-4ae5-822e-f2ba3407f8c3" containerID="aad16521e545b47220d129800350fdcc125f757be95983f5ba227c76e6fc6f45" exitCode=0 Sep 10 23:41:32.486950 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:32.486854 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0c5rb8" event={"ID":"539900e1-0b7d-4ae5-822e-f2ba3407f8c3","Type":"ContainerDied","Data":"aad16521e545b47220d129800350fdcc125f757be95983f5ba227c76e6fc6f45"} Sep 10 23:41:32.486950 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:32.486905 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0c5rb8" event={"ID":"539900e1-0b7d-4ae5-822e-f2ba3407f8c3","Type":"ContainerStarted","Data":"e14a0e002831bb4ef1ff2290985e73e26516f7a1eb4ea68c329426ddb75a6626"} Sep 10 23:41:32.677940 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:32.677915 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-55d55f8464-vjvhz"] Sep 10 23:41:32.679927 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:41:32.679901 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ae1eec4_b5f9_457c_854c_316bda56d48a.slice/crio-9380e8ecfb1674c41619c4b58e6b39ee26737d45afd9031e65c65e8da5a61c8b WatchSource:0}: Error finding container 9380e8ecfb1674c41619c4b58e6b39ee26737d45afd9031e65c65e8da5a61c8b: Status 404 returned error can't find the container with id 9380e8ecfb1674c41619c4b58e6b39ee26737d45afd9031e65c65e8da5a61c8b Sep 10 23:41:33.494740 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:33.494698 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-55d55f8464-vjvhz" event={"ID":"0ae1eec4-b5f9-457c-854c-316bda56d48a","Type":"ContainerStarted","Data":"9380e8ecfb1674c41619c4b58e6b39ee26737d45afd9031e65c65e8da5a61c8b"} Sep 10 23:41:33.497452 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:33.497423 2574 generic.go:356] "Generic (PLEG): container finished" podID="539900e1-0b7d-4ae5-822e-f2ba3407f8c3" containerID="fc44534c35c6524e72dfaccce28247b5cb0556e22a1f4cdcb60a041254b74e8a" exitCode=0 Sep 10 23:41:33.497586 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:33.497520 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0c5rb8" event={"ID":"539900e1-0b7d-4ae5-822e-f2ba3407f8c3","Type":"ContainerDied","Data":"fc44534c35c6524e72dfaccce28247b5cb0556e22a1f4cdcb60a041254b74e8a"} Sep 10 23:41:34.505830 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:34.505797 2574 generic.go:356] "Generic (PLEG): container finished" podID="539900e1-0b7d-4ae5-822e-f2ba3407f8c3" containerID="15546d4f2a49aa53a595ca5836319d9c8aff09a834afad76d4a581490d44dc91" exitCode=0 Sep 10 23:41:34.506222 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:34.505870 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0c5rb8" event={"ID":"539900e1-0b7d-4ae5-822e-f2ba3407f8c3","Type":"ContainerDied","Data":"15546d4f2a49aa53a595ca5836319d9c8aff09a834afad76d4a581490d44dc91"} Sep 10 23:41:35.677531 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:35.677505 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0c5rb8" Sep 10 23:41:35.775505 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:35.775398 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/539900e1-0b7d-4ae5-822e-f2ba3407f8c3-bundle\") pod \"539900e1-0b7d-4ae5-822e-f2ba3407f8c3\" (UID: \"539900e1-0b7d-4ae5-822e-f2ba3407f8c3\") " Sep 10 23:41:35.775505 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:35.775502 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pzn9z\" (UniqueName: \"kubernetes.io/projected/539900e1-0b7d-4ae5-822e-f2ba3407f8c3-kube-api-access-pzn9z\") pod \"539900e1-0b7d-4ae5-822e-f2ba3407f8c3\" (UID: \"539900e1-0b7d-4ae5-822e-f2ba3407f8c3\") " Sep 10 23:41:35.775780 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:35.775533 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/539900e1-0b7d-4ae5-822e-f2ba3407f8c3-util\") pod \"539900e1-0b7d-4ae5-822e-f2ba3407f8c3\" (UID: \"539900e1-0b7d-4ae5-822e-f2ba3407f8c3\") " Sep 10 23:41:35.776221 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:35.776191 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/539900e1-0b7d-4ae5-822e-f2ba3407f8c3-bundle" (OuterVolumeSpecName: "bundle") pod "539900e1-0b7d-4ae5-822e-f2ba3407f8c3" (UID: "539900e1-0b7d-4ae5-822e-f2ba3407f8c3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:41:35.777763 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:35.777716 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/539900e1-0b7d-4ae5-822e-f2ba3407f8c3-kube-api-access-pzn9z" (OuterVolumeSpecName: "kube-api-access-pzn9z") pod "539900e1-0b7d-4ae5-822e-f2ba3407f8c3" (UID: "539900e1-0b7d-4ae5-822e-f2ba3407f8c3"). InnerVolumeSpecName "kube-api-access-pzn9z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 23:41:35.781336 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:35.781312 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/539900e1-0b7d-4ae5-822e-f2ba3407f8c3-util" (OuterVolumeSpecName: "util") pod "539900e1-0b7d-4ae5-822e-f2ba3407f8c3" (UID: "539900e1-0b7d-4ae5-822e-f2ba3407f8c3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:41:35.876306 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:35.876269 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzn9z\" (UniqueName: \"kubernetes.io/projected/539900e1-0b7d-4ae5-822e-f2ba3407f8c3-kube-api-access-pzn9z\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:41:35.876306 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:35.876299 2574 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/539900e1-0b7d-4ae5-822e-f2ba3407f8c3-util\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:41:35.876306 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:35.876310 2574 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/539900e1-0b7d-4ae5-822e-f2ba3407f8c3-bundle\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:41:36.518354 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:36.518317 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-55d55f8464-vjvhz" event={"ID":"0ae1eec4-b5f9-457c-854c-316bda56d48a","Type":"ContainerStarted","Data":"e596270cb40defbc2d852de33fffb0ba11e0df1c9fd2da80080b0f0a4119689a"} Sep 10 23:41:36.520248 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:36.520224 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0c5rb8" Sep 10 23:41:36.520361 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:36.520244 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0c5rb8" event={"ID":"539900e1-0b7d-4ae5-822e-f2ba3407f8c3","Type":"ContainerDied","Data":"e14a0e002831bb4ef1ff2290985e73e26516f7a1eb4ea68c329426ddb75a6626"} Sep 10 23:41:36.520361 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:36.520283 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e14a0e002831bb4ef1ff2290985e73e26516f7a1eb4ea68c329426ddb75a6626" Sep 10 23:41:37.527509 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:37.527477 2574 generic.go:356] "Generic (PLEG): container finished" podID="0ae1eec4-b5f9-457c-854c-316bda56d48a" containerID="e596270cb40defbc2d852de33fffb0ba11e0df1c9fd2da80080b0f0a4119689a" exitCode=0 Sep 10 23:41:37.527945 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:37.527559 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-55d55f8464-vjvhz" event={"ID":"0ae1eec4-b5f9-457c-854c-316bda56d48a","Type":"ContainerDied","Data":"e596270cb40defbc2d852de33fffb0ba11e0df1c9fd2da80080b0f0a4119689a"} Sep 10 23:41:38.534520 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:38.534481 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-55d55f8464-vjvhz" event={"ID":"0ae1eec4-b5f9-457c-854c-316bda56d48a","Type":"ContainerStarted","Data":"0c7092ba8b8b960a65b227ed54e8301ad094b1df2257f3cc9cb49922549ca1ba"} Sep 10 23:41:38.534976 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:38.534562 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-55d55f8464-vjvhz" Sep 10 23:41:38.575371 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:38.575312 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-55d55f8464-vjvhz" podStartSLOduration=1.915251364 podStartE2EDuration="6.575295043s" totalImagesPullingTime="3.576406079s" totalInitContainerRuntime="1.0836376s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:41:32 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:41:38.573607466 +0000 UTC m=+289.961210247" watchObservedRunningTime="2026-09-10 23:41:38.575295043 +0000 UTC m=+289.962897803" Sep 10 23:41:49.541435 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:49.541407 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-55d55f8464-vjvhz" Sep 10 23:41:52.206255 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:52.206222 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-55f89b848f-6pl58"] Sep 10 23:41:52.206614 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:52.206567 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="539900e1-0b7d-4ae5-822e-f2ba3407f8c3" containerName="pull" Sep 10 23:41:52.206614 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:52.206580 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="539900e1-0b7d-4ae5-822e-f2ba3407f8c3" containerName="pull" Sep 10 23:41:52.206614 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:52.206588 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="539900e1-0b7d-4ae5-822e-f2ba3407f8c3" containerName="util" Sep 10 23:41:52.206614 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:52.206593 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="539900e1-0b7d-4ae5-822e-f2ba3407f8c3" containerName="util" Sep 10 23:41:52.206614 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:52.206604 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="539900e1-0b7d-4ae5-822e-f2ba3407f8c3" containerName="extract" Sep 10 23:41:52.206614 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:52.206610 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="539900e1-0b7d-4ae5-822e-f2ba3407f8c3" containerName="extract" Sep 10 23:41:52.206829 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:52.206705 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="539900e1-0b7d-4ae5-822e-f2ba3407f8c3" containerName="extract" Sep 10 23:41:52.209549 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:52.209530 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-55f89b848f-6pl58" Sep 10 23:41:52.215877 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:52.215853 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Sep 10 23:41:52.215996 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:52.215948 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 10 23:41:52.216767 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:52.216752 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 10 23:41:52.216859 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:52.216792 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Sep 10 23:41:52.216859 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:52.216795 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-npjkn\"" Sep 10 23:41:52.251341 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:52.251312 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-55f89b848f-6pl58"] Sep 10 23:41:52.314859 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:52.314824 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnf9m\" (UniqueName: \"kubernetes.io/projected/c9537e38-fd66-4e10-8f78-81987455a894-kube-api-access-xnf9m\") pod \"kube-auth-proxy-55f89b848f-6pl58\" (UID: \"c9537e38-fd66-4e10-8f78-81987455a894\") " pod="openshift-ingress/kube-auth-proxy-55f89b848f-6pl58" Sep 10 23:41:52.314993 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:52.314868 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c9537e38-fd66-4e10-8f78-81987455a894-tmp\") pod \"kube-auth-proxy-55f89b848f-6pl58\" (UID: \"c9537e38-fd66-4e10-8f78-81987455a894\") " pod="openshift-ingress/kube-auth-proxy-55f89b848f-6pl58" Sep 10 23:41:52.314993 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:52.314950 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9537e38-fd66-4e10-8f78-81987455a894-tls-certs\") pod \"kube-auth-proxy-55f89b848f-6pl58\" (UID: \"c9537e38-fd66-4e10-8f78-81987455a894\") " pod="openshift-ingress/kube-auth-proxy-55f89b848f-6pl58" Sep 10 23:41:52.416214 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:52.416183 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnf9m\" (UniqueName: \"kubernetes.io/projected/c9537e38-fd66-4e10-8f78-81987455a894-kube-api-access-xnf9m\") pod \"kube-auth-proxy-55f89b848f-6pl58\" (UID: \"c9537e38-fd66-4e10-8f78-81987455a894\") " pod="openshift-ingress/kube-auth-proxy-55f89b848f-6pl58" Sep 10 23:41:52.416374 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:52.416224 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c9537e38-fd66-4e10-8f78-81987455a894-tmp\") pod \"kube-auth-proxy-55f89b848f-6pl58\" (UID: \"c9537e38-fd66-4e10-8f78-81987455a894\") " pod="openshift-ingress/kube-auth-proxy-55f89b848f-6pl58" Sep 10 23:41:52.416374 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:52.416270 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9537e38-fd66-4e10-8f78-81987455a894-tls-certs\") pod \"kube-auth-proxy-55f89b848f-6pl58\" (UID: \"c9537e38-fd66-4e10-8f78-81987455a894\") " pod="openshift-ingress/kube-auth-proxy-55f89b848f-6pl58" Sep 10 23:41:52.418685 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:52.418658 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c9537e38-fd66-4e10-8f78-81987455a894-tmp\") pod \"kube-auth-proxy-55f89b848f-6pl58\" (UID: \"c9537e38-fd66-4e10-8f78-81987455a894\") " pod="openshift-ingress/kube-auth-proxy-55f89b848f-6pl58" Sep 10 23:41:52.418822 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:52.418805 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9537e38-fd66-4e10-8f78-81987455a894-tls-certs\") pod \"kube-auth-proxy-55f89b848f-6pl58\" (UID: \"c9537e38-fd66-4e10-8f78-81987455a894\") " pod="openshift-ingress/kube-auth-proxy-55f89b848f-6pl58" Sep 10 23:41:52.440797 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:52.440771 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnf9m\" (UniqueName: \"kubernetes.io/projected/c9537e38-fd66-4e10-8f78-81987455a894-kube-api-access-xnf9m\") pod \"kube-auth-proxy-55f89b848f-6pl58\" (UID: \"c9537e38-fd66-4e10-8f78-81987455a894\") " pod="openshift-ingress/kube-auth-proxy-55f89b848f-6pl58" Sep 10 23:41:52.519013 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:52.518935 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-55f89b848f-6pl58" Sep 10 23:41:52.808195 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:52.808174 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-55f89b848f-6pl58"] Sep 10 23:41:52.810305 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:41:52.810271 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9537e38_fd66_4e10_8f78_81987455a894.slice/crio-c662e8fb30ce36aec68928af8d473caf31cfe1faa74bbcf0649c043758d96765 WatchSource:0}: Error finding container c662e8fb30ce36aec68928af8d473caf31cfe1faa74bbcf0649c043758d96765: Status 404 returned error can't find the container with id c662e8fb30ce36aec68928af8d473caf31cfe1faa74bbcf0649c043758d96765 Sep 10 23:41:52.812399 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:52.812380 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 23:41:53.608301 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:53.608257 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-55f89b848f-6pl58" event={"ID":"c9537e38-fd66-4e10-8f78-81987455a894","Type":"ContainerStarted","Data":"c662e8fb30ce36aec68928af8d473caf31cfe1faa74bbcf0649c043758d96765"} Sep 10 23:41:54.729274 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:54.729242 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-z92tj"] Sep 10 23:41:54.733234 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:54.733214 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-z92tj" Sep 10 23:41:54.736855 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:54.736830 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 10 23:41:54.736994 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:54.736837 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Sep 10 23:41:54.737190 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:54.737168 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Sep 10 23:41:54.737334 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:54.737175 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Sep 10 23:41:54.737421 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:54.737178 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-hh6zz\"" Sep 10 23:41:54.758679 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:54.758630 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-z92tj"] Sep 10 23:41:54.836902 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:54.836862 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/22543c02-5514-477e-9bfc-81968fc76a65-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-z92tj\" (UID: \"22543c02-5514-477e-9bfc-81968fc76a65\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-z92tj" Sep 10 23:41:54.836902 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:54.836907 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/22543c02-5514-477e-9bfc-81968fc76a65-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-z92tj\" (UID: \"22543c02-5514-477e-9bfc-81968fc76a65\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-z92tj" Sep 10 23:41:54.837163 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:54.836970 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22543c02-5514-477e-9bfc-81968fc76a65-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-z92tj\" (UID: \"22543c02-5514-477e-9bfc-81968fc76a65\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-z92tj" Sep 10 23:41:54.837163 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:54.837009 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dt654\" (UniqueName: \"kubernetes.io/projected/22543c02-5514-477e-9bfc-81968fc76a65-kube-api-access-dt654\") pod \"istiod-openshift-gateway-7b7995d5cd-z92tj\" (UID: \"22543c02-5514-477e-9bfc-81968fc76a65\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-z92tj" Sep 10 23:41:54.837163 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:54.837034 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/22543c02-5514-477e-9bfc-81968fc76a65-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-z92tj\" (UID: \"22543c02-5514-477e-9bfc-81968fc76a65\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-z92tj" Sep 10 23:41:54.837163 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:54.837067 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/22543c02-5514-477e-9bfc-81968fc76a65-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-z92tj\" (UID: \"22543c02-5514-477e-9bfc-81968fc76a65\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-z92tj" Sep 10 23:41:54.837163 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:54.837107 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/22543c02-5514-477e-9bfc-81968fc76a65-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-z92tj\" (UID: \"22543c02-5514-477e-9bfc-81968fc76a65\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-z92tj" Sep 10 23:41:54.938208 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:54.938170 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/22543c02-5514-477e-9bfc-81968fc76a65-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-z92tj\" (UID: \"22543c02-5514-477e-9bfc-81968fc76a65\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-z92tj" Sep 10 23:41:54.938405 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:54.938241 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/22543c02-5514-477e-9bfc-81968fc76a65-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-z92tj\" (UID: \"22543c02-5514-477e-9bfc-81968fc76a65\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-z92tj" Sep 10 23:41:54.938405 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:54.938268 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/22543c02-5514-477e-9bfc-81968fc76a65-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-z92tj\" (UID: \"22543c02-5514-477e-9bfc-81968fc76a65\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-z92tj" Sep 10 23:41:54.938405 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:54.938313 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22543c02-5514-477e-9bfc-81968fc76a65-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-z92tj\" (UID: \"22543c02-5514-477e-9bfc-81968fc76a65\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-z92tj" Sep 10 23:41:54.938405 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:54.938349 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dt654\" (UniqueName: \"kubernetes.io/projected/22543c02-5514-477e-9bfc-81968fc76a65-kube-api-access-dt654\") pod \"istiod-openshift-gateway-7b7995d5cd-z92tj\" (UID: \"22543c02-5514-477e-9bfc-81968fc76a65\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-z92tj" Sep 10 23:41:54.938405 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:54.938385 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/22543c02-5514-477e-9bfc-81968fc76a65-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-z92tj\" (UID: \"22543c02-5514-477e-9bfc-81968fc76a65\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-z92tj" Sep 10 23:41:54.938682 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:54.938422 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/22543c02-5514-477e-9bfc-81968fc76a65-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-z92tj\" (UID: \"22543c02-5514-477e-9bfc-81968fc76a65\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-z92tj" Sep 10 23:41:54.939398 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:54.939317 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/22543c02-5514-477e-9bfc-81968fc76a65-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-z92tj\" (UID: \"22543c02-5514-477e-9bfc-81968fc76a65\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-z92tj" Sep 10 23:41:54.941010 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:54.940977 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/22543c02-5514-477e-9bfc-81968fc76a65-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-z92tj\" (UID: \"22543c02-5514-477e-9bfc-81968fc76a65\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-z92tj" Sep 10 23:41:54.941148 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:54.941126 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/22543c02-5514-477e-9bfc-81968fc76a65-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-z92tj\" (UID: \"22543c02-5514-477e-9bfc-81968fc76a65\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-z92tj" Sep 10 23:41:54.941148 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:54.941140 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/22543c02-5514-477e-9bfc-81968fc76a65-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-z92tj\" (UID: \"22543c02-5514-477e-9bfc-81968fc76a65\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-z92tj" Sep 10 23:41:54.941300 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:54.941214 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/22543c02-5514-477e-9bfc-81968fc76a65-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-z92tj\" (UID: \"22543c02-5514-477e-9bfc-81968fc76a65\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-z92tj" Sep 10 23:41:54.947700 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:54.947674 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dt654\" (UniqueName: \"kubernetes.io/projected/22543c02-5514-477e-9bfc-81968fc76a65-kube-api-access-dt654\") pod \"istiod-openshift-gateway-7b7995d5cd-z92tj\" (UID: \"22543c02-5514-477e-9bfc-81968fc76a65\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-z92tj" Sep 10 23:41:54.947810 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:54.947779 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22543c02-5514-477e-9bfc-81968fc76a65-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-z92tj\" (UID: \"22543c02-5514-477e-9bfc-81968fc76a65\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-z92tj" Sep 10 23:41:55.045518 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:55.045432 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-z92tj" Sep 10 23:41:55.595854 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:55.595731 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-z92tj"] Sep 10 23:41:55.597999 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:41:55.597965 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22543c02_5514_477e_9bfc_81968fc76a65.slice/crio-e39804217492402a0d91555010f91e7d7c8e36700c316c6d6d537cdd33c2b3ce WatchSource:0}: Error finding container e39804217492402a0d91555010f91e7d7c8e36700c316c6d6d537cdd33c2b3ce: Status 404 returned error can't find the container with id e39804217492402a0d91555010f91e7d7c8e36700c316c6d6d537cdd33c2b3ce Sep 10 23:41:55.620794 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:55.620761 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-z92tj" event={"ID":"22543c02-5514-477e-9bfc-81968fc76a65","Type":"ContainerStarted","Data":"e39804217492402a0d91555010f91e7d7c8e36700c316c6d6d537cdd33c2b3ce"} Sep 10 23:41:55.622190 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:55.622164 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-55f89b848f-6pl58" event={"ID":"c9537e38-fd66-4e10-8f78-81987455a894","Type":"ContainerStarted","Data":"56978049df0e6cae89e790615039c09316387af1c06efc8b68b88dc3b4da55f4"} Sep 10 23:41:55.639527 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:55.639483 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-55f89b848f-6pl58" podStartSLOduration=1.064735485 podStartE2EDuration="3.639468785s" totalImagesPullingTime="2.5747333s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:41:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:41:55.638823388 +0000 UTC m=+307.026426148" watchObservedRunningTime="2026-09-10 23:41:55.639468785 +0000 UTC m=+307.027071623" Sep 10 23:41:58.341172 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:58.341127 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 23:41:58.341465 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:58.341202 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 23:41:58.645010 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:58.644921 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-z92tj" event={"ID":"22543c02-5514-477e-9bfc-81968fc76a65","Type":"ContainerStarted","Data":"ea4fdd35013aed300503270514e43887faccb50322a7c39d17b2f37c2fc67b02"} Sep 10 23:41:58.645010 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:58.644976 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-z92tj" Sep 10 23:41:58.670236 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:58.670177 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-z92tj" podStartSLOduration=1.929301947 podStartE2EDuration="4.670157577s" totalImagesPullingTime="2.74085563s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:41:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:41:58.66860409 +0000 UTC m=+310.056206860" watchObservedRunningTime="2026-09-10 23:41:58.670157577 +0000 UTC m=+310.057760351" Sep 10 23:41:59.652355 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:59.652315 2574 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-z92tj container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 10 23:41:59.652777 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:41:59.652376 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-z92tj" podUID="22543c02-5514-477e-9bfc-81968fc76a65" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 10 23:42:02.651406 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:02.651375 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-z92tj" Sep 10 23:42:09.870792 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:09.870746 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-cdc5j"] Sep 10 23:42:09.878501 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:09.878469 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cdc5j" Sep 10 23:42:09.888055 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:09.888029 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-cdc5j"] Sep 10 23:42:09.982878 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:09.982832 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcxp8\" (UniqueName: \"kubernetes.io/projected/d88224c2-9ecd-42a5-ac77-820af96f724a-kube-api-access-mcxp8\") pod \"istiod-openshift-gateway-5677ccb49b-cdc5j\" (UID: \"d88224c2-9ecd-42a5-ac77-820af96f724a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cdc5j" Sep 10 23:42:09.982878 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:09.982876 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d88224c2-9ecd-42a5-ac77-820af96f724a-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-cdc5j\" (UID: \"d88224c2-9ecd-42a5-ac77-820af96f724a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cdc5j" Sep 10 23:42:09.983136 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:09.982956 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d88224c2-9ecd-42a5-ac77-820af96f724a-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-cdc5j\" (UID: \"d88224c2-9ecd-42a5-ac77-820af96f724a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cdc5j" Sep 10 23:42:09.983136 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:09.983024 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d88224c2-9ecd-42a5-ac77-820af96f724a-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-cdc5j\" (UID: \"d88224c2-9ecd-42a5-ac77-820af96f724a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cdc5j" Sep 10 23:42:09.983136 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:09.983054 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d88224c2-9ecd-42a5-ac77-820af96f724a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-cdc5j\" (UID: \"d88224c2-9ecd-42a5-ac77-820af96f724a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cdc5j" Sep 10 23:42:09.983136 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:09.983095 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d88224c2-9ecd-42a5-ac77-820af96f724a-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-cdc5j\" (UID: \"d88224c2-9ecd-42a5-ac77-820af96f724a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cdc5j" Sep 10 23:42:09.983136 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:09.983118 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d88224c2-9ecd-42a5-ac77-820af96f724a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-cdc5j\" (UID: \"d88224c2-9ecd-42a5-ac77-820af96f724a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cdc5j" Sep 10 23:42:10.084461 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:10.084426 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcxp8\" (UniqueName: \"kubernetes.io/projected/d88224c2-9ecd-42a5-ac77-820af96f724a-kube-api-access-mcxp8\") pod \"istiod-openshift-gateway-5677ccb49b-cdc5j\" (UID: \"d88224c2-9ecd-42a5-ac77-820af96f724a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cdc5j" Sep 10 23:42:10.084461 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:10.084463 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d88224c2-9ecd-42a5-ac77-820af96f724a-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-cdc5j\" (UID: \"d88224c2-9ecd-42a5-ac77-820af96f724a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cdc5j" Sep 10 23:42:10.084748 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:10.084493 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d88224c2-9ecd-42a5-ac77-820af96f724a-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-cdc5j\" (UID: \"d88224c2-9ecd-42a5-ac77-820af96f724a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cdc5j" Sep 10 23:42:10.084748 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:10.084516 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d88224c2-9ecd-42a5-ac77-820af96f724a-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-cdc5j\" (UID: \"d88224c2-9ecd-42a5-ac77-820af96f724a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cdc5j" Sep 10 23:42:10.084748 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:10.084533 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d88224c2-9ecd-42a5-ac77-820af96f724a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-cdc5j\" (UID: \"d88224c2-9ecd-42a5-ac77-820af96f724a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cdc5j" Sep 10 23:42:10.084748 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:10.084563 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d88224c2-9ecd-42a5-ac77-820af96f724a-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-cdc5j\" (UID: \"d88224c2-9ecd-42a5-ac77-820af96f724a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cdc5j" Sep 10 23:42:10.084748 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:10.084585 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d88224c2-9ecd-42a5-ac77-820af96f724a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-cdc5j\" (UID: \"d88224c2-9ecd-42a5-ac77-820af96f724a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cdc5j" Sep 10 23:42:10.085415 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:10.085386 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d88224c2-9ecd-42a5-ac77-820af96f724a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-cdc5j\" (UID: \"d88224c2-9ecd-42a5-ac77-820af96f724a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cdc5j" Sep 10 23:42:10.087074 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:10.087048 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d88224c2-9ecd-42a5-ac77-820af96f724a-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-cdc5j\" (UID: \"d88224c2-9ecd-42a5-ac77-820af96f724a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cdc5j" Sep 10 23:42:10.087074 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:10.087067 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d88224c2-9ecd-42a5-ac77-820af96f724a-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-cdc5j\" (UID: \"d88224c2-9ecd-42a5-ac77-820af96f724a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cdc5j" Sep 10 23:42:10.087195 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:10.087154 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d88224c2-9ecd-42a5-ac77-820af96f724a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-cdc5j\" (UID: \"d88224c2-9ecd-42a5-ac77-820af96f724a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cdc5j" Sep 10 23:42:10.087231 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:10.087197 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d88224c2-9ecd-42a5-ac77-820af96f724a-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-cdc5j\" (UID: \"d88224c2-9ecd-42a5-ac77-820af96f724a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cdc5j" Sep 10 23:42:10.092244 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:10.092223 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d88224c2-9ecd-42a5-ac77-820af96f724a-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-cdc5j\" (UID: \"d88224c2-9ecd-42a5-ac77-820af96f724a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cdc5j" Sep 10 23:42:10.092755 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:10.092737 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcxp8\" (UniqueName: \"kubernetes.io/projected/d88224c2-9ecd-42a5-ac77-820af96f724a-kube-api-access-mcxp8\") pod \"istiod-openshift-gateway-5677ccb49b-cdc5j\" (UID: \"d88224c2-9ecd-42a5-ac77-820af96f724a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cdc5j" Sep 10 23:42:10.189177 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:10.189095 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cdc5j" Sep 10 23:42:10.461342 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:10.461210 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-cdc5j"] Sep 10 23:42:10.464002 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:42:10.463971 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd88224c2_9ecd_42a5_ac77_820af96f724a.slice/crio-59e8c9295c189846f41fd6438661456bfd65559839a7d080599903b4260223ef WatchSource:0}: Error finding container 59e8c9295c189846f41fd6438661456bfd65559839a7d080599903b4260223ef: Status 404 returned error can't find the container with id 59e8c9295c189846f41fd6438661456bfd65559839a7d080599903b4260223ef Sep 10 23:42:10.466926 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:10.466894 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 23:42:10.466993 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:10.466954 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 23:42:10.704769 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:10.704735 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cdc5j" event={"ID":"d88224c2-9ecd-42a5-ac77-820af96f724a","Type":"ContainerStarted","Data":"b6106ffff4e52280ee5bc598dee509268f3fe196cbdea0f9adc10942f82db44f"} Sep 10 23:42:10.704950 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:10.704791 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cdc5j" event={"ID":"d88224c2-9ecd-42a5-ac77-820af96f724a","Type":"ContainerStarted","Data":"59e8c9295c189846f41fd6438661456bfd65559839a7d080599903b4260223ef"} Sep 10 23:42:10.704950 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:10.704816 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cdc5j" Sep 10 23:42:10.725391 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:10.725293 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cdc5j" podStartSLOduration=1.725264774 podStartE2EDuration="1.725264774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:42:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:42:10.723932836 +0000 UTC m=+322.111535596" watchObservedRunningTime="2026-09-10 23:42:10.725264774 +0000 UTC m=+322.112867533" Sep 10 23:42:11.712945 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:11.712916 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-cdc5j" Sep 10 23:42:11.831371 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:11.831335 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-z92tj"] Sep 10 23:42:11.831586 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:11.831558 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-z92tj" podUID="22543c02-5514-477e-9bfc-81968fc76a65" containerName="discovery" containerID="cri-o://ea4fdd35013aed300503270514e43887faccb50322a7c39d17b2f37c2fc67b02" gracePeriod=30 Sep 10 23:42:12.079540 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:12.079517 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-z92tj" Sep 10 23:42:12.203403 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:12.203376 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/22543c02-5514-477e-9bfc-81968fc76a65-istio-csr-dns-cert\") pod \"22543c02-5514-477e-9bfc-81968fc76a65\" (UID: \"22543c02-5514-477e-9bfc-81968fc76a65\") " Sep 10 23:42:12.203552 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:12.203420 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/22543c02-5514-477e-9bfc-81968fc76a65-local-certs\") pod \"22543c02-5514-477e-9bfc-81968fc76a65\" (UID: \"22543c02-5514-477e-9bfc-81968fc76a65\") " Sep 10 23:42:12.203552 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:12.203464 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/22543c02-5514-477e-9bfc-81968fc76a65-istio-csr-ca-configmap\") pod \"22543c02-5514-477e-9bfc-81968fc76a65\" (UID: \"22543c02-5514-477e-9bfc-81968fc76a65\") " Sep 10 23:42:12.203552 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:12.203493 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22543c02-5514-477e-9bfc-81968fc76a65-istio-token\") pod \"22543c02-5514-477e-9bfc-81968fc76a65\" (UID: \"22543c02-5514-477e-9bfc-81968fc76a65\") " Sep 10 23:42:12.203552 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:12.203513 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/22543c02-5514-477e-9bfc-81968fc76a65-cacerts\") pod \"22543c02-5514-477e-9bfc-81968fc76a65\" (UID: \"22543c02-5514-477e-9bfc-81968fc76a65\") " Sep 10 23:42:12.203552 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:12.203548 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dt654\" (UniqueName: \"kubernetes.io/projected/22543c02-5514-477e-9bfc-81968fc76a65-kube-api-access-dt654\") pod \"22543c02-5514-477e-9bfc-81968fc76a65\" (UID: \"22543c02-5514-477e-9bfc-81968fc76a65\") " Sep 10 23:42:12.203845 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:12.203677 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/22543c02-5514-477e-9bfc-81968fc76a65-istio-kubeconfig\") pod \"22543c02-5514-477e-9bfc-81968fc76a65\" (UID: \"22543c02-5514-477e-9bfc-81968fc76a65\") " Sep 10 23:42:12.203905 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:12.203884 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22543c02-5514-477e-9bfc-81968fc76a65-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "22543c02-5514-477e-9bfc-81968fc76a65" (UID: "22543c02-5514-477e-9bfc-81968fc76a65"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 23:42:12.204040 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:12.204012 2574 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/22543c02-5514-477e-9bfc-81968fc76a65-istio-csr-ca-configmap\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:42:12.205861 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:12.205833 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22543c02-5514-477e-9bfc-81968fc76a65-kube-api-access-dt654" (OuterVolumeSpecName: "kube-api-access-dt654") pod "22543c02-5514-477e-9bfc-81968fc76a65" (UID: "22543c02-5514-477e-9bfc-81968fc76a65"). InnerVolumeSpecName "kube-api-access-dt654". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 23:42:12.205995 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:12.205909 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22543c02-5514-477e-9bfc-81968fc76a65-istio-token" (OuterVolumeSpecName: "istio-token") pod "22543c02-5514-477e-9bfc-81968fc76a65" (UID: "22543c02-5514-477e-9bfc-81968fc76a65"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 23:42:12.206104 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:12.206085 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22543c02-5514-477e-9bfc-81968fc76a65-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "22543c02-5514-477e-9bfc-81968fc76a65" (UID: "22543c02-5514-477e-9bfc-81968fc76a65"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 23:42:12.206161 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:12.206106 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22543c02-5514-477e-9bfc-81968fc76a65-local-certs" (OuterVolumeSpecName: "local-certs") pod "22543c02-5514-477e-9bfc-81968fc76a65" (UID: "22543c02-5514-477e-9bfc-81968fc76a65"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:42:12.206161 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:12.206106 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22543c02-5514-477e-9bfc-81968fc76a65-cacerts" (OuterVolumeSpecName: "cacerts") pod "22543c02-5514-477e-9bfc-81968fc76a65" (UID: "22543c02-5514-477e-9bfc-81968fc76a65"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 23:42:12.206262 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:12.206184 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22543c02-5514-477e-9bfc-81968fc76a65-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "22543c02-5514-477e-9bfc-81968fc76a65" (UID: "22543c02-5514-477e-9bfc-81968fc76a65"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 23:42:12.304983 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:12.304917 2574 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/22543c02-5514-477e-9bfc-81968fc76a65-istio-kubeconfig\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:42:12.304983 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:12.304940 2574 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/22543c02-5514-477e-9bfc-81968fc76a65-istio-csr-dns-cert\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:42:12.304983 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:12.304951 2574 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/22543c02-5514-477e-9bfc-81968fc76a65-local-certs\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:42:12.304983 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:12.304960 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22543c02-5514-477e-9bfc-81968fc76a65-istio-token\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:42:12.304983 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:12.304968 2574 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/22543c02-5514-477e-9bfc-81968fc76a65-cacerts\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:42:12.304983 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:12.304976 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dt654\" (UniqueName: \"kubernetes.io/projected/22543c02-5514-477e-9bfc-81968fc76a65-kube-api-access-dt654\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:42:12.718484 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:12.718443 2574 generic.go:356] "Generic (PLEG): container finished" podID="22543c02-5514-477e-9bfc-81968fc76a65" containerID="ea4fdd35013aed300503270514e43887faccb50322a7c39d17b2f37c2fc67b02" exitCode=0 Sep 10 23:42:12.718974 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:12.718502 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-z92tj" Sep 10 23:42:12.718974 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:12.718519 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-z92tj" event={"ID":"22543c02-5514-477e-9bfc-81968fc76a65","Type":"ContainerDied","Data":"ea4fdd35013aed300503270514e43887faccb50322a7c39d17b2f37c2fc67b02"} Sep 10 23:42:12.718974 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:12.718580 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-z92tj" event={"ID":"22543c02-5514-477e-9bfc-81968fc76a65","Type":"ContainerDied","Data":"e39804217492402a0d91555010f91e7d7c8e36700c316c6d6d537cdd33c2b3ce"} Sep 10 23:42:12.718974 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:12.718607 2574 scope.go:117] "RemoveContainer" containerID="ea4fdd35013aed300503270514e43887faccb50322a7c39d17b2f37c2fc67b02" Sep 10 23:42:12.730276 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:12.730249 2574 scope.go:117] "RemoveContainer" containerID="ea4fdd35013aed300503270514e43887faccb50322a7c39d17b2f37c2fc67b02" Sep 10 23:42:12.730591 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:42:12.730566 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea4fdd35013aed300503270514e43887faccb50322a7c39d17b2f37c2fc67b02\": container with ID starting with ea4fdd35013aed300503270514e43887faccb50322a7c39d17b2f37c2fc67b02 not found: ID does not exist" containerID="ea4fdd35013aed300503270514e43887faccb50322a7c39d17b2f37c2fc67b02" Sep 10 23:42:12.730716 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:12.730600 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea4fdd35013aed300503270514e43887faccb50322a7c39d17b2f37c2fc67b02"} err="failed to get container status \"ea4fdd35013aed300503270514e43887faccb50322a7c39d17b2f37c2fc67b02\": rpc error: code = NotFound desc = could not find container \"ea4fdd35013aed300503270514e43887faccb50322a7c39d17b2f37c2fc67b02\": container with ID starting with ea4fdd35013aed300503270514e43887faccb50322a7c39d17b2f37c2fc67b02 not found: ID does not exist" Sep 10 23:42:12.745891 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:12.745867 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-z92tj"] Sep 10 23:42:12.750957 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:12.750934 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-z92tj"] Sep 10 23:42:12.801393 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:12.801360 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-7ql2b"] Sep 10 23:42:12.801746 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:12.801734 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22543c02-5514-477e-9bfc-81968fc76a65" containerName="discovery" Sep 10 23:42:12.801808 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:12.801748 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="22543c02-5514-477e-9bfc-81968fc76a65" containerName="discovery" Sep 10 23:42:12.801847 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:12.801809 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="22543c02-5514-477e-9bfc-81968fc76a65" containerName="discovery" Sep 10 23:42:12.805969 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:12.805951 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-7ql2b" Sep 10 23:42:12.808770 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:12.808747 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-6qlld\"" Sep 10 23:42:12.808911 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:12.808896 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Sep 10 23:42:12.825385 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:12.825363 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-9xxkk"] Sep 10 23:42:12.828605 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:12.828583 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-7ql2b"] Sep 10 23:42:12.828728 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:12.828682 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-9xxkk" Sep 10 23:42:12.831705 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:12.831681 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Sep 10 23:42:12.831815 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:12.831716 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-mrtff\"" Sep 10 23:42:12.841613 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:12.841585 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-9xxkk"] Sep 10 23:42:12.910651 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:12.910618 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjmzs\" (UniqueName: \"kubernetes.io/projected/9a68eaf6-bccb-4fd2-808d-c7546b395981-kube-api-access-wjmzs\") pod \"llmisvc-controller-manager-777c6bd7d5-9xxkk\" (UID: \"9a68eaf6-bccb-4fd2-808d-c7546b395981\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-9xxkk" Sep 10 23:42:12.910805 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:12.910701 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/16430c51-3465-4ed3-a41c-c759af1e0eb1-cert\") pod \"kserve-controller-manager-d6f4b65b8-7ql2b\" (UID: \"16430c51-3465-4ed3-a41c-c759af1e0eb1\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-7ql2b" Sep 10 23:42:12.910805 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:12.910733 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lwpx\" (UniqueName: \"kubernetes.io/projected/16430c51-3465-4ed3-a41c-c759af1e0eb1-kube-api-access-5lwpx\") pod \"kserve-controller-manager-d6f4b65b8-7ql2b\" (UID: \"16430c51-3465-4ed3-a41c-c759af1e0eb1\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-7ql2b" Sep 10 23:42:12.910805 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:12.910748 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9a68eaf6-bccb-4fd2-808d-c7546b395981-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-9xxkk\" (UID: \"9a68eaf6-bccb-4fd2-808d-c7546b395981\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-9xxkk" Sep 10 23:42:13.011427 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:13.011339 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/16430c51-3465-4ed3-a41c-c759af1e0eb1-cert\") pod \"kserve-controller-manager-d6f4b65b8-7ql2b\" (UID: \"16430c51-3465-4ed3-a41c-c759af1e0eb1\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-7ql2b" Sep 10 23:42:13.011427 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:13.011386 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lwpx\" (UniqueName: \"kubernetes.io/projected/16430c51-3465-4ed3-a41c-c759af1e0eb1-kube-api-access-5lwpx\") pod \"kserve-controller-manager-d6f4b65b8-7ql2b\" (UID: \"16430c51-3465-4ed3-a41c-c759af1e0eb1\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-7ql2b" Sep 10 23:42:13.011427 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:13.011405 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9a68eaf6-bccb-4fd2-808d-c7546b395981-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-9xxkk\" (UID: \"9a68eaf6-bccb-4fd2-808d-c7546b395981\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-9xxkk" Sep 10 23:42:13.011728 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:13.011474 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjmzs\" (UniqueName: \"kubernetes.io/projected/9a68eaf6-bccb-4fd2-808d-c7546b395981-kube-api-access-wjmzs\") pod \"llmisvc-controller-manager-777c6bd7d5-9xxkk\" (UID: \"9a68eaf6-bccb-4fd2-808d-c7546b395981\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-9xxkk" Sep 10 23:42:13.011728 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:42:13.011528 2574 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Sep 10 23:42:13.011728 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:42:13.011583 2574 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Sep 10 23:42:13.011728 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:42:13.011668 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16430c51-3465-4ed3-a41c-c759af1e0eb1-cert podName:16430c51-3465-4ed3-a41c-c759af1e0eb1 nodeName:}" failed. No retries permitted until 2026-09-10 23:42:13.511616541 +0000 UTC m=+324.899219297 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/16430c51-3465-4ed3-a41c-c759af1e0eb1-cert") pod "kserve-controller-manager-d6f4b65b8-7ql2b" (UID: "16430c51-3465-4ed3-a41c-c759af1e0eb1") : secret "kserve-webhook-server-cert" not found Sep 10 23:42:13.011728 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:42:13.011691 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a68eaf6-bccb-4fd2-808d-c7546b395981-cert podName:9a68eaf6-bccb-4fd2-808d-c7546b395981 nodeName:}" failed. No retries permitted until 2026-09-10 23:42:13.511681606 +0000 UTC m=+324.899284345 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/9a68eaf6-bccb-4fd2-808d-c7546b395981-cert") pod "llmisvc-controller-manager-777c6bd7d5-9xxkk" (UID: "9a68eaf6-bccb-4fd2-808d-c7546b395981") : secret "llmisvc-webhook-server-cert" not found Sep 10 23:42:13.031275 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:13.031251 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lwpx\" (UniqueName: \"kubernetes.io/projected/16430c51-3465-4ed3-a41c-c759af1e0eb1-kube-api-access-5lwpx\") pod \"kserve-controller-manager-d6f4b65b8-7ql2b\" (UID: \"16430c51-3465-4ed3-a41c-c759af1e0eb1\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-7ql2b" Sep 10 23:42:13.031416 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:13.031390 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjmzs\" (UniqueName: \"kubernetes.io/projected/9a68eaf6-bccb-4fd2-808d-c7546b395981-kube-api-access-wjmzs\") pod \"llmisvc-controller-manager-777c6bd7d5-9xxkk\" (UID: \"9a68eaf6-bccb-4fd2-808d-c7546b395981\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-9xxkk" Sep 10 23:42:13.095157 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:13.095119 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22543c02-5514-477e-9bfc-81968fc76a65" path="/var/lib/kubelet/pods/22543c02-5514-477e-9bfc-81968fc76a65/volumes" Sep 10 23:42:13.515744 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:13.515704 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/16430c51-3465-4ed3-a41c-c759af1e0eb1-cert\") pod \"kserve-controller-manager-d6f4b65b8-7ql2b\" (UID: \"16430c51-3465-4ed3-a41c-c759af1e0eb1\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-7ql2b" Sep 10 23:42:13.515917 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:13.515771 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9a68eaf6-bccb-4fd2-808d-c7546b395981-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-9xxkk\" (UID: \"9a68eaf6-bccb-4fd2-808d-c7546b395981\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-9xxkk" Sep 10 23:42:13.518225 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:13.518199 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/16430c51-3465-4ed3-a41c-c759af1e0eb1-cert\") pod \"kserve-controller-manager-d6f4b65b8-7ql2b\" (UID: \"16430c51-3465-4ed3-a41c-c759af1e0eb1\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-7ql2b" Sep 10 23:42:13.518319 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:13.518243 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9a68eaf6-bccb-4fd2-808d-c7546b395981-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-9xxkk\" (UID: \"9a68eaf6-bccb-4fd2-808d-c7546b395981\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-9xxkk" Sep 10 23:42:13.716611 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:13.716570 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-7ql2b" Sep 10 23:42:13.738876 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:13.738850 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-9xxkk" Sep 10 23:42:13.991825 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:13.991796 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-7ql2b"] Sep 10 23:42:13.993728 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:42:13.993696 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16430c51_3465_4ed3_a41c_c759af1e0eb1.slice/crio-da5e058333cdd36ecfde1f26799cfce3ecc0daac2501a7f86eff3a29d6f561c5 WatchSource:0}: Error finding container da5e058333cdd36ecfde1f26799cfce3ecc0daac2501a7f86eff3a29d6f561c5: Status 404 returned error can't find the container with id da5e058333cdd36ecfde1f26799cfce3ecc0daac2501a7f86eff3a29d6f561c5 Sep 10 23:42:14.020970 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:14.020905 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-9xxkk"] Sep 10 23:42:14.024414 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:42:14.024380 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod9a68eaf6_bccb_4fd2_808d_c7546b395981.slice/crio-387727445c3bc5b40024dc4966259fad981e2aaeef096be6345da13b83abe0d8 WatchSource:0}: Error finding container 387727445c3bc5b40024dc4966259fad981e2aaeef096be6345da13b83abe0d8: Status 404 returned error can't find the container with id 387727445c3bc5b40024dc4966259fad981e2aaeef096be6345da13b83abe0d8 Sep 10 23:42:14.735135 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:14.735089 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-9xxkk" event={"ID":"9a68eaf6-bccb-4fd2-808d-c7546b395981","Type":"ContainerStarted","Data":"387727445c3bc5b40024dc4966259fad981e2aaeef096be6345da13b83abe0d8"} Sep 10 23:42:14.736758 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:14.736706 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-7ql2b" event={"ID":"16430c51-3465-4ed3-a41c-c759af1e0eb1","Type":"ContainerStarted","Data":"da5e058333cdd36ecfde1f26799cfce3ecc0daac2501a7f86eff3a29d6f561c5"} Sep 10 23:42:17.756313 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:17.756278 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-9xxkk" event={"ID":"9a68eaf6-bccb-4fd2-808d-c7546b395981","Type":"ContainerStarted","Data":"f2ba3f8613996e5f6f5804668bebc1086387dc591b703d80929e460052f2ade0"} Sep 10 23:42:17.756774 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:17.756366 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-9xxkk" Sep 10 23:42:17.758025 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:17.758000 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-7ql2b" event={"ID":"16430c51-3465-4ed3-a41c-c759af1e0eb1","Type":"ContainerStarted","Data":"bdeeb62d57b3c3c504ff34fad09cd2817e9a46b8791147061ee93d797ddd6e37"} Sep 10 23:42:17.758147 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:17.758073 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-7ql2b" Sep 10 23:42:17.775529 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:17.775490 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-9xxkk" podStartSLOduration=2.8338790659999997 podStartE2EDuration="5.775474861s" totalImagesPullingTime="2.941595795s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:42:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:42:17.77357958 +0000 UTC m=+329.161182331" watchObservedRunningTime="2026-09-10 23:42:17.775474861 +0000 UTC m=+329.163077610" Sep 10 23:42:17.791654 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:17.791588 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-7ql2b" podStartSLOduration=2.824669057 podStartE2EDuration="5.791572486s" totalImagesPullingTime="2.966903429s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:42:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:42:17.790782557 +0000 UTC m=+329.178385316" watchObservedRunningTime="2026-09-10 23:42:17.791572486 +0000 UTC m=+329.179175247" Sep 10 23:42:43.623968 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:43.623882 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-76f4fbdcf5-vgxpp"] Sep 10 23:42:43.627555 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:43.627529 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76f4fbdcf5-vgxpp" Sep 10 23:42:43.640234 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:43.640209 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-76f4fbdcf5-vgxpp"] Sep 10 23:42:43.670776 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:43.670748 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btzg5\" (UniqueName: \"kubernetes.io/projected/a6155b8d-dffb-4ffa-b03b-97754b95bce4-kube-api-access-btzg5\") pod \"console-76f4fbdcf5-vgxpp\" (UID: \"a6155b8d-dffb-4ffa-b03b-97754b95bce4\") " pod="openshift-console/console-76f4fbdcf5-vgxpp" Sep 10 23:42:43.670915 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:43.670791 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a6155b8d-dffb-4ffa-b03b-97754b95bce4-service-ca\") pod \"console-76f4fbdcf5-vgxpp\" (UID: \"a6155b8d-dffb-4ffa-b03b-97754b95bce4\") " pod="openshift-console/console-76f4fbdcf5-vgxpp" Sep 10 23:42:43.671672 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:43.670811 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6155b8d-dffb-4ffa-b03b-97754b95bce4-trusted-ca-bundle\") pod \"console-76f4fbdcf5-vgxpp\" (UID: \"a6155b8d-dffb-4ffa-b03b-97754b95bce4\") " pod="openshift-console/console-76f4fbdcf5-vgxpp" Sep 10 23:42:43.671672 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:43.671275 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a6155b8d-dffb-4ffa-b03b-97754b95bce4-console-oauth-config\") pod \"console-76f4fbdcf5-vgxpp\" (UID: \"a6155b8d-dffb-4ffa-b03b-97754b95bce4\") " pod="openshift-console/console-76f4fbdcf5-vgxpp" Sep 10 23:42:43.671672 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:43.671399 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a6155b8d-dffb-4ffa-b03b-97754b95bce4-oauth-serving-cert\") pod \"console-76f4fbdcf5-vgxpp\" (UID: \"a6155b8d-dffb-4ffa-b03b-97754b95bce4\") " pod="openshift-console/console-76f4fbdcf5-vgxpp" Sep 10 23:42:43.671672 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:43.671452 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a6155b8d-dffb-4ffa-b03b-97754b95bce4-console-serving-cert\") pod \"console-76f4fbdcf5-vgxpp\" (UID: \"a6155b8d-dffb-4ffa-b03b-97754b95bce4\") " pod="openshift-console/console-76f4fbdcf5-vgxpp" Sep 10 23:42:43.671672 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:43.671499 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a6155b8d-dffb-4ffa-b03b-97754b95bce4-console-config\") pod \"console-76f4fbdcf5-vgxpp\" (UID: \"a6155b8d-dffb-4ffa-b03b-97754b95bce4\") " pod="openshift-console/console-76f4fbdcf5-vgxpp" Sep 10 23:42:43.772010 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:43.771977 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a6155b8d-dffb-4ffa-b03b-97754b95bce4-console-oauth-config\") pod \"console-76f4fbdcf5-vgxpp\" (UID: \"a6155b8d-dffb-4ffa-b03b-97754b95bce4\") " pod="openshift-console/console-76f4fbdcf5-vgxpp" Sep 10 23:42:43.772010 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:43.772018 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a6155b8d-dffb-4ffa-b03b-97754b95bce4-oauth-serving-cert\") pod \"console-76f4fbdcf5-vgxpp\" (UID: \"a6155b8d-dffb-4ffa-b03b-97754b95bce4\") " pod="openshift-console/console-76f4fbdcf5-vgxpp" Sep 10 23:42:43.772275 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:43.772040 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a6155b8d-dffb-4ffa-b03b-97754b95bce4-console-serving-cert\") pod \"console-76f4fbdcf5-vgxpp\" (UID: \"a6155b8d-dffb-4ffa-b03b-97754b95bce4\") " pod="openshift-console/console-76f4fbdcf5-vgxpp" Sep 10 23:42:43.772275 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:43.772064 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a6155b8d-dffb-4ffa-b03b-97754b95bce4-console-config\") pod \"console-76f4fbdcf5-vgxpp\" (UID: \"a6155b8d-dffb-4ffa-b03b-97754b95bce4\") " pod="openshift-console/console-76f4fbdcf5-vgxpp" Sep 10 23:42:43.772275 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:43.772196 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btzg5\" (UniqueName: \"kubernetes.io/projected/a6155b8d-dffb-4ffa-b03b-97754b95bce4-kube-api-access-btzg5\") pod \"console-76f4fbdcf5-vgxpp\" (UID: \"a6155b8d-dffb-4ffa-b03b-97754b95bce4\") " pod="openshift-console/console-76f4fbdcf5-vgxpp" Sep 10 23:42:43.772275 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:43.772270 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a6155b8d-dffb-4ffa-b03b-97754b95bce4-service-ca\") pod \"console-76f4fbdcf5-vgxpp\" (UID: \"a6155b8d-dffb-4ffa-b03b-97754b95bce4\") " pod="openshift-console/console-76f4fbdcf5-vgxpp" Sep 10 23:42:43.772534 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:43.772298 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6155b8d-dffb-4ffa-b03b-97754b95bce4-trusted-ca-bundle\") pod \"console-76f4fbdcf5-vgxpp\" (UID: \"a6155b8d-dffb-4ffa-b03b-97754b95bce4\") " pod="openshift-console/console-76f4fbdcf5-vgxpp" Sep 10 23:42:43.772934 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:43.772911 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a6155b8d-dffb-4ffa-b03b-97754b95bce4-oauth-serving-cert\") pod \"console-76f4fbdcf5-vgxpp\" (UID: \"a6155b8d-dffb-4ffa-b03b-97754b95bce4\") " pod="openshift-console/console-76f4fbdcf5-vgxpp" Sep 10 23:42:43.773031 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:43.772962 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a6155b8d-dffb-4ffa-b03b-97754b95bce4-console-config\") pod \"console-76f4fbdcf5-vgxpp\" (UID: \"a6155b8d-dffb-4ffa-b03b-97754b95bce4\") " pod="openshift-console/console-76f4fbdcf5-vgxpp" Sep 10 23:42:43.773136 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:43.773114 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a6155b8d-dffb-4ffa-b03b-97754b95bce4-service-ca\") pod \"console-76f4fbdcf5-vgxpp\" (UID: \"a6155b8d-dffb-4ffa-b03b-97754b95bce4\") " pod="openshift-console/console-76f4fbdcf5-vgxpp" Sep 10 23:42:43.773308 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:43.773285 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6155b8d-dffb-4ffa-b03b-97754b95bce4-trusted-ca-bundle\") pod \"console-76f4fbdcf5-vgxpp\" (UID: \"a6155b8d-dffb-4ffa-b03b-97754b95bce4\") " pod="openshift-console/console-76f4fbdcf5-vgxpp" Sep 10 23:42:43.774651 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:43.774611 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a6155b8d-dffb-4ffa-b03b-97754b95bce4-console-serving-cert\") pod \"console-76f4fbdcf5-vgxpp\" (UID: \"a6155b8d-dffb-4ffa-b03b-97754b95bce4\") " pod="openshift-console/console-76f4fbdcf5-vgxpp" Sep 10 23:42:43.774765 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:43.774743 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a6155b8d-dffb-4ffa-b03b-97754b95bce4-console-oauth-config\") pod \"console-76f4fbdcf5-vgxpp\" (UID: \"a6155b8d-dffb-4ffa-b03b-97754b95bce4\") " pod="openshift-console/console-76f4fbdcf5-vgxpp" Sep 10 23:42:43.782766 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:43.782745 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-btzg5\" (UniqueName: \"kubernetes.io/projected/a6155b8d-dffb-4ffa-b03b-97754b95bce4-kube-api-access-btzg5\") pod \"console-76f4fbdcf5-vgxpp\" (UID: \"a6155b8d-dffb-4ffa-b03b-97754b95bce4\") " pod="openshift-console/console-76f4fbdcf5-vgxpp" Sep 10 23:42:43.936541 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:43.936444 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76f4fbdcf5-vgxpp" Sep 10 23:42:44.209082 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:44.209056 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-76f4fbdcf5-vgxpp"] Sep 10 23:42:44.211021 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:42:44.210993 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6155b8d_dffb_4ffa_b03b_97754b95bce4.slice/crio-c0d135cf56f9c690c8c223a1a2a33fed8ae0b6154c32818f1d9501fe8771ad8a WatchSource:0}: Error finding container c0d135cf56f9c690c8c223a1a2a33fed8ae0b6154c32818f1d9501fe8771ad8a: Status 404 returned error can't find the container with id c0d135cf56f9c690c8c223a1a2a33fed8ae0b6154c32818f1d9501fe8771ad8a Sep 10 23:42:44.890284 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:44.890246 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76f4fbdcf5-vgxpp" event={"ID":"a6155b8d-dffb-4ffa-b03b-97754b95bce4","Type":"ContainerStarted","Data":"f62be1b30cc5f4e50ee612267d6510885ba3259516f50a331f4da31bdb755a80"} Sep 10 23:42:44.890788 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:44.890295 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76f4fbdcf5-vgxpp" event={"ID":"a6155b8d-dffb-4ffa-b03b-97754b95bce4","Type":"ContainerStarted","Data":"c0d135cf56f9c690c8c223a1a2a33fed8ae0b6154c32818f1d9501fe8771ad8a"} Sep 10 23:42:44.909469 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:44.909414 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-76f4fbdcf5-vgxpp" podStartSLOduration=1.90939478 podStartE2EDuration="1.90939478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:42:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:42:44.909124659 +0000 UTC m=+356.296727417" watchObservedRunningTime="2026-09-10 23:42:44.90939478 +0000 UTC m=+356.296997550" Sep 10 23:42:48.765489 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:48.765455 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-9xxkk" Sep 10 23:42:48.768510 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:48.768490 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-7ql2b" Sep 10 23:42:53.937012 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:53.936984 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-76f4fbdcf5-vgxpp" Sep 10 23:42:53.937400 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:53.937034 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-76f4fbdcf5-vgxpp" Sep 10 23:42:53.941767 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:53.941749 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-76f4fbdcf5-vgxpp" Sep 10 23:42:54.944077 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:54.944047 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-76f4fbdcf5-vgxpp" Sep 10 23:42:54.990547 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:42:54.990513 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-778d864cdc-v6xfr"] Sep 10 23:43:07.147871 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:07.147839 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d758spt"] Sep 10 23:43:07.155005 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:07.154984 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d758spt" Sep 10 23:43:07.157364 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:07.157337 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 10 23:43:07.157493 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:07.157383 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-hdmcv\"" Sep 10 23:43:07.157493 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:07.157412 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 10 23:43:07.161698 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:07.161674 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d758spt"] Sep 10 23:43:07.242377 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:07.242350 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9fv8j"] Sep 10 23:43:07.246107 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:07.246091 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9fv8j" Sep 10 23:43:07.253995 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:07.253969 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9fv8j"] Sep 10 23:43:07.275674 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:07.275622 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4c7f94f5-db28-432e-984b-27c2326a3243-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d758spt\" (UID: \"4c7f94f5-db28-432e-984b-27c2326a3243\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d758spt" Sep 10 23:43:07.275787 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:07.275694 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbthg\" (UniqueName: \"kubernetes.io/projected/4c7f94f5-db28-432e-984b-27c2326a3243-kube-api-access-dbthg\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d758spt\" (UID: \"4c7f94f5-db28-432e-984b-27c2326a3243\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d758spt" Sep 10 23:43:07.275787 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:07.275730 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4c7f94f5-db28-432e-984b-27c2326a3243-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d758spt\" (UID: \"4c7f94f5-db28-432e-984b-27c2326a3243\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d758spt" Sep 10 23:43:07.341918 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:07.341884 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6brhqg"] Sep 10 23:43:07.345541 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:07.345524 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6brhqg" Sep 10 23:43:07.354284 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:07.354264 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6brhqg"] Sep 10 23:43:07.377131 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:07.377108 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4c7f94f5-db28-432e-984b-27c2326a3243-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d758spt\" (UID: \"4c7f94f5-db28-432e-984b-27c2326a3243\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d758spt" Sep 10 23:43:07.377237 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:07.377133 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbthg\" (UniqueName: \"kubernetes.io/projected/4c7f94f5-db28-432e-984b-27c2326a3243-kube-api-access-dbthg\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d758spt\" (UID: \"4c7f94f5-db28-432e-984b-27c2326a3243\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d758spt" Sep 10 23:43:07.377237 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:07.377156 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9a4328b5-5156-41b7-8d72-9a570bdeac3e-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9fv8j\" (UID: \"9a4328b5-5156-41b7-8d72-9a570bdeac3e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9fv8j" Sep 10 23:43:07.377237 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:07.377191 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4c7f94f5-db28-432e-984b-27c2326a3243-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d758spt\" (UID: \"4c7f94f5-db28-432e-984b-27c2326a3243\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d758spt" Sep 10 23:43:07.377237 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:07.377218 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9a4328b5-5156-41b7-8d72-9a570bdeac3e-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9fv8j\" (UID: \"9a4328b5-5156-41b7-8d72-9a570bdeac3e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9fv8j" Sep 10 23:43:07.377397 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:07.377307 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhfgr\" (UniqueName: \"kubernetes.io/projected/9a4328b5-5156-41b7-8d72-9a570bdeac3e-kube-api-access-nhfgr\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9fv8j\" (UID: \"9a4328b5-5156-41b7-8d72-9a570bdeac3e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9fv8j" Sep 10 23:43:07.377513 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:07.377494 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4c7f94f5-db28-432e-984b-27c2326a3243-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d758spt\" (UID: \"4c7f94f5-db28-432e-984b-27c2326a3243\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d758spt" Sep 10 23:43:07.377548 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:07.377508 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4c7f94f5-db28-432e-984b-27c2326a3243-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d758spt\" (UID: \"4c7f94f5-db28-432e-984b-27c2326a3243\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d758spt" Sep 10 23:43:07.384814 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:07.384794 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbthg\" (UniqueName: \"kubernetes.io/projected/4c7f94f5-db28-432e-984b-27c2326a3243-kube-api-access-dbthg\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d758spt\" (UID: \"4c7f94f5-db28-432e-984b-27c2326a3243\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d758spt" Sep 10 23:43:07.445291 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:07.445231 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avmltz"] Sep 10 23:43:07.448931 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:07.448913 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avmltz" Sep 10 23:43:07.459914 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:07.459892 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avmltz"] Sep 10 23:43:07.464932 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:07.464912 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d758spt" Sep 10 23:43:07.478590 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:07.478563 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gngcz\" (UniqueName: \"kubernetes.io/projected/b2275045-789c-463e-b411-83006161901b-kube-api-access-gngcz\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6brhqg\" (UID: \"b2275045-789c-463e-b411-83006161901b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6brhqg" Sep 10 23:43:07.478716 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:07.478617 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9a4328b5-5156-41b7-8d72-9a570bdeac3e-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9fv8j\" (UID: \"9a4328b5-5156-41b7-8d72-9a570bdeac3e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9fv8j" Sep 10 23:43:07.478777 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:07.478720 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b2275045-789c-463e-b411-83006161901b-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6brhqg\" (UID: \"b2275045-789c-463e-b411-83006161901b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6brhqg" Sep 10 23:43:07.478831 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:07.478773 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9a4328b5-5156-41b7-8d72-9a570bdeac3e-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9fv8j\" (UID: \"9a4328b5-5156-41b7-8d72-9a570bdeac3e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9fv8j" Sep 10 23:43:07.478831 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:07.478803 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b2275045-789c-463e-b411-83006161901b-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6brhqg\" (UID: \"b2275045-789c-463e-b411-83006161901b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6brhqg" Sep 10 23:43:07.478933 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:07.478906 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhfgr\" (UniqueName: \"kubernetes.io/projected/9a4328b5-5156-41b7-8d72-9a570bdeac3e-kube-api-access-nhfgr\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9fv8j\" (UID: \"9a4328b5-5156-41b7-8d72-9a570bdeac3e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9fv8j" Sep 10 23:43:07.478987 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:07.478965 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9a4328b5-5156-41b7-8d72-9a570bdeac3e-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9fv8j\" (UID: \"9a4328b5-5156-41b7-8d72-9a570bdeac3e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9fv8j" Sep 10 23:43:07.479231 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:07.479206 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9a4328b5-5156-41b7-8d72-9a570bdeac3e-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9fv8j\" (UID: \"9a4328b5-5156-41b7-8d72-9a570bdeac3e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9fv8j" Sep 10 23:43:07.486539 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:07.486515 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhfgr\" (UniqueName: \"kubernetes.io/projected/9a4328b5-5156-41b7-8d72-9a570bdeac3e-kube-api-access-nhfgr\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9fv8j\" (UID: \"9a4328b5-5156-41b7-8d72-9a570bdeac3e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9fv8j" Sep 10 23:43:07.555723 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:07.555675 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9fv8j" Sep 10 23:43:07.580335 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:07.580297 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/778e1ad0-62ce-43e5-80c0-a1a3652f60e8-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avmltz\" (UID: \"778e1ad0-62ce-43e5-80c0-a1a3652f60e8\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avmltz" Sep 10 23:43:07.580450 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:07.580351 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/778e1ad0-62ce-43e5-80c0-a1a3652f60e8-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avmltz\" (UID: \"778e1ad0-62ce-43e5-80c0-a1a3652f60e8\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avmltz" Sep 10 23:43:07.580526 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:07.580505 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gngcz\" (UniqueName: \"kubernetes.io/projected/b2275045-789c-463e-b411-83006161901b-kube-api-access-gngcz\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6brhqg\" (UID: \"b2275045-789c-463e-b411-83006161901b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6brhqg" Sep 10 23:43:07.580581 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:07.580562 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b2275045-789c-463e-b411-83006161901b-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6brhqg\" (UID: \"b2275045-789c-463e-b411-83006161901b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6brhqg" Sep 10 23:43:07.580634 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:07.580588 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b2275045-789c-463e-b411-83006161901b-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6brhqg\" (UID: \"b2275045-789c-463e-b411-83006161901b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6brhqg" Sep 10 23:43:07.580634 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:07.580608 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lckt\" (UniqueName: \"kubernetes.io/projected/778e1ad0-62ce-43e5-80c0-a1a3652f60e8-kube-api-access-6lckt\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avmltz\" (UID: \"778e1ad0-62ce-43e5-80c0-a1a3652f60e8\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avmltz" Sep 10 23:43:07.580941 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:07.580925 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b2275045-789c-463e-b411-83006161901b-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6brhqg\" (UID: \"b2275045-789c-463e-b411-83006161901b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6brhqg" Sep 10 23:43:07.581001 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:07.580947 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b2275045-789c-463e-b411-83006161901b-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6brhqg\" (UID: \"b2275045-789c-463e-b411-83006161901b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6brhqg" Sep 10 23:43:07.589031 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:07.589006 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gngcz\" (UniqueName: \"kubernetes.io/projected/b2275045-789c-463e-b411-83006161901b-kube-api-access-gngcz\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6brhqg\" (UID: \"b2275045-789c-463e-b411-83006161901b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6brhqg" Sep 10 23:43:07.655360 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:07.655320 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6brhqg" Sep 10 23:43:07.682180 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:07.682141 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lckt\" (UniqueName: \"kubernetes.io/projected/778e1ad0-62ce-43e5-80c0-a1a3652f60e8-kube-api-access-6lckt\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avmltz\" (UID: \"778e1ad0-62ce-43e5-80c0-a1a3652f60e8\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avmltz" Sep 10 23:43:07.682302 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:07.682195 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/778e1ad0-62ce-43e5-80c0-a1a3652f60e8-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avmltz\" (UID: \"778e1ad0-62ce-43e5-80c0-a1a3652f60e8\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avmltz" Sep 10 23:43:07.682302 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:07.682227 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/778e1ad0-62ce-43e5-80c0-a1a3652f60e8-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avmltz\" (UID: \"778e1ad0-62ce-43e5-80c0-a1a3652f60e8\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avmltz" Sep 10 23:43:07.682650 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:07.682616 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/778e1ad0-62ce-43e5-80c0-a1a3652f60e8-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avmltz\" (UID: \"778e1ad0-62ce-43e5-80c0-a1a3652f60e8\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avmltz" Sep 10 23:43:07.682817 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:07.682796 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/778e1ad0-62ce-43e5-80c0-a1a3652f60e8-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avmltz\" (UID: \"778e1ad0-62ce-43e5-80c0-a1a3652f60e8\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avmltz" Sep 10 23:43:07.690288 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:07.690262 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lckt\" (UniqueName: \"kubernetes.io/projected/778e1ad0-62ce-43e5-80c0-a1a3652f60e8-kube-api-access-6lckt\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avmltz\" (UID: \"778e1ad0-62ce-43e5-80c0-a1a3652f60e8\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avmltz" Sep 10 23:43:07.739852 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:07.739822 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d758spt"] Sep 10 23:43:07.741627 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:43:07.741597 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c7f94f5_db28_432e_984b_27c2326a3243.slice/crio-bdea552a8d8913974aba4ae20f1bb2b973426a081bfd887b85f09b7756a9f7fb WatchSource:0}: Error finding container bdea552a8d8913974aba4ae20f1bb2b973426a081bfd887b85f09b7756a9f7fb: Status 404 returned error can't find the container with id bdea552a8d8913974aba4ae20f1bb2b973426a081bfd887b85f09b7756a9f7fb Sep 10 23:43:07.758955 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:07.758933 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avmltz" Sep 10 23:43:07.829401 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:07.829377 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9fv8j"] Sep 10 23:43:07.831596 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:43:07.831570 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a4328b5_5156_41b7_8d72_9a570bdeac3e.slice/crio-1f5ae2d1ce720c5219c8d738e490efe737af2c4d27dd0bce059998aeacbd3578 WatchSource:0}: Error finding container 1f5ae2d1ce720c5219c8d738e490efe737af2c4d27dd0bce059998aeacbd3578: Status 404 returned error can't find the container with id 1f5ae2d1ce720c5219c8d738e490efe737af2c4d27dd0bce059998aeacbd3578 Sep 10 23:43:07.928476 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:07.928450 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6brhqg"] Sep 10 23:43:07.940200 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:43:07.940172 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2275045_789c_463e_b411_83006161901b.slice/crio-ea0967ce6a943f8411ab10f98826704a14c59505059792153be349fcca05ae74 WatchSource:0}: Error finding container ea0967ce6a943f8411ab10f98826704a14c59505059792153be349fcca05ae74: Status 404 returned error can't find the container with id ea0967ce6a943f8411ab10f98826704a14c59505059792153be349fcca05ae74 Sep 10 23:43:08.006605 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:08.006565 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6brhqg" event={"ID":"b2275045-789c-463e-b411-83006161901b","Type":"ContainerStarted","Data":"ea0967ce6a943f8411ab10f98826704a14c59505059792153be349fcca05ae74"} Sep 10 23:43:08.008450 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:08.008421 2574 generic.go:356] "Generic (PLEG): container finished" podID="4c7f94f5-db28-432e-984b-27c2326a3243" containerID="883e8d2a5bc30b85283c0a40842ed35ef4cbdaa92cdf696c2b9e20f3198b0c06" exitCode=0 Sep 10 23:43:08.008556 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:08.008504 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d758spt" event={"ID":"4c7f94f5-db28-432e-984b-27c2326a3243","Type":"ContainerDied","Data":"883e8d2a5bc30b85283c0a40842ed35ef4cbdaa92cdf696c2b9e20f3198b0c06"} Sep 10 23:43:08.008617 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:08.008556 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d758spt" event={"ID":"4c7f94f5-db28-432e-984b-27c2326a3243","Type":"ContainerStarted","Data":"bdea552a8d8913974aba4ae20f1bb2b973426a081bfd887b85f09b7756a9f7fb"} Sep 10 23:43:08.010520 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:08.010497 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9fv8j" event={"ID":"9a4328b5-5156-41b7-8d72-9a570bdeac3e","Type":"ContainerStarted","Data":"ada58e7f3282875f1c649700033a9eb5c26ee17a2a66b5d760bf5011557fa3f0"} Sep 10 23:43:08.010607 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:08.010539 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9fv8j" event={"ID":"9a4328b5-5156-41b7-8d72-9a570bdeac3e","Type":"ContainerStarted","Data":"1f5ae2d1ce720c5219c8d738e490efe737af2c4d27dd0bce059998aeacbd3578"} Sep 10 23:43:08.034338 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:08.034305 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avmltz"] Sep 10 23:43:08.043774 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:43:08.043735 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod778e1ad0_62ce_43e5_80c0_a1a3652f60e8.slice/crio-44531fc159b1442eefc020a72d0e0a773274d625fcc98ce224a50cbd219d67f2 WatchSource:0}: Error finding container 44531fc159b1442eefc020a72d0e0a773274d625fcc98ce224a50cbd219d67f2: Status 404 returned error can't find the container with id 44531fc159b1442eefc020a72d0e0a773274d625fcc98ce224a50cbd219d67f2 Sep 10 23:43:09.017677 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:09.017627 2574 generic.go:356] "Generic (PLEG): container finished" podID="b2275045-789c-463e-b411-83006161901b" containerID="f2fef40b4251e82f2069b66a3ed2824d326bcefd5dc98076654561f86bf24928" exitCode=0 Sep 10 23:43:09.017677 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:09.017670 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6brhqg" event={"ID":"b2275045-789c-463e-b411-83006161901b","Type":"ContainerDied","Data":"f2fef40b4251e82f2069b66a3ed2824d326bcefd5dc98076654561f86bf24928"} Sep 10 23:43:09.019697 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:09.019676 2574 generic.go:356] "Generic (PLEG): container finished" podID="4c7f94f5-db28-432e-984b-27c2326a3243" containerID="b308311404c6fe670deef6f477c4458b97c2d063ca75a694c85bc8adb677e2ce" exitCode=0 Sep 10 23:43:09.019789 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:09.019763 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d758spt" event={"ID":"4c7f94f5-db28-432e-984b-27c2326a3243","Type":"ContainerDied","Data":"b308311404c6fe670deef6f477c4458b97c2d063ca75a694c85bc8adb677e2ce"} Sep 10 23:43:09.022073 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:09.022050 2574 generic.go:356] "Generic (PLEG): container finished" podID="9a4328b5-5156-41b7-8d72-9a570bdeac3e" containerID="ada58e7f3282875f1c649700033a9eb5c26ee17a2a66b5d760bf5011557fa3f0" exitCode=0 Sep 10 23:43:09.022073 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:09.022073 2574 generic.go:356] "Generic (PLEG): container finished" podID="9a4328b5-5156-41b7-8d72-9a570bdeac3e" containerID="cc4905094e3e18a547b14825159bc1651813f92f8bf5963956bc5c76fa088b30" exitCode=0 Sep 10 23:43:09.022346 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:09.022134 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9fv8j" event={"ID":"9a4328b5-5156-41b7-8d72-9a570bdeac3e","Type":"ContainerDied","Data":"ada58e7f3282875f1c649700033a9eb5c26ee17a2a66b5d760bf5011557fa3f0"} Sep 10 23:43:09.022346 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:09.022172 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9fv8j" event={"ID":"9a4328b5-5156-41b7-8d72-9a570bdeac3e","Type":"ContainerDied","Data":"cc4905094e3e18a547b14825159bc1651813f92f8bf5963956bc5c76fa088b30"} Sep 10 23:43:09.023918 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:09.023897 2574 generic.go:356] "Generic (PLEG): container finished" podID="778e1ad0-62ce-43e5-80c0-a1a3652f60e8" containerID="75fe1279db8305c034e79c0f96ffb269ce16d167cdd778e711e9b7cf1977328d" exitCode=0 Sep 10 23:43:09.024001 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:09.023944 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avmltz" event={"ID":"778e1ad0-62ce-43e5-80c0-a1a3652f60e8","Type":"ContainerDied","Data":"75fe1279db8305c034e79c0f96ffb269ce16d167cdd778e711e9b7cf1977328d"} Sep 10 23:43:09.024001 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:09.023984 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avmltz" event={"ID":"778e1ad0-62ce-43e5-80c0-a1a3652f60e8","Type":"ContainerStarted","Data":"44531fc159b1442eefc020a72d0e0a773274d625fcc98ce224a50cbd219d67f2"} Sep 10 23:43:10.032182 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:10.032145 2574 generic.go:356] "Generic (PLEG): container finished" podID="b2275045-789c-463e-b411-83006161901b" containerID="525e4a50f86b922eecf4dae937e82be7235b4db60c229776784bf550769f4d11" exitCode=0 Sep 10 23:43:10.032635 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:10.032196 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6brhqg" event={"ID":"b2275045-789c-463e-b411-83006161901b","Type":"ContainerDied","Data":"525e4a50f86b922eecf4dae937e82be7235b4db60c229776784bf550769f4d11"} Sep 10 23:43:10.034525 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:10.034500 2574 generic.go:356] "Generic (PLEG): container finished" podID="4c7f94f5-db28-432e-984b-27c2326a3243" containerID="a49dc145182cb3c931335112fe23d6ed94d7cd10fe4dc9fd5d860e69c756d6ed" exitCode=0 Sep 10 23:43:10.034616 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:10.034581 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d758spt" event={"ID":"4c7f94f5-db28-432e-984b-27c2326a3243","Type":"ContainerDied","Data":"a49dc145182cb3c931335112fe23d6ed94d7cd10fe4dc9fd5d860e69c756d6ed"} Sep 10 23:43:10.044061 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:10.044037 2574 generic.go:356] "Generic (PLEG): container finished" podID="9a4328b5-5156-41b7-8d72-9a570bdeac3e" containerID="6345fb1437c55170b1789e73a5449468078df993ae2bcd5dfc38b5a76c2155f1" exitCode=0 Sep 10 23:43:10.044154 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:10.044117 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9fv8j" event={"ID":"9a4328b5-5156-41b7-8d72-9a570bdeac3e","Type":"ContainerDied","Data":"6345fb1437c55170b1789e73a5449468078df993ae2bcd5dfc38b5a76c2155f1"} Sep 10 23:43:10.045886 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:10.045862 2574 generic.go:356] "Generic (PLEG): container finished" podID="778e1ad0-62ce-43e5-80c0-a1a3652f60e8" containerID="f6230c620a72644c038bf71dc01fe457c1a89bbaf6c9dd193a7461f333dc9024" exitCode=0 Sep 10 23:43:10.045989 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:10.045952 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avmltz" event={"ID":"778e1ad0-62ce-43e5-80c0-a1a3652f60e8","Type":"ContainerDied","Data":"f6230c620a72644c038bf71dc01fe457c1a89bbaf6c9dd193a7461f333dc9024"} Sep 10 23:43:11.052899 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:11.052867 2574 generic.go:356] "Generic (PLEG): container finished" podID="778e1ad0-62ce-43e5-80c0-a1a3652f60e8" containerID="455778b2dac728c600a7e6bd196e193b6ae50a7bc5bcb9726c4b5591e6887188" exitCode=0 Sep 10 23:43:11.053307 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:11.052951 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avmltz" event={"ID":"778e1ad0-62ce-43e5-80c0-a1a3652f60e8","Type":"ContainerDied","Data":"455778b2dac728c600a7e6bd196e193b6ae50a7bc5bcb9726c4b5591e6887188"} Sep 10 23:43:11.054934 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:11.054908 2574 generic.go:356] "Generic (PLEG): container finished" podID="b2275045-789c-463e-b411-83006161901b" containerID="31aa303b5eab79a6355b5ee5f328961d3e52ea0367182890d0f6e9a5e55ae635" exitCode=0 Sep 10 23:43:11.055042 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:11.054966 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6brhqg" event={"ID":"b2275045-789c-463e-b411-83006161901b","Type":"ContainerDied","Data":"31aa303b5eab79a6355b5ee5f328961d3e52ea0367182890d0f6e9a5e55ae635"} Sep 10 23:43:11.211992 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:11.211966 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9fv8j" Sep 10 23:43:11.214973 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:11.214951 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d758spt" Sep 10 23:43:11.317491 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:11.317405 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dbthg\" (UniqueName: \"kubernetes.io/projected/4c7f94f5-db28-432e-984b-27c2326a3243-kube-api-access-dbthg\") pod \"4c7f94f5-db28-432e-984b-27c2326a3243\" (UID: \"4c7f94f5-db28-432e-984b-27c2326a3243\") " Sep 10 23:43:11.317491 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:11.317441 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4c7f94f5-db28-432e-984b-27c2326a3243-util\") pod \"4c7f94f5-db28-432e-984b-27c2326a3243\" (UID: \"4c7f94f5-db28-432e-984b-27c2326a3243\") " Sep 10 23:43:11.317491 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:11.317462 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9a4328b5-5156-41b7-8d72-9a570bdeac3e-util\") pod \"9a4328b5-5156-41b7-8d72-9a570bdeac3e\" (UID: \"9a4328b5-5156-41b7-8d72-9a570bdeac3e\") " Sep 10 23:43:11.317802 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:11.317505 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4c7f94f5-db28-432e-984b-27c2326a3243-bundle\") pod \"4c7f94f5-db28-432e-984b-27c2326a3243\" (UID: \"4c7f94f5-db28-432e-984b-27c2326a3243\") " Sep 10 23:43:11.317802 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:11.317537 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9a4328b5-5156-41b7-8d72-9a570bdeac3e-bundle\") pod \"9a4328b5-5156-41b7-8d72-9a570bdeac3e\" (UID: \"9a4328b5-5156-41b7-8d72-9a570bdeac3e\") " Sep 10 23:43:11.317802 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:11.317566 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nhfgr\" (UniqueName: \"kubernetes.io/projected/9a4328b5-5156-41b7-8d72-9a570bdeac3e-kube-api-access-nhfgr\") pod \"9a4328b5-5156-41b7-8d72-9a570bdeac3e\" (UID: \"9a4328b5-5156-41b7-8d72-9a570bdeac3e\") " Sep 10 23:43:11.318348 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:11.318122 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c7f94f5-db28-432e-984b-27c2326a3243-bundle" (OuterVolumeSpecName: "bundle") pod "4c7f94f5-db28-432e-984b-27c2326a3243" (UID: "4c7f94f5-db28-432e-984b-27c2326a3243"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:43:11.318348 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:11.318164 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a4328b5-5156-41b7-8d72-9a570bdeac3e-bundle" (OuterVolumeSpecName: "bundle") pod "9a4328b5-5156-41b7-8d72-9a570bdeac3e" (UID: "9a4328b5-5156-41b7-8d72-9a570bdeac3e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:43:11.319850 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:11.319823 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a4328b5-5156-41b7-8d72-9a570bdeac3e-kube-api-access-nhfgr" (OuterVolumeSpecName: "kube-api-access-nhfgr") pod "9a4328b5-5156-41b7-8d72-9a570bdeac3e" (UID: "9a4328b5-5156-41b7-8d72-9a570bdeac3e"). InnerVolumeSpecName "kube-api-access-nhfgr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 23:43:11.319948 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:11.319857 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c7f94f5-db28-432e-984b-27c2326a3243-kube-api-access-dbthg" (OuterVolumeSpecName: "kube-api-access-dbthg") pod "4c7f94f5-db28-432e-984b-27c2326a3243" (UID: "4c7f94f5-db28-432e-984b-27c2326a3243"). InnerVolumeSpecName "kube-api-access-dbthg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 23:43:11.323399 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:11.323376 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a4328b5-5156-41b7-8d72-9a570bdeac3e-util" (OuterVolumeSpecName: "util") pod "9a4328b5-5156-41b7-8d72-9a570bdeac3e" (UID: "9a4328b5-5156-41b7-8d72-9a570bdeac3e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:43:11.323483 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:11.323417 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c7f94f5-db28-432e-984b-27c2326a3243-util" (OuterVolumeSpecName: "util") pod "4c7f94f5-db28-432e-984b-27c2326a3243" (UID: "4c7f94f5-db28-432e-984b-27c2326a3243"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:43:11.419056 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:11.419019 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbthg\" (UniqueName: \"kubernetes.io/projected/4c7f94f5-db28-432e-984b-27c2326a3243-kube-api-access-dbthg\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:43:11.419056 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:11.419052 2574 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4c7f94f5-db28-432e-984b-27c2326a3243-util\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:43:11.419056 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:11.419063 2574 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9a4328b5-5156-41b7-8d72-9a570bdeac3e-util\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:43:11.419266 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:11.419071 2574 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4c7f94f5-db28-432e-984b-27c2326a3243-bundle\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:43:11.419266 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:11.419078 2574 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9a4328b5-5156-41b7-8d72-9a570bdeac3e-bundle\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:43:11.419266 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:11.419086 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhfgr\" (UniqueName: \"kubernetes.io/projected/9a4328b5-5156-41b7-8d72-9a570bdeac3e-kube-api-access-nhfgr\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:43:12.061962 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:12.061922 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9fv8j" Sep 10 23:43:12.061962 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:12.061942 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9fv8j" event={"ID":"9a4328b5-5156-41b7-8d72-9a570bdeac3e","Type":"ContainerDied","Data":"1f5ae2d1ce720c5219c8d738e490efe737af2c4d27dd0bce059998aeacbd3578"} Sep 10 23:43:12.062476 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:12.061995 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1f5ae2d1ce720c5219c8d738e490efe737af2c4d27dd0bce059998aeacbd3578" Sep 10 23:43:12.063959 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:12.063938 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d758spt" Sep 10 23:43:12.064102 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:12.063955 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d758spt" event={"ID":"4c7f94f5-db28-432e-984b-27c2326a3243","Type":"ContainerDied","Data":"bdea552a8d8913974aba4ae20f1bb2b973426a081bfd887b85f09b7756a9f7fb"} Sep 10 23:43:12.064102 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:12.063993 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bdea552a8d8913974aba4ae20f1bb2b973426a081bfd887b85f09b7756a9f7fb" Sep 10 23:43:12.222876 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:12.222853 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avmltz" Sep 10 23:43:12.225691 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:12.225671 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6brhqg" Sep 10 23:43:12.326964 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:12.326881 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b2275045-789c-463e-b411-83006161901b-bundle\") pod \"b2275045-789c-463e-b411-83006161901b\" (UID: \"b2275045-789c-463e-b411-83006161901b\") " Sep 10 23:43:12.326964 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:12.326931 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gngcz\" (UniqueName: \"kubernetes.io/projected/b2275045-789c-463e-b411-83006161901b-kube-api-access-gngcz\") pod \"b2275045-789c-463e-b411-83006161901b\" (UID: \"b2275045-789c-463e-b411-83006161901b\") " Sep 10 23:43:12.327173 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:12.326975 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/778e1ad0-62ce-43e5-80c0-a1a3652f60e8-bundle\") pod \"778e1ad0-62ce-43e5-80c0-a1a3652f60e8\" (UID: \"778e1ad0-62ce-43e5-80c0-a1a3652f60e8\") " Sep 10 23:43:12.327173 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:12.327016 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/778e1ad0-62ce-43e5-80c0-a1a3652f60e8-util\") pod \"778e1ad0-62ce-43e5-80c0-a1a3652f60e8\" (UID: \"778e1ad0-62ce-43e5-80c0-a1a3652f60e8\") " Sep 10 23:43:12.327173 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:12.327041 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6lckt\" (UniqueName: \"kubernetes.io/projected/778e1ad0-62ce-43e5-80c0-a1a3652f60e8-kube-api-access-6lckt\") pod \"778e1ad0-62ce-43e5-80c0-a1a3652f60e8\" (UID: \"778e1ad0-62ce-43e5-80c0-a1a3652f60e8\") " Sep 10 23:43:12.327173 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:12.327102 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b2275045-789c-463e-b411-83006161901b-util\") pod \"b2275045-789c-463e-b411-83006161901b\" (UID: \"b2275045-789c-463e-b411-83006161901b\") " Sep 10 23:43:12.327555 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:12.327529 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2275045-789c-463e-b411-83006161901b-bundle" (OuterVolumeSpecName: "bundle") pod "b2275045-789c-463e-b411-83006161901b" (UID: "b2275045-789c-463e-b411-83006161901b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:43:12.327845 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:12.327820 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/778e1ad0-62ce-43e5-80c0-a1a3652f60e8-bundle" (OuterVolumeSpecName: "bundle") pod "778e1ad0-62ce-43e5-80c0-a1a3652f60e8" (UID: "778e1ad0-62ce-43e5-80c0-a1a3652f60e8"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:43:12.329153 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:12.329127 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2275045-789c-463e-b411-83006161901b-kube-api-access-gngcz" (OuterVolumeSpecName: "kube-api-access-gngcz") pod "b2275045-789c-463e-b411-83006161901b" (UID: "b2275045-789c-463e-b411-83006161901b"). InnerVolumeSpecName "kube-api-access-gngcz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 23:43:12.329153 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:12.329143 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/778e1ad0-62ce-43e5-80c0-a1a3652f60e8-kube-api-access-6lckt" (OuterVolumeSpecName: "kube-api-access-6lckt") pod "778e1ad0-62ce-43e5-80c0-a1a3652f60e8" (UID: "778e1ad0-62ce-43e5-80c0-a1a3652f60e8"). InnerVolumeSpecName "kube-api-access-6lckt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 23:43:12.332683 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:12.332664 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/778e1ad0-62ce-43e5-80c0-a1a3652f60e8-util" (OuterVolumeSpecName: "util") pod "778e1ad0-62ce-43e5-80c0-a1a3652f60e8" (UID: "778e1ad0-62ce-43e5-80c0-a1a3652f60e8"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:43:12.335616 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:12.335592 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2275045-789c-463e-b411-83006161901b-util" (OuterVolumeSpecName: "util") pod "b2275045-789c-463e-b411-83006161901b" (UID: "b2275045-789c-463e-b411-83006161901b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:43:12.427787 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:12.427765 2574 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/778e1ad0-62ce-43e5-80c0-a1a3652f60e8-util\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:43:12.427787 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:12.427784 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6lckt\" (UniqueName: \"kubernetes.io/projected/778e1ad0-62ce-43e5-80c0-a1a3652f60e8-kube-api-access-6lckt\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:43:12.427901 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:12.427795 2574 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b2275045-789c-463e-b411-83006161901b-util\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:43:12.427901 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:12.427804 2574 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b2275045-789c-463e-b411-83006161901b-bundle\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:43:12.427901 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:12.427813 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gngcz\" (UniqueName: \"kubernetes.io/projected/b2275045-789c-463e-b411-83006161901b-kube-api-access-gngcz\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:43:12.427901 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:12.427821 2574 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/778e1ad0-62ce-43e5-80c0-a1a3652f60e8-bundle\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:43:13.070923 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:13.070895 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avmltz" Sep 10 23:43:13.071369 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:13.070928 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847avmltz" event={"ID":"778e1ad0-62ce-43e5-80c0-a1a3652f60e8","Type":"ContainerDied","Data":"44531fc159b1442eefc020a72d0e0a773274d625fcc98ce224a50cbd219d67f2"} Sep 10 23:43:13.071369 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:13.070977 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="44531fc159b1442eefc020a72d0e0a773274d625fcc98ce224a50cbd219d67f2" Sep 10 23:43:13.072994 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:13.072976 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6brhqg" Sep 10 23:43:13.072994 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:13.072983 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6brhqg" event={"ID":"b2275045-789c-463e-b411-83006161901b","Type":"ContainerDied","Data":"ea0967ce6a943f8411ab10f98826704a14c59505059792153be349fcca05ae74"} Sep 10 23:43:13.073180 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:13.073021 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ea0967ce6a943f8411ab10f98826704a14c59505059792153be349fcca05ae74" Sep 10 23:43:19.577434 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:19.577395 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-sv4gw"] Sep 10 23:43:19.577855 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:19.577807 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2275045-789c-463e-b411-83006161901b" containerName="util" Sep 10 23:43:19.577855 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:19.577823 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2275045-789c-463e-b411-83006161901b" containerName="util" Sep 10 23:43:19.577855 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:19.577834 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c7f94f5-db28-432e-984b-27c2326a3243" containerName="extract" Sep 10 23:43:19.577855 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:19.577839 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c7f94f5-db28-432e-984b-27c2326a3243" containerName="extract" Sep 10 23:43:19.577855 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:19.577851 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a4328b5-5156-41b7-8d72-9a570bdeac3e" containerName="util" Sep 10 23:43:19.577855 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:19.577857 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a4328b5-5156-41b7-8d72-9a570bdeac3e" containerName="util" Sep 10 23:43:19.578045 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:19.577863 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="778e1ad0-62ce-43e5-80c0-a1a3652f60e8" containerName="util" Sep 10 23:43:19.578045 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:19.577867 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="778e1ad0-62ce-43e5-80c0-a1a3652f60e8" containerName="util" Sep 10 23:43:19.578045 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:19.577876 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="778e1ad0-62ce-43e5-80c0-a1a3652f60e8" containerName="pull" Sep 10 23:43:19.578045 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:19.577881 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="778e1ad0-62ce-43e5-80c0-a1a3652f60e8" containerName="pull" Sep 10 23:43:19.578045 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:19.577892 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a4328b5-5156-41b7-8d72-9a570bdeac3e" containerName="extract" Sep 10 23:43:19.578045 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:19.577901 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a4328b5-5156-41b7-8d72-9a570bdeac3e" containerName="extract" Sep 10 23:43:19.578045 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:19.577909 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c7f94f5-db28-432e-984b-27c2326a3243" containerName="util" Sep 10 23:43:19.578045 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:19.577913 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c7f94f5-db28-432e-984b-27c2326a3243" containerName="util" Sep 10 23:43:19.578045 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:19.577920 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2275045-789c-463e-b411-83006161901b" containerName="pull" Sep 10 23:43:19.578045 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:19.577925 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2275045-789c-463e-b411-83006161901b" containerName="pull" Sep 10 23:43:19.578045 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:19.577933 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a4328b5-5156-41b7-8d72-9a570bdeac3e" containerName="pull" Sep 10 23:43:19.578045 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:19.577938 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a4328b5-5156-41b7-8d72-9a570bdeac3e" containerName="pull" Sep 10 23:43:19.578045 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:19.577952 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c7f94f5-db28-432e-984b-27c2326a3243" containerName="pull" Sep 10 23:43:19.578045 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:19.577957 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c7f94f5-db28-432e-984b-27c2326a3243" containerName="pull" Sep 10 23:43:19.578045 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:19.577963 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2275045-789c-463e-b411-83006161901b" containerName="extract" Sep 10 23:43:19.578045 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:19.577970 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2275045-789c-463e-b411-83006161901b" containerName="extract" Sep 10 23:43:19.578045 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:19.577980 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="778e1ad0-62ce-43e5-80c0-a1a3652f60e8" containerName="extract" Sep 10 23:43:19.578045 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:19.577988 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="778e1ad0-62ce-43e5-80c0-a1a3652f60e8" containerName="extract" Sep 10 23:43:19.578045 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:19.578043 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a4328b5-5156-41b7-8d72-9a570bdeac3e" containerName="extract" Sep 10 23:43:19.578548 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:19.578053 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="778e1ad0-62ce-43e5-80c0-a1a3652f60e8" containerName="extract" Sep 10 23:43:19.578548 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:19.578061 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2275045-789c-463e-b411-83006161901b" containerName="extract" Sep 10 23:43:19.578548 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:19.578069 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c7f94f5-db28-432e-984b-27c2326a3243" containerName="extract" Sep 10 23:43:19.582540 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:19.582522 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-sv4gw" Sep 10 23:43:19.586043 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:19.586022 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-5lmkm\"" Sep 10 23:43:19.596629 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:19.596609 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Sep 10 23:43:19.596786 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:19.596707 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Sep 10 23:43:19.596864 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:19.596846 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Sep 10 23:43:19.599292 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:19.599271 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-sv4gw"] Sep 10 23:43:19.694587 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:19.694549 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8d79j\" (UniqueName: \"kubernetes.io/projected/c7ab7425-af9d-4839-aba0-e4ca2d298860-kube-api-access-8d79j\") pod \"dns-operator-controller-manager-7b84cd7c9c-sv4gw\" (UID: \"c7ab7425-af9d-4839-aba0-e4ca2d298860\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-sv4gw" Sep 10 23:43:19.795223 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:19.795185 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8d79j\" (UniqueName: \"kubernetes.io/projected/c7ab7425-af9d-4839-aba0-e4ca2d298860-kube-api-access-8d79j\") pod \"dns-operator-controller-manager-7b84cd7c9c-sv4gw\" (UID: \"c7ab7425-af9d-4839-aba0-e4ca2d298860\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-sv4gw" Sep 10 23:43:19.824653 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:19.824624 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8d79j\" (UniqueName: \"kubernetes.io/projected/c7ab7425-af9d-4839-aba0-e4ca2d298860-kube-api-access-8d79j\") pod \"dns-operator-controller-manager-7b84cd7c9c-sv4gw\" (UID: \"c7ab7425-af9d-4839-aba0-e4ca2d298860\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-sv4gw" Sep 10 23:43:19.893224 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:19.893156 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-sv4gw" Sep 10 23:43:20.010169 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:20.010109 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-778d864cdc-v6xfr" podUID="402e5517-4211-49e2-95a2-d9ea715f4a6c" containerName="console" containerID="cri-o://6c4f1d40f39a74469b215ff17af4efd2b6e5cfcb758ac41ae3bef268b495922a" gracePeriod=15 Sep 10 23:43:20.178011 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:20.177841 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-sv4gw"] Sep 10 23:43:20.242186 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:20.242162 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-778d864cdc-v6xfr" Sep 10 23:43:20.399618 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:20.399584 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/402e5517-4211-49e2-95a2-d9ea715f4a6c-service-ca\") pod \"402e5517-4211-49e2-95a2-d9ea715f4a6c\" (UID: \"402e5517-4211-49e2-95a2-d9ea715f4a6c\") " Sep 10 23:43:20.399618 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:20.399623 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/402e5517-4211-49e2-95a2-d9ea715f4a6c-console-config\") pod \"402e5517-4211-49e2-95a2-d9ea715f4a6c\" (UID: \"402e5517-4211-49e2-95a2-d9ea715f4a6c\") " Sep 10 23:43:20.399880 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:20.399676 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/402e5517-4211-49e2-95a2-d9ea715f4a6c-console-oauth-config\") pod \"402e5517-4211-49e2-95a2-d9ea715f4a6c\" (UID: \"402e5517-4211-49e2-95a2-d9ea715f4a6c\") " Sep 10 23:43:20.399880 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:20.399704 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/402e5517-4211-49e2-95a2-d9ea715f4a6c-trusted-ca-bundle\") pod \"402e5517-4211-49e2-95a2-d9ea715f4a6c\" (UID: \"402e5517-4211-49e2-95a2-d9ea715f4a6c\") " Sep 10 23:43:20.399880 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:20.399748 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/402e5517-4211-49e2-95a2-d9ea715f4a6c-oauth-serving-cert\") pod \"402e5517-4211-49e2-95a2-d9ea715f4a6c\" (UID: \"402e5517-4211-49e2-95a2-d9ea715f4a6c\") " Sep 10 23:43:20.399880 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:20.399780 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vzzmh\" (UniqueName: \"kubernetes.io/projected/402e5517-4211-49e2-95a2-d9ea715f4a6c-kube-api-access-vzzmh\") pod \"402e5517-4211-49e2-95a2-d9ea715f4a6c\" (UID: \"402e5517-4211-49e2-95a2-d9ea715f4a6c\") " Sep 10 23:43:20.399880 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:20.399853 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/402e5517-4211-49e2-95a2-d9ea715f4a6c-console-serving-cert\") pod \"402e5517-4211-49e2-95a2-d9ea715f4a6c\" (UID: \"402e5517-4211-49e2-95a2-d9ea715f4a6c\") " Sep 10 23:43:20.400131 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:20.400087 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/402e5517-4211-49e2-95a2-d9ea715f4a6c-service-ca" (OuterVolumeSpecName: "service-ca") pod "402e5517-4211-49e2-95a2-d9ea715f4a6c" (UID: "402e5517-4211-49e2-95a2-d9ea715f4a6c"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 23:43:20.400183 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:20.400127 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/402e5517-4211-49e2-95a2-d9ea715f4a6c-console-config" (OuterVolumeSpecName: "console-config") pod "402e5517-4211-49e2-95a2-d9ea715f4a6c" (UID: "402e5517-4211-49e2-95a2-d9ea715f4a6c"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 23:43:20.400244 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:20.400207 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/402e5517-4211-49e2-95a2-d9ea715f4a6c-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "402e5517-4211-49e2-95a2-d9ea715f4a6c" (UID: "402e5517-4211-49e2-95a2-d9ea715f4a6c"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 23:43:20.400492 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:20.400465 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/402e5517-4211-49e2-95a2-d9ea715f4a6c-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "402e5517-4211-49e2-95a2-d9ea715f4a6c" (UID: "402e5517-4211-49e2-95a2-d9ea715f4a6c"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 23:43:20.401972 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:20.401940 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/402e5517-4211-49e2-95a2-d9ea715f4a6c-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "402e5517-4211-49e2-95a2-d9ea715f4a6c" (UID: "402e5517-4211-49e2-95a2-d9ea715f4a6c"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 23:43:20.402033 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:20.401952 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/402e5517-4211-49e2-95a2-d9ea715f4a6c-kube-api-access-vzzmh" (OuterVolumeSpecName: "kube-api-access-vzzmh") pod "402e5517-4211-49e2-95a2-d9ea715f4a6c" (UID: "402e5517-4211-49e2-95a2-d9ea715f4a6c"). InnerVolumeSpecName "kube-api-access-vzzmh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 23:43:20.402033 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:20.402000 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/402e5517-4211-49e2-95a2-d9ea715f4a6c-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "402e5517-4211-49e2-95a2-d9ea715f4a6c" (UID: "402e5517-4211-49e2-95a2-d9ea715f4a6c"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 23:43:20.500915 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:20.500880 2574 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/402e5517-4211-49e2-95a2-d9ea715f4a6c-console-serving-cert\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:43:20.500915 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:20.500911 2574 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/402e5517-4211-49e2-95a2-d9ea715f4a6c-service-ca\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:43:20.500915 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:20.500921 2574 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/402e5517-4211-49e2-95a2-d9ea715f4a6c-console-config\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:43:20.501130 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:20.500931 2574 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/402e5517-4211-49e2-95a2-d9ea715f4a6c-console-oauth-config\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:43:20.501130 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:20.500940 2574 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/402e5517-4211-49e2-95a2-d9ea715f4a6c-trusted-ca-bundle\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:43:20.501130 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:20.500949 2574 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/402e5517-4211-49e2-95a2-d9ea715f4a6c-oauth-serving-cert\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:43:20.501130 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:20.500957 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzzmh\" (UniqueName: \"kubernetes.io/projected/402e5517-4211-49e2-95a2-d9ea715f4a6c-kube-api-access-vzzmh\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:43:21.119066 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:21.119034 2574 generic.go:356] "Generic (PLEG): container finished" podID="402e5517-4211-49e2-95a2-d9ea715f4a6c" containerID="6c4f1d40f39a74469b215ff17af4efd2b6e5cfcb758ac41ae3bef268b495922a" exitCode=0 Sep 10 23:43:21.119514 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:21.119091 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-778d864cdc-v6xfr" Sep 10 23:43:21.119514 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:21.119124 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-778d864cdc-v6xfr" event={"ID":"402e5517-4211-49e2-95a2-d9ea715f4a6c","Type":"ContainerDied","Data":"6c4f1d40f39a74469b215ff17af4efd2b6e5cfcb758ac41ae3bef268b495922a"} Sep 10 23:43:21.119514 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:21.119188 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-778d864cdc-v6xfr" event={"ID":"402e5517-4211-49e2-95a2-d9ea715f4a6c","Type":"ContainerDied","Data":"8d071a0e4669d44924ff9d47614ac663a0c8655ce4fb6137b1b7797fdabae3fc"} Sep 10 23:43:21.119514 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:21.119216 2574 scope.go:117] "RemoveContainer" containerID="6c4f1d40f39a74469b215ff17af4efd2b6e5cfcb758ac41ae3bef268b495922a" Sep 10 23:43:21.121082 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:21.121053 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-sv4gw" event={"ID":"c7ab7425-af9d-4839-aba0-e4ca2d298860","Type":"ContainerStarted","Data":"7aeb77802bbe6b1aad7f940842cd363f1329bf09e7f54ffae212d3f63efd58fc"} Sep 10 23:43:21.130780 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:21.130759 2574 scope.go:117] "RemoveContainer" containerID="6c4f1d40f39a74469b215ff17af4efd2b6e5cfcb758ac41ae3bef268b495922a" Sep 10 23:43:21.131101 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:43:21.131082 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c4f1d40f39a74469b215ff17af4efd2b6e5cfcb758ac41ae3bef268b495922a\": container with ID starting with 6c4f1d40f39a74469b215ff17af4efd2b6e5cfcb758ac41ae3bef268b495922a not found: ID does not exist" containerID="6c4f1d40f39a74469b215ff17af4efd2b6e5cfcb758ac41ae3bef268b495922a" Sep 10 23:43:21.131157 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:21.131110 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c4f1d40f39a74469b215ff17af4efd2b6e5cfcb758ac41ae3bef268b495922a"} err="failed to get container status \"6c4f1d40f39a74469b215ff17af4efd2b6e5cfcb758ac41ae3bef268b495922a\": rpc error: code = NotFound desc = could not find container \"6c4f1d40f39a74469b215ff17af4efd2b6e5cfcb758ac41ae3bef268b495922a\": container with ID starting with 6c4f1d40f39a74469b215ff17af4efd2b6e5cfcb758ac41ae3bef268b495922a not found: ID does not exist" Sep 10 23:43:21.155037 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:21.155008 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-778d864cdc-v6xfr"] Sep 10 23:43:21.164176 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:21.164152 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-778d864cdc-v6xfr"] Sep 10 23:43:23.096816 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:23.094727 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="402e5517-4211-49e2-95a2-d9ea715f4a6c" path="/var/lib/kubelet/pods/402e5517-4211-49e2-95a2-d9ea715f4a6c/volumes" Sep 10 23:43:24.143488 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:24.143450 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-sv4gw" event={"ID":"c7ab7425-af9d-4839-aba0-e4ca2d298860","Type":"ContainerStarted","Data":"5cd16355fd63697e27ee8846f00c452de746e10ff410ec9ecc573419fdf788c8"} Sep 10 23:43:24.143960 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:24.143520 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-sv4gw" Sep 10 23:43:24.548783 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:24.548728 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-sv4gw" podStartSLOduration=2.579489668 podStartE2EDuration="5.548710419s" totalImagesPullingTime="2.969220751s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:43:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:43:24.185661934 +0000 UTC m=+395.573264687" watchObservedRunningTime="2026-09-10 23:43:24.548710419 +0000 UTC m=+395.936313201" Sep 10 23:43:24.549395 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:24.549380 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-v57r4"] Sep 10 23:43:24.549782 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:24.549769 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="402e5517-4211-49e2-95a2-d9ea715f4a6c" containerName="console" Sep 10 23:43:24.549827 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:24.549784 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="402e5517-4211-49e2-95a2-d9ea715f4a6c" containerName="console" Sep 10 23:43:24.549857 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:24.549850 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="402e5517-4211-49e2-95a2-d9ea715f4a6c" containerName="console" Sep 10 23:43:24.552952 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:24.552935 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-v57r4" Sep 10 23:43:24.556324 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:24.556308 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-5wwxc\"" Sep 10 23:43:24.582521 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:24.582491 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-v57r4"] Sep 10 23:43:24.741731 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:24.741695 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgnn2\" (UniqueName: \"kubernetes.io/projected/0f5e4782-ea19-453f-955d-f8f10137f89b-kube-api-access-hgnn2\") pod \"kuadrant-operator-controller-manager-76f8f686bf-v57r4\" (UID: \"0f5e4782-ea19-453f-955d-f8f10137f89b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-v57r4" Sep 10 23:43:24.741731 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:24.741735 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0f5e4782-ea19-453f-955d-f8f10137f89b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-v57r4\" (UID: \"0f5e4782-ea19-453f-955d-f8f10137f89b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-v57r4" Sep 10 23:43:24.843082 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:24.842996 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgnn2\" (UniqueName: \"kubernetes.io/projected/0f5e4782-ea19-453f-955d-f8f10137f89b-kube-api-access-hgnn2\") pod \"kuadrant-operator-controller-manager-76f8f686bf-v57r4\" (UID: \"0f5e4782-ea19-453f-955d-f8f10137f89b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-v57r4" Sep 10 23:43:24.843082 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:24.843036 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0f5e4782-ea19-453f-955d-f8f10137f89b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-v57r4\" (UID: \"0f5e4782-ea19-453f-955d-f8f10137f89b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-v57r4" Sep 10 23:43:24.843423 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:24.843406 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0f5e4782-ea19-453f-955d-f8f10137f89b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-v57r4\" (UID: \"0f5e4782-ea19-453f-955d-f8f10137f89b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-v57r4" Sep 10 23:43:24.856265 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:24.856244 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgnn2\" (UniqueName: \"kubernetes.io/projected/0f5e4782-ea19-453f-955d-f8f10137f89b-kube-api-access-hgnn2\") pod \"kuadrant-operator-controller-manager-76f8f686bf-v57r4\" (UID: \"0f5e4782-ea19-453f-955d-f8f10137f89b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-v57r4" Sep 10 23:43:24.863115 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:24.863087 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-v57r4" Sep 10 23:43:25.167203 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:43:25.167160 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f5e4782_ea19_453f_955d_f8f10137f89b.slice/crio-d0bbfd3be197b02413922c2278da584850b850f90dd9a3b19cc4e25e8402ff65 WatchSource:0}: Error finding container d0bbfd3be197b02413922c2278da584850b850f90dd9a3b19cc4e25e8402ff65: Status 404 returned error can't find the container with id d0bbfd3be197b02413922c2278da584850b850f90dd9a3b19cc4e25e8402ff65 Sep 10 23:43:25.167983 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:25.167957 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-v57r4"] Sep 10 23:43:26.155567 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:26.155527 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-v57r4" event={"ID":"0f5e4782-ea19-453f-955d-f8f10137f89b","Type":"ContainerStarted","Data":"d0bbfd3be197b02413922c2278da584850b850f90dd9a3b19cc4e25e8402ff65"} Sep 10 23:43:27.326968 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:27.326933 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-dml9m"] Sep 10 23:43:27.331685 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:27.331662 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-dml9m" Sep 10 23:43:27.333828 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:27.333808 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-7f2pr\"" Sep 10 23:43:27.340589 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:27.340562 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-dml9m"] Sep 10 23:43:27.363686 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:27.363657 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nd4rd\" (UniqueName: \"kubernetes.io/projected/264aca46-d6b4-4981-9392-7092015b0d7e-kube-api-access-nd4rd\") pod \"authorino-operator-76b478bd78-dml9m\" (UID: \"264aca46-d6b4-4981-9392-7092015b0d7e\") " pod="kuadrant-system/authorino-operator-76b478bd78-dml9m" Sep 10 23:43:27.464899 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:27.464873 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nd4rd\" (UniqueName: \"kubernetes.io/projected/264aca46-d6b4-4981-9392-7092015b0d7e-kube-api-access-nd4rd\") pod \"authorino-operator-76b478bd78-dml9m\" (UID: \"264aca46-d6b4-4981-9392-7092015b0d7e\") " pod="kuadrant-system/authorino-operator-76b478bd78-dml9m" Sep 10 23:43:27.473702 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:27.473674 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nd4rd\" (UniqueName: \"kubernetes.io/projected/264aca46-d6b4-4981-9392-7092015b0d7e-kube-api-access-nd4rd\") pod \"authorino-operator-76b478bd78-dml9m\" (UID: \"264aca46-d6b4-4981-9392-7092015b0d7e\") " pod="kuadrant-system/authorino-operator-76b478bd78-dml9m" Sep 10 23:43:27.643234 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:27.643168 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-dml9m" Sep 10 23:43:27.921265 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:27.921173 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-dml9m"] Sep 10 23:43:27.933777 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:43:27.933731 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod264aca46_d6b4_4981_9392_7092015b0d7e.slice/crio-0690e168819180e839e4d51c187620fca3b033055e68f98b88a414654fb08ea0 WatchSource:0}: Error finding container 0690e168819180e839e4d51c187620fca3b033055e68f98b88a414654fb08ea0: Status 404 returned error can't find the container with id 0690e168819180e839e4d51c187620fca3b033055e68f98b88a414654fb08ea0 Sep 10 23:43:28.169567 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:28.169533 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-dml9m" event={"ID":"264aca46-d6b4-4981-9392-7092015b0d7e","Type":"ContainerStarted","Data":"0690e168819180e839e4d51c187620fca3b033055e68f98b88a414654fb08ea0"} Sep 10 23:43:31.194724 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:31.194598 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-v57r4" event={"ID":"0f5e4782-ea19-453f-955d-f8f10137f89b","Type":"ContainerStarted","Data":"f17ff9878c5fe382fc05f3a045e194909a94c40553e89db0fd871dc702633169"} Sep 10 23:43:31.194724 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:31.194705 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-v57r4" Sep 10 23:43:31.196295 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:31.196265 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-dml9m" event={"ID":"264aca46-d6b4-4981-9392-7092015b0d7e","Type":"ContainerStarted","Data":"64b0a7e89130f83f8c6d65660743939d410bf6a3edf5ef19a89d29239f9f1105"} Sep 10 23:43:31.196386 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:31.196324 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-76b478bd78-dml9m" Sep 10 23:43:31.212623 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:31.212577 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-v57r4" podStartSLOduration=1.6183607100000001 podStartE2EDuration="7.212558151s" totalImagesPullingTime="5.594197441s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:43:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:43:31.211825644 +0000 UTC m=+402.599428404" watchObservedRunningTime="2026-09-10 23:43:31.212558151 +0000 UTC m=+402.600160911" Sep 10 23:43:31.230856 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:31.230806 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-76b478bd78-dml9m" podStartSLOduration=1.417952308 podStartE2EDuration="4.230789327s" totalImagesPullingTime="2.812837019s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:43:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:43:31.228836716 +0000 UTC m=+402.616439476" watchObservedRunningTime="2026-09-10 23:43:31.230789327 +0000 UTC m=+402.618392147" Sep 10 23:43:35.151422 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:35.151386 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-sv4gw" Sep 10 23:43:42.204164 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:42.204129 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-76b478bd78-dml9m" Sep 10 23:43:42.204672 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:42.204204 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-v57r4" Sep 10 23:43:44.318701 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:44.318662 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kbflj"] Sep 10 23:43:44.322818 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:44.322795 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kbflj" Sep 10 23:43:44.338464 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:44.338439 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kbflj"] Sep 10 23:43:44.352917 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:44.352877 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kbflj"] Sep 10 23:43:44.353437 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:43:44.353401 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[extensions-socket-volume kube-api-access-vnww7], unattached volumes=[], failed to process volumes=[extensions-socket-volume kube-api-access-vnww7]: context canceled" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kbflj" podUID="31559d24-ec43-401a-bf1b-30be36579a99" Sep 10 23:43:44.363482 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:44.363447 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kbflj"] Sep 10 23:43:44.378866 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:44.378830 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-v57r4"] Sep 10 23:43:44.379101 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:44.379076 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-v57r4" podUID="0f5e4782-ea19-453f-955d-f8f10137f89b" containerName="manager" containerID="cri-o://f17ff9878c5fe382fc05f3a045e194909a94c40553e89db0fd871dc702633169" gracePeriod=2 Sep 10 23:43:44.397495 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:44.397460 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-v57r4"] Sep 10 23:43:44.400060 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:43:44.400019 2574 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"0f5e4782-ea19-453f-955d-f8f10137f89b\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-76f8f686bf-v57r4\": pods \"kuadrant-operator-controller-manager-76f8f686bf-v57r4\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-v57r4" Sep 10 23:43:44.419300 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:44.419271 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nwvhw"] Sep 10 23:43:44.420207 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:44.420189 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f5e4782-ea19-453f-955d-f8f10137f89b" containerName="manager" Sep 10 23:43:44.420270 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:44.420210 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f5e4782-ea19-453f-955d-f8f10137f89b" containerName="manager" Sep 10 23:43:44.420309 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:44.420290 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f5e4782-ea19-453f-955d-f8f10137f89b" containerName="manager" Sep 10 23:43:44.423634 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:44.423607 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nwvhw" Sep 10 23:43:44.441972 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:44.441923 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nwvhw"] Sep 10 23:43:44.473806 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:43:44.473771 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-v57r4\" is forbidden: User \"system:node:ip-10-0-138-128.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-128.ec2.internal' and this object" podUID="0f5e4782-ea19-453f-955d-f8f10137f89b" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-v57r4" Sep 10 23:43:44.514408 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:44.514368 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdxx8\" (UniqueName: \"kubernetes.io/projected/2121b776-4f30-4dfa-b40f-ef35c1308a74-kube-api-access-gdxx8\") pod \"kuadrant-operator-controller-manager-76f8f686bf-nwvhw\" (UID: \"2121b776-4f30-4dfa-b40f-ef35c1308a74\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nwvhw" Sep 10 23:43:44.514598 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:44.514469 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2121b776-4f30-4dfa-b40f-ef35c1308a74-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-nwvhw\" (UID: \"2121b776-4f30-4dfa-b40f-ef35c1308a74\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nwvhw" Sep 10 23:43:44.614949 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:44.614917 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2121b776-4f30-4dfa-b40f-ef35c1308a74-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-nwvhw\" (UID: \"2121b776-4f30-4dfa-b40f-ef35c1308a74\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nwvhw" Sep 10 23:43:44.615073 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:44.615030 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdxx8\" (UniqueName: \"kubernetes.io/projected/2121b776-4f30-4dfa-b40f-ef35c1308a74-kube-api-access-gdxx8\") pod \"kuadrant-operator-controller-manager-76f8f686bf-nwvhw\" (UID: \"2121b776-4f30-4dfa-b40f-ef35c1308a74\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nwvhw" Sep 10 23:43:44.615298 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:44.615276 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2121b776-4f30-4dfa-b40f-ef35c1308a74-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-nwvhw\" (UID: \"2121b776-4f30-4dfa-b40f-ef35c1308a74\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nwvhw" Sep 10 23:43:44.623353 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:44.623329 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdxx8\" (UniqueName: \"kubernetes.io/projected/2121b776-4f30-4dfa-b40f-ef35c1308a74-kube-api-access-gdxx8\") pod \"kuadrant-operator-controller-manager-76f8f686bf-nwvhw\" (UID: \"2121b776-4f30-4dfa-b40f-ef35c1308a74\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nwvhw" Sep 10 23:43:44.627130 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:44.627110 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-v57r4" Sep 10 23:43:44.629418 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:43:44.629386 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-v57r4\" is forbidden: User \"system:node:ip-10-0-138-128.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-128.ec2.internal' and this object" podUID="0f5e4782-ea19-453f-955d-f8f10137f89b" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-v57r4" Sep 10 23:43:44.715581 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:44.715552 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hgnn2\" (UniqueName: \"kubernetes.io/projected/0f5e4782-ea19-453f-955d-f8f10137f89b-kube-api-access-hgnn2\") pod \"0f5e4782-ea19-453f-955d-f8f10137f89b\" (UID: \"0f5e4782-ea19-453f-955d-f8f10137f89b\") " Sep 10 23:43:44.715801 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:44.715588 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0f5e4782-ea19-453f-955d-f8f10137f89b-extensions-socket-volume\") pod \"0f5e4782-ea19-453f-955d-f8f10137f89b\" (UID: \"0f5e4782-ea19-453f-955d-f8f10137f89b\") " Sep 10 23:43:44.716129 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:44.716104 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f5e4782-ea19-453f-955d-f8f10137f89b-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "0f5e4782-ea19-453f-955d-f8f10137f89b" (UID: "0f5e4782-ea19-453f-955d-f8f10137f89b"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:43:44.717867 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:44.717845 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f5e4782-ea19-453f-955d-f8f10137f89b-kube-api-access-hgnn2" (OuterVolumeSpecName: "kube-api-access-hgnn2") pod "0f5e4782-ea19-453f-955d-f8f10137f89b" (UID: "0f5e4782-ea19-453f-955d-f8f10137f89b"). InnerVolumeSpecName "kube-api-access-hgnn2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 23:43:44.769090 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:44.769054 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nwvhw" Sep 10 23:43:44.816833 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:44.816773 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgnn2\" (UniqueName: \"kubernetes.io/projected/0f5e4782-ea19-453f-955d-f8f10137f89b-kube-api-access-hgnn2\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:43:44.816833 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:44.816808 2574 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0f5e4782-ea19-453f-955d-f8f10137f89b-extensions-socket-volume\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:43:45.064673 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:45.064623 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nwvhw"] Sep 10 23:43:45.065549 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:43:45.065520 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2121b776_4f30_4dfa_b40f_ef35c1308a74.slice/crio-33f699a7a3a4019e295ba85a85d37c454877c752994e764cca7627a9acd09d96 WatchSource:0}: Error finding container 33f699a7a3a4019e295ba85a85d37c454877c752994e764cca7627a9acd09d96: Status 404 returned error can't find the container with id 33f699a7a3a4019e295ba85a85d37c454877c752994e764cca7627a9acd09d96 Sep 10 23:43:45.094463 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:45.094434 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f5e4782-ea19-453f-955d-f8f10137f89b" path="/var/lib/kubelet/pods/0f5e4782-ea19-453f-955d-f8f10137f89b/volumes" Sep 10 23:43:45.094819 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:45.094798 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31559d24-ec43-401a-bf1b-30be36579a99" path="/var/lib/kubelet/pods/31559d24-ec43-401a-bf1b-30be36579a99/volumes" Sep 10 23:43:45.278237 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:45.278202 2574 generic.go:356] "Generic (PLEG): container finished" podID="0f5e4782-ea19-453f-955d-f8f10137f89b" containerID="f17ff9878c5fe382fc05f3a045e194909a94c40553e89db0fd871dc702633169" exitCode=0 Sep 10 23:43:45.278430 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:45.278265 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-v57r4" Sep 10 23:43:45.278430 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:45.278295 2574 scope.go:117] "RemoveContainer" containerID="f17ff9878c5fe382fc05f3a045e194909a94c40553e89db0fd871dc702633169" Sep 10 23:43:45.280586 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:45.280550 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nwvhw" event={"ID":"2121b776-4f30-4dfa-b40f-ef35c1308a74","Type":"ContainerStarted","Data":"3b215378f43922fb7fa81439e2b0401e4a6018580d0488a8f584952e64cb5145"} Sep 10 23:43:45.280709 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:45.280614 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nwvhw" event={"ID":"2121b776-4f30-4dfa-b40f-ef35c1308a74","Type":"ContainerStarted","Data":"33f699a7a3a4019e295ba85a85d37c454877c752994e764cca7627a9acd09d96"} Sep 10 23:43:45.280709 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:45.280699 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nwvhw" Sep 10 23:43:45.280842 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:45.280731 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kbflj" Sep 10 23:43:45.288205 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:45.288184 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kbflj" Sep 10 23:43:45.289253 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:45.289237 2574 scope.go:117] "RemoveContainer" containerID="f17ff9878c5fe382fc05f3a045e194909a94c40553e89db0fd871dc702633169" Sep 10 23:43:45.289530 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:43:45.289510 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f17ff9878c5fe382fc05f3a045e194909a94c40553e89db0fd871dc702633169\": container with ID starting with f17ff9878c5fe382fc05f3a045e194909a94c40553e89db0fd871dc702633169 not found: ID does not exist" containerID="f17ff9878c5fe382fc05f3a045e194909a94c40553e89db0fd871dc702633169" Sep 10 23:43:45.289599 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:45.289539 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f17ff9878c5fe382fc05f3a045e194909a94c40553e89db0fd871dc702633169"} err="failed to get container status \"f17ff9878c5fe382fc05f3a045e194909a94c40553e89db0fd871dc702633169\": rpc error: code = NotFound desc = could not find container \"f17ff9878c5fe382fc05f3a045e194909a94c40553e89db0fd871dc702633169\": container with ID starting with f17ff9878c5fe382fc05f3a045e194909a94c40553e89db0fd871dc702633169 not found: ID does not exist" Sep 10 23:43:45.345937 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:43:45.345850 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5769f99d9f-kbflj\" is forbidden: User \"system:node:ip-10-0-138-128.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-128.ec2.internal' and this object" podUID="31559d24-ec43-401a-bf1b-30be36579a99" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kbflj" Sep 10 23:43:45.350074 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:45.349989 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nwvhw" podStartSLOduration=1.349972453 podStartE2EDuration="1.349972453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:43:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:43:45.343794919 +0000 UTC m=+416.731397678" watchObservedRunningTime="2026-09-10 23:43:45.349972453 +0000 UTC m=+416.737575189" Sep 10 23:43:46.286462 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:46.286436 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kbflj" Sep 10 23:43:46.288409 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:43:46.288385 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5769f99d9f-kbflj\" is forbidden: User \"system:node:ip-10-0-138-128.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-128.ec2.internal' and this object" podUID="31559d24-ec43-401a-bf1b-30be36579a99" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kbflj" Sep 10 23:43:46.294268 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:43:46.294239 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5769f99d9f-kbflj\" is forbidden: User \"system:node:ip-10-0-138-128.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-128.ec2.internal' and this object" podUID="31559d24-ec43-401a-bf1b-30be36579a99" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kbflj" Sep 10 23:43:56.288533 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:43:56.288499 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nwvhw" Sep 10 23:44:00.435774 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:44:00.435740 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nwvhw"] Sep 10 23:44:00.436235 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:44:00.435962 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nwvhw" podUID="2121b776-4f30-4dfa-b40f-ef35c1308a74" containerName="manager" containerID="cri-o://3b215378f43922fb7fa81439e2b0401e4a6018580d0488a8f584952e64cb5145" gracePeriod=10 Sep 10 23:44:00.728720 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:44:00.728689 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nwvhw" Sep 10 23:44:00.858096 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:44:00.858053 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gdxx8\" (UniqueName: \"kubernetes.io/projected/2121b776-4f30-4dfa-b40f-ef35c1308a74-kube-api-access-gdxx8\") pod \"2121b776-4f30-4dfa-b40f-ef35c1308a74\" (UID: \"2121b776-4f30-4dfa-b40f-ef35c1308a74\") " Sep 10 23:44:00.858096 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:44:00.858105 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2121b776-4f30-4dfa-b40f-ef35c1308a74-extensions-socket-volume\") pod \"2121b776-4f30-4dfa-b40f-ef35c1308a74\" (UID: \"2121b776-4f30-4dfa-b40f-ef35c1308a74\") " Sep 10 23:44:00.858484 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:44:00.858460 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2121b776-4f30-4dfa-b40f-ef35c1308a74-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "2121b776-4f30-4dfa-b40f-ef35c1308a74" (UID: "2121b776-4f30-4dfa-b40f-ef35c1308a74"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:44:00.860221 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:44:00.860201 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2121b776-4f30-4dfa-b40f-ef35c1308a74-kube-api-access-gdxx8" (OuterVolumeSpecName: "kube-api-access-gdxx8") pod "2121b776-4f30-4dfa-b40f-ef35c1308a74" (UID: "2121b776-4f30-4dfa-b40f-ef35c1308a74"). InnerVolumeSpecName "kube-api-access-gdxx8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 23:44:00.959119 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:44:00.959086 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdxx8\" (UniqueName: \"kubernetes.io/projected/2121b776-4f30-4dfa-b40f-ef35c1308a74-kube-api-access-gdxx8\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:44:00.959119 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:44:00.959117 2574 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2121b776-4f30-4dfa-b40f-ef35c1308a74-extensions-socket-volume\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:44:01.371691 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:44:01.371593 2574 generic.go:356] "Generic (PLEG): container finished" podID="2121b776-4f30-4dfa-b40f-ef35c1308a74" containerID="3b215378f43922fb7fa81439e2b0401e4a6018580d0488a8f584952e64cb5145" exitCode=0 Sep 10 23:44:01.371691 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:44:01.371663 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nwvhw" Sep 10 23:44:01.371861 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:44:01.371684 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nwvhw" event={"ID":"2121b776-4f30-4dfa-b40f-ef35c1308a74","Type":"ContainerDied","Data":"3b215378f43922fb7fa81439e2b0401e4a6018580d0488a8f584952e64cb5145"} Sep 10 23:44:01.371861 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:44:01.371745 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nwvhw" event={"ID":"2121b776-4f30-4dfa-b40f-ef35c1308a74","Type":"ContainerDied","Data":"33f699a7a3a4019e295ba85a85d37c454877c752994e764cca7627a9acd09d96"} Sep 10 23:44:01.371861 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:44:01.371770 2574 scope.go:117] "RemoveContainer" containerID="3b215378f43922fb7fa81439e2b0401e4a6018580d0488a8f584952e64cb5145" Sep 10 23:44:01.380587 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:44:01.380568 2574 scope.go:117] "RemoveContainer" containerID="3b215378f43922fb7fa81439e2b0401e4a6018580d0488a8f584952e64cb5145" Sep 10 23:44:01.380865 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:44:01.380845 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b215378f43922fb7fa81439e2b0401e4a6018580d0488a8f584952e64cb5145\": container with ID starting with 3b215378f43922fb7fa81439e2b0401e4a6018580d0488a8f584952e64cb5145 not found: ID does not exist" containerID="3b215378f43922fb7fa81439e2b0401e4a6018580d0488a8f584952e64cb5145" Sep 10 23:44:01.380938 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:44:01.380876 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b215378f43922fb7fa81439e2b0401e4a6018580d0488a8f584952e64cb5145"} err="failed to get container status \"3b215378f43922fb7fa81439e2b0401e4a6018580d0488a8f584952e64cb5145\": rpc error: code = NotFound desc = could not find container \"3b215378f43922fb7fa81439e2b0401e4a6018580d0488a8f584952e64cb5145\": container with ID starting with 3b215378f43922fb7fa81439e2b0401e4a6018580d0488a8f584952e64cb5145 not found: ID does not exist" Sep 10 23:44:01.393674 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:44:01.393634 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nwvhw"] Sep 10 23:44:01.395898 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:44:01.395880 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nwvhw"] Sep 10 23:44:03.093863 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:44:03.093827 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2121b776-4f30-4dfa-b40f-ef35c1308a74" path="/var/lib/kubelet/pods/2121b776-4f30-4dfa-b40f-ef35c1308a74/volumes" Sep 10 23:44:16.729453 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:44:16.729368 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2jhsv"] Sep 10 23:44:16.730438 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:44:16.729938 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2121b776-4f30-4dfa-b40f-ef35c1308a74" containerName="manager" Sep 10 23:44:16.730438 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:44:16.729957 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="2121b776-4f30-4dfa-b40f-ef35c1308a74" containerName="manager" Sep 10 23:44:16.730438 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:44:16.730090 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="2121b776-4f30-4dfa-b40f-ef35c1308a74" containerName="manager" Sep 10 23:44:16.733220 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:44:16.733196 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2jhsv" Sep 10 23:44:16.735746 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:44:16.735721 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-zxv8k\"" Sep 10 23:44:16.747547 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:44:16.747521 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2jhsv"] Sep 10 23:44:16.795107 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:44:16.795076 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1034bfd-cf35-4a42-90ac-6518eed5ff4f-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2jhsv\" (UID: \"e1034bfd-cf35-4a42-90ac-6518eed5ff4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2jhsv" Sep 10 23:44:16.795232 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:44:16.795111 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1034bfd-cf35-4a42-90ac-6518eed5ff4f-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2jhsv\" (UID: \"e1034bfd-cf35-4a42-90ac-6518eed5ff4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2jhsv" Sep 10 23:44:16.795232 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:44:16.795215 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1034bfd-cf35-4a42-90ac-6518eed5ff4f-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2jhsv\" (UID: \"e1034bfd-cf35-4a42-90ac-6518eed5ff4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2jhsv" Sep 10 23:44:16.795331 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:44:16.795246 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1034bfd-cf35-4a42-90ac-6518eed5ff4f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2jhsv\" (UID: \"e1034bfd-cf35-4a42-90ac-6518eed5ff4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2jhsv" Sep 10 23:44:16.795331 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:44:16.795308 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1034bfd-cf35-4a42-90ac-6518eed5ff4f-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2jhsv\" (UID: \"e1034bfd-cf35-4a42-90ac-6518eed5ff4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2jhsv" Sep 10 23:44:16.795415 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:44:16.795339 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1034bfd-cf35-4a42-90ac-6518eed5ff4f-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2jhsv\" (UID: \"e1034bfd-cf35-4a42-90ac-6518eed5ff4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2jhsv" Sep 10 23:44:16.795415 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:44:16.795355 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1034bfd-cf35-4a42-90ac-6518eed5ff4f-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2jhsv\" (UID: \"e1034bfd-cf35-4a42-90ac-6518eed5ff4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2jhsv" Sep 10 23:44:16.795415 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:44:16.795392 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1034bfd-cf35-4a42-90ac-6518eed5ff4f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2jhsv\" (UID: \"e1034bfd-cf35-4a42-90ac-6518eed5ff4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2jhsv" Sep 10 23:44:16.795415 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:44:16.795408 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8v2mm\" (UniqueName: \"kubernetes.io/projected/e1034bfd-cf35-4a42-90ac-6518eed5ff4f-kube-api-access-8v2mm\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2jhsv\" (UID: \"e1034bfd-cf35-4a42-90ac-6518eed5ff4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2jhsv" Sep 10 23:44:16.896307 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:44:16.896268 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1034bfd-cf35-4a42-90ac-6518eed5ff4f-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2jhsv\" (UID: \"e1034bfd-cf35-4a42-90ac-6518eed5ff4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2jhsv" Sep 10 23:44:16.896547 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:44:16.896312 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1034bfd-cf35-4a42-90ac-6518eed5ff4f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2jhsv\" (UID: \"e1034bfd-cf35-4a42-90ac-6518eed5ff4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2jhsv" Sep 10 23:44:16.896547 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:44:16.896347 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1034bfd-cf35-4a42-90ac-6518eed5ff4f-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2jhsv\" (UID: \"e1034bfd-cf35-4a42-90ac-6518eed5ff4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2jhsv" Sep 10 23:44:16.896547 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:44:16.896376 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1034bfd-cf35-4a42-90ac-6518eed5ff4f-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2jhsv\" (UID: \"e1034bfd-cf35-4a42-90ac-6518eed5ff4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2jhsv" Sep 10 23:44:16.896547 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:44:16.896399 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1034bfd-cf35-4a42-90ac-6518eed5ff4f-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2jhsv\" (UID: \"e1034bfd-cf35-4a42-90ac-6518eed5ff4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2jhsv" Sep 10 23:44:16.896547 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:44:16.896435 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1034bfd-cf35-4a42-90ac-6518eed5ff4f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2jhsv\" (UID: \"e1034bfd-cf35-4a42-90ac-6518eed5ff4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2jhsv" Sep 10 23:44:16.896547 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:44:16.896460 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8v2mm\" (UniqueName: \"kubernetes.io/projected/e1034bfd-cf35-4a42-90ac-6518eed5ff4f-kube-api-access-8v2mm\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2jhsv\" (UID: \"e1034bfd-cf35-4a42-90ac-6518eed5ff4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2jhsv" Sep 10 23:44:16.896547 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:44:16.896511 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1034bfd-cf35-4a42-90ac-6518eed5ff4f-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2jhsv\" (UID: \"e1034bfd-cf35-4a42-90ac-6518eed5ff4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2jhsv" Sep 10 23:44:16.896547 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:44:16.896543 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1034bfd-cf35-4a42-90ac-6518eed5ff4f-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2jhsv\" (UID: \"e1034bfd-cf35-4a42-90ac-6518eed5ff4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2jhsv" Sep 10 23:44:16.897047 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:44:16.896746 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1034bfd-cf35-4a42-90ac-6518eed5ff4f-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2jhsv\" (UID: \"e1034bfd-cf35-4a42-90ac-6518eed5ff4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2jhsv" Sep 10 23:44:16.897047 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:44:16.896795 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1034bfd-cf35-4a42-90ac-6518eed5ff4f-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2jhsv\" (UID: \"e1034bfd-cf35-4a42-90ac-6518eed5ff4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2jhsv" Sep 10 23:44:16.897047 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:44:16.896968 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1034bfd-cf35-4a42-90ac-6518eed5ff4f-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2jhsv\" (UID: \"e1034bfd-cf35-4a42-90ac-6518eed5ff4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2jhsv" Sep 10 23:44:16.897047 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:44:16.897034 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1034bfd-cf35-4a42-90ac-6518eed5ff4f-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2jhsv\" (UID: \"e1034bfd-cf35-4a42-90ac-6518eed5ff4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2jhsv" Sep 10 23:44:16.897282 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:44:16.897263 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1034bfd-cf35-4a42-90ac-6518eed5ff4f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2jhsv\" (UID: \"e1034bfd-cf35-4a42-90ac-6518eed5ff4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2jhsv" Sep 10 23:44:16.899209 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:44:16.899190 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1034bfd-cf35-4a42-90ac-6518eed5ff4f-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2jhsv\" (UID: \"e1034bfd-cf35-4a42-90ac-6518eed5ff4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2jhsv" Sep 10 23:44:16.899525 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:44:16.899502 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1034bfd-cf35-4a42-90ac-6518eed5ff4f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2jhsv\" (UID: \"e1034bfd-cf35-4a42-90ac-6518eed5ff4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2jhsv" Sep 10 23:44:16.907464 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:44:16.907444 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1034bfd-cf35-4a42-90ac-6518eed5ff4f-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2jhsv\" (UID: \"e1034bfd-cf35-4a42-90ac-6518eed5ff4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2jhsv" Sep 10 23:44:16.907464 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:44:16.907456 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8v2mm\" (UniqueName: \"kubernetes.io/projected/e1034bfd-cf35-4a42-90ac-6518eed5ff4f-kube-api-access-8v2mm\") pod \"maas-default-gateway-openshift-default-78bb78ff78-2jhsv\" (UID: \"e1034bfd-cf35-4a42-90ac-6518eed5ff4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2jhsv" Sep 10 23:44:17.043233 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:44:17.043145 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2jhsv" Sep 10 23:44:17.327338 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:44:17.327311 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2jhsv"] Sep 10 23:44:17.328865 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:44:17.328840 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1034bfd_cf35_4a42_90ac_6518eed5ff4f.slice/crio-affebabe93ffb5ee5d86d993e4a32d2d2b0eeee9dc35f60bee018bd969b2caac WatchSource:0}: Error finding container affebabe93ffb5ee5d86d993e4a32d2d2b0eeee9dc35f60bee018bd969b2caac: Status 404 returned error can't find the container with id affebabe93ffb5ee5d86d993e4a32d2d2b0eeee9dc35f60bee018bd969b2caac Sep 10 23:44:17.461437 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:44:17.461404 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2jhsv" event={"ID":"e1034bfd-cf35-4a42-90ac-6518eed5ff4f","Type":"ContainerStarted","Data":"affebabe93ffb5ee5d86d993e4a32d2d2b0eeee9dc35f60bee018bd969b2caac"} Sep 10 23:44:20.030913 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:44:20.030876 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 23:44:20.031207 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:44:20.030946 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 23:44:20.031207 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:44:20.030978 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 23:44:20.484419 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:44:20.484382 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2jhsv" event={"ID":"e1034bfd-cf35-4a42-90ac-6518eed5ff4f","Type":"ContainerStarted","Data":"b0c654488206d2f368323a719bbeef4aa8163da20d86c9121e82c1fda63e47a2"} Sep 10 23:44:20.527685 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:44:20.527620 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2jhsv" podStartSLOduration=1.8280738479999998 podStartE2EDuration="4.527606335s" totalImagesPullingTime="2.699532487s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:44:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:44:20.525184102 +0000 UTC m=+451.912786863" watchObservedRunningTime="2026-09-10 23:44:20.527606335 +0000 UTC m=+451.915209094" Sep 10 23:44:21.043481 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:44:21.043441 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2jhsv" Sep 10 23:44:21.048265 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:44:21.048238 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2jhsv" Sep 10 23:44:21.490068 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:44:21.490033 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2jhsv" Sep 10 23:44:21.491215 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:44:21.491194 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2jhsv" Sep 10 23:44:21.992421 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:44:21.992383 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-56r7l"] Sep 10 23:44:21.996090 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:44:21.996065 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-56r7l" Sep 10 23:44:21.998631 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:44:21.998608 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Sep 10 23:44:21.998749 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:44:21.998682 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-fdzdl\"" Sep 10 23:44:22.007347 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:44:22.007322 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-56r7l"] Sep 10 23:44:22.034716 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:44:22.034690 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqvqx\" (UniqueName: \"kubernetes.io/projected/3c63a0c1-85ba-47d5-9450-91f8809eb2f0-kube-api-access-vqvqx\") pod \"limitador-limitador-84994f7ffc-56r7l\" (UID: \"3c63a0c1-85ba-47d5-9450-91f8809eb2f0\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-56r7l" Sep 10 23:44:22.034826 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:44:22.034789 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3c63a0c1-85ba-47d5-9450-91f8809eb2f0-config-file\") pod \"limitador-limitador-84994f7ffc-56r7l\" (UID: \"3c63a0c1-85ba-47d5-9450-91f8809eb2f0\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-56r7l" Sep 10 23:44:22.036843 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:44:22.036821 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-56r7l"] Sep 10 23:44:22.135307 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:44:22.135272 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqvqx\" (UniqueName: \"kubernetes.io/projected/3c63a0c1-85ba-47d5-9450-91f8809eb2f0-kube-api-access-vqvqx\") pod \"limitador-limitador-84994f7ffc-56r7l\" (UID: \"3c63a0c1-85ba-47d5-9450-91f8809eb2f0\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-56r7l" Sep 10 23:44:22.135781 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:44:22.135401 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3c63a0c1-85ba-47d5-9450-91f8809eb2f0-config-file\") pod \"limitador-limitador-84994f7ffc-56r7l\" (UID: \"3c63a0c1-85ba-47d5-9450-91f8809eb2f0\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-56r7l" Sep 10 23:44:22.135982 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:44:22.135962 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3c63a0c1-85ba-47d5-9450-91f8809eb2f0-config-file\") pod \"limitador-limitador-84994f7ffc-56r7l\" (UID: \"3c63a0c1-85ba-47d5-9450-91f8809eb2f0\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-56r7l" Sep 10 23:44:22.143714 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:44:22.143685 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqvqx\" (UniqueName: \"kubernetes.io/projected/3c63a0c1-85ba-47d5-9450-91f8809eb2f0-kube-api-access-vqvqx\") pod \"limitador-limitador-84994f7ffc-56r7l\" (UID: \"3c63a0c1-85ba-47d5-9450-91f8809eb2f0\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-56r7l" Sep 10 23:44:22.307671 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:44:22.307570 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-56r7l" Sep 10 23:44:22.582873 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:44:22.582844 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-56r7l"] Sep 10 23:44:22.584340 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:44:22.584307 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c63a0c1_85ba_47d5_9450_91f8809eb2f0.slice/crio-dc0b3e8dab18a48a354a7611bf189981d11b8d1b613483b3ae8a14c22cd3c25e WatchSource:0}: Error finding container dc0b3e8dab18a48a354a7611bf189981d11b8d1b613483b3ae8a14c22cd3c25e: Status 404 returned error can't find the container with id dc0b3e8dab18a48a354a7611bf189981d11b8d1b613483b3ae8a14c22cd3c25e Sep 10 23:44:23.502230 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:44:23.502199 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-56r7l" event={"ID":"3c63a0c1-85ba-47d5-9450-91f8809eb2f0","Type":"ContainerStarted","Data":"dc0b3e8dab18a48a354a7611bf189981d11b8d1b613483b3ae8a14c22cd3c25e"} Sep 10 23:44:24.509886 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:44:24.509853 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-56r7l" event={"ID":"3c63a0c1-85ba-47d5-9450-91f8809eb2f0","Type":"ContainerStarted","Data":"346a53bc29a3b4a6b0e31a70614a4b4285182e4160c33b5fa161a35ec288c821"} Sep 10 23:44:24.510276 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:44:24.509912 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-56r7l" Sep 10 23:44:24.526877 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:44:24.526832 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-56r7l" podStartSLOduration=2.619027644 podStartE2EDuration="3.526816252s" totalImagesPullingTime="907.788608ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:44:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:44:24.524526676 +0000 UTC m=+455.912129436" watchObservedRunningTime="2026-09-10 23:44:24.526816252 +0000 UTC m=+455.914419012" Sep 10 23:44:35.516724 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:44:35.516694 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-56r7l" Sep 10 23:45:50.914202 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:45:50.914125 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-wbdst"] Sep 10 23:45:50.917630 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:45:50.917607 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-wbdst" Sep 10 23:45:50.919538 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:45:50.919517 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-r8rth\"" Sep 10 23:45:50.919744 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:45:50.919723 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Sep 10 23:45:50.928080 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:45:50.928057 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-wbdst"] Sep 10 23:45:51.022763 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:45:51.022728 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/645e6258-646a-4a26-83e8-4cb7af5347d7-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-wbdst\" (UID: \"645e6258-646a-4a26-83e8-4cb7af5347d7\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-wbdst" Sep 10 23:45:51.022763 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:45:51.022760 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zvhv\" (UniqueName: \"kubernetes.io/projected/645e6258-646a-4a26-83e8-4cb7af5347d7-kube-api-access-2zvhv\") pod \"payload-pre-processing-59fbc47bf8-wbdst\" (UID: \"645e6258-646a-4a26-83e8-4cb7af5347d7\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-wbdst" Sep 10 23:45:51.024241 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:45:51.024220 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-56r7l"] Sep 10 23:45:51.124038 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:45:51.123994 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/645e6258-646a-4a26-83e8-4cb7af5347d7-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-wbdst\" (UID: \"645e6258-646a-4a26-83e8-4cb7af5347d7\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-wbdst" Sep 10 23:45:51.124038 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:45:51.124043 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zvhv\" (UniqueName: \"kubernetes.io/projected/645e6258-646a-4a26-83e8-4cb7af5347d7-kube-api-access-2zvhv\") pod \"payload-pre-processing-59fbc47bf8-wbdst\" (UID: \"645e6258-646a-4a26-83e8-4cb7af5347d7\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-wbdst" Sep 10 23:45:51.124856 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:45:51.124821 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/645e6258-646a-4a26-83e8-4cb7af5347d7-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-wbdst\" (UID: \"645e6258-646a-4a26-83e8-4cb7af5347d7\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-wbdst" Sep 10 23:45:51.134036 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:45:51.134011 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zvhv\" (UniqueName: \"kubernetes.io/projected/645e6258-646a-4a26-83e8-4cb7af5347d7-kube-api-access-2zvhv\") pod \"payload-pre-processing-59fbc47bf8-wbdst\" (UID: \"645e6258-646a-4a26-83e8-4cb7af5347d7\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-wbdst" Sep 10 23:45:51.228552 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:45:51.228515 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-wbdst" Sep 10 23:45:51.511083 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:45:51.511053 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-wbdst"] Sep 10 23:45:51.512482 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:45:51.512451 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod645e6258_646a_4a26_83e8_4cb7af5347d7.slice/crio-8d63ff0522f155d304c20e14bc1d3bb16f58151abe0b12672bdac5ef80d63e87 WatchSource:0}: Error finding container 8d63ff0522f155d304c20e14bc1d3bb16f58151abe0b12672bdac5ef80d63e87: Status 404 returned error can't find the container with id 8d63ff0522f155d304c20e14bc1d3bb16f58151abe0b12672bdac5ef80d63e87 Sep 10 23:45:51.999239 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:45:51.999200 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-wbdst" event={"ID":"645e6258-646a-4a26-83e8-4cb7af5347d7","Type":"ContainerStarted","Data":"8d63ff0522f155d304c20e14bc1d3bb16f58151abe0b12672bdac5ef80d63e87"} Sep 10 23:45:54.013624 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:45:54.013587 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-wbdst" event={"ID":"645e6258-646a-4a26-83e8-4cb7af5347d7","Type":"ContainerStarted","Data":"2dcbcb5cd771e739c4073ad0212d8dfcc96ed236d0261aaa107036076bf193cb"} Sep 10 23:45:54.014133 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:45:54.013707 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-wbdst" Sep 10 23:45:54.030603 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:45:54.030550 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-wbdst" podStartSLOduration=1.940574084 podStartE2EDuration="4.030531917s" totalImagesPullingTime="2.089957833s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:45:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:45:54.029135061 +0000 UTC m=+545.416737822" watchObservedRunningTime="2026-09-10 23:45:54.030531917 +0000 UTC m=+545.418134718" Sep 10 23:46:00.051005 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:46:00.050972 2574 generic.go:356] "Generic (PLEG): container finished" podID="e1034bfd-cf35-4a42-90ac-6518eed5ff4f" containerID="b0c654488206d2f368323a719bbeef4aa8163da20d86c9121e82c1fda63e47a2" exitCode=0 Sep 10 23:46:00.051412 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:46:00.051041 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2jhsv" event={"ID":"e1034bfd-cf35-4a42-90ac-6518eed5ff4f","Type":"ContainerDied","Data":"b0c654488206d2f368323a719bbeef4aa8163da20d86c9121e82c1fda63e47a2"} Sep 10 23:46:00.051481 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:46:00.051465 2574 scope.go:117] "RemoveContainer" containerID="b0c654488206d2f368323a719bbeef4aa8163da20d86c9121e82c1fda63e47a2" Sep 10 23:46:00.052502 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:46:00.052469 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 23:46:00.052606 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:46:00.052554 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 23:46:00.052688 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:46:00.052602 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 23:46:01.058924 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:46:01.058887 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2jhsv" event={"ID":"e1034bfd-cf35-4a42-90ac-6518eed5ff4f","Type":"ContainerStarted","Data":"f85f5ed999db6ae6016840627b80003170b2a69cfbd4d8c6dd4846f439d51d60"} Sep 10 23:46:02.043824 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:46:02.043789 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2jhsv" Sep 10 23:46:02.044611 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:46:02.044585 2574 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-2jhsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Sep 10 23:46:02.044767 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:46:02.044656 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2jhsv" podUID="e1034bfd-cf35-4a42-90ac-6518eed5ff4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Sep 10 23:46:03.044487 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:46:03.044445 2574 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-2jhsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Sep 10 23:46:03.044889 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:46:03.044526 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2jhsv" podUID="e1034bfd-cf35-4a42-90ac-6518eed5ff4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Sep 10 23:46:04.044188 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:46:04.044150 2574 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-2jhsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Sep 10 23:46:04.044373 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:46:04.044222 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2jhsv" podUID="e1034bfd-cf35-4a42-90ac-6518eed5ff4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Sep 10 23:46:05.023081 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:46:05.023043 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-wbdst" Sep 10 23:46:05.044122 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:46:05.044094 2574 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-2jhsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Sep 10 23:46:05.044269 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:46:05.044141 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2jhsv" podUID="e1034bfd-cf35-4a42-90ac-6518eed5ff4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Sep 10 23:46:06.043501 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:46:06.043467 2574 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-2jhsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Sep 10 23:46:06.043910 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:46:06.043524 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2jhsv" podUID="e1034bfd-cf35-4a42-90ac-6518eed5ff4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Sep 10 23:46:06.490674 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:46:06.490624 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2jhsv" Sep 10 23:46:07.047404 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:46:07.047370 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2jhsv" Sep 10 23:46:07.111765 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:46:07.111734 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-2jhsv" Sep 10 23:46:14.933349 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:46:14.933314 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nm2sz"] Sep 10 23:46:14.936851 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:46:14.936834 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nm2sz" Sep 10 23:46:14.939674 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:46:14.939650 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 10 23:46:14.939674 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:46:14.939660 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-rmfcp\"" Sep 10 23:46:14.939879 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:46:14.939694 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Sep 10 23:46:14.939879 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:46:14.939733 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 10 23:46:14.945145 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:46:14.945120 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nm2sz"] Sep 10 23:46:15.052198 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:46:15.052169 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sglwt\" (UniqueName: \"kubernetes.io/projected/df5991e1-0d2b-4b74-864f-d263b942fbea-kube-api-access-sglwt\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nm2sz\" (UID: \"df5991e1-0d2b-4b74-864f-d263b942fbea\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nm2sz" Sep 10 23:46:15.052355 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:46:15.052236 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df5991e1-0d2b-4b74-864f-d263b942fbea-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nm2sz\" (UID: \"df5991e1-0d2b-4b74-864f-d263b942fbea\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nm2sz" Sep 10 23:46:15.052355 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:46:15.052273 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df5991e1-0d2b-4b74-864f-d263b942fbea-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nm2sz\" (UID: \"df5991e1-0d2b-4b74-864f-d263b942fbea\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nm2sz" Sep 10 23:46:15.052355 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:46:15.052337 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df5991e1-0d2b-4b74-864f-d263b942fbea-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nm2sz\" (UID: \"df5991e1-0d2b-4b74-864f-d263b942fbea\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nm2sz" Sep 10 23:46:15.052355 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:46:15.052354 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df5991e1-0d2b-4b74-864f-d263b942fbea-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nm2sz\" (UID: \"df5991e1-0d2b-4b74-864f-d263b942fbea\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nm2sz" Sep 10 23:46:15.052535 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:46:15.052387 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df5991e1-0d2b-4b74-864f-d263b942fbea-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nm2sz\" (UID: \"df5991e1-0d2b-4b74-864f-d263b942fbea\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nm2sz" Sep 10 23:46:15.153068 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:46:15.153042 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df5991e1-0d2b-4b74-864f-d263b942fbea-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nm2sz\" (UID: \"df5991e1-0d2b-4b74-864f-d263b942fbea\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nm2sz" Sep 10 23:46:15.153068 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:46:15.153070 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df5991e1-0d2b-4b74-864f-d263b942fbea-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nm2sz\" (UID: \"df5991e1-0d2b-4b74-864f-d263b942fbea\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nm2sz" Sep 10 23:46:15.153286 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:46:15.153092 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df5991e1-0d2b-4b74-864f-d263b942fbea-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nm2sz\" (UID: \"df5991e1-0d2b-4b74-864f-d263b942fbea\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nm2sz" Sep 10 23:46:15.153286 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:46:15.153197 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df5991e1-0d2b-4b74-864f-d263b942fbea-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nm2sz\" (UID: \"df5991e1-0d2b-4b74-864f-d263b942fbea\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nm2sz" Sep 10 23:46:15.153286 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:46:15.153272 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df5991e1-0d2b-4b74-864f-d263b942fbea-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nm2sz\" (UID: \"df5991e1-0d2b-4b74-864f-d263b942fbea\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nm2sz" Sep 10 23:46:15.153441 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:46:15.153360 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sglwt\" (UniqueName: \"kubernetes.io/projected/df5991e1-0d2b-4b74-864f-d263b942fbea-kube-api-access-sglwt\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nm2sz\" (UID: \"df5991e1-0d2b-4b74-864f-d263b942fbea\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nm2sz" Sep 10 23:46:15.153504 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:46:15.153441 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df5991e1-0d2b-4b74-864f-d263b942fbea-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nm2sz\" (UID: \"df5991e1-0d2b-4b74-864f-d263b942fbea\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nm2sz" Sep 10 23:46:15.153599 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:46:15.153579 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df5991e1-0d2b-4b74-864f-d263b942fbea-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nm2sz\" (UID: \"df5991e1-0d2b-4b74-864f-d263b942fbea\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nm2sz" Sep 10 23:46:15.153718 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:46:15.153697 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df5991e1-0d2b-4b74-864f-d263b942fbea-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nm2sz\" (UID: \"df5991e1-0d2b-4b74-864f-d263b942fbea\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nm2sz" Sep 10 23:46:15.155349 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:46:15.155332 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df5991e1-0d2b-4b74-864f-d263b942fbea-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nm2sz\" (UID: \"df5991e1-0d2b-4b74-864f-d263b942fbea\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nm2sz" Sep 10 23:46:15.155509 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:46:15.155491 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df5991e1-0d2b-4b74-864f-d263b942fbea-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nm2sz\" (UID: \"df5991e1-0d2b-4b74-864f-d263b942fbea\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nm2sz" Sep 10 23:46:15.160909 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:46:15.160884 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sglwt\" (UniqueName: \"kubernetes.io/projected/df5991e1-0d2b-4b74-864f-d263b942fbea-kube-api-access-sglwt\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-nm2sz\" (UID: \"df5991e1-0d2b-4b74-864f-d263b942fbea\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nm2sz" Sep 10 23:46:15.248980 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:46:15.248955 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nm2sz" Sep 10 23:46:15.518787 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:46:15.518762 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nm2sz"] Sep 10 23:46:15.520928 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:46:15.520901 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf5991e1_0d2b_4b74_864f_d263b942fbea.slice/crio-a6cef5444ebb4e7d50429f887f2bb7b54792c0e437d5ab67a1dc22244010bf5f WatchSource:0}: Error finding container a6cef5444ebb4e7d50429f887f2bb7b54792c0e437d5ab67a1dc22244010bf5f: Status 404 returned error can't find the container with id a6cef5444ebb4e7d50429f887f2bb7b54792c0e437d5ab67a1dc22244010bf5f Sep 10 23:46:15.577802 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:46:15.577766 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-56r7l"] Sep 10 23:46:16.161161 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:46:16.161124 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nm2sz" event={"ID":"df5991e1-0d2b-4b74-864f-d263b942fbea","Type":"ContainerStarted","Data":"a6cef5444ebb4e7d50429f887f2bb7b54792c0e437d5ab67a1dc22244010bf5f"} Sep 10 23:46:18.183239 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:46:18.183197 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nm2sz" event={"ID":"df5991e1-0d2b-4b74-864f-d263b942fbea","Type":"ContainerStarted","Data":"d7f5485ba93dcfe2121bd9ff882099595eb6c31b6b6ed84e463422fc5082fb4a"} Sep 10 23:46:18.200617 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:46:18.200551 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nm2sz" podStartSLOduration=2.455200286 podStartE2EDuration="4.200531564s" totalImagesPullingTime="1.745331278s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:46:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:46:18.199767946 +0000 UTC m=+569.587370704" watchObservedRunningTime="2026-09-10 23:46:18.200531564 +0000 UTC m=+569.588134335" Sep 10 23:46:25.249466 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:46:25.249428 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nm2sz" Sep 10 23:46:25.249962 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:46:25.249508 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nm2sz" Sep 10 23:46:25.262069 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:46:25.262047 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nm2sz" Sep 10 23:46:26.240385 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:46:26.240358 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-nm2sz" Sep 10 23:46:30.276989 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:46:30.276957 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-56r7l"] Sep 10 23:46:48.177509 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:46:48.177475 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-56r7l"] Sep 10 23:46:55.977863 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:46:55.977828 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-56r7l"] Sep 10 23:47:25.184269 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:47:25.184186 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-56r7l"] Sep 10 23:47:31.481301 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:47:31.481266 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-56r7l"] Sep 10 23:47:53.190889 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:47:53.190854 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-56r7l"] Sep 10 23:49:19.911387 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:19.911352 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-1fe290-openshift-default-644c884748-msx97"] Sep 10 23:49:19.915223 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:19.915203 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1fe290-openshift-default-644c884748-msx97" Sep 10 23:49:19.917351 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:19.917331 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-1fe290-openshift-default-dockercfg-grprj\"" Sep 10 23:49:19.928334 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:19.928310 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-1fe290-openshift-default-644c884748-msx97"] Sep 10 23:49:20.043498 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:20.043458 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4cee31c3-1da1-4483-a73c-1d0e23fb8063-istio-podinfo\") pod \"e2e-inf-a-1fe290-openshift-default-644c884748-msx97\" (UID: \"4cee31c3-1da1-4483-a73c-1d0e23fb8063\") " pod="openshift-ingress/e2e-inf-a-1fe290-openshift-default-644c884748-msx97" Sep 10 23:49:20.043498 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:20.043501 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4cee31c3-1da1-4483-a73c-1d0e23fb8063-istio-token\") pod \"e2e-inf-a-1fe290-openshift-default-644c884748-msx97\" (UID: \"4cee31c3-1da1-4483-a73c-1d0e23fb8063\") " pod="openshift-ingress/e2e-inf-a-1fe290-openshift-default-644c884748-msx97" Sep 10 23:49:20.043759 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:20.043666 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cee31c3-1da1-4483-a73c-1d0e23fb8063-credential-socket\") pod \"e2e-inf-a-1fe290-openshift-default-644c884748-msx97\" (UID: \"4cee31c3-1da1-4483-a73c-1d0e23fb8063\") " pod="openshift-ingress/e2e-inf-a-1fe290-openshift-default-644c884748-msx97" Sep 10 23:49:20.043759 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:20.043708 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4cee31c3-1da1-4483-a73c-1d0e23fb8063-istiod-ca-cert\") pod \"e2e-inf-a-1fe290-openshift-default-644c884748-msx97\" (UID: \"4cee31c3-1da1-4483-a73c-1d0e23fb8063\") " pod="openshift-ingress/e2e-inf-a-1fe290-openshift-default-644c884748-msx97" Sep 10 23:49:20.043842 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:20.043762 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4cee31c3-1da1-4483-a73c-1d0e23fb8063-istio-data\") pod \"e2e-inf-a-1fe290-openshift-default-644c884748-msx97\" (UID: \"4cee31c3-1da1-4483-a73c-1d0e23fb8063\") " pod="openshift-ingress/e2e-inf-a-1fe290-openshift-default-644c884748-msx97" Sep 10 23:49:20.043842 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:20.043790 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4cee31c3-1da1-4483-a73c-1d0e23fb8063-workload-certs\") pod \"e2e-inf-a-1fe290-openshift-default-644c884748-msx97\" (UID: \"4cee31c3-1da1-4483-a73c-1d0e23fb8063\") " pod="openshift-ingress/e2e-inf-a-1fe290-openshift-default-644c884748-msx97" Sep 10 23:49:20.043842 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:20.043809 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4cee31c3-1da1-4483-a73c-1d0e23fb8063-istio-envoy\") pod \"e2e-inf-a-1fe290-openshift-default-644c884748-msx97\" (UID: \"4cee31c3-1da1-4483-a73c-1d0e23fb8063\") " pod="openshift-ingress/e2e-inf-a-1fe290-openshift-default-644c884748-msx97" Sep 10 23:49:20.043842 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:20.043828 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cee31c3-1da1-4483-a73c-1d0e23fb8063-workload-socket\") pod \"e2e-inf-a-1fe290-openshift-default-644c884748-msx97\" (UID: \"4cee31c3-1da1-4483-a73c-1d0e23fb8063\") " pod="openshift-ingress/e2e-inf-a-1fe290-openshift-default-644c884748-msx97" Sep 10 23:49:20.043973 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:20.043856 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v67w4\" (UniqueName: \"kubernetes.io/projected/4cee31c3-1da1-4483-a73c-1d0e23fb8063-kube-api-access-v67w4\") pod \"e2e-inf-a-1fe290-openshift-default-644c884748-msx97\" (UID: \"4cee31c3-1da1-4483-a73c-1d0e23fb8063\") " pod="openshift-ingress/e2e-inf-a-1fe290-openshift-default-644c884748-msx97" Sep 10 23:49:20.144518 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:20.144480 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cee31c3-1da1-4483-a73c-1d0e23fb8063-credential-socket\") pod \"e2e-inf-a-1fe290-openshift-default-644c884748-msx97\" (UID: \"4cee31c3-1da1-4483-a73c-1d0e23fb8063\") " pod="openshift-ingress/e2e-inf-a-1fe290-openshift-default-644c884748-msx97" Sep 10 23:49:20.144518 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:20.144514 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4cee31c3-1da1-4483-a73c-1d0e23fb8063-istiod-ca-cert\") pod \"e2e-inf-a-1fe290-openshift-default-644c884748-msx97\" (UID: \"4cee31c3-1da1-4483-a73c-1d0e23fb8063\") " pod="openshift-ingress/e2e-inf-a-1fe290-openshift-default-644c884748-msx97" Sep 10 23:49:20.144772 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:20.144562 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4cee31c3-1da1-4483-a73c-1d0e23fb8063-istio-data\") pod \"e2e-inf-a-1fe290-openshift-default-644c884748-msx97\" (UID: \"4cee31c3-1da1-4483-a73c-1d0e23fb8063\") " pod="openshift-ingress/e2e-inf-a-1fe290-openshift-default-644c884748-msx97" Sep 10 23:49:20.144772 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:20.144581 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4cee31c3-1da1-4483-a73c-1d0e23fb8063-workload-certs\") pod \"e2e-inf-a-1fe290-openshift-default-644c884748-msx97\" (UID: \"4cee31c3-1da1-4483-a73c-1d0e23fb8063\") " pod="openshift-ingress/e2e-inf-a-1fe290-openshift-default-644c884748-msx97" Sep 10 23:49:20.144772 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:20.144597 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4cee31c3-1da1-4483-a73c-1d0e23fb8063-istio-envoy\") pod \"e2e-inf-a-1fe290-openshift-default-644c884748-msx97\" (UID: \"4cee31c3-1da1-4483-a73c-1d0e23fb8063\") " pod="openshift-ingress/e2e-inf-a-1fe290-openshift-default-644c884748-msx97" Sep 10 23:49:20.144772 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:20.144621 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cee31c3-1da1-4483-a73c-1d0e23fb8063-workload-socket\") pod \"e2e-inf-a-1fe290-openshift-default-644c884748-msx97\" (UID: \"4cee31c3-1da1-4483-a73c-1d0e23fb8063\") " pod="openshift-ingress/e2e-inf-a-1fe290-openshift-default-644c884748-msx97" Sep 10 23:49:20.144772 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:20.144674 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v67w4\" (UniqueName: \"kubernetes.io/projected/4cee31c3-1da1-4483-a73c-1d0e23fb8063-kube-api-access-v67w4\") pod \"e2e-inf-a-1fe290-openshift-default-644c884748-msx97\" (UID: \"4cee31c3-1da1-4483-a73c-1d0e23fb8063\") " pod="openshift-ingress/e2e-inf-a-1fe290-openshift-default-644c884748-msx97" Sep 10 23:49:20.144772 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:20.144716 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4cee31c3-1da1-4483-a73c-1d0e23fb8063-istio-podinfo\") pod \"e2e-inf-a-1fe290-openshift-default-644c884748-msx97\" (UID: \"4cee31c3-1da1-4483-a73c-1d0e23fb8063\") " pod="openshift-ingress/e2e-inf-a-1fe290-openshift-default-644c884748-msx97" Sep 10 23:49:20.144772 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:20.144753 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4cee31c3-1da1-4483-a73c-1d0e23fb8063-istio-token\") pod \"e2e-inf-a-1fe290-openshift-default-644c884748-msx97\" (UID: \"4cee31c3-1da1-4483-a73c-1d0e23fb8063\") " pod="openshift-ingress/e2e-inf-a-1fe290-openshift-default-644c884748-msx97" Sep 10 23:49:20.145145 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:20.144932 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cee31c3-1da1-4483-a73c-1d0e23fb8063-credential-socket\") pod \"e2e-inf-a-1fe290-openshift-default-644c884748-msx97\" (UID: \"4cee31c3-1da1-4483-a73c-1d0e23fb8063\") " pod="openshift-ingress/e2e-inf-a-1fe290-openshift-default-644c884748-msx97" Sep 10 23:49:20.145145 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:20.144946 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4cee31c3-1da1-4483-a73c-1d0e23fb8063-workload-certs\") pod \"e2e-inf-a-1fe290-openshift-default-644c884748-msx97\" (UID: \"4cee31c3-1da1-4483-a73c-1d0e23fb8063\") " pod="openshift-ingress/e2e-inf-a-1fe290-openshift-default-644c884748-msx97" Sep 10 23:49:20.145145 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:20.144977 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4cee31c3-1da1-4483-a73c-1d0e23fb8063-istio-data\") pod \"e2e-inf-a-1fe290-openshift-default-644c884748-msx97\" (UID: \"4cee31c3-1da1-4483-a73c-1d0e23fb8063\") " pod="openshift-ingress/e2e-inf-a-1fe290-openshift-default-644c884748-msx97" Sep 10 23:49:20.145325 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:20.145159 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cee31c3-1da1-4483-a73c-1d0e23fb8063-workload-socket\") pod \"e2e-inf-a-1fe290-openshift-default-644c884748-msx97\" (UID: \"4cee31c3-1da1-4483-a73c-1d0e23fb8063\") " pod="openshift-ingress/e2e-inf-a-1fe290-openshift-default-644c884748-msx97" Sep 10 23:49:20.145443 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:20.145421 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4cee31c3-1da1-4483-a73c-1d0e23fb8063-istiod-ca-cert\") pod \"e2e-inf-a-1fe290-openshift-default-644c884748-msx97\" (UID: \"4cee31c3-1da1-4483-a73c-1d0e23fb8063\") " pod="openshift-ingress/e2e-inf-a-1fe290-openshift-default-644c884748-msx97" Sep 10 23:49:20.147102 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:20.147079 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4cee31c3-1da1-4483-a73c-1d0e23fb8063-istio-envoy\") pod \"e2e-inf-a-1fe290-openshift-default-644c884748-msx97\" (UID: \"4cee31c3-1da1-4483-a73c-1d0e23fb8063\") " pod="openshift-ingress/e2e-inf-a-1fe290-openshift-default-644c884748-msx97" Sep 10 23:49:20.147234 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:20.147216 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4cee31c3-1da1-4483-a73c-1d0e23fb8063-istio-podinfo\") pod \"e2e-inf-a-1fe290-openshift-default-644c884748-msx97\" (UID: \"4cee31c3-1da1-4483-a73c-1d0e23fb8063\") " pod="openshift-ingress/e2e-inf-a-1fe290-openshift-default-644c884748-msx97" Sep 10 23:49:20.153048 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:20.153021 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4cee31c3-1da1-4483-a73c-1d0e23fb8063-istio-token\") pod \"e2e-inf-a-1fe290-openshift-default-644c884748-msx97\" (UID: \"4cee31c3-1da1-4483-a73c-1d0e23fb8063\") " pod="openshift-ingress/e2e-inf-a-1fe290-openshift-default-644c884748-msx97" Sep 10 23:49:20.153632 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:20.153614 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v67w4\" (UniqueName: \"kubernetes.io/projected/4cee31c3-1da1-4483-a73c-1d0e23fb8063-kube-api-access-v67w4\") pod \"e2e-inf-a-1fe290-openshift-default-644c884748-msx97\" (UID: \"4cee31c3-1da1-4483-a73c-1d0e23fb8063\") " pod="openshift-ingress/e2e-inf-a-1fe290-openshift-default-644c884748-msx97" Sep 10 23:49:20.226742 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:20.226705 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1fe290-openshift-default-644c884748-msx97" Sep 10 23:49:20.507240 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:49:20.507208 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cee31c3_1da1_4483_a73c_1d0e23fb8063.slice/crio-caddce181bb5ce5a03dd5d72c27e9cdaf688716fb6fc14c20c4bf42ce8ddbee8 WatchSource:0}: Error finding container caddce181bb5ce5a03dd5d72c27e9cdaf688716fb6fc14c20c4bf42ce8ddbee8: Status 404 returned error can't find the container with id caddce181bb5ce5a03dd5d72c27e9cdaf688716fb6fc14c20c4bf42ce8ddbee8 Sep 10 23:49:20.509484 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:20.509444 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 23:49:20.509590 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:20.509527 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 23:49:20.509590 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:20.509574 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 23:49:20.510366 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:20.510339 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-1fe290-openshift-default-644c884748-msx97"] Sep 10 23:49:21.242565 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:21.242529 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1fe290-openshift-default-644c884748-msx97" event={"ID":"4cee31c3-1da1-4483-a73c-1d0e23fb8063","Type":"ContainerStarted","Data":"f26d88deff51be711fcb129535330522310814d64fe4bd8fb0933ee0ff9f16c1"} Sep 10 23:49:21.243056 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:21.242580 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1fe290-openshift-default-644c884748-msx97" event={"ID":"4cee31c3-1da1-4483-a73c-1d0e23fb8063","Type":"ContainerStarted","Data":"caddce181bb5ce5a03dd5d72c27e9cdaf688716fb6fc14c20c4bf42ce8ddbee8"} Sep 10 23:49:21.263860 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:21.263805 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-1fe290-openshift-default-644c884748-msx97" podStartSLOduration=2.2637869139999998 podStartE2EDuration="2.263786914s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:49:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:49:21.262836904 +0000 UTC m=+752.650439687" watchObservedRunningTime="2026-09-10 23:49:21.263786914 +0000 UTC m=+752.651389673" Sep 10 23:49:22.227383 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:22.227347 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-1fe290-openshift-default-644c884748-msx97" Sep 10 23:49:22.232294 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:22.232266 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-1fe290-openshift-default-644c884748-msx97" Sep 10 23:49:22.247766 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:22.247732 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-1fe290-openshift-default-644c884748-msx97" Sep 10 23:49:22.248561 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:22.248533 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-1fe290-openshift-default-644c884748-msx97" Sep 10 23:49:23.094436 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:23.094407 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-56r7l"] Sep 10 23:49:25.184803 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:25.184772 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1fe290-68d856fdd6-wsm2c"] Sep 10 23:49:25.188537 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:25.188514 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1fe290-68d856fdd6-wsm2c" Sep 10 23:49:25.191037 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:25.191014 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-1fe290-dockercfg-z46jc\"" Sep 10 23:49:25.191037 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:25.191028 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-1fe290\"" Sep 10 23:49:25.198472 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:25.198444 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1fe290-68d856fdd6-wsm2c"] Sep 10 23:49:25.290624 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:25.290587 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p64sl\" (UniqueName: \"kubernetes.io/projected/cf7b6c3f-2561-4739-afa5-98e98f6b7ebf-kube-api-access-p64sl\") pod \"payload-pre-processing-e2e-inf-a-1fe290-68d856fdd6-wsm2c\" (UID: \"cf7b6c3f-2561-4739-afa5-98e98f6b7ebf\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1fe290-68d856fdd6-wsm2c" Sep 10 23:49:25.290820 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:25.290777 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf7b6c3f-2561-4739-afa5-98e98f6b7ebf-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-1fe290-68d856fdd6-wsm2c\" (UID: \"cf7b6c3f-2561-4739-afa5-98e98f6b7ebf\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1fe290-68d856fdd6-wsm2c" Sep 10 23:49:25.391418 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:25.391381 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf7b6c3f-2561-4739-afa5-98e98f6b7ebf-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-1fe290-68d856fdd6-wsm2c\" (UID: \"cf7b6c3f-2561-4739-afa5-98e98f6b7ebf\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1fe290-68d856fdd6-wsm2c" Sep 10 23:49:25.391584 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:25.391440 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p64sl\" (UniqueName: \"kubernetes.io/projected/cf7b6c3f-2561-4739-afa5-98e98f6b7ebf-kube-api-access-p64sl\") pod \"payload-pre-processing-e2e-inf-a-1fe290-68d856fdd6-wsm2c\" (UID: \"cf7b6c3f-2561-4739-afa5-98e98f6b7ebf\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1fe290-68d856fdd6-wsm2c" Sep 10 23:49:25.392171 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:25.392150 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf7b6c3f-2561-4739-afa5-98e98f6b7ebf-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-1fe290-68d856fdd6-wsm2c\" (UID: \"cf7b6c3f-2561-4739-afa5-98e98f6b7ebf\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1fe290-68d856fdd6-wsm2c" Sep 10 23:49:25.399978 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:25.399945 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p64sl\" (UniqueName: \"kubernetes.io/projected/cf7b6c3f-2561-4739-afa5-98e98f6b7ebf-kube-api-access-p64sl\") pod \"payload-pre-processing-e2e-inf-a-1fe290-68d856fdd6-wsm2c\" (UID: \"cf7b6c3f-2561-4739-afa5-98e98f6b7ebf\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1fe290-68d856fdd6-wsm2c" Sep 10 23:49:25.501891 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:25.501854 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1fe290-68d856fdd6-wsm2c" Sep 10 23:49:25.779942 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:25.779912 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1fe290-68d856fdd6-wsm2c"] Sep 10 23:49:25.782268 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:49:25.782237 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf7b6c3f_2561_4739_afa5_98e98f6b7ebf.slice/crio-7ac6c893c8d7b57871423ea06d000b02ba2146312c0544110130b400eac64f4f WatchSource:0}: Error finding container 7ac6c893c8d7b57871423ea06d000b02ba2146312c0544110130b400eac64f4f: Status 404 returned error can't find the container with id 7ac6c893c8d7b57871423ea06d000b02ba2146312c0544110130b400eac64f4f Sep 10 23:49:26.283542 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:26.283494 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1fe290-68d856fdd6-wsm2c" event={"ID":"cf7b6c3f-2561-4739-afa5-98e98f6b7ebf","Type":"ContainerStarted","Data":"83ab22a388d5ff81363e120c34bbb3560cba324700f3d9b0afc7cf1b3e251b0a"} Sep 10 23:49:26.284029 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:26.283606 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1fe290-68d856fdd6-wsm2c" event={"ID":"cf7b6c3f-2561-4739-afa5-98e98f6b7ebf","Type":"ContainerStarted","Data":"7ac6c893c8d7b57871423ea06d000b02ba2146312c0544110130b400eac64f4f"} Sep 10 23:49:26.301566 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:26.301513 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1fe290-68d856fdd6-wsm2c" podStartSLOduration=1.301495283 podStartE2EDuration="1.301495283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:49:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:49:26.299270961 +0000 UTC m=+757.686873721" watchObservedRunningTime="2026-09-10 23:49:26.301495283 +0000 UTC m=+757.689098044" Sep 10 23:49:26.518832 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:26.518778 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-74f1c1-openshift-default-5545549c98-z628g"] Sep 10 23:49:26.522568 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:26.522539 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-74f1c1-openshift-default-5545549c98-z628g" Sep 10 23:49:26.525239 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:26.525212 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-74f1c1-openshift-default-dockercfg-q9vtm\"" Sep 10 23:49:26.534060 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:26.533993 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-74f1c1-openshift-default-5545549c98-z628g"] Sep 10 23:49:26.602414 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:26.602392 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5970ef76-234b-4f98-9f7b-22db89263e2a-istio-envoy\") pod \"e2e-autores-74f1c1-openshift-default-5545549c98-z628g\" (UID: \"5970ef76-234b-4f98-9f7b-22db89263e2a\") " pod="openshift-ingress/e2e-autores-74f1c1-openshift-default-5545549c98-z628g" Sep 10 23:49:26.602552 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:26.602425 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5970ef76-234b-4f98-9f7b-22db89263e2a-istio-data\") pod \"e2e-autores-74f1c1-openshift-default-5545549c98-z628g\" (UID: \"5970ef76-234b-4f98-9f7b-22db89263e2a\") " pod="openshift-ingress/e2e-autores-74f1c1-openshift-default-5545549c98-z628g" Sep 10 23:49:26.602552 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:26.602445 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5970ef76-234b-4f98-9f7b-22db89263e2a-credential-socket\") pod \"e2e-autores-74f1c1-openshift-default-5545549c98-z628g\" (UID: \"5970ef76-234b-4f98-9f7b-22db89263e2a\") " pod="openshift-ingress/e2e-autores-74f1c1-openshift-default-5545549c98-z628g" Sep 10 23:49:26.602552 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:26.602493 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5970ef76-234b-4f98-9f7b-22db89263e2a-workload-certs\") pod \"e2e-autores-74f1c1-openshift-default-5545549c98-z628g\" (UID: \"5970ef76-234b-4f98-9f7b-22db89263e2a\") " pod="openshift-ingress/e2e-autores-74f1c1-openshift-default-5545549c98-z628g" Sep 10 23:49:26.602552 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:26.602533 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jw9d\" (UniqueName: \"kubernetes.io/projected/5970ef76-234b-4f98-9f7b-22db89263e2a-kube-api-access-4jw9d\") pod \"e2e-autores-74f1c1-openshift-default-5545549c98-z628g\" (UID: \"5970ef76-234b-4f98-9f7b-22db89263e2a\") " pod="openshift-ingress/e2e-autores-74f1c1-openshift-default-5545549c98-z628g" Sep 10 23:49:26.602740 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:26.602603 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5970ef76-234b-4f98-9f7b-22db89263e2a-istiod-ca-cert\") pod \"e2e-autores-74f1c1-openshift-default-5545549c98-z628g\" (UID: \"5970ef76-234b-4f98-9f7b-22db89263e2a\") " pod="openshift-ingress/e2e-autores-74f1c1-openshift-default-5545549c98-z628g" Sep 10 23:49:26.602740 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:26.602665 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5970ef76-234b-4f98-9f7b-22db89263e2a-istio-podinfo\") pod \"e2e-autores-74f1c1-openshift-default-5545549c98-z628g\" (UID: \"5970ef76-234b-4f98-9f7b-22db89263e2a\") " pod="openshift-ingress/e2e-autores-74f1c1-openshift-default-5545549c98-z628g" Sep 10 23:49:26.602740 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:26.602695 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5970ef76-234b-4f98-9f7b-22db89263e2a-workload-socket\") pod \"e2e-autores-74f1c1-openshift-default-5545549c98-z628g\" (UID: \"5970ef76-234b-4f98-9f7b-22db89263e2a\") " pod="openshift-ingress/e2e-autores-74f1c1-openshift-default-5545549c98-z628g" Sep 10 23:49:26.602740 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:26.602724 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5970ef76-234b-4f98-9f7b-22db89263e2a-istio-token\") pod \"e2e-autores-74f1c1-openshift-default-5545549c98-z628g\" (UID: \"5970ef76-234b-4f98-9f7b-22db89263e2a\") " pod="openshift-ingress/e2e-autores-74f1c1-openshift-default-5545549c98-z628g" Sep 10 23:49:26.703611 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:26.703576 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5970ef76-234b-4f98-9f7b-22db89263e2a-istio-podinfo\") pod \"e2e-autores-74f1c1-openshift-default-5545549c98-z628g\" (UID: \"5970ef76-234b-4f98-9f7b-22db89263e2a\") " pod="openshift-ingress/e2e-autores-74f1c1-openshift-default-5545549c98-z628g" Sep 10 23:49:26.703804 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:26.703625 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5970ef76-234b-4f98-9f7b-22db89263e2a-workload-socket\") pod \"e2e-autores-74f1c1-openshift-default-5545549c98-z628g\" (UID: \"5970ef76-234b-4f98-9f7b-22db89263e2a\") " pod="openshift-ingress/e2e-autores-74f1c1-openshift-default-5545549c98-z628g" Sep 10 23:49:26.703804 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:26.703694 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5970ef76-234b-4f98-9f7b-22db89263e2a-istio-token\") pod \"e2e-autores-74f1c1-openshift-default-5545549c98-z628g\" (UID: \"5970ef76-234b-4f98-9f7b-22db89263e2a\") " pod="openshift-ingress/e2e-autores-74f1c1-openshift-default-5545549c98-z628g" Sep 10 23:49:26.703804 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:26.703751 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5970ef76-234b-4f98-9f7b-22db89263e2a-istio-envoy\") pod \"e2e-autores-74f1c1-openshift-default-5545549c98-z628g\" (UID: \"5970ef76-234b-4f98-9f7b-22db89263e2a\") " pod="openshift-ingress/e2e-autores-74f1c1-openshift-default-5545549c98-z628g" Sep 10 23:49:26.703961 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:26.703872 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5970ef76-234b-4f98-9f7b-22db89263e2a-istio-data\") pod \"e2e-autores-74f1c1-openshift-default-5545549c98-z628g\" (UID: \"5970ef76-234b-4f98-9f7b-22db89263e2a\") " pod="openshift-ingress/e2e-autores-74f1c1-openshift-default-5545549c98-z628g" Sep 10 23:49:26.703961 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:26.703929 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5970ef76-234b-4f98-9f7b-22db89263e2a-credential-socket\") pod \"e2e-autores-74f1c1-openshift-default-5545549c98-z628g\" (UID: \"5970ef76-234b-4f98-9f7b-22db89263e2a\") " pod="openshift-ingress/e2e-autores-74f1c1-openshift-default-5545549c98-z628g" Sep 10 23:49:26.704084 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:26.703960 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5970ef76-234b-4f98-9f7b-22db89263e2a-workload-certs\") pod \"e2e-autores-74f1c1-openshift-default-5545549c98-z628g\" (UID: \"5970ef76-234b-4f98-9f7b-22db89263e2a\") " pod="openshift-ingress/e2e-autores-74f1c1-openshift-default-5545549c98-z628g" Sep 10 23:49:26.704084 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:26.704004 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jw9d\" (UniqueName: \"kubernetes.io/projected/5970ef76-234b-4f98-9f7b-22db89263e2a-kube-api-access-4jw9d\") pod \"e2e-autores-74f1c1-openshift-default-5545549c98-z628g\" (UID: \"5970ef76-234b-4f98-9f7b-22db89263e2a\") " pod="openshift-ingress/e2e-autores-74f1c1-openshift-default-5545549c98-z628g" Sep 10 23:49:26.704084 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:26.704077 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5970ef76-234b-4f98-9f7b-22db89263e2a-istiod-ca-cert\") pod \"e2e-autores-74f1c1-openshift-default-5545549c98-z628g\" (UID: \"5970ef76-234b-4f98-9f7b-22db89263e2a\") " pod="openshift-ingress/e2e-autores-74f1c1-openshift-default-5545549c98-z628g" Sep 10 23:49:26.704243 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:26.704205 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5970ef76-234b-4f98-9f7b-22db89263e2a-workload-socket\") pod \"e2e-autores-74f1c1-openshift-default-5545549c98-z628g\" (UID: \"5970ef76-234b-4f98-9f7b-22db89263e2a\") " pod="openshift-ingress/e2e-autores-74f1c1-openshift-default-5545549c98-z628g" Sep 10 23:49:26.704589 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:26.704559 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5970ef76-234b-4f98-9f7b-22db89263e2a-credential-socket\") pod \"e2e-autores-74f1c1-openshift-default-5545549c98-z628g\" (UID: \"5970ef76-234b-4f98-9f7b-22db89263e2a\") " pod="openshift-ingress/e2e-autores-74f1c1-openshift-default-5545549c98-z628g" Sep 10 23:49:26.704589 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:26.704577 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5970ef76-234b-4f98-9f7b-22db89263e2a-istio-data\") pod \"e2e-autores-74f1c1-openshift-default-5545549c98-z628g\" (UID: \"5970ef76-234b-4f98-9f7b-22db89263e2a\") " pod="openshift-ingress/e2e-autores-74f1c1-openshift-default-5545549c98-z628g" Sep 10 23:49:26.704829 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:26.704672 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5970ef76-234b-4f98-9f7b-22db89263e2a-workload-certs\") pod \"e2e-autores-74f1c1-openshift-default-5545549c98-z628g\" (UID: \"5970ef76-234b-4f98-9f7b-22db89263e2a\") " pod="openshift-ingress/e2e-autores-74f1c1-openshift-default-5545549c98-z628g" Sep 10 23:49:26.704884 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:26.704835 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5970ef76-234b-4f98-9f7b-22db89263e2a-istiod-ca-cert\") pod \"e2e-autores-74f1c1-openshift-default-5545549c98-z628g\" (UID: \"5970ef76-234b-4f98-9f7b-22db89263e2a\") " pod="openshift-ingress/e2e-autores-74f1c1-openshift-default-5545549c98-z628g" Sep 10 23:49:26.706086 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:26.706057 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5970ef76-234b-4f98-9f7b-22db89263e2a-istio-envoy\") pod \"e2e-autores-74f1c1-openshift-default-5545549c98-z628g\" (UID: \"5970ef76-234b-4f98-9f7b-22db89263e2a\") " pod="openshift-ingress/e2e-autores-74f1c1-openshift-default-5545549c98-z628g" Sep 10 23:49:26.706281 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:26.706262 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5970ef76-234b-4f98-9f7b-22db89263e2a-istio-podinfo\") pod \"e2e-autores-74f1c1-openshift-default-5545549c98-z628g\" (UID: \"5970ef76-234b-4f98-9f7b-22db89263e2a\") " pod="openshift-ingress/e2e-autores-74f1c1-openshift-default-5545549c98-z628g" Sep 10 23:49:26.711173 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:26.711149 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5970ef76-234b-4f98-9f7b-22db89263e2a-istio-token\") pod \"e2e-autores-74f1c1-openshift-default-5545549c98-z628g\" (UID: \"5970ef76-234b-4f98-9f7b-22db89263e2a\") " pod="openshift-ingress/e2e-autores-74f1c1-openshift-default-5545549c98-z628g" Sep 10 23:49:26.711359 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:26.711339 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jw9d\" (UniqueName: \"kubernetes.io/projected/5970ef76-234b-4f98-9f7b-22db89263e2a-kube-api-access-4jw9d\") pod \"e2e-autores-74f1c1-openshift-default-5545549c98-z628g\" (UID: \"5970ef76-234b-4f98-9f7b-22db89263e2a\") " pod="openshift-ingress/e2e-autores-74f1c1-openshift-default-5545549c98-z628g" Sep 10 23:49:26.835299 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:26.835239 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-74f1c1-openshift-default-5545549c98-z628g" Sep 10 23:49:27.114219 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:27.114162 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-74f1c1-openshift-default-5545549c98-z628g"] Sep 10 23:49:27.116900 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:49:27.116857 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5970ef76_234b_4f98_9f7b_22db89263e2a.slice/crio-0aa2f2fa4ef7de17b0b5b2182502fa3fc5fbd7d7bc130851d71af09db8a5bbf3 WatchSource:0}: Error finding container 0aa2f2fa4ef7de17b0b5b2182502fa3fc5fbd7d7bc130851d71af09db8a5bbf3: Status 404 returned error can't find the container with id 0aa2f2fa4ef7de17b0b5b2182502fa3fc5fbd7d7bc130851d71af09db8a5bbf3 Sep 10 23:49:27.119345 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:27.119303 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 23:49:27.119427 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:27.119399 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 23:49:27.119467 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:27.119444 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 23:49:27.294319 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:27.294277 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-74f1c1-openshift-default-5545549c98-z628g" event={"ID":"5970ef76-234b-4f98-9f7b-22db89263e2a","Type":"ContainerStarted","Data":"8f0a0304ff1487247e7b71b8006745327b1e9f45b9e3513bda575e0ab97c66db"} Sep 10 23:49:27.295023 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:27.294999 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1fe290-68d856fdd6-wsm2c" Sep 10 23:49:27.295180 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:27.295057 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-74f1c1-openshift-default-5545549c98-z628g" event={"ID":"5970ef76-234b-4f98-9f7b-22db89263e2a","Type":"ContainerStarted","Data":"0aa2f2fa4ef7de17b0b5b2182502fa3fc5fbd7d7bc130851d71af09db8a5bbf3"} Sep 10 23:49:27.324999 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:27.324948 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-74f1c1-openshift-default-5545549c98-z628g" podStartSLOduration=1.324934727 podStartE2EDuration="1.324934727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:49:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:49:27.323032968 +0000 UTC m=+758.710635738" watchObservedRunningTime="2026-09-10 23:49:27.324934727 +0000 UTC m=+758.712537486" Sep 10 23:49:27.836255 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:27.836221 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-74f1c1-openshift-default-5545549c98-z628g" Sep 10 23:49:28.537984 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:28.537950 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-e027297d-openshift-default-5cc9d886d4-bggs4"] Sep 10 23:49:28.542158 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:28.542133 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e027297d-openshift-default-5cc9d886d4-bggs4" Sep 10 23:49:28.544310 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:28.544286 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-e027297d-openshift-default-dockercfg-5cwp5\"" Sep 10 23:49:28.555658 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:28.555616 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-e027297d-openshift-default-5cc9d886d4-bggs4"] Sep 10 23:49:28.623376 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:28.623332 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b61c82a2-ced4-4c1c-8b44-4f94593d5abb-istio-token\") pod \"e2e-ait-e027297d-openshift-default-5cc9d886d4-bggs4\" (UID: \"b61c82a2-ced4-4c1c-8b44-4f94593d5abb\") " pod="openshift-ingress/e2e-ait-e027297d-openshift-default-5cc9d886d4-bggs4" Sep 10 23:49:28.623561 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:28.623385 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b61c82a2-ced4-4c1c-8b44-4f94593d5abb-workload-socket\") pod \"e2e-ait-e027297d-openshift-default-5cc9d886d4-bggs4\" (UID: \"b61c82a2-ced4-4c1c-8b44-4f94593d5abb\") " pod="openshift-ingress/e2e-ait-e027297d-openshift-default-5cc9d886d4-bggs4" Sep 10 23:49:28.623561 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:28.623479 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b61c82a2-ced4-4c1c-8b44-4f94593d5abb-credential-socket\") pod \"e2e-ait-e027297d-openshift-default-5cc9d886d4-bggs4\" (UID: \"b61c82a2-ced4-4c1c-8b44-4f94593d5abb\") " pod="openshift-ingress/e2e-ait-e027297d-openshift-default-5cc9d886d4-bggs4" Sep 10 23:49:28.623561 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:28.623530 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b61c82a2-ced4-4c1c-8b44-4f94593d5abb-istiod-ca-cert\") pod \"e2e-ait-e027297d-openshift-default-5cc9d886d4-bggs4\" (UID: \"b61c82a2-ced4-4c1c-8b44-4f94593d5abb\") " pod="openshift-ingress/e2e-ait-e027297d-openshift-default-5cc9d886d4-bggs4" Sep 10 23:49:28.623749 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:28.623562 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b61c82a2-ced4-4c1c-8b44-4f94593d5abb-istio-envoy\") pod \"e2e-ait-e027297d-openshift-default-5cc9d886d4-bggs4\" (UID: \"b61c82a2-ced4-4c1c-8b44-4f94593d5abb\") " pod="openshift-ingress/e2e-ait-e027297d-openshift-default-5cc9d886d4-bggs4" Sep 10 23:49:28.623749 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:28.623612 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b61c82a2-ced4-4c1c-8b44-4f94593d5abb-workload-certs\") pod \"e2e-ait-e027297d-openshift-default-5cc9d886d4-bggs4\" (UID: \"b61c82a2-ced4-4c1c-8b44-4f94593d5abb\") " pod="openshift-ingress/e2e-ait-e027297d-openshift-default-5cc9d886d4-bggs4" Sep 10 23:49:28.623749 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:28.623658 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gbcg\" (UniqueName: \"kubernetes.io/projected/b61c82a2-ced4-4c1c-8b44-4f94593d5abb-kube-api-access-5gbcg\") pod \"e2e-ait-e027297d-openshift-default-5cc9d886d4-bggs4\" (UID: \"b61c82a2-ced4-4c1c-8b44-4f94593d5abb\") " pod="openshift-ingress/e2e-ait-e027297d-openshift-default-5cc9d886d4-bggs4" Sep 10 23:49:28.623749 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:28.623729 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b61c82a2-ced4-4c1c-8b44-4f94593d5abb-istio-data\") pod \"e2e-ait-e027297d-openshift-default-5cc9d886d4-bggs4\" (UID: \"b61c82a2-ced4-4c1c-8b44-4f94593d5abb\") " pod="openshift-ingress/e2e-ait-e027297d-openshift-default-5cc9d886d4-bggs4" Sep 10 23:49:28.623913 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:28.623748 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b61c82a2-ced4-4c1c-8b44-4f94593d5abb-istio-podinfo\") pod \"e2e-ait-e027297d-openshift-default-5cc9d886d4-bggs4\" (UID: \"b61c82a2-ced4-4c1c-8b44-4f94593d5abb\") " pod="openshift-ingress/e2e-ait-e027297d-openshift-default-5cc9d886d4-bggs4" Sep 10 23:49:28.725028 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:28.724993 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b61c82a2-ced4-4c1c-8b44-4f94593d5abb-istio-token\") pod \"e2e-ait-e027297d-openshift-default-5cc9d886d4-bggs4\" (UID: \"b61c82a2-ced4-4c1c-8b44-4f94593d5abb\") " pod="openshift-ingress/e2e-ait-e027297d-openshift-default-5cc9d886d4-bggs4" Sep 10 23:49:28.725028 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:28.725039 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b61c82a2-ced4-4c1c-8b44-4f94593d5abb-workload-socket\") pod \"e2e-ait-e027297d-openshift-default-5cc9d886d4-bggs4\" (UID: \"b61c82a2-ced4-4c1c-8b44-4f94593d5abb\") " pod="openshift-ingress/e2e-ait-e027297d-openshift-default-5cc9d886d4-bggs4" Sep 10 23:49:28.725264 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:28.725081 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b61c82a2-ced4-4c1c-8b44-4f94593d5abb-credential-socket\") pod \"e2e-ait-e027297d-openshift-default-5cc9d886d4-bggs4\" (UID: \"b61c82a2-ced4-4c1c-8b44-4f94593d5abb\") " pod="openshift-ingress/e2e-ait-e027297d-openshift-default-5cc9d886d4-bggs4" Sep 10 23:49:28.725264 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:28.725115 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b61c82a2-ced4-4c1c-8b44-4f94593d5abb-istiod-ca-cert\") pod \"e2e-ait-e027297d-openshift-default-5cc9d886d4-bggs4\" (UID: \"b61c82a2-ced4-4c1c-8b44-4f94593d5abb\") " pod="openshift-ingress/e2e-ait-e027297d-openshift-default-5cc9d886d4-bggs4" Sep 10 23:49:28.725264 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:28.725142 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b61c82a2-ced4-4c1c-8b44-4f94593d5abb-istio-envoy\") pod \"e2e-ait-e027297d-openshift-default-5cc9d886d4-bggs4\" (UID: \"b61c82a2-ced4-4c1c-8b44-4f94593d5abb\") " pod="openshift-ingress/e2e-ait-e027297d-openshift-default-5cc9d886d4-bggs4" Sep 10 23:49:28.725264 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:28.725175 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b61c82a2-ced4-4c1c-8b44-4f94593d5abb-workload-certs\") pod \"e2e-ait-e027297d-openshift-default-5cc9d886d4-bggs4\" (UID: \"b61c82a2-ced4-4c1c-8b44-4f94593d5abb\") " pod="openshift-ingress/e2e-ait-e027297d-openshift-default-5cc9d886d4-bggs4" Sep 10 23:49:28.725470 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:28.725259 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gbcg\" (UniqueName: \"kubernetes.io/projected/b61c82a2-ced4-4c1c-8b44-4f94593d5abb-kube-api-access-5gbcg\") pod \"e2e-ait-e027297d-openshift-default-5cc9d886d4-bggs4\" (UID: \"b61c82a2-ced4-4c1c-8b44-4f94593d5abb\") " pod="openshift-ingress/e2e-ait-e027297d-openshift-default-5cc9d886d4-bggs4" Sep 10 23:49:28.725470 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:28.725344 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b61c82a2-ced4-4c1c-8b44-4f94593d5abb-istio-data\") pod \"e2e-ait-e027297d-openshift-default-5cc9d886d4-bggs4\" (UID: \"b61c82a2-ced4-4c1c-8b44-4f94593d5abb\") " pod="openshift-ingress/e2e-ait-e027297d-openshift-default-5cc9d886d4-bggs4" Sep 10 23:49:28.725470 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:28.725373 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b61c82a2-ced4-4c1c-8b44-4f94593d5abb-istio-podinfo\") pod \"e2e-ait-e027297d-openshift-default-5cc9d886d4-bggs4\" (UID: \"b61c82a2-ced4-4c1c-8b44-4f94593d5abb\") " pod="openshift-ingress/e2e-ait-e027297d-openshift-default-5cc9d886d4-bggs4" Sep 10 23:49:28.725702 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:28.725528 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b61c82a2-ced4-4c1c-8b44-4f94593d5abb-workload-socket\") pod \"e2e-ait-e027297d-openshift-default-5cc9d886d4-bggs4\" (UID: \"b61c82a2-ced4-4c1c-8b44-4f94593d5abb\") " pod="openshift-ingress/e2e-ait-e027297d-openshift-default-5cc9d886d4-bggs4" Sep 10 23:49:28.725702 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:28.725591 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b61c82a2-ced4-4c1c-8b44-4f94593d5abb-credential-socket\") pod \"e2e-ait-e027297d-openshift-default-5cc9d886d4-bggs4\" (UID: \"b61c82a2-ced4-4c1c-8b44-4f94593d5abb\") " pod="openshift-ingress/e2e-ait-e027297d-openshift-default-5cc9d886d4-bggs4" Sep 10 23:49:28.725702 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:28.725655 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b61c82a2-ced4-4c1c-8b44-4f94593d5abb-workload-certs\") pod \"e2e-ait-e027297d-openshift-default-5cc9d886d4-bggs4\" (UID: \"b61c82a2-ced4-4c1c-8b44-4f94593d5abb\") " pod="openshift-ingress/e2e-ait-e027297d-openshift-default-5cc9d886d4-bggs4" Sep 10 23:49:28.725871 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:28.725752 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b61c82a2-ced4-4c1c-8b44-4f94593d5abb-istio-data\") pod \"e2e-ait-e027297d-openshift-default-5cc9d886d4-bggs4\" (UID: \"b61c82a2-ced4-4c1c-8b44-4f94593d5abb\") " pod="openshift-ingress/e2e-ait-e027297d-openshift-default-5cc9d886d4-bggs4" Sep 10 23:49:28.726088 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:28.726067 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b61c82a2-ced4-4c1c-8b44-4f94593d5abb-istiod-ca-cert\") pod \"e2e-ait-e027297d-openshift-default-5cc9d886d4-bggs4\" (UID: \"b61c82a2-ced4-4c1c-8b44-4f94593d5abb\") " pod="openshift-ingress/e2e-ait-e027297d-openshift-default-5cc9d886d4-bggs4" Sep 10 23:49:28.727492 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:28.727470 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b61c82a2-ced4-4c1c-8b44-4f94593d5abb-istio-envoy\") pod \"e2e-ait-e027297d-openshift-default-5cc9d886d4-bggs4\" (UID: \"b61c82a2-ced4-4c1c-8b44-4f94593d5abb\") " pod="openshift-ingress/e2e-ait-e027297d-openshift-default-5cc9d886d4-bggs4" Sep 10 23:49:28.727826 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:28.727809 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b61c82a2-ced4-4c1c-8b44-4f94593d5abb-istio-podinfo\") pod \"e2e-ait-e027297d-openshift-default-5cc9d886d4-bggs4\" (UID: \"b61c82a2-ced4-4c1c-8b44-4f94593d5abb\") " pod="openshift-ingress/e2e-ait-e027297d-openshift-default-5cc9d886d4-bggs4" Sep 10 23:49:28.740657 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:28.740616 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b61c82a2-ced4-4c1c-8b44-4f94593d5abb-istio-token\") pod \"e2e-ait-e027297d-openshift-default-5cc9d886d4-bggs4\" (UID: \"b61c82a2-ced4-4c1c-8b44-4f94593d5abb\") " pod="openshift-ingress/e2e-ait-e027297d-openshift-default-5cc9d886d4-bggs4" Sep 10 23:49:28.742033 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:28.742008 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gbcg\" (UniqueName: \"kubernetes.io/projected/b61c82a2-ced4-4c1c-8b44-4f94593d5abb-kube-api-access-5gbcg\") pod \"e2e-ait-e027297d-openshift-default-5cc9d886d4-bggs4\" (UID: \"b61c82a2-ced4-4c1c-8b44-4f94593d5abb\") " pod="openshift-ingress/e2e-ait-e027297d-openshift-default-5cc9d886d4-bggs4" Sep 10 23:49:28.841395 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:28.841320 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-74f1c1-openshift-default-5545549c98-z628g" Sep 10 23:49:28.854827 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:28.854801 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e027297d-openshift-default-5cc9d886d4-bggs4" Sep 10 23:49:29.157336 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:29.157301 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-e027297d-openshift-default-5cc9d886d4-bggs4"] Sep 10 23:49:29.157728 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:49:29.157705 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb61c82a2_ced4_4c1c_8b44_4f94593d5abb.slice/crio-da7962a56e32bd719ff491d5138a73e5e4387f618086817475f23340423e189e WatchSource:0}: Error finding container da7962a56e32bd719ff491d5138a73e5e4387f618086817475f23340423e189e: Status 404 returned error can't find the container with id da7962a56e32bd719ff491d5138a73e5e4387f618086817475f23340423e189e Sep 10 23:49:29.159933 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:29.159903 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 23:49:29.159997 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:29.159963 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 23:49:29.159997 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:29.159992 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 23:49:29.311276 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:29.311232 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e027297d-openshift-default-5cc9d886d4-bggs4" event={"ID":"b61c82a2-ced4-4c1c-8b44-4f94593d5abb","Type":"ContainerStarted","Data":"b18d24bfb72bed217f40b2d61999712cc42e3108886e685f90a866207a089b32"} Sep 10 23:49:29.311485 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:29.311339 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e027297d-openshift-default-5cc9d886d4-bggs4" event={"ID":"b61c82a2-ced4-4c1c-8b44-4f94593d5abb","Type":"ContainerStarted","Data":"da7962a56e32bd719ff491d5138a73e5e4387f618086817475f23340423e189e"} Sep 10 23:49:29.311781 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:29.311756 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-74f1c1-openshift-default-5545549c98-z628g" Sep 10 23:49:29.312762 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:29.312738 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-74f1c1-openshift-default-5545549c98-z628g" Sep 10 23:49:29.337479 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:29.337420 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-e027297d-openshift-default-5cc9d886d4-bggs4" podStartSLOduration=1.337399257 podStartE2EDuration="1.337399257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:49:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:49:29.335200092 +0000 UTC m=+760.722802851" watchObservedRunningTime="2026-09-10 23:49:29.337399257 +0000 UTC m=+760.725002018" Sep 10 23:49:29.855613 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:29.855572 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-e027297d-openshift-default-5cc9d886d4-bggs4" Sep 10 23:49:30.860602 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:30.860572 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-e027297d-openshift-default-5cc9d886d4-bggs4" Sep 10 23:49:31.327005 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:31.326972 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-e027297d-openshift-default-5cc9d886d4-bggs4" Sep 10 23:49:31.328289 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:31.328267 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-e027297d-openshift-default-5cc9d886d4-bggs4" Sep 10 23:49:32.962085 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:32.962049 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e027297d-94968cf8b-lrn5f"] Sep 10 23:49:32.978665 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:32.965871 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e027297d-94968cf8b-lrn5f" Sep 10 23:49:32.984279 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:32.984022 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-e027297d-dockercfg-cjwv2\"" Sep 10 23:49:32.989604 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:32.989576 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-e027297d\"" Sep 10 23:49:32.990602 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:32.990556 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e027297d-94968cf8b-lrn5f"] Sep 10 23:49:33.096663 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:33.095730 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d9a0a5f-e1bf-4f7d-8de5-0379379bc937-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-e027297d-94968cf8b-lrn5f\" (UID: \"4d9a0a5f-e1bf-4f7d-8de5-0379379bc937\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e027297d-94968cf8b-lrn5f" Sep 10 23:49:33.096663 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:33.095780 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzg5w\" (UniqueName: \"kubernetes.io/projected/4d9a0a5f-e1bf-4f7d-8de5-0379379bc937-kube-api-access-gzg5w\") pod \"payload-pre-processing-e2e-ait-e027297d-94968cf8b-lrn5f\" (UID: \"4d9a0a5f-e1bf-4f7d-8de5-0379379bc937\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e027297d-94968cf8b-lrn5f" Sep 10 23:49:33.196966 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:33.196919 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d9a0a5f-e1bf-4f7d-8de5-0379379bc937-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-e027297d-94968cf8b-lrn5f\" (UID: \"4d9a0a5f-e1bf-4f7d-8de5-0379379bc937\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e027297d-94968cf8b-lrn5f" Sep 10 23:49:33.196966 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:33.196968 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzg5w\" (UniqueName: \"kubernetes.io/projected/4d9a0a5f-e1bf-4f7d-8de5-0379379bc937-kube-api-access-gzg5w\") pod \"payload-pre-processing-e2e-ait-e027297d-94968cf8b-lrn5f\" (UID: \"4d9a0a5f-e1bf-4f7d-8de5-0379379bc937\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e027297d-94968cf8b-lrn5f" Sep 10 23:49:33.197669 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:33.197623 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d9a0a5f-e1bf-4f7d-8de5-0379379bc937-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-e027297d-94968cf8b-lrn5f\" (UID: \"4d9a0a5f-e1bf-4f7d-8de5-0379379bc937\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e027297d-94968cf8b-lrn5f" Sep 10 23:49:33.222888 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:33.222820 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzg5w\" (UniqueName: \"kubernetes.io/projected/4d9a0a5f-e1bf-4f7d-8de5-0379379bc937-kube-api-access-gzg5w\") pod \"payload-pre-processing-e2e-ait-e027297d-94968cf8b-lrn5f\" (UID: \"4d9a0a5f-e1bf-4f7d-8de5-0379379bc937\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e027297d-94968cf8b-lrn5f" Sep 10 23:49:33.302318 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:33.302280 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e027297d-94968cf8b-lrn5f" Sep 10 23:49:33.591506 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:33.591239 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e027297d-94968cf8b-lrn5f"] Sep 10 23:49:33.596269 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:49:33.596233 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d9a0a5f_e1bf_4f7d_8de5_0379379bc937.slice/crio-6d28a0909eae96ee3f4d84af92274996ffde7352276c076a3d46f0fdac6be586 WatchSource:0}: Error finding container 6d28a0909eae96ee3f4d84af92274996ffde7352276c076a3d46f0fdac6be586: Status 404 returned error can't find the container with id 6d28a0909eae96ee3f4d84af92274996ffde7352276c076a3d46f0fdac6be586 Sep 10 23:49:34.347016 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:34.346970 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e027297d-94968cf8b-lrn5f" event={"ID":"4d9a0a5f-e1bf-4f7d-8de5-0379379bc937","Type":"ContainerStarted","Data":"9dd4da36e989d53f53ed245216faa595cab7c1f4e6492503417950d3e5508f57"} Sep 10 23:49:34.347454 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:34.347029 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e027297d-94968cf8b-lrn5f" event={"ID":"4d9a0a5f-e1bf-4f7d-8de5-0379379bc937","Type":"ContainerStarted","Data":"6d28a0909eae96ee3f4d84af92274996ffde7352276c076a3d46f0fdac6be586"} Sep 10 23:49:34.347454 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:34.347051 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-e027297d-94968cf8b-lrn5f" Sep 10 23:49:34.365536 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:34.365484 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-e027297d-94968cf8b-lrn5f" podStartSLOduration=2.36546851 podStartE2EDuration="2.36546851s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:49:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:49:34.362958794 +0000 UTC m=+765.750561573" watchObservedRunningTime="2026-09-10 23:49:34.36546851 +0000 UTC m=+765.753071268" Sep 10 23:49:36.249064 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:36.249032 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-56r7l"] Sep 10 23:49:38.290141 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:38.289763 2574 patch_prober.go:28] interesting pod/e2e-inf-a-1fe290-openshift-default-644c884748-msx97 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 10 23:49:38.290141 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:38.289833 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-1fe290-openshift-default-644c884748-msx97" podUID="4cee31c3-1da1-4483-a73c-1d0e23fb8063" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 10 23:49:38.322675 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:38.318799 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1fe290-68d856fdd6-wsm2c" Sep 10 23:49:41.419518 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:41.419479 2574 generic.go:356] "Generic (PLEG): container finished" podID="4cee31c3-1da1-4483-a73c-1d0e23fb8063" containerID="f26d88deff51be711fcb129535330522310814d64fe4bd8fb0933ee0ff9f16c1" exitCode=0 Sep 10 23:49:41.420169 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:41.419734 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1fe290-openshift-default-644c884748-msx97" event={"ID":"4cee31c3-1da1-4483-a73c-1d0e23fb8063","Type":"ContainerDied","Data":"f26d88deff51be711fcb129535330522310814d64fe4bd8fb0933ee0ff9f16c1"} Sep 10 23:49:41.420717 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:41.420698 2574 scope.go:117] "RemoveContainer" containerID="f26d88deff51be711fcb129535330522310814d64fe4bd8fb0933ee0ff9f16c1" Sep 10 23:49:41.421835 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:41.421778 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 23:49:41.421953 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:41.421873 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 23:49:41.421953 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:41.421905 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 23:49:42.431675 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:42.431511 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1fe290-openshift-default-644c884748-msx97" event={"ID":"4cee31c3-1da1-4483-a73c-1d0e23fb8063","Type":"ContainerStarted","Data":"e1f6c5a8c0b05204295429ca0cbf03540112c45bb23e2aca8f002070d7217e71"} Sep 10 23:49:43.227082 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:43.227047 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-1fe290-openshift-default-644c884748-msx97" Sep 10 23:49:43.227292 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:43.227268 2574 patch_prober.go:28] interesting pod/e2e-inf-a-1fe290-openshift-default-644c884748-msx97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Sep 10 23:49:43.227342 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:43.227326 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1fe290-openshift-default-644c884748-msx97" podUID="4cee31c3-1da1-4483-a73c-1d0e23fb8063" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Sep 10 23:49:44.227721 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:44.227683 2574 patch_prober.go:28] interesting pod/e2e-inf-a-1fe290-openshift-default-644c884748-msx97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Sep 10 23:49:44.228126 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:44.227744 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1fe290-openshift-default-644c884748-msx97" podUID="4cee31c3-1da1-4483-a73c-1d0e23fb8063" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Sep 10 23:49:45.227371 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:45.227324 2574 patch_prober.go:28] interesting pod/e2e-inf-a-1fe290-openshift-default-644c884748-msx97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Sep 10 23:49:45.227564 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:45.227415 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1fe290-openshift-default-644c884748-msx97" podUID="4cee31c3-1da1-4483-a73c-1d0e23fb8063" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Sep 10 23:49:45.352782 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:45.352733 2574 patch_prober.go:28] interesting pod/e2e-autores-74f1c1-openshift-default-5545549c98-z628g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 10 23:49:45.353258 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:45.352800 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-74f1c1-openshift-default-5545549c98-z628g" podUID="5970ef76-234b-4f98-9f7b-22db89263e2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 10 23:49:45.358569 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:45.358546 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-e027297d-94968cf8b-lrn5f" Sep 10 23:49:46.227486 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:46.227439 2574 patch_prober.go:28] interesting pod/e2e-inf-a-1fe290-openshift-default-644c884748-msx97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Sep 10 23:49:46.227719 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:46.227495 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1fe290-openshift-default-644c884748-msx97" podUID="4cee31c3-1da1-4483-a73c-1d0e23fb8063" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Sep 10 23:49:47.106827 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:47.106466 2574 patch_prober.go:28] interesting pod/e2e-ait-e027297d-openshift-default-5cc9d886d4-bggs4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": read tcp 10.134.0.2:52874->10.134.0.54:15021: read: connection reset by peer" start-of-body= Sep 10 23:49:47.106827 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:47.106523 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-e027297d-openshift-default-5cc9d886d4-bggs4" podUID="b61c82a2-ced4-4c1c-8b44-4f94593d5abb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": read tcp 10.134.0.2:52874->10.134.0.54:15021: read: connection reset by peer" Sep 10 23:49:47.227258 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:47.227223 2574 patch_prober.go:28] interesting pod/e2e-inf-a-1fe290-openshift-default-644c884748-msx97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Sep 10 23:49:47.227428 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:47.227281 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1fe290-openshift-default-644c884748-msx97" podUID="4cee31c3-1da1-4483-a73c-1d0e23fb8063" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Sep 10 23:49:47.503338 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:47.503310 2574 generic.go:356] "Generic (PLEG): container finished" podID="b61c82a2-ced4-4c1c-8b44-4f94593d5abb" containerID="b18d24bfb72bed217f40b2d61999712cc42e3108886e685f90a866207a089b32" exitCode=0 Sep 10 23:49:47.503827 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:47.503395 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e027297d-openshift-default-5cc9d886d4-bggs4" event={"ID":"b61c82a2-ced4-4c1c-8b44-4f94593d5abb","Type":"ContainerDied","Data":"b18d24bfb72bed217f40b2d61999712cc42e3108886e685f90a866207a089b32"} Sep 10 23:49:47.503918 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:47.503866 2574 scope.go:117] "RemoveContainer" containerID="b18d24bfb72bed217f40b2d61999712cc42e3108886e685f90a866207a089b32" Sep 10 23:49:47.505036 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:47.505003 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 23:49:47.505118 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:47.505100 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 23:49:47.505165 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:47.505145 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 23:49:48.227819 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:48.227776 2574 patch_prober.go:28] interesting pod/e2e-inf-a-1fe290-openshift-default-644c884748-msx97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Sep 10 23:49:48.228216 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:48.227845 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1fe290-openshift-default-644c884748-msx97" podUID="4cee31c3-1da1-4483-a73c-1d0e23fb8063" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Sep 10 23:49:48.511044 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:48.510962 2574 generic.go:356] "Generic (PLEG): container finished" podID="5970ef76-234b-4f98-9f7b-22db89263e2a" containerID="8f0a0304ff1487247e7b71b8006745327b1e9f45b9e3513bda575e0ab97c66db" exitCode=0 Sep 10 23:49:48.511193 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:48.511037 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-74f1c1-openshift-default-5545549c98-z628g" event={"ID":"5970ef76-234b-4f98-9f7b-22db89263e2a","Type":"ContainerDied","Data":"8f0a0304ff1487247e7b71b8006745327b1e9f45b9e3513bda575e0ab97c66db"} Sep 10 23:49:48.511558 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:48.511532 2574 scope.go:117] "RemoveContainer" containerID="8f0a0304ff1487247e7b71b8006745327b1e9f45b9e3513bda575e0ab97c66db" Sep 10 23:49:48.512844 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:48.512814 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 23:49:48.512937 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:48.512897 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 23:49:48.512937 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:48.512933 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 23:49:48.513410 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:48.513381 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e027297d-openshift-default-5cc9d886d4-bggs4" event={"ID":"b61c82a2-ced4-4c1c-8b44-4f94593d5abb","Type":"ContainerStarted","Data":"7f19b40920477c764fd70e4b81a99aa8c63674290bf3500f3a24ab260802f58d"} Sep 10 23:49:48.786294 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:48.785094 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-7f598e-openshift-default-97747f75b-tz94v"] Sep 10 23:49:48.811588 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:48.811553 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7f598e-openshift-default-97747f75b-tz94v" Sep 10 23:49:48.812427 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:48.812395 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-7f598e-openshift-default-97747f75b-tz94v"] Sep 10 23:49:48.816443 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:48.816412 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-7f598e-openshift-default-dockercfg-scqx7\"" Sep 10 23:49:48.855872 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:48.855845 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-e027297d-openshift-default-5cc9d886d4-bggs4" Sep 10 23:49:48.856179 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:48.856153 2574 patch_prober.go:28] interesting pod/e2e-ait-e027297d-openshift-default-5cc9d886d4-bggs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Sep 10 23:49:48.856281 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:48.856223 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e027297d-openshift-default-5cc9d886d4-bggs4" podUID="b61c82a2-ced4-4c1c-8b44-4f94593d5abb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Sep 10 23:49:48.920420 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:48.920381 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7f0cc66-80ee-4f43-8c01-e38ffb30ec32-istio-podinfo\") pod \"e2e-inf-b-7f598e-openshift-default-97747f75b-tz94v\" (UID: \"f7f0cc66-80ee-4f43-8c01-e38ffb30ec32\") " pod="openshift-ingress/e2e-inf-b-7f598e-openshift-default-97747f75b-tz94v" Sep 10 23:49:48.920617 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:48.920436 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tsjx\" (UniqueName: \"kubernetes.io/projected/f7f0cc66-80ee-4f43-8c01-e38ffb30ec32-kube-api-access-8tsjx\") pod \"e2e-inf-b-7f598e-openshift-default-97747f75b-tz94v\" (UID: \"f7f0cc66-80ee-4f43-8c01-e38ffb30ec32\") " pod="openshift-ingress/e2e-inf-b-7f598e-openshift-default-97747f75b-tz94v" Sep 10 23:49:48.920617 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:48.920543 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7f0cc66-80ee-4f43-8c01-e38ffb30ec32-credential-socket\") pod \"e2e-inf-b-7f598e-openshift-default-97747f75b-tz94v\" (UID: \"f7f0cc66-80ee-4f43-8c01-e38ffb30ec32\") " pod="openshift-ingress/e2e-inf-b-7f598e-openshift-default-97747f75b-tz94v" Sep 10 23:49:48.920617 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:48.920597 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7f0cc66-80ee-4f43-8c01-e38ffb30ec32-istio-data\") pod \"e2e-inf-b-7f598e-openshift-default-97747f75b-tz94v\" (UID: \"f7f0cc66-80ee-4f43-8c01-e38ffb30ec32\") " pod="openshift-ingress/e2e-inf-b-7f598e-openshift-default-97747f75b-tz94v" Sep 10 23:49:48.920843 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:48.920680 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7f0cc66-80ee-4f43-8c01-e38ffb30ec32-workload-socket\") pod \"e2e-inf-b-7f598e-openshift-default-97747f75b-tz94v\" (UID: \"f7f0cc66-80ee-4f43-8c01-e38ffb30ec32\") " pod="openshift-ingress/e2e-inf-b-7f598e-openshift-default-97747f75b-tz94v" Sep 10 23:49:48.920843 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:48.920753 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7f0cc66-80ee-4f43-8c01-e38ffb30ec32-workload-certs\") pod \"e2e-inf-b-7f598e-openshift-default-97747f75b-tz94v\" (UID: \"f7f0cc66-80ee-4f43-8c01-e38ffb30ec32\") " pod="openshift-ingress/e2e-inf-b-7f598e-openshift-default-97747f75b-tz94v" Sep 10 23:49:48.920843 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:48.920795 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7f0cc66-80ee-4f43-8c01-e38ffb30ec32-istiod-ca-cert\") pod \"e2e-inf-b-7f598e-openshift-default-97747f75b-tz94v\" (UID: \"f7f0cc66-80ee-4f43-8c01-e38ffb30ec32\") " pod="openshift-ingress/e2e-inf-b-7f598e-openshift-default-97747f75b-tz94v" Sep 10 23:49:48.921017 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:48.920854 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7f0cc66-80ee-4f43-8c01-e38ffb30ec32-istio-envoy\") pod \"e2e-inf-b-7f598e-openshift-default-97747f75b-tz94v\" (UID: \"f7f0cc66-80ee-4f43-8c01-e38ffb30ec32\") " pod="openshift-ingress/e2e-inf-b-7f598e-openshift-default-97747f75b-tz94v" Sep 10 23:49:48.921017 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:48.920907 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7f0cc66-80ee-4f43-8c01-e38ffb30ec32-istio-token\") pod \"e2e-inf-b-7f598e-openshift-default-97747f75b-tz94v\" (UID: \"f7f0cc66-80ee-4f43-8c01-e38ffb30ec32\") " pod="openshift-ingress/e2e-inf-b-7f598e-openshift-default-97747f75b-tz94v" Sep 10 23:49:49.021779 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:49.021746 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7f0cc66-80ee-4f43-8c01-e38ffb30ec32-workload-socket\") pod \"e2e-inf-b-7f598e-openshift-default-97747f75b-tz94v\" (UID: \"f7f0cc66-80ee-4f43-8c01-e38ffb30ec32\") " pod="openshift-ingress/e2e-inf-b-7f598e-openshift-default-97747f75b-tz94v" Sep 10 23:49:49.021951 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:49.021801 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7f0cc66-80ee-4f43-8c01-e38ffb30ec32-workload-certs\") pod \"e2e-inf-b-7f598e-openshift-default-97747f75b-tz94v\" (UID: \"f7f0cc66-80ee-4f43-8c01-e38ffb30ec32\") " pod="openshift-ingress/e2e-inf-b-7f598e-openshift-default-97747f75b-tz94v" Sep 10 23:49:49.021951 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:49.021840 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7f0cc66-80ee-4f43-8c01-e38ffb30ec32-istiod-ca-cert\") pod \"e2e-inf-b-7f598e-openshift-default-97747f75b-tz94v\" (UID: \"f7f0cc66-80ee-4f43-8c01-e38ffb30ec32\") " pod="openshift-ingress/e2e-inf-b-7f598e-openshift-default-97747f75b-tz94v" Sep 10 23:49:49.021951 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:49.021873 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7f0cc66-80ee-4f43-8c01-e38ffb30ec32-istio-envoy\") pod \"e2e-inf-b-7f598e-openshift-default-97747f75b-tz94v\" (UID: \"f7f0cc66-80ee-4f43-8c01-e38ffb30ec32\") " pod="openshift-ingress/e2e-inf-b-7f598e-openshift-default-97747f75b-tz94v" Sep 10 23:49:49.021951 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:49.021911 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7f0cc66-80ee-4f43-8c01-e38ffb30ec32-istio-token\") pod \"e2e-inf-b-7f598e-openshift-default-97747f75b-tz94v\" (UID: \"f7f0cc66-80ee-4f43-8c01-e38ffb30ec32\") " pod="openshift-ingress/e2e-inf-b-7f598e-openshift-default-97747f75b-tz94v" Sep 10 23:49:49.022196 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:49.021976 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7f0cc66-80ee-4f43-8c01-e38ffb30ec32-istio-podinfo\") pod \"e2e-inf-b-7f598e-openshift-default-97747f75b-tz94v\" (UID: \"f7f0cc66-80ee-4f43-8c01-e38ffb30ec32\") " pod="openshift-ingress/e2e-inf-b-7f598e-openshift-default-97747f75b-tz94v" Sep 10 23:49:49.022196 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:49.022013 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tsjx\" (UniqueName: \"kubernetes.io/projected/f7f0cc66-80ee-4f43-8c01-e38ffb30ec32-kube-api-access-8tsjx\") pod \"e2e-inf-b-7f598e-openshift-default-97747f75b-tz94v\" (UID: \"f7f0cc66-80ee-4f43-8c01-e38ffb30ec32\") " pod="openshift-ingress/e2e-inf-b-7f598e-openshift-default-97747f75b-tz94v" Sep 10 23:49:49.022196 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:49.022068 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7f0cc66-80ee-4f43-8c01-e38ffb30ec32-credential-socket\") pod \"e2e-inf-b-7f598e-openshift-default-97747f75b-tz94v\" (UID: \"f7f0cc66-80ee-4f43-8c01-e38ffb30ec32\") " pod="openshift-ingress/e2e-inf-b-7f598e-openshift-default-97747f75b-tz94v" Sep 10 23:49:49.022196 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:49.022097 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7f0cc66-80ee-4f43-8c01-e38ffb30ec32-istio-data\") pod \"e2e-inf-b-7f598e-openshift-default-97747f75b-tz94v\" (UID: \"f7f0cc66-80ee-4f43-8c01-e38ffb30ec32\") " pod="openshift-ingress/e2e-inf-b-7f598e-openshift-default-97747f75b-tz94v" Sep 10 23:49:49.022196 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:49.022156 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7f0cc66-80ee-4f43-8c01-e38ffb30ec32-workload-certs\") pod \"e2e-inf-b-7f598e-openshift-default-97747f75b-tz94v\" (UID: \"f7f0cc66-80ee-4f43-8c01-e38ffb30ec32\") " pod="openshift-ingress/e2e-inf-b-7f598e-openshift-default-97747f75b-tz94v" Sep 10 23:49:49.022471 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:49.022203 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7f0cc66-80ee-4f43-8c01-e38ffb30ec32-workload-socket\") pod \"e2e-inf-b-7f598e-openshift-default-97747f75b-tz94v\" (UID: \"f7f0cc66-80ee-4f43-8c01-e38ffb30ec32\") " pod="openshift-ingress/e2e-inf-b-7f598e-openshift-default-97747f75b-tz94v" Sep 10 23:49:49.022613 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:49.022592 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7f0cc66-80ee-4f43-8c01-e38ffb30ec32-istiod-ca-cert\") pod \"e2e-inf-b-7f598e-openshift-default-97747f75b-tz94v\" (UID: \"f7f0cc66-80ee-4f43-8c01-e38ffb30ec32\") " pod="openshift-ingress/e2e-inf-b-7f598e-openshift-default-97747f75b-tz94v" Sep 10 23:49:49.022697 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:49.022610 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7f0cc66-80ee-4f43-8c01-e38ffb30ec32-istio-data\") pod \"e2e-inf-b-7f598e-openshift-default-97747f75b-tz94v\" (UID: \"f7f0cc66-80ee-4f43-8c01-e38ffb30ec32\") " pod="openshift-ingress/e2e-inf-b-7f598e-openshift-default-97747f75b-tz94v" Sep 10 23:49:49.022816 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:49.022795 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7f0cc66-80ee-4f43-8c01-e38ffb30ec32-credential-socket\") pod \"e2e-inf-b-7f598e-openshift-default-97747f75b-tz94v\" (UID: \"f7f0cc66-80ee-4f43-8c01-e38ffb30ec32\") " pod="openshift-ingress/e2e-inf-b-7f598e-openshift-default-97747f75b-tz94v" Sep 10 23:49:49.024493 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:49.024473 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7f0cc66-80ee-4f43-8c01-e38ffb30ec32-istio-envoy\") pod \"e2e-inf-b-7f598e-openshift-default-97747f75b-tz94v\" (UID: \"f7f0cc66-80ee-4f43-8c01-e38ffb30ec32\") " pod="openshift-ingress/e2e-inf-b-7f598e-openshift-default-97747f75b-tz94v" Sep 10 23:49:49.024539 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:49.024510 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7f0cc66-80ee-4f43-8c01-e38ffb30ec32-istio-podinfo\") pod \"e2e-inf-b-7f598e-openshift-default-97747f75b-tz94v\" (UID: \"f7f0cc66-80ee-4f43-8c01-e38ffb30ec32\") " pod="openshift-ingress/e2e-inf-b-7f598e-openshift-default-97747f75b-tz94v" Sep 10 23:49:49.030278 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:49.030253 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7f0cc66-80ee-4f43-8c01-e38ffb30ec32-istio-token\") pod \"e2e-inf-b-7f598e-openshift-default-97747f75b-tz94v\" (UID: \"f7f0cc66-80ee-4f43-8c01-e38ffb30ec32\") " pod="openshift-ingress/e2e-inf-b-7f598e-openshift-default-97747f75b-tz94v" Sep 10 23:49:49.031228 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:49.031205 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tsjx\" (UniqueName: \"kubernetes.io/projected/f7f0cc66-80ee-4f43-8c01-e38ffb30ec32-kube-api-access-8tsjx\") pod \"e2e-inf-b-7f598e-openshift-default-97747f75b-tz94v\" (UID: \"f7f0cc66-80ee-4f43-8c01-e38ffb30ec32\") " pod="openshift-ingress/e2e-inf-b-7f598e-openshift-default-97747f75b-tz94v" Sep 10 23:49:49.134679 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:49.134595 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-7f598e-openshift-default-dockercfg-scqx7\"" Sep 10 23:49:49.143400 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:49.143376 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7f598e-openshift-default-97747f75b-tz94v" Sep 10 23:49:49.233205 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:49.233174 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-1fe290-openshift-default-644c884748-msx97" Sep 10 23:49:49.234338 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:49.234320 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-1fe290-openshift-default-644c884748-msx97" Sep 10 23:49:49.235692 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:49.235627 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-1fe290-openshift-default-644c884748-msx97" Sep 10 23:49:49.458543 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:49:49.458511 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7f0cc66_80ee_4f43_8c01_e38ffb30ec32.slice/crio-37f2ea978d0d55cfcc5d306780844f7fbf8aa85192f1732dff0e7b4009f6bbbb WatchSource:0}: Error finding container 37f2ea978d0d55cfcc5d306780844f7fbf8aa85192f1732dff0e7b4009f6bbbb: Status 404 returned error can't find the container with id 37f2ea978d0d55cfcc5d306780844f7fbf8aa85192f1732dff0e7b4009f6bbbb Sep 10 23:49:49.458741 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:49.458715 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-7f598e-openshift-default-97747f75b-tz94v"] Sep 10 23:49:49.460899 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:49.460868 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 23:49:49.460959 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:49.460930 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 23:49:49.461002 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:49.460960 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 23:49:49.525688 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:49.525495 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-74f1c1-openshift-default-5545549c98-z628g" event={"ID":"5970ef76-234b-4f98-9f7b-22db89263e2a","Type":"ContainerStarted","Data":"3aa30afa9c76003c052aea2703b56618e1f7189d3d1f4b7969b9ea55f35011f0"} Sep 10 23:49:49.527453 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:49.527390 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7f598e-openshift-default-97747f75b-tz94v" event={"ID":"f7f0cc66-80ee-4f43-8c01-e38ffb30ec32","Type":"ContainerStarted","Data":"37f2ea978d0d55cfcc5d306780844f7fbf8aa85192f1732dff0e7b4009f6bbbb"} Sep 10 23:49:49.836665 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:49.836558 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-74f1c1-openshift-default-5545549c98-z628g" Sep 10 23:49:49.836819 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:49.836715 2574 patch_prober.go:28] interesting pod/e2e-autores-74f1c1-openshift-default-5545549c98-z628g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Sep 10 23:49:49.836819 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:49.836765 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-74f1c1-openshift-default-5545549c98-z628g" podUID="5970ef76-234b-4f98-9f7b-22db89263e2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Sep 10 23:49:49.855603 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:49.855469 2574 patch_prober.go:28] interesting pod/e2e-ait-e027297d-openshift-default-5cc9d886d4-bggs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Sep 10 23:49:49.855603 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:49.855548 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e027297d-openshift-default-5cc9d886d4-bggs4" podUID="b61c82a2-ced4-4c1c-8b44-4f94593d5abb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Sep 10 23:49:50.537944 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:50.537906 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7f598e-openshift-default-97747f75b-tz94v" event={"ID":"f7f0cc66-80ee-4f43-8c01-e38ffb30ec32","Type":"ContainerStarted","Data":"9344c7ba58e4214c7d1716bd492314f72bf2bb398eb77fd28c2d9b4d0891ec5a"} Sep 10 23:49:50.563025 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:50.562955 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-7f598e-openshift-default-97747f75b-tz94v" podStartSLOduration=2.562937046 podStartE2EDuration="2.562937046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:49:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:49:50.561704785 +0000 UTC m=+781.949307569" watchObservedRunningTime="2026-09-10 23:49:50.562937046 +0000 UTC m=+781.950539816" Sep 10 23:49:50.835795 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:50.835708 2574 patch_prober.go:28] interesting pod/e2e-autores-74f1c1-openshift-default-5545549c98-z628g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Sep 10 23:49:50.835795 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:50.835769 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-74f1c1-openshift-default-5545549c98-z628g" podUID="5970ef76-234b-4f98-9f7b-22db89263e2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Sep 10 23:49:50.855530 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:50.855501 2574 patch_prober.go:28] interesting pod/e2e-ait-e027297d-openshift-default-5cc9d886d4-bggs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Sep 10 23:49:50.855711 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:50.855549 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e027297d-openshift-default-5cc9d886d4-bggs4" podUID="b61c82a2-ced4-4c1c-8b44-4f94593d5abb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Sep 10 23:49:51.144215 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:51.144122 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-7f598e-openshift-default-97747f75b-tz94v" Sep 10 23:49:51.148920 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:51.148893 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-7f598e-openshift-default-97747f75b-tz94v" Sep 10 23:49:51.543956 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:51.543917 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-7f598e-openshift-default-97747f75b-tz94v" Sep 10 23:49:51.544922 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:51.544901 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-7f598e-openshift-default-97747f75b-tz94v" Sep 10 23:49:51.836187 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:51.836097 2574 patch_prober.go:28] interesting pod/e2e-autores-74f1c1-openshift-default-5545549c98-z628g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Sep 10 23:49:51.836187 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:51.836178 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-74f1c1-openshift-default-5545549c98-z628g" podUID="5970ef76-234b-4f98-9f7b-22db89263e2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Sep 10 23:49:51.856046 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:51.856008 2574 patch_prober.go:28] interesting pod/e2e-ait-e027297d-openshift-default-5cc9d886d4-bggs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Sep 10 23:49:51.856208 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:51.856072 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e027297d-openshift-default-5cc9d886d4-bggs4" podUID="b61c82a2-ced4-4c1c-8b44-4f94593d5abb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Sep 10 23:49:52.178875 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:52.178771 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-56r7l"] Sep 10 23:49:52.835916 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:52.835882 2574 patch_prober.go:28] interesting pod/e2e-autores-74f1c1-openshift-default-5545549c98-z628g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Sep 10 23:49:52.836475 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:52.836435 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-74f1c1-openshift-default-5545549c98-z628g" podUID="5970ef76-234b-4f98-9f7b-22db89263e2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Sep 10 23:49:52.855792 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:52.855768 2574 patch_prober.go:28] interesting pod/e2e-ait-e027297d-openshift-default-5cc9d886d4-bggs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Sep 10 23:49:52.855910 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:52.855817 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e027297d-openshift-default-5cc9d886d4-bggs4" podUID="b61c82a2-ced4-4c1c-8b44-4f94593d5abb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Sep 10 23:49:53.835929 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:53.835892 2574 patch_prober.go:28] interesting pod/e2e-autores-74f1c1-openshift-default-5545549c98-z628g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Sep 10 23:49:53.836353 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:53.835956 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-74f1c1-openshift-default-5545549c98-z628g" podUID="5970ef76-234b-4f98-9f7b-22db89263e2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Sep 10 23:49:53.859450 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:53.859422 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-e027297d-openshift-default-5cc9d886d4-bggs4" Sep 10 23:49:53.859834 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:53.859811 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-e027297d-openshift-default-5cc9d886d4-bggs4" Sep 10 23:49:53.860548 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:53.860532 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-e027297d-openshift-default-5cc9d886d4-bggs4" Sep 10 23:49:54.836300 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:54.836262 2574 patch_prober.go:28] interesting pod/e2e-autores-74f1c1-openshift-default-5545549c98-z628g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Sep 10 23:49:54.836794 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:54.836333 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-74f1c1-openshift-default-5545549c98-z628g" podUID="5970ef76-234b-4f98-9f7b-22db89263e2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Sep 10 23:49:54.925275 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:54.925235 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-7f598e-5cb99c745c-6n4tm"] Sep 10 23:49:54.932533 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:54.932506 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7f598e-5cb99c745c-6n4tm" Sep 10 23:49:54.934791 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:54.934766 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 23:49:54.934932 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:54.934843 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 10 23:49:54.934932 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:54.934886 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-7f598e-serving-cert\"" Sep 10 23:49:54.935070 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:54.934843 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-9tsnw\"" Sep 10 23:49:54.935350 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:54.935331 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 10 23:49:54.943178 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:54.943147 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-7f598e-5cb99c745c-6n4tm"] Sep 10 23:49:54.991432 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:54.991395 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p44r6\" (UniqueName: \"kubernetes.io/projected/1305be23-ac85-4b42-a83f-2b0b8fa9d435-kube-api-access-p44r6\") pod \"maas-api-e2e-inf-b-7f598e-5cb99c745c-6n4tm\" (UID: \"1305be23-ac85-4b42-a83f-2b0b8fa9d435\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7f598e-5cb99c745c-6n4tm" Sep 10 23:49:54.991676 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:54.991482 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1305be23-ac85-4b42-a83f-2b0b8fa9d435-openshift-service-ca\") pod \"maas-api-e2e-inf-b-7f598e-5cb99c745c-6n4tm\" (UID: \"1305be23-ac85-4b42-a83f-2b0b8fa9d435\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7f598e-5cb99c745c-6n4tm" Sep 10 23:49:54.991676 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:54.991543 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1305be23-ac85-4b42-a83f-2b0b8fa9d435-maas-api-tls\") pod \"maas-api-e2e-inf-b-7f598e-5cb99c745c-6n4tm\" (UID: \"1305be23-ac85-4b42-a83f-2b0b8fa9d435\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7f598e-5cb99c745c-6n4tm" Sep 10 23:49:54.991676 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:54.991584 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1305be23-ac85-4b42-a83f-2b0b8fa9d435-metrics-certs\") pod \"maas-api-e2e-inf-b-7f598e-5cb99c745c-6n4tm\" (UID: \"1305be23-ac85-4b42-a83f-2b0b8fa9d435\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7f598e-5cb99c745c-6n4tm" Sep 10 23:49:55.092326 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:55.092226 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p44r6\" (UniqueName: \"kubernetes.io/projected/1305be23-ac85-4b42-a83f-2b0b8fa9d435-kube-api-access-p44r6\") pod \"maas-api-e2e-inf-b-7f598e-5cb99c745c-6n4tm\" (UID: \"1305be23-ac85-4b42-a83f-2b0b8fa9d435\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7f598e-5cb99c745c-6n4tm" Sep 10 23:49:55.092326 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:55.092315 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1305be23-ac85-4b42-a83f-2b0b8fa9d435-openshift-service-ca\") pod \"maas-api-e2e-inf-b-7f598e-5cb99c745c-6n4tm\" (UID: \"1305be23-ac85-4b42-a83f-2b0b8fa9d435\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7f598e-5cb99c745c-6n4tm" Sep 10 23:49:55.092548 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:55.092366 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1305be23-ac85-4b42-a83f-2b0b8fa9d435-maas-api-tls\") pod \"maas-api-e2e-inf-b-7f598e-5cb99c745c-6n4tm\" (UID: \"1305be23-ac85-4b42-a83f-2b0b8fa9d435\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7f598e-5cb99c745c-6n4tm" Sep 10 23:49:55.092548 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:55.092429 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1305be23-ac85-4b42-a83f-2b0b8fa9d435-metrics-certs\") pod \"maas-api-e2e-inf-b-7f598e-5cb99c745c-6n4tm\" (UID: \"1305be23-ac85-4b42-a83f-2b0b8fa9d435\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7f598e-5cb99c745c-6n4tm" Sep 10 23:49:55.093183 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:55.093161 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1305be23-ac85-4b42-a83f-2b0b8fa9d435-openshift-service-ca\") pod \"maas-api-e2e-inf-b-7f598e-5cb99c745c-6n4tm\" (UID: \"1305be23-ac85-4b42-a83f-2b0b8fa9d435\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7f598e-5cb99c745c-6n4tm" Sep 10 23:49:55.095091 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:55.095060 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1305be23-ac85-4b42-a83f-2b0b8fa9d435-metrics-certs\") pod \"maas-api-e2e-inf-b-7f598e-5cb99c745c-6n4tm\" (UID: \"1305be23-ac85-4b42-a83f-2b0b8fa9d435\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7f598e-5cb99c745c-6n4tm" Sep 10 23:49:55.095265 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:55.095228 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1305be23-ac85-4b42-a83f-2b0b8fa9d435-maas-api-tls\") pod \"maas-api-e2e-inf-b-7f598e-5cb99c745c-6n4tm\" (UID: \"1305be23-ac85-4b42-a83f-2b0b8fa9d435\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7f598e-5cb99c745c-6n4tm" Sep 10 23:49:55.101504 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:55.101464 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p44r6\" (UniqueName: \"kubernetes.io/projected/1305be23-ac85-4b42-a83f-2b0b8fa9d435-kube-api-access-p44r6\") pod \"maas-api-e2e-inf-b-7f598e-5cb99c745c-6n4tm\" (UID: \"1305be23-ac85-4b42-a83f-2b0b8fa9d435\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7f598e-5cb99c745c-6n4tm" Sep 10 23:49:55.248752 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:55.248712 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7f598e-5cb99c745c-6n4tm" Sep 10 23:49:55.536910 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:55.536866 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-7f598e-5cb99c745c-6n4tm"] Sep 10 23:49:55.538573 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:49:55.538536 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1305be23_ac85_4b42_a83f_2b0b8fa9d435.slice/crio-05444731b115192da28878d037e433350965c6d8b6b5ba479dfa25d0e07657b8 WatchSource:0}: Error finding container 05444731b115192da28878d037e433350965c6d8b6b5ba479dfa25d0e07657b8: Status 404 returned error can't find the container with id 05444731b115192da28878d037e433350965c6d8b6b5ba479dfa25d0e07657b8 Sep 10 23:49:55.540227 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:55.540202 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 23:49:55.569730 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:55.569695 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7f598e-5cb99c745c-6n4tm" event={"ID":"1305be23-ac85-4b42-a83f-2b0b8fa9d435","Type":"ContainerStarted","Data":"05444731b115192da28878d037e433350965c6d8b6b5ba479dfa25d0e07657b8"} Sep 10 23:49:55.840424 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:55.840352 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-74f1c1-openshift-default-5545549c98-z628g" Sep 10 23:49:55.840778 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:55.840635 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-74f1c1-openshift-default-5545549c98-z628g" Sep 10 23:49:55.841423 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:55.841405 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-74f1c1-openshift-default-5545549c98-z628g" Sep 10 23:49:58.602457 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:58.602415 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7f598e-5cb99c745c-6n4tm" event={"ID":"1305be23-ac85-4b42-a83f-2b0b8fa9d435","Type":"ContainerStarted","Data":"09532062b76c06f36ee005b9a5820e00f4b19cf559d02da6789796a1dadaf059"} Sep 10 23:49:58.602932 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:49:58.602487 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7f598e-5cb99c745c-6n4tm" Sep 10 23:50:04.613488 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:04.613456 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7f598e-5cb99c745c-6n4tm" Sep 10 23:50:04.636851 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:04.636791 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7f598e-5cb99c745c-6n4tm" podStartSLOduration=8.280863622 podStartE2EDuration="10.636769663s" totalImagesPullingTime="2.355906041s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:49:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:49:58.768757654 +0000 UTC m=+790.156360434" watchObservedRunningTime="2026-09-10 23:50:04.636769663 +0000 UTC m=+796.024372423" Sep 10 23:50:08.695685 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:08.695628 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-56r7l"] Sep 10 23:50:10.676395 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:10.676363 2574 generic.go:356] "Generic (PLEG): container finished" podID="f7f0cc66-80ee-4f43-8c01-e38ffb30ec32" containerID="9344c7ba58e4214c7d1716bd492314f72bf2bb398eb77fd28c2d9b4d0891ec5a" exitCode=0 Sep 10 23:50:10.676810 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:10.676436 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7f598e-openshift-default-97747f75b-tz94v" event={"ID":"f7f0cc66-80ee-4f43-8c01-e38ffb30ec32","Type":"ContainerDied","Data":"9344c7ba58e4214c7d1716bd492314f72bf2bb398eb77fd28c2d9b4d0891ec5a"} Sep 10 23:50:10.676865 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:10.676850 2574 scope.go:117] "RemoveContainer" containerID="9344c7ba58e4214c7d1716bd492314f72bf2bb398eb77fd28c2d9b4d0891ec5a" Sep 10 23:50:10.678109 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:10.678081 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 23:50:10.678182 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:10.678151 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 23:50:10.678221 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:10.678183 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 23:50:11.697598 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:11.697506 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7f598e-openshift-default-97747f75b-tz94v" event={"ID":"f7f0cc66-80ee-4f43-8c01-e38ffb30ec32","Type":"ContainerStarted","Data":"801fe559f65712d9cad406723bb140bf555e8b4482e281bae1f9cba25b583f80"} Sep 10 23:50:12.143746 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:12.143658 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-7f598e-openshift-default-97747f75b-tz94v" Sep 10 23:50:12.143885 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:12.143845 2574 patch_prober.go:28] interesting pod/e2e-inf-b-7f598e-openshift-default-97747f75b-tz94v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Sep 10 23:50:12.143927 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:12.143897 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7f598e-openshift-default-97747f75b-tz94v" podUID="f7f0cc66-80ee-4f43-8c01-e38ffb30ec32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Sep 10 23:50:12.754279 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:12.754238 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e027297d-94968cf8b-lrn5f"] Sep 10 23:50:12.754753 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:12.754524 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-e027297d-94968cf8b-lrn5f" podUID="4d9a0a5f-e1bf-4f7d-8de5-0379379bc937" containerName="payload-pre-processing" containerID="cri-o://9dd4da36e989d53f53ed245216faa595cab7c1f4e6492503417950d3e5508f57" gracePeriod=30 Sep 10 23:50:13.060248 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:13.060220 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-e027297d-openshift-default-5cc9d886d4-bggs4"] Sep 10 23:50:13.060454 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:13.060425 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-e027297d-openshift-default-5cc9d886d4-bggs4" podUID="b61c82a2-ced4-4c1c-8b44-4f94593d5abb" containerName="istio-proxy" containerID="cri-o://7f19b40920477c764fd70e4b81a99aa8c63674290bf3500f3a24ab260802f58d" gracePeriod=30 Sep 10 23:50:13.064042 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:13.064019 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e027297d-94968cf8b-lrn5f" Sep 10 23:50:13.144812 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:13.144778 2574 patch_prober.go:28] interesting pod/e2e-inf-b-7f598e-openshift-default-97747f75b-tz94v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Sep 10 23:50:13.144972 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:13.144831 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7f598e-openshift-default-97747f75b-tz94v" podUID="f7f0cc66-80ee-4f43-8c01-e38ffb30ec32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Sep 10 23:50:13.203665 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:13.203000 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d9a0a5f-e1bf-4f7d-8de5-0379379bc937-plugins-config-volume\") pod \"4d9a0a5f-e1bf-4f7d-8de5-0379379bc937\" (UID: \"4d9a0a5f-e1bf-4f7d-8de5-0379379bc937\") " Sep 10 23:50:13.203665 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:13.203083 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gzg5w\" (UniqueName: \"kubernetes.io/projected/4d9a0a5f-e1bf-4f7d-8de5-0379379bc937-kube-api-access-gzg5w\") pod \"4d9a0a5f-e1bf-4f7d-8de5-0379379bc937\" (UID: \"4d9a0a5f-e1bf-4f7d-8de5-0379379bc937\") " Sep 10 23:50:13.204281 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:13.204249 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d9a0a5f-e1bf-4f7d-8de5-0379379bc937-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4d9a0a5f-e1bf-4f7d-8de5-0379379bc937" (UID: "4d9a0a5f-e1bf-4f7d-8de5-0379379bc937"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 23:50:13.208956 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:13.208924 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d9a0a5f-e1bf-4f7d-8de5-0379379bc937-kube-api-access-gzg5w" (OuterVolumeSpecName: "kube-api-access-gzg5w") pod "4d9a0a5f-e1bf-4f7d-8de5-0379379bc937" (UID: "4d9a0a5f-e1bf-4f7d-8de5-0379379bc937"). InnerVolumeSpecName "kube-api-access-gzg5w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 23:50:13.304877 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:13.304795 2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d9a0a5f-e1bf-4f7d-8de5-0379379bc937-plugins-config-volume\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:50:13.304877 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:13.304827 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzg5w\" (UniqueName: \"kubernetes.io/projected/4d9a0a5f-e1bf-4f7d-8de5-0379379bc937-kube-api-access-gzg5w\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:50:13.715994 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:13.715959 2574 generic.go:356] "Generic (PLEG): container finished" podID="4d9a0a5f-e1bf-4f7d-8de5-0379379bc937" containerID="9dd4da36e989d53f53ed245216faa595cab7c1f4e6492503417950d3e5508f57" exitCode=0 Sep 10 23:50:13.716180 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:13.716011 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e027297d-94968cf8b-lrn5f" Sep 10 23:50:13.716180 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:13.716046 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e027297d-94968cf8b-lrn5f" event={"ID":"4d9a0a5f-e1bf-4f7d-8de5-0379379bc937","Type":"ContainerDied","Data":"9dd4da36e989d53f53ed245216faa595cab7c1f4e6492503417950d3e5508f57"} Sep 10 23:50:13.716180 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:13.716107 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e027297d-94968cf8b-lrn5f" event={"ID":"4d9a0a5f-e1bf-4f7d-8de5-0379379bc937","Type":"ContainerDied","Data":"6d28a0909eae96ee3f4d84af92274996ffde7352276c076a3d46f0fdac6be586"} Sep 10 23:50:13.716180 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:13.716137 2574 scope.go:117] "RemoveContainer" containerID="9dd4da36e989d53f53ed245216faa595cab7c1f4e6492503417950d3e5508f57" Sep 10 23:50:13.727010 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:13.726987 2574 scope.go:117] "RemoveContainer" containerID="9dd4da36e989d53f53ed245216faa595cab7c1f4e6492503417950d3e5508f57" Sep 10 23:50:13.727319 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:50:13.727299 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9dd4da36e989d53f53ed245216faa595cab7c1f4e6492503417950d3e5508f57\": container with ID starting with 9dd4da36e989d53f53ed245216faa595cab7c1f4e6492503417950d3e5508f57 not found: ID does not exist" containerID="9dd4da36e989d53f53ed245216faa595cab7c1f4e6492503417950d3e5508f57" Sep 10 23:50:13.727363 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:13.727330 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9dd4da36e989d53f53ed245216faa595cab7c1f4e6492503417950d3e5508f57"} err="failed to get container status \"9dd4da36e989d53f53ed245216faa595cab7c1f4e6492503417950d3e5508f57\": rpc error: code = NotFound desc = could not find container \"9dd4da36e989d53f53ed245216faa595cab7c1f4e6492503417950d3e5508f57\": container with ID starting with 9dd4da36e989d53f53ed245216faa595cab7c1f4e6492503417950d3e5508f57 not found: ID does not exist" Sep 10 23:50:13.740864 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:13.740824 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e027297d-94968cf8b-lrn5f"] Sep 10 23:50:13.748615 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:13.748586 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e027297d-94968cf8b-lrn5f"] Sep 10 23:50:14.126692 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:14.126592 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-6a8995da-gw-openshift-default-6c66554d4d-hdr6p"] Sep 10 23:50:14.127442 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:14.127414 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d9a0a5f-e1bf-4f7d-8de5-0379379bc937" containerName="payload-pre-processing" Sep 10 23:50:14.127621 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:14.127603 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d9a0a5f-e1bf-4f7d-8de5-0379379bc937" containerName="payload-pre-processing" Sep 10 23:50:14.127792 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:14.127777 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d9a0a5f-e1bf-4f7d-8de5-0379379bc937" containerName="payload-pre-processing" Sep 10 23:50:14.132835 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:14.132813 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-6a8995da-gw-openshift-default-6c66554d4d-hdr6p" Sep 10 23:50:14.135983 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:14.135959 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-6a8995da-gw-openshift-default-dockercfg-6vh6k\"" Sep 10 23:50:14.144288 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:14.144260 2574 patch_prober.go:28] interesting pod/e2e-inf-b-7f598e-openshift-default-97747f75b-tz94v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Sep 10 23:50:14.144400 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:14.144308 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7f598e-openshift-default-97747f75b-tz94v" podUID="f7f0cc66-80ee-4f43-8c01-e38ffb30ec32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Sep 10 23:50:14.150298 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:14.150270 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-6a8995da-gw-openshift-default-6c66554d4d-hdr6p"] Sep 10 23:50:14.315674 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:14.315608 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jp6fc\" (UniqueName: \"kubernetes.io/projected/dfb8b1ec-79f0-4db8-871e-fbd82b1623e6-kube-api-access-jp6fc\") pod \"e2e-migrate-6a8995da-gw-openshift-default-6c66554d4d-hdr6p\" (UID: \"dfb8b1ec-79f0-4db8-871e-fbd82b1623e6\") " pod="openshift-ingress/e2e-migrate-6a8995da-gw-openshift-default-6c66554d4d-hdr6p" Sep 10 23:50:14.315868 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:14.315687 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfb8b1ec-79f0-4db8-871e-fbd82b1623e6-workload-certs\") pod \"e2e-migrate-6a8995da-gw-openshift-default-6c66554d4d-hdr6p\" (UID: \"dfb8b1ec-79f0-4db8-871e-fbd82b1623e6\") " pod="openshift-ingress/e2e-migrate-6a8995da-gw-openshift-default-6c66554d4d-hdr6p" Sep 10 23:50:14.315868 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:14.315726 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfb8b1ec-79f0-4db8-871e-fbd82b1623e6-istio-envoy\") pod \"e2e-migrate-6a8995da-gw-openshift-default-6c66554d4d-hdr6p\" (UID: \"dfb8b1ec-79f0-4db8-871e-fbd82b1623e6\") " pod="openshift-ingress/e2e-migrate-6a8995da-gw-openshift-default-6c66554d4d-hdr6p" Sep 10 23:50:14.315868 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:14.315751 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfb8b1ec-79f0-4db8-871e-fbd82b1623e6-istio-podinfo\") pod \"e2e-migrate-6a8995da-gw-openshift-default-6c66554d4d-hdr6p\" (UID: \"dfb8b1ec-79f0-4db8-871e-fbd82b1623e6\") " pod="openshift-ingress/e2e-migrate-6a8995da-gw-openshift-default-6c66554d4d-hdr6p" Sep 10 23:50:14.315868 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:14.315826 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfb8b1ec-79f0-4db8-871e-fbd82b1623e6-workload-socket\") pod \"e2e-migrate-6a8995da-gw-openshift-default-6c66554d4d-hdr6p\" (UID: \"dfb8b1ec-79f0-4db8-871e-fbd82b1623e6\") " pod="openshift-ingress/e2e-migrate-6a8995da-gw-openshift-default-6c66554d4d-hdr6p" Sep 10 23:50:14.316024 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:14.315898 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfb8b1ec-79f0-4db8-871e-fbd82b1623e6-istio-token\") pod \"e2e-migrate-6a8995da-gw-openshift-default-6c66554d4d-hdr6p\" (UID: \"dfb8b1ec-79f0-4db8-871e-fbd82b1623e6\") " pod="openshift-ingress/e2e-migrate-6a8995da-gw-openshift-default-6c66554d4d-hdr6p" Sep 10 23:50:14.316024 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:14.315919 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfb8b1ec-79f0-4db8-871e-fbd82b1623e6-istiod-ca-cert\") pod \"e2e-migrate-6a8995da-gw-openshift-default-6c66554d4d-hdr6p\" (UID: \"dfb8b1ec-79f0-4db8-871e-fbd82b1623e6\") " pod="openshift-ingress/e2e-migrate-6a8995da-gw-openshift-default-6c66554d4d-hdr6p" Sep 10 23:50:14.316024 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:14.315963 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfb8b1ec-79f0-4db8-871e-fbd82b1623e6-credential-socket\") pod \"e2e-migrate-6a8995da-gw-openshift-default-6c66554d4d-hdr6p\" (UID: \"dfb8b1ec-79f0-4db8-871e-fbd82b1623e6\") " pod="openshift-ingress/e2e-migrate-6a8995da-gw-openshift-default-6c66554d4d-hdr6p" Sep 10 23:50:14.316024 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:14.315985 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfb8b1ec-79f0-4db8-871e-fbd82b1623e6-istio-data\") pod \"e2e-migrate-6a8995da-gw-openshift-default-6c66554d4d-hdr6p\" (UID: \"dfb8b1ec-79f0-4db8-871e-fbd82b1623e6\") " pod="openshift-ingress/e2e-migrate-6a8995da-gw-openshift-default-6c66554d4d-hdr6p" Sep 10 23:50:14.417398 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:14.417297 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfb8b1ec-79f0-4db8-871e-fbd82b1623e6-workload-certs\") pod \"e2e-migrate-6a8995da-gw-openshift-default-6c66554d4d-hdr6p\" (UID: \"dfb8b1ec-79f0-4db8-871e-fbd82b1623e6\") " pod="openshift-ingress/e2e-migrate-6a8995da-gw-openshift-default-6c66554d4d-hdr6p" Sep 10 23:50:14.417398 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:14.417347 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfb8b1ec-79f0-4db8-871e-fbd82b1623e6-istio-envoy\") pod \"e2e-migrate-6a8995da-gw-openshift-default-6c66554d4d-hdr6p\" (UID: \"dfb8b1ec-79f0-4db8-871e-fbd82b1623e6\") " pod="openshift-ingress/e2e-migrate-6a8995da-gw-openshift-default-6c66554d4d-hdr6p" Sep 10 23:50:14.417398 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:14.417381 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfb8b1ec-79f0-4db8-871e-fbd82b1623e6-istio-podinfo\") pod \"e2e-migrate-6a8995da-gw-openshift-default-6c66554d4d-hdr6p\" (UID: \"dfb8b1ec-79f0-4db8-871e-fbd82b1623e6\") " pod="openshift-ingress/e2e-migrate-6a8995da-gw-openshift-default-6c66554d4d-hdr6p" Sep 10 23:50:14.417718 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:14.417409 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfb8b1ec-79f0-4db8-871e-fbd82b1623e6-workload-socket\") pod \"e2e-migrate-6a8995da-gw-openshift-default-6c66554d4d-hdr6p\" (UID: \"dfb8b1ec-79f0-4db8-871e-fbd82b1623e6\") " pod="openshift-ingress/e2e-migrate-6a8995da-gw-openshift-default-6c66554d4d-hdr6p" Sep 10 23:50:14.417718 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:14.417455 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfb8b1ec-79f0-4db8-871e-fbd82b1623e6-istio-token\") pod \"e2e-migrate-6a8995da-gw-openshift-default-6c66554d4d-hdr6p\" (UID: \"dfb8b1ec-79f0-4db8-871e-fbd82b1623e6\") " pod="openshift-ingress/e2e-migrate-6a8995da-gw-openshift-default-6c66554d4d-hdr6p" Sep 10 23:50:14.417718 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:14.417476 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfb8b1ec-79f0-4db8-871e-fbd82b1623e6-istiod-ca-cert\") pod \"e2e-migrate-6a8995da-gw-openshift-default-6c66554d4d-hdr6p\" (UID: \"dfb8b1ec-79f0-4db8-871e-fbd82b1623e6\") " pod="openshift-ingress/e2e-migrate-6a8995da-gw-openshift-default-6c66554d4d-hdr6p" Sep 10 23:50:14.417718 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:14.417533 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfb8b1ec-79f0-4db8-871e-fbd82b1623e6-credential-socket\") pod \"e2e-migrate-6a8995da-gw-openshift-default-6c66554d4d-hdr6p\" (UID: \"dfb8b1ec-79f0-4db8-871e-fbd82b1623e6\") " pod="openshift-ingress/e2e-migrate-6a8995da-gw-openshift-default-6c66554d4d-hdr6p" Sep 10 23:50:14.417718 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:14.417562 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfb8b1ec-79f0-4db8-871e-fbd82b1623e6-istio-data\") pod \"e2e-migrate-6a8995da-gw-openshift-default-6c66554d4d-hdr6p\" (UID: \"dfb8b1ec-79f0-4db8-871e-fbd82b1623e6\") " pod="openshift-ingress/e2e-migrate-6a8995da-gw-openshift-default-6c66554d4d-hdr6p" Sep 10 23:50:14.417988 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:14.417722 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jp6fc\" (UniqueName: \"kubernetes.io/projected/dfb8b1ec-79f0-4db8-871e-fbd82b1623e6-kube-api-access-jp6fc\") pod \"e2e-migrate-6a8995da-gw-openshift-default-6c66554d4d-hdr6p\" (UID: \"dfb8b1ec-79f0-4db8-871e-fbd82b1623e6\") " pod="openshift-ingress/e2e-migrate-6a8995da-gw-openshift-default-6c66554d4d-hdr6p" Sep 10 23:50:14.417988 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:14.417816 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfb8b1ec-79f0-4db8-871e-fbd82b1623e6-workload-certs\") pod \"e2e-migrate-6a8995da-gw-openshift-default-6c66554d4d-hdr6p\" (UID: \"dfb8b1ec-79f0-4db8-871e-fbd82b1623e6\") " pod="openshift-ingress/e2e-migrate-6a8995da-gw-openshift-default-6c66554d4d-hdr6p" Sep 10 23:50:14.417988 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:14.417875 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfb8b1ec-79f0-4db8-871e-fbd82b1623e6-workload-socket\") pod \"e2e-migrate-6a8995da-gw-openshift-default-6c66554d4d-hdr6p\" (UID: \"dfb8b1ec-79f0-4db8-871e-fbd82b1623e6\") " pod="openshift-ingress/e2e-migrate-6a8995da-gw-openshift-default-6c66554d4d-hdr6p" Sep 10 23:50:14.418205 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:14.418087 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfb8b1ec-79f0-4db8-871e-fbd82b1623e6-credential-socket\") pod \"e2e-migrate-6a8995da-gw-openshift-default-6c66554d4d-hdr6p\" (UID: \"dfb8b1ec-79f0-4db8-871e-fbd82b1623e6\") " pod="openshift-ingress/e2e-migrate-6a8995da-gw-openshift-default-6c66554d4d-hdr6p" Sep 10 23:50:14.418320 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:14.418291 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfb8b1ec-79f0-4db8-871e-fbd82b1623e6-istio-data\") pod \"e2e-migrate-6a8995da-gw-openshift-default-6c66554d4d-hdr6p\" (UID: \"dfb8b1ec-79f0-4db8-871e-fbd82b1623e6\") " pod="openshift-ingress/e2e-migrate-6a8995da-gw-openshift-default-6c66554d4d-hdr6p" Sep 10 23:50:14.418891 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:14.418843 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfb8b1ec-79f0-4db8-871e-fbd82b1623e6-istiod-ca-cert\") pod \"e2e-migrate-6a8995da-gw-openshift-default-6c66554d4d-hdr6p\" (UID: \"dfb8b1ec-79f0-4db8-871e-fbd82b1623e6\") " pod="openshift-ingress/e2e-migrate-6a8995da-gw-openshift-default-6c66554d4d-hdr6p" Sep 10 23:50:14.420319 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:14.420292 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfb8b1ec-79f0-4db8-871e-fbd82b1623e6-istio-podinfo\") pod \"e2e-migrate-6a8995da-gw-openshift-default-6c66554d4d-hdr6p\" (UID: \"dfb8b1ec-79f0-4db8-871e-fbd82b1623e6\") " pod="openshift-ingress/e2e-migrate-6a8995da-gw-openshift-default-6c66554d4d-hdr6p" Sep 10 23:50:14.420429 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:14.420337 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfb8b1ec-79f0-4db8-871e-fbd82b1623e6-istio-envoy\") pod \"e2e-migrate-6a8995da-gw-openshift-default-6c66554d4d-hdr6p\" (UID: \"dfb8b1ec-79f0-4db8-871e-fbd82b1623e6\") " pod="openshift-ingress/e2e-migrate-6a8995da-gw-openshift-default-6c66554d4d-hdr6p" Sep 10 23:50:14.430784 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:14.430753 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jp6fc\" (UniqueName: \"kubernetes.io/projected/dfb8b1ec-79f0-4db8-871e-fbd82b1623e6-kube-api-access-jp6fc\") pod \"e2e-migrate-6a8995da-gw-openshift-default-6c66554d4d-hdr6p\" (UID: \"dfb8b1ec-79f0-4db8-871e-fbd82b1623e6\") " pod="openshift-ingress/e2e-migrate-6a8995da-gw-openshift-default-6c66554d4d-hdr6p" Sep 10 23:50:14.430979 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:14.430962 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfb8b1ec-79f0-4db8-871e-fbd82b1623e6-istio-token\") pod \"e2e-migrate-6a8995da-gw-openshift-default-6c66554d4d-hdr6p\" (UID: \"dfb8b1ec-79f0-4db8-871e-fbd82b1623e6\") " pod="openshift-ingress/e2e-migrate-6a8995da-gw-openshift-default-6c66554d4d-hdr6p" Sep 10 23:50:14.444120 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:14.444100 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-6a8995da-gw-openshift-default-6c66554d4d-hdr6p" Sep 10 23:50:14.748924 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:14.748899 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-6a8995da-gw-openshift-default-6c66554d4d-hdr6p"] Sep 10 23:50:14.750035 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:50:14.749998 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfb8b1ec_79f0_4db8_871e_fbd82b1623e6.slice/crio-b4ece5b368bd0c93efe42ca1455c2cfc6d49da0267f33f8bf5a7c3e327accb45 WatchSource:0}: Error finding container b4ece5b368bd0c93efe42ca1455c2cfc6d49da0267f33f8bf5a7c3e327accb45: Status 404 returned error can't find the container with id b4ece5b368bd0c93efe42ca1455c2cfc6d49da0267f33f8bf5a7c3e327accb45 Sep 10 23:50:14.752293 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:14.752258 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 23:50:14.752385 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:14.752324 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 23:50:14.752385 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:14.752355 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 23:50:15.099625 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:15.099523 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d9a0a5f-e1bf-4f7d-8de5-0379379bc937" path="/var/lib/kubelet/pods/4d9a0a5f-e1bf-4f7d-8de5-0379379bc937/volumes" Sep 10 23:50:15.143761 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:15.143722 2574 patch_prober.go:28] interesting pod/e2e-inf-b-7f598e-openshift-default-97747f75b-tz94v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Sep 10 23:50:15.144164 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:15.143779 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7f598e-openshift-default-97747f75b-tz94v" podUID="f7f0cc66-80ee-4f43-8c01-e38ffb30ec32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Sep 10 23:50:15.731625 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:15.731585 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-6a8995da-gw-openshift-default-6c66554d4d-hdr6p" event={"ID":"dfb8b1ec-79f0-4db8-871e-fbd82b1623e6","Type":"ContainerStarted","Data":"0dab95069fa9edd9fec5e0ec662a78d540465f00111d252e4341b9d0d5a167f7"} Sep 10 23:50:15.731815 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:15.731671 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-6a8995da-gw-openshift-default-6c66554d4d-hdr6p" event={"ID":"dfb8b1ec-79f0-4db8-871e-fbd82b1623e6","Type":"ContainerStarted","Data":"b4ece5b368bd0c93efe42ca1455c2cfc6d49da0267f33f8bf5a7c3e327accb45"} Sep 10 23:50:15.757252 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:15.757204 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-6a8995da-gw-openshift-default-6c66554d4d-hdr6p" podStartSLOduration=1.757190373 podStartE2EDuration="1.757190373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:50:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:50:15.755829714 +0000 UTC m=+807.143432495" watchObservedRunningTime="2026-09-10 23:50:15.757190373 +0000 UTC m=+807.144793132" Sep 10 23:50:16.144523 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:16.144431 2574 patch_prober.go:28] interesting pod/e2e-inf-b-7f598e-openshift-default-97747f75b-tz94v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Sep 10 23:50:16.144523 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:16.144490 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7f598e-openshift-default-97747f75b-tz94v" podUID="f7f0cc66-80ee-4f43-8c01-e38ffb30ec32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Sep 10 23:50:16.444292 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:16.444194 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-6a8995da-gw-openshift-default-6c66554d4d-hdr6p" Sep 10 23:50:16.450010 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:16.449980 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-6a8995da-gw-openshift-default-6c66554d4d-hdr6p" Sep 10 23:50:16.738100 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:16.738067 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-6a8995da-gw-openshift-default-6c66554d4d-hdr6p" Sep 10 23:50:16.739042 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:16.739012 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-6a8995da-gw-openshift-default-6c66554d4d-hdr6p" Sep 10 23:50:17.285840 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:17.285805 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-7f598e-openshift-default-97747f75b-tz94v" Sep 10 23:50:17.286704 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:17.286679 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-7f598e-openshift-default-97747f75b-tz94v" Sep 10 23:50:17.288512 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:17.288492 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-7f598e-openshift-default-97747f75b-tz94v" Sep 10 23:50:18.330388 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:18.330321 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e027297d-openshift-default-5cc9d886d4-bggs4" Sep 10 23:50:18.355231 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:18.355194 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b61c82a2-ced4-4c1c-8b44-4f94593d5abb-credential-socket\") pod \"b61c82a2-ced4-4c1c-8b44-4f94593d5abb\" (UID: \"b61c82a2-ced4-4c1c-8b44-4f94593d5abb\") " Sep 10 23:50:18.356463 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:18.355257 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b61c82a2-ced4-4c1c-8b44-4f94593d5abb-workload-certs\") pod \"b61c82a2-ced4-4c1c-8b44-4f94593d5abb\" (UID: \"b61c82a2-ced4-4c1c-8b44-4f94593d5abb\") " Sep 10 23:50:18.356463 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:18.355337 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b61c82a2-ced4-4c1c-8b44-4f94593d5abb-istio-data\") pod \"b61c82a2-ced4-4c1c-8b44-4f94593d5abb\" (UID: \"b61c82a2-ced4-4c1c-8b44-4f94593d5abb\") " Sep 10 23:50:18.356463 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:18.355367 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b61c82a2-ced4-4c1c-8b44-4f94593d5abb-istiod-ca-cert\") pod \"b61c82a2-ced4-4c1c-8b44-4f94593d5abb\" (UID: \"b61c82a2-ced4-4c1c-8b44-4f94593d5abb\") " Sep 10 23:50:18.356463 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:18.355395 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b61c82a2-ced4-4c1c-8b44-4f94593d5abb-istio-envoy\") pod \"b61c82a2-ced4-4c1c-8b44-4f94593d5abb\" (UID: \"b61c82a2-ced4-4c1c-8b44-4f94593d5abb\") " Sep 10 23:50:18.356463 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:18.355428 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b61c82a2-ced4-4c1c-8b44-4f94593d5abb-workload-socket\") pod \"b61c82a2-ced4-4c1c-8b44-4f94593d5abb\" (UID: \"b61c82a2-ced4-4c1c-8b44-4f94593d5abb\") " Sep 10 23:50:18.356463 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:18.355475 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b61c82a2-ced4-4c1c-8b44-4f94593d5abb-istio-token\") pod \"b61c82a2-ced4-4c1c-8b44-4f94593d5abb\" (UID: \"b61c82a2-ced4-4c1c-8b44-4f94593d5abb\") " Sep 10 23:50:18.356463 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:18.355525 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5gbcg\" (UniqueName: \"kubernetes.io/projected/b61c82a2-ced4-4c1c-8b44-4f94593d5abb-kube-api-access-5gbcg\") pod \"b61c82a2-ced4-4c1c-8b44-4f94593d5abb\" (UID: \"b61c82a2-ced4-4c1c-8b44-4f94593d5abb\") " Sep 10 23:50:18.356463 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:18.355553 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b61c82a2-ced4-4c1c-8b44-4f94593d5abb-istio-podinfo\") pod \"b61c82a2-ced4-4c1c-8b44-4f94593d5abb\" (UID: \"b61c82a2-ced4-4c1c-8b44-4f94593d5abb\") " Sep 10 23:50:18.356463 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:18.355808 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b61c82a2-ced4-4c1c-8b44-4f94593d5abb-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b61c82a2-ced4-4c1c-8b44-4f94593d5abb" (UID: "b61c82a2-ced4-4c1c-8b44-4f94593d5abb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:50:18.356463 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:18.356018 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b61c82a2-ced4-4c1c-8b44-4f94593d5abb-workload-certs\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:50:18.356463 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:18.356037 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b61c82a2-ced4-4c1c-8b44-4f94593d5abb-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b61c82a2-ced4-4c1c-8b44-4f94593d5abb" (UID: "b61c82a2-ced4-4c1c-8b44-4f94593d5abb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:50:18.356463 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:18.356075 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b61c82a2-ced4-4c1c-8b44-4f94593d5abb-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b61c82a2-ced4-4c1c-8b44-4f94593d5abb" (UID: "b61c82a2-ced4-4c1c-8b44-4f94593d5abb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 23:50:18.357312 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:18.357274 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b61c82a2-ced4-4c1c-8b44-4f94593d5abb-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b61c82a2-ced4-4c1c-8b44-4f94593d5abb" (UID: "b61c82a2-ced4-4c1c-8b44-4f94593d5abb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:50:18.357498 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:18.357463 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b61c82a2-ced4-4c1c-8b44-4f94593d5abb-istio-data" (OuterVolumeSpecName: "istio-data") pod "b61c82a2-ced4-4c1c-8b44-4f94593d5abb" (UID: "b61c82a2-ced4-4c1c-8b44-4f94593d5abb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:50:18.358471 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:18.358428 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b61c82a2-ced4-4c1c-8b44-4f94593d5abb-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b61c82a2-ced4-4c1c-8b44-4f94593d5abb" (UID: "b61c82a2-ced4-4c1c-8b44-4f94593d5abb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 23:50:18.358581 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:18.358536 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b61c82a2-ced4-4c1c-8b44-4f94593d5abb-istio-token" (OuterVolumeSpecName: "istio-token") pod "b61c82a2-ced4-4c1c-8b44-4f94593d5abb" (UID: "b61c82a2-ced4-4c1c-8b44-4f94593d5abb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 23:50:18.358913 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:18.358878 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b61c82a2-ced4-4c1c-8b44-4f94593d5abb-kube-api-access-5gbcg" (OuterVolumeSpecName: "kube-api-access-5gbcg") pod "b61c82a2-ced4-4c1c-8b44-4f94593d5abb" (UID: "b61c82a2-ced4-4c1c-8b44-4f94593d5abb"). InnerVolumeSpecName "kube-api-access-5gbcg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 23:50:18.359533 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:18.359503 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b61c82a2-ced4-4c1c-8b44-4f94593d5abb-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b61c82a2-ced4-4c1c-8b44-4f94593d5abb" (UID: "b61c82a2-ced4-4c1c-8b44-4f94593d5abb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:50:18.456705 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:18.456626 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b61c82a2-ced4-4c1c-8b44-4f94593d5abb-credential-socket\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:50:18.456705 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:18.456698 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b61c82a2-ced4-4c1c-8b44-4f94593d5abb-istio-data\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:50:18.456705 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:18.456708 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b61c82a2-ced4-4c1c-8b44-4f94593d5abb-istiod-ca-cert\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:50:18.456705 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:18.456715 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b61c82a2-ced4-4c1c-8b44-4f94593d5abb-istio-envoy\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:50:18.456983 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:18.456727 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b61c82a2-ced4-4c1c-8b44-4f94593d5abb-workload-socket\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:50:18.456983 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:18.456737 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b61c82a2-ced4-4c1c-8b44-4f94593d5abb-istio-token\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:50:18.456983 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:18.456746 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5gbcg\" (UniqueName: \"kubernetes.io/projected/b61c82a2-ced4-4c1c-8b44-4f94593d5abb-kube-api-access-5gbcg\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:50:18.456983 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:18.456754 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b61c82a2-ced4-4c1c-8b44-4f94593d5abb-istio-podinfo\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:50:18.752717 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:18.752681 2574 generic.go:356] "Generic (PLEG): container finished" podID="b61c82a2-ced4-4c1c-8b44-4f94593d5abb" containerID="7f19b40920477c764fd70e4b81a99aa8c63674290bf3500f3a24ab260802f58d" exitCode=0 Sep 10 23:50:18.752875 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:18.752735 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e027297d-openshift-default-5cc9d886d4-bggs4" Sep 10 23:50:18.752875 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:18.752773 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e027297d-openshift-default-5cc9d886d4-bggs4" event={"ID":"b61c82a2-ced4-4c1c-8b44-4f94593d5abb","Type":"ContainerDied","Data":"7f19b40920477c764fd70e4b81a99aa8c63674290bf3500f3a24ab260802f58d"} Sep 10 23:50:18.752875 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:18.752829 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e027297d-openshift-default-5cc9d886d4-bggs4" event={"ID":"b61c82a2-ced4-4c1c-8b44-4f94593d5abb","Type":"ContainerDied","Data":"da7962a56e32bd719ff491d5138a73e5e4387f618086817475f23340423e189e"} Sep 10 23:50:18.752875 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:18.752858 2574 scope.go:117] "RemoveContainer" containerID="7f19b40920477c764fd70e4b81a99aa8c63674290bf3500f3a24ab260802f58d" Sep 10 23:50:18.774895 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:18.774823 2574 scope.go:117] "RemoveContainer" containerID="b18d24bfb72bed217f40b2d61999712cc42e3108886e685f90a866207a089b32" Sep 10 23:50:18.788360 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:18.788278 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-e027297d-openshift-default-5cc9d886d4-bggs4"] Sep 10 23:50:18.793505 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:18.793460 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-e027297d-openshift-default-5cc9d886d4-bggs4"] Sep 10 23:50:18.806857 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:18.806482 2574 scope.go:117] "RemoveContainer" containerID="7f19b40920477c764fd70e4b81a99aa8c63674290bf3500f3a24ab260802f58d" Sep 10 23:50:18.807489 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:50:18.807436 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f19b40920477c764fd70e4b81a99aa8c63674290bf3500f3a24ab260802f58d\": container with ID starting with 7f19b40920477c764fd70e4b81a99aa8c63674290bf3500f3a24ab260802f58d not found: ID does not exist" containerID="7f19b40920477c764fd70e4b81a99aa8c63674290bf3500f3a24ab260802f58d" Sep 10 23:50:18.807618 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:18.807481 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f19b40920477c764fd70e4b81a99aa8c63674290bf3500f3a24ab260802f58d"} err="failed to get container status \"7f19b40920477c764fd70e4b81a99aa8c63674290bf3500f3a24ab260802f58d\": rpc error: code = NotFound desc = could not find container \"7f19b40920477c764fd70e4b81a99aa8c63674290bf3500f3a24ab260802f58d\": container with ID starting with 7f19b40920477c764fd70e4b81a99aa8c63674290bf3500f3a24ab260802f58d not found: ID does not exist" Sep 10 23:50:18.807618 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:18.807510 2574 scope.go:117] "RemoveContainer" containerID="b18d24bfb72bed217f40b2d61999712cc42e3108886e685f90a866207a089b32" Sep 10 23:50:18.807978 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:50:18.807910 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b18d24bfb72bed217f40b2d61999712cc42e3108886e685f90a866207a089b32\": container with ID starting with b18d24bfb72bed217f40b2d61999712cc42e3108886e685f90a866207a089b32 not found: ID does not exist" containerID="b18d24bfb72bed217f40b2d61999712cc42e3108886e685f90a866207a089b32" Sep 10 23:50:18.807978 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:18.807940 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b18d24bfb72bed217f40b2d61999712cc42e3108886e685f90a866207a089b32"} err="failed to get container status \"b18d24bfb72bed217f40b2d61999712cc42e3108886e685f90a866207a089b32\": rpc error: code = NotFound desc = could not find container \"b18d24bfb72bed217f40b2d61999712cc42e3108886e685f90a866207a089b32\": container with ID starting with b18d24bfb72bed217f40b2d61999712cc42e3108886e685f90a866207a089b32 not found: ID does not exist" Sep 10 23:50:19.095131 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:19.095045 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b61c82a2-ced4-4c1c-8b44-4f94593d5abb" path="/var/lib/kubelet/pods/b61c82a2-ced4-4c1c-8b44-4f94593d5abb/volumes" Sep 10 23:50:23.697673 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:23.697620 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-56r7l"] Sep 10 23:50:32.779862 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:32.779819 2574 patch_prober.go:28] interesting pod/e2e-migrate-6a8995da-gw-openshift-default-6c66554d4d-hdr6p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 10 23:50:32.780244 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:32.779895 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-6a8995da-gw-openshift-default-6c66554d4d-hdr6p" podUID="dfb8b1ec-79f0-4db8-871e-fbd82b1623e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 10 23:50:33.848577 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:33.848541 2574 generic.go:356] "Generic (PLEG): container finished" podID="dfb8b1ec-79f0-4db8-871e-fbd82b1623e6" containerID="0dab95069fa9edd9fec5e0ec662a78d540465f00111d252e4341b9d0d5a167f7" exitCode=0 Sep 10 23:50:33.849028 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:33.848613 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-6a8995da-gw-openshift-default-6c66554d4d-hdr6p" event={"ID":"dfb8b1ec-79f0-4db8-871e-fbd82b1623e6","Type":"ContainerDied","Data":"0dab95069fa9edd9fec5e0ec662a78d540465f00111d252e4341b9d0d5a167f7"} Sep 10 23:50:33.849131 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:33.849112 2574 scope.go:117] "RemoveContainer" containerID="0dab95069fa9edd9fec5e0ec662a78d540465f00111d252e4341b9d0d5a167f7" Sep 10 23:50:33.850204 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:33.850176 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 23:50:33.850279 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:33.850266 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 23:50:33.850320 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:33.850301 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 23:50:34.856981 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:34.856941 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-6a8995da-gw-openshift-default-6c66554d4d-hdr6p" event={"ID":"dfb8b1ec-79f0-4db8-871e-fbd82b1623e6","Type":"ContainerStarted","Data":"d5065280c9e9fb0ba27dd1d0c2bc16dd18b8ba2812013777d3743267e12e57bb"} Sep 10 23:50:35.444453 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:35.444415 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-6a8995da-gw-openshift-default-6c66554d4d-hdr6p" Sep 10 23:50:35.444717 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:35.444694 2574 patch_prober.go:28] interesting pod/e2e-migrate-6a8995da-gw-openshift-default-6c66554d4d-hdr6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Sep 10 23:50:35.444780 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:35.444759 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-6a8995da-gw-openshift-default-6c66554d4d-hdr6p" podUID="dfb8b1ec-79f0-4db8-871e-fbd82b1623e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Sep 10 23:50:36.444972 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:36.444926 2574 patch_prober.go:28] interesting pod/e2e-migrate-6a8995da-gw-openshift-default-6c66554d4d-hdr6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Sep 10 23:50:36.445375 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:36.445003 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-6a8995da-gw-openshift-default-6c66554d4d-hdr6p" podUID="dfb8b1ec-79f0-4db8-871e-fbd82b1623e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Sep 10 23:50:37.445454 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:37.445419 2574 patch_prober.go:28] interesting pod/e2e-migrate-6a8995da-gw-openshift-default-6c66554d4d-hdr6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Sep 10 23:50:37.445838 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:37.445473 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-6a8995da-gw-openshift-default-6c66554d4d-hdr6p" podUID="dfb8b1ec-79f0-4db8-871e-fbd82b1623e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Sep 10 23:50:38.444823 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:38.444782 2574 patch_prober.go:28] interesting pod/e2e-migrate-6a8995da-gw-openshift-default-6c66554d4d-hdr6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Sep 10 23:50:38.445013 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:38.444842 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-6a8995da-gw-openshift-default-6c66554d4d-hdr6p" podUID="dfb8b1ec-79f0-4db8-871e-fbd82b1623e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Sep 10 23:50:39.445131 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:39.445093 2574 patch_prober.go:28] interesting pod/e2e-migrate-6a8995da-gw-openshift-default-6c66554d4d-hdr6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Sep 10 23:50:39.445558 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:39.445185 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-6a8995da-gw-openshift-default-6c66554d4d-hdr6p" podUID="dfb8b1ec-79f0-4db8-871e-fbd82b1623e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Sep 10 23:50:40.448628 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:40.448593 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-6a8995da-gw-openshift-default-6c66554d4d-hdr6p" Sep 10 23:50:40.449110 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:40.448905 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-6a8995da-gw-openshift-default-6c66554d4d-hdr6p" Sep 10 23:50:40.450098 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:40.450078 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-6a8995da-gw-openshift-default-6c66554d4d-hdr6p" Sep 10 23:50:40.930432 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:40.930350 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-56r7l"] Sep 10 23:50:49.075123 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:49.075089 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-6a8995da-gw-openshift-default-6c66554d4d-hdr6p"] Sep 10 23:50:49.075521 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:49.075358 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-6a8995da-gw-openshift-default-6c66554d4d-hdr6p" podUID="dfb8b1ec-79f0-4db8-871e-fbd82b1623e6" containerName="istio-proxy" containerID="cri-o://d5065280c9e9fb0ba27dd1d0c2bc16dd18b8ba2812013777d3743267e12e57bb" gracePeriod=30 Sep 10 23:50:49.788859 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:49.788825 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-5706c553-openshift-default-5f9795bb96-pwgxg"] Sep 10 23:50:49.789379 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:49.789359 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b61c82a2-ced4-4c1c-8b44-4f94593d5abb" containerName="istio-proxy" Sep 10 23:50:49.789379 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:49.789380 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="b61c82a2-ced4-4c1c-8b44-4f94593d5abb" containerName="istio-proxy" Sep 10 23:50:49.789534 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:49.789396 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b61c82a2-ced4-4c1c-8b44-4f94593d5abb" containerName="istio-proxy" Sep 10 23:50:49.789534 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:49.789404 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="b61c82a2-ced4-4c1c-8b44-4f94593d5abb" containerName="istio-proxy" Sep 10 23:50:49.789659 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:49.789533 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="b61c82a2-ced4-4c1c-8b44-4f94593d5abb" containerName="istio-proxy" Sep 10 23:50:49.789659 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:49.789546 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="b61c82a2-ced4-4c1c-8b44-4f94593d5abb" containerName="istio-proxy" Sep 10 23:50:49.794366 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:49.794345 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5706c553-openshift-default-5f9795bb96-pwgxg" Sep 10 23:50:49.797174 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:49.797152 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-5706c553-openshift-default-dockercfg-gt9nl\"" Sep 10 23:50:49.806207 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:49.806178 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-5706c553-openshift-default-5f9795bb96-pwgxg"] Sep 10 23:50:49.990895 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:49.990862 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d86ed847-7911-4c26-a728-4e87c31e756d-workload-socket\") pod \"e2e-ait-5706c553-openshift-default-5f9795bb96-pwgxg\" (UID: \"d86ed847-7911-4c26-a728-4e87c31e756d\") " pod="openshift-ingress/e2e-ait-5706c553-openshift-default-5f9795bb96-pwgxg" Sep 10 23:50:49.991080 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:49.990915 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d86ed847-7911-4c26-a728-4e87c31e756d-istio-envoy\") pod \"e2e-ait-5706c553-openshift-default-5f9795bb96-pwgxg\" (UID: \"d86ed847-7911-4c26-a728-4e87c31e756d\") " pod="openshift-ingress/e2e-ait-5706c553-openshift-default-5f9795bb96-pwgxg" Sep 10 23:50:49.991080 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:49.991043 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d86ed847-7911-4c26-a728-4e87c31e756d-credential-socket\") pod \"e2e-ait-5706c553-openshift-default-5f9795bb96-pwgxg\" (UID: \"d86ed847-7911-4c26-a728-4e87c31e756d\") " pod="openshift-ingress/e2e-ait-5706c553-openshift-default-5f9795bb96-pwgxg" Sep 10 23:50:49.991182 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:49.991098 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8g8j\" (UniqueName: \"kubernetes.io/projected/d86ed847-7911-4c26-a728-4e87c31e756d-kube-api-access-h8g8j\") pod \"e2e-ait-5706c553-openshift-default-5f9795bb96-pwgxg\" (UID: \"d86ed847-7911-4c26-a728-4e87c31e756d\") " pod="openshift-ingress/e2e-ait-5706c553-openshift-default-5f9795bb96-pwgxg" Sep 10 23:50:49.991241 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:49.991201 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d86ed847-7911-4c26-a728-4e87c31e756d-istiod-ca-cert\") pod \"e2e-ait-5706c553-openshift-default-5f9795bb96-pwgxg\" (UID: \"d86ed847-7911-4c26-a728-4e87c31e756d\") " pod="openshift-ingress/e2e-ait-5706c553-openshift-default-5f9795bb96-pwgxg" Sep 10 23:50:49.991295 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:49.991254 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d86ed847-7911-4c26-a728-4e87c31e756d-istio-data\") pod \"e2e-ait-5706c553-openshift-default-5f9795bb96-pwgxg\" (UID: \"d86ed847-7911-4c26-a728-4e87c31e756d\") " pod="openshift-ingress/e2e-ait-5706c553-openshift-default-5f9795bb96-pwgxg" Sep 10 23:50:49.991376 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:49.991349 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d86ed847-7911-4c26-a728-4e87c31e756d-istio-podinfo\") pod \"e2e-ait-5706c553-openshift-default-5f9795bb96-pwgxg\" (UID: \"d86ed847-7911-4c26-a728-4e87c31e756d\") " pod="openshift-ingress/e2e-ait-5706c553-openshift-default-5f9795bb96-pwgxg" Sep 10 23:50:49.991433 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:49.991419 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d86ed847-7911-4c26-a728-4e87c31e756d-istio-token\") pod \"e2e-ait-5706c553-openshift-default-5f9795bb96-pwgxg\" (UID: \"d86ed847-7911-4c26-a728-4e87c31e756d\") " pod="openshift-ingress/e2e-ait-5706c553-openshift-default-5f9795bb96-pwgxg" Sep 10 23:50:49.991490 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:49.991474 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d86ed847-7911-4c26-a728-4e87c31e756d-workload-certs\") pod \"e2e-ait-5706c553-openshift-default-5f9795bb96-pwgxg\" (UID: \"d86ed847-7911-4c26-a728-4e87c31e756d\") " pod="openshift-ingress/e2e-ait-5706c553-openshift-default-5f9795bb96-pwgxg" Sep 10 23:50:50.092312 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:50.092211 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d86ed847-7911-4c26-a728-4e87c31e756d-istio-podinfo\") pod \"e2e-ait-5706c553-openshift-default-5f9795bb96-pwgxg\" (UID: \"d86ed847-7911-4c26-a728-4e87c31e756d\") " pod="openshift-ingress/e2e-ait-5706c553-openshift-default-5f9795bb96-pwgxg" Sep 10 23:50:50.092985 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:50.092956 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d86ed847-7911-4c26-a728-4e87c31e756d-istio-token\") pod \"e2e-ait-5706c553-openshift-default-5f9795bb96-pwgxg\" (UID: \"d86ed847-7911-4c26-a728-4e87c31e756d\") " pod="openshift-ingress/e2e-ait-5706c553-openshift-default-5f9795bb96-pwgxg" Sep 10 23:50:50.093190 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:50.093164 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d86ed847-7911-4c26-a728-4e87c31e756d-workload-certs\") pod \"e2e-ait-5706c553-openshift-default-5f9795bb96-pwgxg\" (UID: \"d86ed847-7911-4c26-a728-4e87c31e756d\") " pod="openshift-ingress/e2e-ait-5706c553-openshift-default-5f9795bb96-pwgxg" Sep 10 23:50:50.093339 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:50.093322 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d86ed847-7911-4c26-a728-4e87c31e756d-workload-socket\") pod \"e2e-ait-5706c553-openshift-default-5f9795bb96-pwgxg\" (UID: \"d86ed847-7911-4c26-a728-4e87c31e756d\") " pod="openshift-ingress/e2e-ait-5706c553-openshift-default-5f9795bb96-pwgxg" Sep 10 23:50:50.093485 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:50.093466 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d86ed847-7911-4c26-a728-4e87c31e756d-istio-envoy\") pod \"e2e-ait-5706c553-openshift-default-5f9795bb96-pwgxg\" (UID: \"d86ed847-7911-4c26-a728-4e87c31e756d\") " pod="openshift-ingress/e2e-ait-5706c553-openshift-default-5f9795bb96-pwgxg" Sep 10 23:50:50.093607 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:50.093594 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d86ed847-7911-4c26-a728-4e87c31e756d-credential-socket\") pod \"e2e-ait-5706c553-openshift-default-5f9795bb96-pwgxg\" (UID: \"d86ed847-7911-4c26-a728-4e87c31e756d\") " pod="openshift-ingress/e2e-ait-5706c553-openshift-default-5f9795bb96-pwgxg" Sep 10 23:50:50.093735 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:50.093718 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8g8j\" (UniqueName: \"kubernetes.io/projected/d86ed847-7911-4c26-a728-4e87c31e756d-kube-api-access-h8g8j\") pod \"e2e-ait-5706c553-openshift-default-5f9795bb96-pwgxg\" (UID: \"d86ed847-7911-4c26-a728-4e87c31e756d\") " pod="openshift-ingress/e2e-ait-5706c553-openshift-default-5f9795bb96-pwgxg" Sep 10 23:50:50.093885 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:50.093868 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d86ed847-7911-4c26-a728-4e87c31e756d-istiod-ca-cert\") pod \"e2e-ait-5706c553-openshift-default-5f9795bb96-pwgxg\" (UID: \"d86ed847-7911-4c26-a728-4e87c31e756d\") " pod="openshift-ingress/e2e-ait-5706c553-openshift-default-5f9795bb96-pwgxg" Sep 10 23:50:50.094067 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:50.094046 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d86ed847-7911-4c26-a728-4e87c31e756d-istio-data\") pod \"e2e-ait-5706c553-openshift-default-5f9795bb96-pwgxg\" (UID: \"d86ed847-7911-4c26-a728-4e87c31e756d\") " pod="openshift-ingress/e2e-ait-5706c553-openshift-default-5f9795bb96-pwgxg" Sep 10 23:50:50.094333 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:50.094137 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d86ed847-7911-4c26-a728-4e87c31e756d-workload-certs\") pod \"e2e-ait-5706c553-openshift-default-5f9795bb96-pwgxg\" (UID: \"d86ed847-7911-4c26-a728-4e87c31e756d\") " pod="openshift-ingress/e2e-ait-5706c553-openshift-default-5f9795bb96-pwgxg" Sep 10 23:50:50.094446 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:50.094370 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d86ed847-7911-4c26-a728-4e87c31e756d-workload-socket\") pod \"e2e-ait-5706c553-openshift-default-5f9795bb96-pwgxg\" (UID: \"d86ed847-7911-4c26-a728-4e87c31e756d\") " pod="openshift-ingress/e2e-ait-5706c553-openshift-default-5f9795bb96-pwgxg" Sep 10 23:50:50.094831 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:50.094809 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d86ed847-7911-4c26-a728-4e87c31e756d-istio-data\") pod \"e2e-ait-5706c553-openshift-default-5f9795bb96-pwgxg\" (UID: \"d86ed847-7911-4c26-a728-4e87c31e756d\") " pod="openshift-ingress/e2e-ait-5706c553-openshift-default-5f9795bb96-pwgxg" Sep 10 23:50:50.095135 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:50.095077 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d86ed847-7911-4c26-a728-4e87c31e756d-istiod-ca-cert\") pod \"e2e-ait-5706c553-openshift-default-5f9795bb96-pwgxg\" (UID: \"d86ed847-7911-4c26-a728-4e87c31e756d\") " pod="openshift-ingress/e2e-ait-5706c553-openshift-default-5f9795bb96-pwgxg" Sep 10 23:50:50.095477 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:50.095452 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d86ed847-7911-4c26-a728-4e87c31e756d-credential-socket\") pod \"e2e-ait-5706c553-openshift-default-5f9795bb96-pwgxg\" (UID: \"d86ed847-7911-4c26-a728-4e87c31e756d\") " pod="openshift-ingress/e2e-ait-5706c553-openshift-default-5f9795bb96-pwgxg" Sep 10 23:50:50.095632 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:50.095605 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d86ed847-7911-4c26-a728-4e87c31e756d-istio-podinfo\") pod \"e2e-ait-5706c553-openshift-default-5f9795bb96-pwgxg\" (UID: \"d86ed847-7911-4c26-a728-4e87c31e756d\") " pod="openshift-ingress/e2e-ait-5706c553-openshift-default-5f9795bb96-pwgxg" Sep 10 23:50:50.097238 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:50.097215 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d86ed847-7911-4c26-a728-4e87c31e756d-istio-envoy\") pod \"e2e-ait-5706c553-openshift-default-5f9795bb96-pwgxg\" (UID: \"d86ed847-7911-4c26-a728-4e87c31e756d\") " pod="openshift-ingress/e2e-ait-5706c553-openshift-default-5f9795bb96-pwgxg" Sep 10 23:50:50.102198 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:50.102169 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8g8j\" (UniqueName: \"kubernetes.io/projected/d86ed847-7911-4c26-a728-4e87c31e756d-kube-api-access-h8g8j\") pod \"e2e-ait-5706c553-openshift-default-5f9795bb96-pwgxg\" (UID: \"d86ed847-7911-4c26-a728-4e87c31e756d\") " pod="openshift-ingress/e2e-ait-5706c553-openshift-default-5f9795bb96-pwgxg" Sep 10 23:50:50.103520 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:50.103498 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d86ed847-7911-4c26-a728-4e87c31e756d-istio-token\") pod \"e2e-ait-5706c553-openshift-default-5f9795bb96-pwgxg\" (UID: \"d86ed847-7911-4c26-a728-4e87c31e756d\") " pod="openshift-ingress/e2e-ait-5706c553-openshift-default-5f9795bb96-pwgxg" Sep 10 23:50:50.110545 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:50.110522 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5706c553-openshift-default-5f9795bb96-pwgxg" Sep 10 23:50:50.400414 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:50.400387 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-5706c553-openshift-default-5f9795bb96-pwgxg"] Sep 10 23:50:50.401273 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:50:50.401248 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd86ed847_7911_4c26_a728_4e87c31e756d.slice/crio-cd3b11db12fa9c3c47a363d4cf6277631504f47286ed007c4b881c51d397d985 WatchSource:0}: Error finding container cd3b11db12fa9c3c47a363d4cf6277631504f47286ed007c4b881c51d397d985: Status 404 returned error can't find the container with id cd3b11db12fa9c3c47a363d4cf6277631504f47286ed007c4b881c51d397d985 Sep 10 23:50:50.403978 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:50.403944 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 23:50:50.404033 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:50.404008 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 23:50:50.404079 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:50.404048 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 23:50:50.974589 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:50.974555 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5706c553-openshift-default-5f9795bb96-pwgxg" event={"ID":"d86ed847-7911-4c26-a728-4e87c31e756d","Type":"ContainerStarted","Data":"1278a3e2424510f0c3d8b10f86072fdd2f0c4b6e59ece4e7c0b69f15ac85c960"} Sep 10 23:50:50.974774 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:50.974617 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5706c553-openshift-default-5f9795bb96-pwgxg" event={"ID":"d86ed847-7911-4c26-a728-4e87c31e756d","Type":"ContainerStarted","Data":"cd3b11db12fa9c3c47a363d4cf6277631504f47286ed007c4b881c51d397d985"} Sep 10 23:50:51.000654 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:51.000581 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-5706c553-openshift-default-5f9795bb96-pwgxg" podStartSLOduration=2.000564465 podStartE2EDuration="2.000564465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:50:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:50:50.998045276 +0000 UTC m=+842.385648035" watchObservedRunningTime="2026-09-10 23:50:51.000564465 +0000 UTC m=+842.388167236" Sep 10 23:50:51.111284 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:51.111250 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-5706c553-openshift-default-5f9795bb96-pwgxg" Sep 10 23:50:51.129264 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:51.129236 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-5706c553-openshift-default-5f9795bb96-pwgxg" Sep 10 23:50:51.980786 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:51.980755 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-5706c553-openshift-default-5f9795bb96-pwgxg" Sep 10 23:50:51.981686 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:51.981665 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-5706c553-openshift-default-5f9795bb96-pwgxg" Sep 10 23:50:54.338061 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:54.337975 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-6a8995da-gw-openshift-default-6c66554d4d-hdr6p" Sep 10 23:50:54.438243 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:54.438200 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfb8b1ec-79f0-4db8-871e-fbd82b1623e6-workload-certs\") pod \"dfb8b1ec-79f0-4db8-871e-fbd82b1623e6\" (UID: \"dfb8b1ec-79f0-4db8-871e-fbd82b1623e6\") " Sep 10 23:50:54.438243 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:54.438243 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfb8b1ec-79f0-4db8-871e-fbd82b1623e6-credential-socket\") pod \"dfb8b1ec-79f0-4db8-871e-fbd82b1623e6\" (UID: \"dfb8b1ec-79f0-4db8-871e-fbd82b1623e6\") " Sep 10 23:50:54.438623 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:54.438286 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfb8b1ec-79f0-4db8-871e-fbd82b1623e6-istio-podinfo\") pod \"dfb8b1ec-79f0-4db8-871e-fbd82b1623e6\" (UID: \"dfb8b1ec-79f0-4db8-871e-fbd82b1623e6\") " Sep 10 23:50:54.438623 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:54.438311 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfb8b1ec-79f0-4db8-871e-fbd82b1623e6-istiod-ca-cert\") pod \"dfb8b1ec-79f0-4db8-871e-fbd82b1623e6\" (UID: \"dfb8b1ec-79f0-4db8-871e-fbd82b1623e6\") " Sep 10 23:50:54.438623 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:54.438341 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfb8b1ec-79f0-4db8-871e-fbd82b1623e6-istio-data\") pod \"dfb8b1ec-79f0-4db8-871e-fbd82b1623e6\" (UID: \"dfb8b1ec-79f0-4db8-871e-fbd82b1623e6\") " Sep 10 23:50:54.438623 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:54.438410 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfb8b1ec-79f0-4db8-871e-fbd82b1623e6-istio-token\") pod \"dfb8b1ec-79f0-4db8-871e-fbd82b1623e6\" (UID: \"dfb8b1ec-79f0-4db8-871e-fbd82b1623e6\") " Sep 10 23:50:54.438623 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:54.438444 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jp6fc\" (UniqueName: \"kubernetes.io/projected/dfb8b1ec-79f0-4db8-871e-fbd82b1623e6-kube-api-access-jp6fc\") pod \"dfb8b1ec-79f0-4db8-871e-fbd82b1623e6\" (UID: \"dfb8b1ec-79f0-4db8-871e-fbd82b1623e6\") " Sep 10 23:50:54.438623 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:54.438492 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfb8b1ec-79f0-4db8-871e-fbd82b1623e6-istio-envoy\") pod \"dfb8b1ec-79f0-4db8-871e-fbd82b1623e6\" (UID: \"dfb8b1ec-79f0-4db8-871e-fbd82b1623e6\") " Sep 10 23:50:54.438623 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:54.438536 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfb8b1ec-79f0-4db8-871e-fbd82b1623e6-workload-socket\") pod \"dfb8b1ec-79f0-4db8-871e-fbd82b1623e6\" (UID: \"dfb8b1ec-79f0-4db8-871e-fbd82b1623e6\") " Sep 10 23:50:54.438623 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:54.438536 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfb8b1ec-79f0-4db8-871e-fbd82b1623e6-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "dfb8b1ec-79f0-4db8-871e-fbd82b1623e6" (UID: "dfb8b1ec-79f0-4db8-871e-fbd82b1623e6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:50:54.438623 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:54.438540 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfb8b1ec-79f0-4db8-871e-fbd82b1623e6-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "dfb8b1ec-79f0-4db8-871e-fbd82b1623e6" (UID: "dfb8b1ec-79f0-4db8-871e-fbd82b1623e6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:50:54.439166 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:54.438690 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfb8b1ec-79f0-4db8-871e-fbd82b1623e6-istio-data" (OuterVolumeSpecName: "istio-data") pod "dfb8b1ec-79f0-4db8-871e-fbd82b1623e6" (UID: "dfb8b1ec-79f0-4db8-871e-fbd82b1623e6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:50:54.439166 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:54.438770 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfb8b1ec-79f0-4db8-871e-fbd82b1623e6-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "dfb8b1ec-79f0-4db8-871e-fbd82b1623e6" (UID: "dfb8b1ec-79f0-4db8-871e-fbd82b1623e6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 23:50:54.439166 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:54.438944 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfb8b1ec-79f0-4db8-871e-fbd82b1623e6-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "dfb8b1ec-79f0-4db8-871e-fbd82b1623e6" (UID: "dfb8b1ec-79f0-4db8-871e-fbd82b1623e6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:50:54.439166 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:54.439040 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfb8b1ec-79f0-4db8-871e-fbd82b1623e6-workload-certs\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:50:54.439166 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:54.439064 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfb8b1ec-79f0-4db8-871e-fbd82b1623e6-credential-socket\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:50:54.439166 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:54.439080 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfb8b1ec-79f0-4db8-871e-fbd82b1623e6-istiod-ca-cert\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:50:54.439166 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:54.439094 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfb8b1ec-79f0-4db8-871e-fbd82b1623e6-istio-data\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:50:54.439166 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:54.439106 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfb8b1ec-79f0-4db8-871e-fbd82b1623e6-workload-socket\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:50:54.440683 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:54.440634 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfb8b1ec-79f0-4db8-871e-fbd82b1623e6-istio-token" (OuterVolumeSpecName: "istio-token") pod "dfb8b1ec-79f0-4db8-871e-fbd82b1623e6" (UID: "dfb8b1ec-79f0-4db8-871e-fbd82b1623e6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 23:50:54.440683 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:54.440662 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dfb8b1ec-79f0-4db8-871e-fbd82b1623e6-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "dfb8b1ec-79f0-4db8-871e-fbd82b1623e6" (UID: "dfb8b1ec-79f0-4db8-871e-fbd82b1623e6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 23:50:54.440841 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:54.440700 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfb8b1ec-79f0-4db8-871e-fbd82b1623e6-kube-api-access-jp6fc" (OuterVolumeSpecName: "kube-api-access-jp6fc") pod "dfb8b1ec-79f0-4db8-871e-fbd82b1623e6" (UID: "dfb8b1ec-79f0-4db8-871e-fbd82b1623e6"). InnerVolumeSpecName "kube-api-access-jp6fc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 23:50:54.440900 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:54.440882 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfb8b1ec-79f0-4db8-871e-fbd82b1623e6-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "dfb8b1ec-79f0-4db8-871e-fbd82b1623e6" (UID: "dfb8b1ec-79f0-4db8-871e-fbd82b1623e6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:50:54.540232 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:54.540192 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jp6fc\" (UniqueName: \"kubernetes.io/projected/dfb8b1ec-79f0-4db8-871e-fbd82b1623e6-kube-api-access-jp6fc\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:50:54.540232 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:54.540232 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfb8b1ec-79f0-4db8-871e-fbd82b1623e6-istio-envoy\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:50:54.540445 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:54.540245 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfb8b1ec-79f0-4db8-871e-fbd82b1623e6-istio-podinfo\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:50:54.540445 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:54.540256 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfb8b1ec-79f0-4db8-871e-fbd82b1623e6-istio-token\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:50:55.004245 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:55.004211 2574 generic.go:356] "Generic (PLEG): container finished" podID="dfb8b1ec-79f0-4db8-871e-fbd82b1623e6" containerID="d5065280c9e9fb0ba27dd1d0c2bc16dd18b8ba2812013777d3743267e12e57bb" exitCode=0 Sep 10 23:50:55.004422 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:55.004265 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-6a8995da-gw-openshift-default-6c66554d4d-hdr6p" Sep 10 23:50:55.004422 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:55.004289 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-6a8995da-gw-openshift-default-6c66554d4d-hdr6p" event={"ID":"dfb8b1ec-79f0-4db8-871e-fbd82b1623e6","Type":"ContainerDied","Data":"d5065280c9e9fb0ba27dd1d0c2bc16dd18b8ba2812013777d3743267e12e57bb"} Sep 10 23:50:55.004422 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:55.004373 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-6a8995da-gw-openshift-default-6c66554d4d-hdr6p" event={"ID":"dfb8b1ec-79f0-4db8-871e-fbd82b1623e6","Type":"ContainerDied","Data":"b4ece5b368bd0c93efe42ca1455c2cfc6d49da0267f33f8bf5a7c3e327accb45"} Sep 10 23:50:55.004422 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:55.004396 2574 scope.go:117] "RemoveContainer" containerID="d5065280c9e9fb0ba27dd1d0c2bc16dd18b8ba2812013777d3743267e12e57bb" Sep 10 23:50:55.014749 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:55.014731 2574 scope.go:117] "RemoveContainer" containerID="0dab95069fa9edd9fec5e0ec662a78d540465f00111d252e4341b9d0d5a167f7" Sep 10 23:50:55.026974 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:55.026877 2574 scope.go:117] "RemoveContainer" containerID="d5065280c9e9fb0ba27dd1d0c2bc16dd18b8ba2812013777d3743267e12e57bb" Sep 10 23:50:55.027557 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:50:55.027523 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5065280c9e9fb0ba27dd1d0c2bc16dd18b8ba2812013777d3743267e12e57bb\": container with ID starting with d5065280c9e9fb0ba27dd1d0c2bc16dd18b8ba2812013777d3743267e12e57bb not found: ID does not exist" containerID="d5065280c9e9fb0ba27dd1d0c2bc16dd18b8ba2812013777d3743267e12e57bb" Sep 10 23:50:55.027699 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:55.027571 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5065280c9e9fb0ba27dd1d0c2bc16dd18b8ba2812013777d3743267e12e57bb"} err="failed to get container status \"d5065280c9e9fb0ba27dd1d0c2bc16dd18b8ba2812013777d3743267e12e57bb\": rpc error: code = NotFound desc = could not find container \"d5065280c9e9fb0ba27dd1d0c2bc16dd18b8ba2812013777d3743267e12e57bb\": container with ID starting with d5065280c9e9fb0ba27dd1d0c2bc16dd18b8ba2812013777d3743267e12e57bb not found: ID does not exist" Sep 10 23:50:55.027699 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:55.027596 2574 scope.go:117] "RemoveContainer" containerID="0dab95069fa9edd9fec5e0ec662a78d540465f00111d252e4341b9d0d5a167f7" Sep 10 23:50:55.028024 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:50:55.028000 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0dab95069fa9edd9fec5e0ec662a78d540465f00111d252e4341b9d0d5a167f7\": container with ID starting with 0dab95069fa9edd9fec5e0ec662a78d540465f00111d252e4341b9d0d5a167f7 not found: ID does not exist" containerID="0dab95069fa9edd9fec5e0ec662a78d540465f00111d252e4341b9d0d5a167f7" Sep 10 23:50:55.028134 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:55.028035 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0dab95069fa9edd9fec5e0ec662a78d540465f00111d252e4341b9d0d5a167f7"} err="failed to get container status \"0dab95069fa9edd9fec5e0ec662a78d540465f00111d252e4341b9d0d5a167f7\": rpc error: code = NotFound desc = could not find container \"0dab95069fa9edd9fec5e0ec662a78d540465f00111d252e4341b9d0d5a167f7\": container with ID starting with 0dab95069fa9edd9fec5e0ec662a78d540465f00111d252e4341b9d0d5a167f7 not found: ID does not exist" Sep 10 23:50:55.030943 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:55.030906 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-6a8995da-gw-openshift-default-6c66554d4d-hdr6p"] Sep 10 23:50:55.033525 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:55.033506 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-6a8995da-gw-openshift-default-6c66554d4d-hdr6p"] Sep 10 23:50:55.094788 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:55.094753 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dfb8b1ec-79f0-4db8-871e-fbd82b1623e6" path="/var/lib/kubelet/pods/dfb8b1ec-79f0-4db8-871e-fbd82b1623e6/volumes" Sep 10 23:50:58.587477 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:50:58.587440 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-56r7l"] Sep 10 23:51:07.975174 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:07.975141 2574 patch_prober.go:28] interesting pod/e2e-ait-5706c553-openshift-default-5f9795bb96-pwgxg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": read tcp 10.134.0.2:47556->10.134.0.59:15021: read: connection reset by peer" start-of-body= Sep 10 23:51:07.975582 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:07.975208 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-5706c553-openshift-default-5f9795bb96-pwgxg" podUID="d86ed847-7911-4c26-a728-4e87c31e756d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": read tcp 10.134.0.2:47556->10.134.0.59:15021: read: connection reset by peer" Sep 10 23:51:08.089379 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:08.089347 2574 generic.go:356] "Generic (PLEG): container finished" podID="d86ed847-7911-4c26-a728-4e87c31e756d" containerID="1278a3e2424510f0c3d8b10f86072fdd2f0c4b6e59ece4e7c0b69f15ac85c960" exitCode=0 Sep 10 23:51:08.089552 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:08.089422 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5706c553-openshift-default-5f9795bb96-pwgxg" event={"ID":"d86ed847-7911-4c26-a728-4e87c31e756d","Type":"ContainerDied","Data":"1278a3e2424510f0c3d8b10f86072fdd2f0c4b6e59ece4e7c0b69f15ac85c960"} Sep 10 23:51:08.089788 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:08.089775 2574 scope.go:117] "RemoveContainer" containerID="1278a3e2424510f0c3d8b10f86072fdd2f0c4b6e59ece4e7c0b69f15ac85c960" Sep 10 23:51:08.090841 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:08.090810 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 23:51:08.090961 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:08.090892 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 23:51:08.090961 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:08.090938 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 23:51:09.102718 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:09.102686 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5706c553-openshift-default-5f9795bb96-pwgxg" event={"ID":"d86ed847-7911-4c26-a728-4e87c31e756d","Type":"ContainerStarted","Data":"3b760a97d9a6db132e039cdced0c013d65d80f690802a7c3d4dbd8cbaa4246cf"} Sep 10 23:51:09.111143 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:09.111120 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-5706c553-openshift-default-5f9795bb96-pwgxg" Sep 10 23:51:09.111293 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:09.111273 2574 patch_prober.go:28] interesting pod/e2e-ait-5706c553-openshift-default-5f9795bb96-pwgxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Sep 10 23:51:09.111341 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:09.111323 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5706c553-openshift-default-5f9795bb96-pwgxg" podUID="d86ed847-7911-4c26-a728-4e87c31e756d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Sep 10 23:51:10.111125 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:10.111090 2574 patch_prober.go:28] interesting pod/e2e-ait-5706c553-openshift-default-5f9795bb96-pwgxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Sep 10 23:51:10.111499 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:10.111146 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5706c553-openshift-default-5f9795bb96-pwgxg" podUID="d86ed847-7911-4c26-a728-4e87c31e756d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Sep 10 23:51:11.111735 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:11.111692 2574 patch_prober.go:28] interesting pod/e2e-ait-5706c553-openshift-default-5f9795bb96-pwgxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Sep 10 23:51:11.112243 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:11.111756 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5706c553-openshift-default-5f9795bb96-pwgxg" podUID="d86ed847-7911-4c26-a728-4e87c31e756d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Sep 10 23:51:12.111941 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:12.111899 2574 patch_prober.go:28] interesting pod/e2e-ait-5706c553-openshift-default-5f9795bb96-pwgxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Sep 10 23:51:12.112358 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:12.111960 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5706c553-openshift-default-5f9795bb96-pwgxg" podUID="d86ed847-7911-4c26-a728-4e87c31e756d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Sep 10 23:51:13.111259 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:13.111225 2574 patch_prober.go:28] interesting pod/e2e-ait-5706c553-openshift-default-5f9795bb96-pwgxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Sep 10 23:51:13.111458 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:13.111285 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5706c553-openshift-default-5f9795bb96-pwgxg" podUID="d86ed847-7911-4c26-a728-4e87c31e756d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Sep 10 23:51:14.111105 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:14.111067 2574 patch_prober.go:28] interesting pod/e2e-ait-5706c553-openshift-default-5f9795bb96-pwgxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Sep 10 23:51:14.111491 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:14.111136 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5706c553-openshift-default-5f9795bb96-pwgxg" podUID="d86ed847-7911-4c26-a728-4e87c31e756d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Sep 10 23:51:15.115346 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:15.115312 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-5706c553-openshift-default-5f9795bb96-pwgxg" Sep 10 23:51:15.115968 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:15.115677 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-5706c553-openshift-default-5f9795bb96-pwgxg" Sep 10 23:51:15.116282 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:15.116261 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-5706c553-openshift-default-5f9795bb96-pwgxg" Sep 10 23:51:15.583831 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:15.583796 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-56r7l"] Sep 10 23:51:25.854880 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:25.854836 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-5706c553-openshift-default-5f9795bb96-pwgxg"] Sep 10 23:51:25.855481 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:25.855055 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-5706c553-openshift-default-5f9795bb96-pwgxg" podUID="d86ed847-7911-4c26-a728-4e87c31e756d" containerName="istio-proxy" containerID="cri-o://3b760a97d9a6db132e039cdced0c013d65d80f690802a7c3d4dbd8cbaa4246cf" gracePeriod=30 Sep 10 23:51:26.550299 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:26.550260 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-a4b2fb82-openshift-default-7d6485799b-6mpln"] Sep 10 23:51:26.550947 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:26.550926 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dfb8b1ec-79f0-4db8-871e-fbd82b1623e6" containerName="istio-proxy" Sep 10 23:51:26.551054 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:26.550951 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="dfb8b1ec-79f0-4db8-871e-fbd82b1623e6" containerName="istio-proxy" Sep 10 23:51:26.551098 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:26.551063 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="dfb8b1ec-79f0-4db8-871e-fbd82b1623e6" containerName="istio-proxy" Sep 10 23:51:26.554731 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:26.554706 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-a4b2fb82-openshift-default-7d6485799b-6mpln" Sep 10 23:51:26.557465 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:26.557442 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-a4b2fb82-openshift-default-dockercfg-94c8m\"" Sep 10 23:51:26.567186 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:26.567159 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-a4b2fb82-openshift-default-7d6485799b-6mpln"] Sep 10 23:51:26.670911 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:26.670813 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f754a97f-d4e6-4276-a1e4-71592585df75-istiod-ca-cert\") pod \"e2e-derive-a4b2fb82-openshift-default-7d6485799b-6mpln\" (UID: \"f754a97f-d4e6-4276-a1e4-71592585df75\") " pod="openshift-ingress/e2e-derive-a4b2fb82-openshift-default-7d6485799b-6mpln" Sep 10 23:51:26.670911 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:26.670858 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f754a97f-d4e6-4276-a1e4-71592585df75-credential-socket\") pod \"e2e-derive-a4b2fb82-openshift-default-7d6485799b-6mpln\" (UID: \"f754a97f-d4e6-4276-a1e4-71592585df75\") " pod="openshift-ingress/e2e-derive-a4b2fb82-openshift-default-7d6485799b-6mpln" Sep 10 23:51:26.671115 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:26.670917 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxth8\" (UniqueName: \"kubernetes.io/projected/f754a97f-d4e6-4276-a1e4-71592585df75-kube-api-access-bxth8\") pod \"e2e-derive-a4b2fb82-openshift-default-7d6485799b-6mpln\" (UID: \"f754a97f-d4e6-4276-a1e4-71592585df75\") " pod="openshift-ingress/e2e-derive-a4b2fb82-openshift-default-7d6485799b-6mpln" Sep 10 23:51:26.671115 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:26.671005 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f754a97f-d4e6-4276-a1e4-71592585df75-istio-envoy\") pod \"e2e-derive-a4b2fb82-openshift-default-7d6485799b-6mpln\" (UID: \"f754a97f-d4e6-4276-a1e4-71592585df75\") " pod="openshift-ingress/e2e-derive-a4b2fb82-openshift-default-7d6485799b-6mpln" Sep 10 23:51:26.671115 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:26.671055 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f754a97f-d4e6-4276-a1e4-71592585df75-workload-certs\") pod \"e2e-derive-a4b2fb82-openshift-default-7d6485799b-6mpln\" (UID: \"f754a97f-d4e6-4276-a1e4-71592585df75\") " pod="openshift-ingress/e2e-derive-a4b2fb82-openshift-default-7d6485799b-6mpln" Sep 10 23:51:26.671115 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:26.671076 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f754a97f-d4e6-4276-a1e4-71592585df75-workload-socket\") pod \"e2e-derive-a4b2fb82-openshift-default-7d6485799b-6mpln\" (UID: \"f754a97f-d4e6-4276-a1e4-71592585df75\") " pod="openshift-ingress/e2e-derive-a4b2fb82-openshift-default-7d6485799b-6mpln" Sep 10 23:51:26.671325 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:26.671133 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f754a97f-d4e6-4276-a1e4-71592585df75-istio-data\") pod \"e2e-derive-a4b2fb82-openshift-default-7d6485799b-6mpln\" (UID: \"f754a97f-d4e6-4276-a1e4-71592585df75\") " pod="openshift-ingress/e2e-derive-a4b2fb82-openshift-default-7d6485799b-6mpln" Sep 10 23:51:26.671325 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:26.671159 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f754a97f-d4e6-4276-a1e4-71592585df75-istio-token\") pod \"e2e-derive-a4b2fb82-openshift-default-7d6485799b-6mpln\" (UID: \"f754a97f-d4e6-4276-a1e4-71592585df75\") " pod="openshift-ingress/e2e-derive-a4b2fb82-openshift-default-7d6485799b-6mpln" Sep 10 23:51:26.671325 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:26.671185 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f754a97f-d4e6-4276-a1e4-71592585df75-istio-podinfo\") pod \"e2e-derive-a4b2fb82-openshift-default-7d6485799b-6mpln\" (UID: \"f754a97f-d4e6-4276-a1e4-71592585df75\") " pod="openshift-ingress/e2e-derive-a4b2fb82-openshift-default-7d6485799b-6mpln" Sep 10 23:51:26.772123 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:26.772090 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f754a97f-d4e6-4276-a1e4-71592585df75-istiod-ca-cert\") pod \"e2e-derive-a4b2fb82-openshift-default-7d6485799b-6mpln\" (UID: \"f754a97f-d4e6-4276-a1e4-71592585df75\") " pod="openshift-ingress/e2e-derive-a4b2fb82-openshift-default-7d6485799b-6mpln" Sep 10 23:51:26.772123 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:26.772127 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f754a97f-d4e6-4276-a1e4-71592585df75-credential-socket\") pod \"e2e-derive-a4b2fb82-openshift-default-7d6485799b-6mpln\" (UID: \"f754a97f-d4e6-4276-a1e4-71592585df75\") " pod="openshift-ingress/e2e-derive-a4b2fb82-openshift-default-7d6485799b-6mpln" Sep 10 23:51:26.772393 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:26.772156 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxth8\" (UniqueName: \"kubernetes.io/projected/f754a97f-d4e6-4276-a1e4-71592585df75-kube-api-access-bxth8\") pod \"e2e-derive-a4b2fb82-openshift-default-7d6485799b-6mpln\" (UID: \"f754a97f-d4e6-4276-a1e4-71592585df75\") " pod="openshift-ingress/e2e-derive-a4b2fb82-openshift-default-7d6485799b-6mpln" Sep 10 23:51:26.772393 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:26.772198 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f754a97f-d4e6-4276-a1e4-71592585df75-istio-envoy\") pod \"e2e-derive-a4b2fb82-openshift-default-7d6485799b-6mpln\" (UID: \"f754a97f-d4e6-4276-a1e4-71592585df75\") " pod="openshift-ingress/e2e-derive-a4b2fb82-openshift-default-7d6485799b-6mpln" Sep 10 23:51:26.772393 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:26.772230 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f754a97f-d4e6-4276-a1e4-71592585df75-workload-certs\") pod \"e2e-derive-a4b2fb82-openshift-default-7d6485799b-6mpln\" (UID: \"f754a97f-d4e6-4276-a1e4-71592585df75\") " pod="openshift-ingress/e2e-derive-a4b2fb82-openshift-default-7d6485799b-6mpln" Sep 10 23:51:26.772393 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:26.772257 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f754a97f-d4e6-4276-a1e4-71592585df75-workload-socket\") pod \"e2e-derive-a4b2fb82-openshift-default-7d6485799b-6mpln\" (UID: \"f754a97f-d4e6-4276-a1e4-71592585df75\") " pod="openshift-ingress/e2e-derive-a4b2fb82-openshift-default-7d6485799b-6mpln" Sep 10 23:51:26.772393 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:26.772297 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f754a97f-d4e6-4276-a1e4-71592585df75-istio-data\") pod \"e2e-derive-a4b2fb82-openshift-default-7d6485799b-6mpln\" (UID: \"f754a97f-d4e6-4276-a1e4-71592585df75\") " pod="openshift-ingress/e2e-derive-a4b2fb82-openshift-default-7d6485799b-6mpln" Sep 10 23:51:26.772393 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:26.772321 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f754a97f-d4e6-4276-a1e4-71592585df75-istio-token\") pod \"e2e-derive-a4b2fb82-openshift-default-7d6485799b-6mpln\" (UID: \"f754a97f-d4e6-4276-a1e4-71592585df75\") " pod="openshift-ingress/e2e-derive-a4b2fb82-openshift-default-7d6485799b-6mpln" Sep 10 23:51:26.772393 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:26.772353 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f754a97f-d4e6-4276-a1e4-71592585df75-istio-podinfo\") pod \"e2e-derive-a4b2fb82-openshift-default-7d6485799b-6mpln\" (UID: \"f754a97f-d4e6-4276-a1e4-71592585df75\") " pod="openshift-ingress/e2e-derive-a4b2fb82-openshift-default-7d6485799b-6mpln" Sep 10 23:51:26.772880 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:26.772845 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f754a97f-d4e6-4276-a1e4-71592585df75-workload-socket\") pod \"e2e-derive-a4b2fb82-openshift-default-7d6485799b-6mpln\" (UID: \"f754a97f-d4e6-4276-a1e4-71592585df75\") " pod="openshift-ingress/e2e-derive-a4b2fb82-openshift-default-7d6485799b-6mpln" Sep 10 23:51:26.773032 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:26.772922 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f754a97f-d4e6-4276-a1e4-71592585df75-workload-certs\") pod \"e2e-derive-a4b2fb82-openshift-default-7d6485799b-6mpln\" (UID: \"f754a97f-d4e6-4276-a1e4-71592585df75\") " pod="openshift-ingress/e2e-derive-a4b2fb82-openshift-default-7d6485799b-6mpln" Sep 10 23:51:26.773032 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:26.772969 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f754a97f-d4e6-4276-a1e4-71592585df75-istio-data\") pod \"e2e-derive-a4b2fb82-openshift-default-7d6485799b-6mpln\" (UID: \"f754a97f-d4e6-4276-a1e4-71592585df75\") " pod="openshift-ingress/e2e-derive-a4b2fb82-openshift-default-7d6485799b-6mpln" Sep 10 23:51:26.773032 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:26.772989 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f754a97f-d4e6-4276-a1e4-71592585df75-credential-socket\") pod \"e2e-derive-a4b2fb82-openshift-default-7d6485799b-6mpln\" (UID: \"f754a97f-d4e6-4276-a1e4-71592585df75\") " pod="openshift-ingress/e2e-derive-a4b2fb82-openshift-default-7d6485799b-6mpln" Sep 10 23:51:26.773165 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:26.773129 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f754a97f-d4e6-4276-a1e4-71592585df75-istiod-ca-cert\") pod \"e2e-derive-a4b2fb82-openshift-default-7d6485799b-6mpln\" (UID: \"f754a97f-d4e6-4276-a1e4-71592585df75\") " pod="openshift-ingress/e2e-derive-a4b2fb82-openshift-default-7d6485799b-6mpln" Sep 10 23:51:26.774789 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:26.774768 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f754a97f-d4e6-4276-a1e4-71592585df75-istio-envoy\") pod \"e2e-derive-a4b2fb82-openshift-default-7d6485799b-6mpln\" (UID: \"f754a97f-d4e6-4276-a1e4-71592585df75\") " pod="openshift-ingress/e2e-derive-a4b2fb82-openshift-default-7d6485799b-6mpln" Sep 10 23:51:26.774976 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:26.774958 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f754a97f-d4e6-4276-a1e4-71592585df75-istio-podinfo\") pod \"e2e-derive-a4b2fb82-openshift-default-7d6485799b-6mpln\" (UID: \"f754a97f-d4e6-4276-a1e4-71592585df75\") " pod="openshift-ingress/e2e-derive-a4b2fb82-openshift-default-7d6485799b-6mpln" Sep 10 23:51:26.780139 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:26.780119 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxth8\" (UniqueName: \"kubernetes.io/projected/f754a97f-d4e6-4276-a1e4-71592585df75-kube-api-access-bxth8\") pod \"e2e-derive-a4b2fb82-openshift-default-7d6485799b-6mpln\" (UID: \"f754a97f-d4e6-4276-a1e4-71592585df75\") " pod="openshift-ingress/e2e-derive-a4b2fb82-openshift-default-7d6485799b-6mpln" Sep 10 23:51:26.780479 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:26.780456 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f754a97f-d4e6-4276-a1e4-71592585df75-istio-token\") pod \"e2e-derive-a4b2fb82-openshift-default-7d6485799b-6mpln\" (UID: \"f754a97f-d4e6-4276-a1e4-71592585df75\") " pod="openshift-ingress/e2e-derive-a4b2fb82-openshift-default-7d6485799b-6mpln" Sep 10 23:51:26.866669 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:26.866610 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-a4b2fb82-openshift-default-7d6485799b-6mpln" Sep 10 23:51:27.144804 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:27.144774 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-a4b2fb82-openshift-default-7d6485799b-6mpln"] Sep 10 23:51:27.145570 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:51:27.145544 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf754a97f_d4e6_4276_a1e4_71592585df75.slice/crio-0e0de5a6647a69cefc75f1869c071ea51dbe68fb1e3e8e14a0755aea2dec31ad WatchSource:0}: Error finding container 0e0de5a6647a69cefc75f1869c071ea51dbe68fb1e3e8e14a0755aea2dec31ad: Status 404 returned error can't find the container with id 0e0de5a6647a69cefc75f1869c071ea51dbe68fb1e3e8e14a0755aea2dec31ad Sep 10 23:51:27.147849 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:27.147814 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 23:51:27.147961 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:27.147881 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 23:51:27.147961 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:27.147912 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 23:51:27.213702 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:27.213673 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-a4b2fb82-openshift-default-7d6485799b-6mpln" event={"ID":"f754a97f-d4e6-4276-a1e4-71592585df75","Type":"ContainerStarted","Data":"0e0de5a6647a69cefc75f1869c071ea51dbe68fb1e3e8e14a0755aea2dec31ad"} Sep 10 23:51:28.221673 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:28.221631 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-a4b2fb82-openshift-default-7d6485799b-6mpln" event={"ID":"f754a97f-d4e6-4276-a1e4-71592585df75","Type":"ContainerStarted","Data":"037777d973ad49cd9348aa9e02b475ba980dcda8e5c9a88cd0be71ce3fa581e6"} Sep 10 23:51:28.242857 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:28.242809 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-a4b2fb82-openshift-default-7d6485799b-6mpln" podStartSLOduration=2.242791224 podStartE2EDuration="2.242791224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:51:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:51:28.240610772 +0000 UTC m=+879.628213562" watchObservedRunningTime="2026-09-10 23:51:28.242791224 +0000 UTC m=+879.630393984" Sep 10 23:51:28.867269 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:28.867213 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-a4b2fb82-openshift-default-7d6485799b-6mpln" Sep 10 23:51:28.872185 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:28.872159 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-a4b2fb82-openshift-default-7d6485799b-6mpln" Sep 10 23:51:29.227056 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:29.227024 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-a4b2fb82-openshift-default-7d6485799b-6mpln" Sep 10 23:51:29.228020 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:29.227995 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-a4b2fb82-openshift-default-7d6485799b-6mpln" Sep 10 23:51:30.117515 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:30.117482 2574 patch_prober.go:28] interesting pod/e2e-ait-5706c553-openshift-default-5f9795bb96-pwgxg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Sep 10 23:51:30.117691 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:30.117537 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-5706c553-openshift-default-5f9795bb96-pwgxg" podUID="d86ed847-7911-4c26-a728-4e87c31e756d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 10 23:51:31.112881 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:31.112854 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5706c553-openshift-default-5f9795bb96-pwgxg" Sep 10 23:51:31.217298 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:31.217264 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d86ed847-7911-4c26-a728-4e87c31e756d-istio-data\") pod \"d86ed847-7911-4c26-a728-4e87c31e756d\" (UID: \"d86ed847-7911-4c26-a728-4e87c31e756d\") " Sep 10 23:51:31.217512 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:31.217337 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d86ed847-7911-4c26-a728-4e87c31e756d-istio-token\") pod \"d86ed847-7911-4c26-a728-4e87c31e756d\" (UID: \"d86ed847-7911-4c26-a728-4e87c31e756d\") " Sep 10 23:51:31.217512 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:31.217359 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d86ed847-7911-4c26-a728-4e87c31e756d-istio-envoy\") pod \"d86ed847-7911-4c26-a728-4e87c31e756d\" (UID: \"d86ed847-7911-4c26-a728-4e87c31e756d\") " Sep 10 23:51:31.217512 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:31.217382 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d86ed847-7911-4c26-a728-4e87c31e756d-workload-socket\") pod \"d86ed847-7911-4c26-a728-4e87c31e756d\" (UID: \"d86ed847-7911-4c26-a728-4e87c31e756d\") " Sep 10 23:51:31.217512 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:31.217408 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d86ed847-7911-4c26-a728-4e87c31e756d-credential-socket\") pod \"d86ed847-7911-4c26-a728-4e87c31e756d\" (UID: \"d86ed847-7911-4c26-a728-4e87c31e756d\") " Sep 10 23:51:31.217512 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:31.217449 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h8g8j\" (UniqueName: \"kubernetes.io/projected/d86ed847-7911-4c26-a728-4e87c31e756d-kube-api-access-h8g8j\") pod \"d86ed847-7911-4c26-a728-4e87c31e756d\" (UID: \"d86ed847-7911-4c26-a728-4e87c31e756d\") " Sep 10 23:51:31.217839 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:31.217523 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d86ed847-7911-4c26-a728-4e87c31e756d-istio-podinfo\") pod \"d86ed847-7911-4c26-a728-4e87c31e756d\" (UID: \"d86ed847-7911-4c26-a728-4e87c31e756d\") " Sep 10 23:51:31.217839 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:31.217570 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d86ed847-7911-4c26-a728-4e87c31e756d-istio-data" (OuterVolumeSpecName: "istio-data") pod "d86ed847-7911-4c26-a728-4e87c31e756d" (UID: "d86ed847-7911-4c26-a728-4e87c31e756d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:51:31.217839 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:31.217584 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d86ed847-7911-4c26-a728-4e87c31e756d-istiod-ca-cert\") pod \"d86ed847-7911-4c26-a728-4e87c31e756d\" (UID: \"d86ed847-7911-4c26-a728-4e87c31e756d\") " Sep 10 23:51:31.217839 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:31.217680 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d86ed847-7911-4c26-a728-4e87c31e756d-workload-certs\") pod \"d86ed847-7911-4c26-a728-4e87c31e756d\" (UID: \"d86ed847-7911-4c26-a728-4e87c31e756d\") " Sep 10 23:51:31.217839 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:31.217753 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d86ed847-7911-4c26-a728-4e87c31e756d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d86ed847-7911-4c26-a728-4e87c31e756d" (UID: "d86ed847-7911-4c26-a728-4e87c31e756d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:51:31.218124 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:31.217877 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d86ed847-7911-4c26-a728-4e87c31e756d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d86ed847-7911-4c26-a728-4e87c31e756d" (UID: "d86ed847-7911-4c26-a728-4e87c31e756d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 23:51:31.218124 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:31.218051 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d86ed847-7911-4c26-a728-4e87c31e756d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d86ed847-7911-4c26-a728-4e87c31e756d" (UID: "d86ed847-7911-4c26-a728-4e87c31e756d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:51:31.218124 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:31.218078 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d86ed847-7911-4c26-a728-4e87c31e756d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d86ed847-7911-4c26-a728-4e87c31e756d" (UID: "d86ed847-7911-4c26-a728-4e87c31e756d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:51:31.218282 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:31.218249 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d86ed847-7911-4c26-a728-4e87c31e756d-istiod-ca-cert\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:51:31.218282 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:31.218269 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d86ed847-7911-4c26-a728-4e87c31e756d-workload-certs\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:51:31.218391 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:31.218282 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d86ed847-7911-4c26-a728-4e87c31e756d-istio-data\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:51:31.218391 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:31.218296 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d86ed847-7911-4c26-a728-4e87c31e756d-workload-socket\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:51:31.218391 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:31.218311 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d86ed847-7911-4c26-a728-4e87c31e756d-credential-socket\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:51:31.219895 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:31.219870 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d86ed847-7911-4c26-a728-4e87c31e756d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d86ed847-7911-4c26-a728-4e87c31e756d" (UID: "d86ed847-7911-4c26-a728-4e87c31e756d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:51:31.220189 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:31.220162 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d86ed847-7911-4c26-a728-4e87c31e756d-istio-token" (OuterVolumeSpecName: "istio-token") pod "d86ed847-7911-4c26-a728-4e87c31e756d" (UID: "d86ed847-7911-4c26-a728-4e87c31e756d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 23:51:31.220269 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:31.220189 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d86ed847-7911-4c26-a728-4e87c31e756d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d86ed847-7911-4c26-a728-4e87c31e756d" (UID: "d86ed847-7911-4c26-a728-4e87c31e756d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 23:51:31.220387 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:31.220371 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d86ed847-7911-4c26-a728-4e87c31e756d-kube-api-access-h8g8j" (OuterVolumeSpecName: "kube-api-access-h8g8j") pod "d86ed847-7911-4c26-a728-4e87c31e756d" (UID: "d86ed847-7911-4c26-a728-4e87c31e756d"). InnerVolumeSpecName "kube-api-access-h8g8j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 23:51:31.242373 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:31.242340 2574 generic.go:356] "Generic (PLEG): container finished" podID="d86ed847-7911-4c26-a728-4e87c31e756d" containerID="3b760a97d9a6db132e039cdced0c013d65d80f690802a7c3d4dbd8cbaa4246cf" exitCode=0 Sep 10 23:51:31.242561 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:31.242394 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5706c553-openshift-default-5f9795bb96-pwgxg" Sep 10 23:51:31.242561 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:31.242397 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5706c553-openshift-default-5f9795bb96-pwgxg" event={"ID":"d86ed847-7911-4c26-a728-4e87c31e756d","Type":"ContainerDied","Data":"3b760a97d9a6db132e039cdced0c013d65d80f690802a7c3d4dbd8cbaa4246cf"} Sep 10 23:51:31.242561 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:31.242458 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5706c553-openshift-default-5f9795bb96-pwgxg" event={"ID":"d86ed847-7911-4c26-a728-4e87c31e756d","Type":"ContainerDied","Data":"cd3b11db12fa9c3c47a363d4cf6277631504f47286ed007c4b881c51d397d985"} Sep 10 23:51:31.242561 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:31.242471 2574 scope.go:117] "RemoveContainer" containerID="3b760a97d9a6db132e039cdced0c013d65d80f690802a7c3d4dbd8cbaa4246cf" Sep 10 23:51:31.253401 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:31.253380 2574 scope.go:117] "RemoveContainer" containerID="1278a3e2424510f0c3d8b10f86072fdd2f0c4b6e59ece4e7c0b69f15ac85c960" Sep 10 23:51:31.265589 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:31.265562 2574 scope.go:117] "RemoveContainer" containerID="3b760a97d9a6db132e039cdced0c013d65d80f690802a7c3d4dbd8cbaa4246cf" Sep 10 23:51:31.265986 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:51:31.265963 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b760a97d9a6db132e039cdced0c013d65d80f690802a7c3d4dbd8cbaa4246cf\": container with ID starting with 3b760a97d9a6db132e039cdced0c013d65d80f690802a7c3d4dbd8cbaa4246cf not found: ID does not exist" containerID="3b760a97d9a6db132e039cdced0c013d65d80f690802a7c3d4dbd8cbaa4246cf" Sep 10 23:51:31.266105 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:31.265994 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b760a97d9a6db132e039cdced0c013d65d80f690802a7c3d4dbd8cbaa4246cf"} err="failed to get container status \"3b760a97d9a6db132e039cdced0c013d65d80f690802a7c3d4dbd8cbaa4246cf\": rpc error: code = NotFound desc = could not find container \"3b760a97d9a6db132e039cdced0c013d65d80f690802a7c3d4dbd8cbaa4246cf\": container with ID starting with 3b760a97d9a6db132e039cdced0c013d65d80f690802a7c3d4dbd8cbaa4246cf not found: ID does not exist" Sep 10 23:51:31.266105 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:31.266025 2574 scope.go:117] "RemoveContainer" containerID="1278a3e2424510f0c3d8b10f86072fdd2f0c4b6e59ece4e7c0b69f15ac85c960" Sep 10 23:51:31.266327 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:51:31.266307 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1278a3e2424510f0c3d8b10f86072fdd2f0c4b6e59ece4e7c0b69f15ac85c960\": container with ID starting with 1278a3e2424510f0c3d8b10f86072fdd2f0c4b6e59ece4e7c0b69f15ac85c960 not found: ID does not exist" containerID="1278a3e2424510f0c3d8b10f86072fdd2f0c4b6e59ece4e7c0b69f15ac85c960" Sep 10 23:51:31.266393 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:31.266333 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1278a3e2424510f0c3d8b10f86072fdd2f0c4b6e59ece4e7c0b69f15ac85c960"} err="failed to get container status \"1278a3e2424510f0c3d8b10f86072fdd2f0c4b6e59ece4e7c0b69f15ac85c960\": rpc error: code = NotFound desc = could not find container \"1278a3e2424510f0c3d8b10f86072fdd2f0c4b6e59ece4e7c0b69f15ac85c960\": container with ID starting with 1278a3e2424510f0c3d8b10f86072fdd2f0c4b6e59ece4e7c0b69f15ac85c960 not found: ID does not exist" Sep 10 23:51:31.269690 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:31.269635 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-5706c553-openshift-default-5f9795bb96-pwgxg"] Sep 10 23:51:31.273108 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:31.273081 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-5706c553-openshift-default-5f9795bb96-pwgxg"] Sep 10 23:51:31.319424 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:31.319384 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d86ed847-7911-4c26-a728-4e87c31e756d-istio-token\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:51:31.319424 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:31.319415 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d86ed847-7911-4c26-a728-4e87c31e756d-istio-envoy\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:51:31.319424 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:31.319424 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8g8j\" (UniqueName: \"kubernetes.io/projected/d86ed847-7911-4c26-a728-4e87c31e756d-kube-api-access-h8g8j\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:51:31.319692 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:31.319438 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d86ed847-7911-4c26-a728-4e87c31e756d-istio-podinfo\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:51:31.702551 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:31.702513 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-56r7l"] Sep 10 23:51:33.097919 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:33.097881 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d86ed847-7911-4c26-a728-4e87c31e756d" path="/var/lib/kubelet/pods/d86ed847-7911-4c26-a728-4e87c31e756d/volumes" Sep 10 23:51:45.590358 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:45.590247 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-56r7l"] Sep 10 23:51:49.370629 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:49.370593 2574 generic.go:356] "Generic (PLEG): container finished" podID="f754a97f-d4e6-4276-a1e4-71592585df75" containerID="037777d973ad49cd9348aa9e02b475ba980dcda8e5c9a88cd0be71ce3fa581e6" exitCode=0 Sep 10 23:51:49.371079 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:49.370671 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-a4b2fb82-openshift-default-7d6485799b-6mpln" event={"ID":"f754a97f-d4e6-4276-a1e4-71592585df75","Type":"ContainerDied","Data":"037777d973ad49cd9348aa9e02b475ba980dcda8e5c9a88cd0be71ce3fa581e6"} Sep 10 23:51:49.371145 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:49.371135 2574 scope.go:117] "RemoveContainer" containerID="037777d973ad49cd9348aa9e02b475ba980dcda8e5c9a88cd0be71ce3fa581e6" Sep 10 23:51:49.372295 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:49.372258 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 23:51:49.372407 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:49.372331 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 23:51:49.372407 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:49.372374 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 23:51:50.383665 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:50.383610 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-a4b2fb82-openshift-default-7d6485799b-6mpln" event={"ID":"f754a97f-d4e6-4276-a1e4-71592585df75","Type":"ContainerStarted","Data":"6c1e85a7641b2628efffbd4c136a0f4e6cceac12cc197e333e735b3ecfa23cc5"} Sep 10 23:51:50.867441 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:50.867400 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-a4b2fb82-openshift-default-7d6485799b-6mpln" Sep 10 23:51:50.867923 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:50.867782 2574 patch_prober.go:28] interesting pod/e2e-derive-a4b2fb82-openshift-default-7d6485799b-6mpln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Sep 10 23:51:50.868140 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:50.868112 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-a4b2fb82-openshift-default-7d6485799b-6mpln" podUID="f754a97f-d4e6-4276-a1e4-71592585df75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Sep 10 23:51:51.867899 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:51.867860 2574 patch_prober.go:28] interesting pod/e2e-derive-a4b2fb82-openshift-default-7d6485799b-6mpln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Sep 10 23:51:51.868497 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:51.868465 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-a4b2fb82-openshift-default-7d6485799b-6mpln" podUID="f754a97f-d4e6-4276-a1e4-71592585df75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Sep 10 23:51:52.867452 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:52.867405 2574 patch_prober.go:28] interesting pod/e2e-derive-a4b2fb82-openshift-default-7d6485799b-6mpln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Sep 10 23:51:52.867686 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:52.867481 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-a4b2fb82-openshift-default-7d6485799b-6mpln" podUID="f754a97f-d4e6-4276-a1e4-71592585df75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Sep 10 23:51:53.867428 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:53.867378 2574 patch_prober.go:28] interesting pod/e2e-derive-a4b2fb82-openshift-default-7d6485799b-6mpln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Sep 10 23:51:53.867956 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:53.867453 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-a4b2fb82-openshift-default-7d6485799b-6mpln" podUID="f754a97f-d4e6-4276-a1e4-71592585df75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Sep 10 23:51:54.867379 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:54.867339 2574 patch_prober.go:28] interesting pod/e2e-derive-a4b2fb82-openshift-default-7d6485799b-6mpln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Sep 10 23:51:54.867717 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:54.867682 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-a4b2fb82-openshift-default-7d6485799b-6mpln" podUID="f754a97f-d4e6-4276-a1e4-71592585df75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Sep 10 23:51:55.870757 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:55.870724 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-a4b2fb82-openshift-default-7d6485799b-6mpln" Sep 10 23:51:55.871170 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:55.871016 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-a4b2fb82-openshift-default-7d6485799b-6mpln" Sep 10 23:51:55.871766 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:51:55.871746 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-a4b2fb82-openshift-default-7d6485799b-6mpln" Sep 10 23:52:01.596801 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:01.596763 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-a4b2fb82-openshift-default-7d6485799b-6mpln"] Sep 10 23:52:01.597356 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:01.597031 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-a4b2fb82-openshift-default-7d6485799b-6mpln" podUID="f754a97f-d4e6-4276-a1e4-71592585df75" containerName="istio-proxy" containerID="cri-o://6c1e85a7641b2628efffbd4c136a0f4e6cceac12cc197e333e735b3ecfa23cc5" gracePeriod=30 Sep 10 23:52:06.287467 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:06.287434 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-56r7l"] Sep 10 23:52:06.858361 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:06.858333 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-a4b2fb82-openshift-default-7d6485799b-6mpln" Sep 10 23:52:06.895033 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:06.894933 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f754a97f-d4e6-4276-a1e4-71592585df75-istio-envoy\") pod \"f754a97f-d4e6-4276-a1e4-71592585df75\" (UID: \"f754a97f-d4e6-4276-a1e4-71592585df75\") " Sep 10 23:52:06.895321 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:06.895298 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f754a97f-d4e6-4276-a1e4-71592585df75-credential-socket\") pod \"f754a97f-d4e6-4276-a1e4-71592585df75\" (UID: \"f754a97f-d4e6-4276-a1e4-71592585df75\") " Sep 10 23:52:06.895482 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:06.895468 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f754a97f-d4e6-4276-a1e4-71592585df75-istio-data\") pod \"f754a97f-d4e6-4276-a1e4-71592585df75\" (UID: \"f754a97f-d4e6-4276-a1e4-71592585df75\") " Sep 10 23:52:06.895618 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:06.895605 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f754a97f-d4e6-4276-a1e4-71592585df75-workload-socket\") pod \"f754a97f-d4e6-4276-a1e4-71592585df75\" (UID: \"f754a97f-d4e6-4276-a1e4-71592585df75\") " Sep 10 23:52:06.895782 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:06.895767 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f754a97f-d4e6-4276-a1e4-71592585df75-istio-podinfo\") pod \"f754a97f-d4e6-4276-a1e4-71592585df75\" (UID: \"f754a97f-d4e6-4276-a1e4-71592585df75\") " Sep 10 23:52:06.895907 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:06.895895 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f754a97f-d4e6-4276-a1e4-71592585df75-istiod-ca-cert\") pod \"f754a97f-d4e6-4276-a1e4-71592585df75\" (UID: \"f754a97f-d4e6-4276-a1e4-71592585df75\") " Sep 10 23:52:06.896020 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:06.896007 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f754a97f-d4e6-4276-a1e4-71592585df75-workload-certs\") pod \"f754a97f-d4e6-4276-a1e4-71592585df75\" (UID: \"f754a97f-d4e6-4276-a1e4-71592585df75\") " Sep 10 23:52:06.896128 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:06.896116 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bxth8\" (UniqueName: \"kubernetes.io/projected/f754a97f-d4e6-4276-a1e4-71592585df75-kube-api-access-bxth8\") pod \"f754a97f-d4e6-4276-a1e4-71592585df75\" (UID: \"f754a97f-d4e6-4276-a1e4-71592585df75\") " Sep 10 23:52:06.896218 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:06.896205 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f754a97f-d4e6-4276-a1e4-71592585df75-istio-token\") pod \"f754a97f-d4e6-4276-a1e4-71592585df75\" (UID: \"f754a97f-d4e6-4276-a1e4-71592585df75\") " Sep 10 23:52:06.897118 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:06.897083 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f754a97f-d4e6-4276-a1e4-71592585df75-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f754a97f-d4e6-4276-a1e4-71592585df75" (UID: "f754a97f-d4e6-4276-a1e4-71592585df75"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:52:06.897342 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:06.897312 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f754a97f-d4e6-4276-a1e4-71592585df75-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f754a97f-d4e6-4276-a1e4-71592585df75" (UID: "f754a97f-d4e6-4276-a1e4-71592585df75"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:52:06.897492 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:06.897472 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f754a97f-d4e6-4276-a1e4-71592585df75-istio-data" (OuterVolumeSpecName: "istio-data") pod "f754a97f-d4e6-4276-a1e4-71592585df75" (UID: "f754a97f-d4e6-4276-a1e4-71592585df75"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:52:06.897759 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:06.897733 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f754a97f-d4e6-4276-a1e4-71592585df75-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f754a97f-d4e6-4276-a1e4-71592585df75" (UID: "f754a97f-d4e6-4276-a1e4-71592585df75"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 23:52:06.898399 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:06.898351 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f754a97f-d4e6-4276-a1e4-71592585df75-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f754a97f-d4e6-4276-a1e4-71592585df75" (UID: "f754a97f-d4e6-4276-a1e4-71592585df75"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:52:06.899447 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:06.899418 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f754a97f-d4e6-4276-a1e4-71592585df75-kube-api-access-bxth8" (OuterVolumeSpecName: "kube-api-access-bxth8") pod "f754a97f-d4e6-4276-a1e4-71592585df75" (UID: "f754a97f-d4e6-4276-a1e4-71592585df75"). InnerVolumeSpecName "kube-api-access-bxth8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 23:52:06.899744 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:06.899719 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f754a97f-d4e6-4276-a1e4-71592585df75-istio-token" (OuterVolumeSpecName: "istio-token") pod "f754a97f-d4e6-4276-a1e4-71592585df75" (UID: "f754a97f-d4e6-4276-a1e4-71592585df75"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 23:52:06.899848 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:06.899741 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f754a97f-d4e6-4276-a1e4-71592585df75-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f754a97f-d4e6-4276-a1e4-71592585df75" (UID: "f754a97f-d4e6-4276-a1e4-71592585df75"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:52:06.899997 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:06.899978 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f754a97f-d4e6-4276-a1e4-71592585df75-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f754a97f-d4e6-4276-a1e4-71592585df75" (UID: "f754a97f-d4e6-4276-a1e4-71592585df75"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 23:52:06.998420 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:06.998369 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f754a97f-d4e6-4276-a1e4-71592585df75-credential-socket\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:52:06.998420 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:06.998414 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f754a97f-d4e6-4276-a1e4-71592585df75-istio-data\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:52:06.998420 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:06.998427 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f754a97f-d4e6-4276-a1e4-71592585df75-workload-socket\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:52:06.998694 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:06.998439 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f754a97f-d4e6-4276-a1e4-71592585df75-istio-podinfo\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:52:06.998694 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:06.998451 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f754a97f-d4e6-4276-a1e4-71592585df75-istiod-ca-cert\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:52:06.998694 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:06.998463 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f754a97f-d4e6-4276-a1e4-71592585df75-workload-certs\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:52:06.998694 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:06.998500 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxth8\" (UniqueName: \"kubernetes.io/projected/f754a97f-d4e6-4276-a1e4-71592585df75-kube-api-access-bxth8\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:52:06.998694 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:06.998512 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f754a97f-d4e6-4276-a1e4-71592585df75-istio-token\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:52:06.998694 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:06.998525 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f754a97f-d4e6-4276-a1e4-71592585df75-istio-envoy\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:52:07.500103 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:07.500072 2574 generic.go:356] "Generic (PLEG): container finished" podID="f754a97f-d4e6-4276-a1e4-71592585df75" containerID="6c1e85a7641b2628efffbd4c136a0f4e6cceac12cc197e333e735b3ecfa23cc5" exitCode=0 Sep 10 23:52:07.500524 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:07.500139 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-a4b2fb82-openshift-default-7d6485799b-6mpln" Sep 10 23:52:07.500524 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:07.500163 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-a4b2fb82-openshift-default-7d6485799b-6mpln" event={"ID":"f754a97f-d4e6-4276-a1e4-71592585df75","Type":"ContainerDied","Data":"6c1e85a7641b2628efffbd4c136a0f4e6cceac12cc197e333e735b3ecfa23cc5"} Sep 10 23:52:07.500524 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:07.500234 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-a4b2fb82-openshift-default-7d6485799b-6mpln" event={"ID":"f754a97f-d4e6-4276-a1e4-71592585df75","Type":"ContainerDied","Data":"0e0de5a6647a69cefc75f1869c071ea51dbe68fb1e3e8e14a0755aea2dec31ad"} Sep 10 23:52:07.500524 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:07.500267 2574 scope.go:117] "RemoveContainer" containerID="6c1e85a7641b2628efffbd4c136a0f4e6cceac12cc197e333e735b3ecfa23cc5" Sep 10 23:52:07.510079 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:07.509845 2574 scope.go:117] "RemoveContainer" containerID="037777d973ad49cd9348aa9e02b475ba980dcda8e5c9a88cd0be71ce3fa581e6" Sep 10 23:52:07.519381 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:07.519356 2574 scope.go:117] "RemoveContainer" containerID="6c1e85a7641b2628efffbd4c136a0f4e6cceac12cc197e333e735b3ecfa23cc5" Sep 10 23:52:07.519462 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:07.519422 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-a4b2fb82-openshift-default-7d6485799b-6mpln"] Sep 10 23:52:07.519704 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:52:07.519684 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c1e85a7641b2628efffbd4c136a0f4e6cceac12cc197e333e735b3ecfa23cc5\": container with ID starting with 6c1e85a7641b2628efffbd4c136a0f4e6cceac12cc197e333e735b3ecfa23cc5 not found: ID does not exist" containerID="6c1e85a7641b2628efffbd4c136a0f4e6cceac12cc197e333e735b3ecfa23cc5" Sep 10 23:52:07.519794 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:07.519716 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c1e85a7641b2628efffbd4c136a0f4e6cceac12cc197e333e735b3ecfa23cc5"} err="failed to get container status \"6c1e85a7641b2628efffbd4c136a0f4e6cceac12cc197e333e735b3ecfa23cc5\": rpc error: code = NotFound desc = could not find container \"6c1e85a7641b2628efffbd4c136a0f4e6cceac12cc197e333e735b3ecfa23cc5\": container with ID starting with 6c1e85a7641b2628efffbd4c136a0f4e6cceac12cc197e333e735b3ecfa23cc5 not found: ID does not exist" Sep 10 23:52:07.519794 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:07.519741 2574 scope.go:117] "RemoveContainer" containerID="037777d973ad49cd9348aa9e02b475ba980dcda8e5c9a88cd0be71ce3fa581e6" Sep 10 23:52:07.520063 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:52:07.520044 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"037777d973ad49cd9348aa9e02b475ba980dcda8e5c9a88cd0be71ce3fa581e6\": container with ID starting with 037777d973ad49cd9348aa9e02b475ba980dcda8e5c9a88cd0be71ce3fa581e6 not found: ID does not exist" containerID="037777d973ad49cd9348aa9e02b475ba980dcda8e5c9a88cd0be71ce3fa581e6" Sep 10 23:52:07.520108 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:07.520070 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"037777d973ad49cd9348aa9e02b475ba980dcda8e5c9a88cd0be71ce3fa581e6"} err="failed to get container status \"037777d973ad49cd9348aa9e02b475ba980dcda8e5c9a88cd0be71ce3fa581e6\": rpc error: code = NotFound desc = could not find container \"037777d973ad49cd9348aa9e02b475ba980dcda8e5c9a88cd0be71ce3fa581e6\": container with ID starting with 037777d973ad49cd9348aa9e02b475ba980dcda8e5c9a88cd0be71ce3fa581e6 not found: ID does not exist" Sep 10 23:52:07.523530 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:07.523507 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-a4b2fb82-openshift-default-7d6485799b-6mpln"] Sep 10 23:52:09.094427 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:09.094395 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f754a97f-d4e6-4276-a1e4-71592585df75" path="/var/lib/kubelet/pods/f754a97f-d4e6-4276-a1e4-71592585df75/volumes" Sep 10 23:52:21.787588 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:21.787551 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-56r7l"] Sep 10 23:52:37.181118 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:37.181086 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-56r7l"] Sep 10 23:52:39.017658 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:39.017609 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-74f1c1-openshift-default-5545549c98-z628g"] Sep 10 23:52:39.018196 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:39.017952 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-74f1c1-openshift-default-5545549c98-z628g" podUID="5970ef76-234b-4f98-9f7b-22db89263e2a" containerName="istio-proxy" containerID="cri-o://3aa30afa9c76003c052aea2703b56618e1f7189d3d1f4b7969b9ea55f35011f0" gracePeriod=30 Sep 10 23:52:39.733973 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:39.733940 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-b88386-openshift-default-6dd749cdb4-cwf4b"] Sep 10 23:52:39.734667 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:39.734630 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d86ed847-7911-4c26-a728-4e87c31e756d" containerName="istio-proxy" Sep 10 23:52:39.734773 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:39.734670 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="d86ed847-7911-4c26-a728-4e87c31e756d" containerName="istio-proxy" Sep 10 23:52:39.734773 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:39.734683 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f754a97f-d4e6-4276-a1e4-71592585df75" containerName="istio-proxy" Sep 10 23:52:39.734773 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:39.734692 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="f754a97f-d4e6-4276-a1e4-71592585df75" containerName="istio-proxy" Sep 10 23:52:39.734773 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:39.734706 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f754a97f-d4e6-4276-a1e4-71592585df75" containerName="istio-proxy" Sep 10 23:52:39.734773 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:39.734713 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="f754a97f-d4e6-4276-a1e4-71592585df75" containerName="istio-proxy" Sep 10 23:52:39.734773 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:39.734742 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dfb8b1ec-79f0-4db8-871e-fbd82b1623e6" containerName="istio-proxy" Sep 10 23:52:39.734773 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:39.734751 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="dfb8b1ec-79f0-4db8-871e-fbd82b1623e6" containerName="istio-proxy" Sep 10 23:52:39.735169 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:39.734877 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="f754a97f-d4e6-4276-a1e4-71592585df75" containerName="istio-proxy" Sep 10 23:52:39.735169 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:39.734898 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="d86ed847-7911-4c26-a728-4e87c31e756d" containerName="istio-proxy" Sep 10 23:52:39.735169 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:39.734906 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="d86ed847-7911-4c26-a728-4e87c31e756d" containerName="istio-proxy" Sep 10 23:52:39.735169 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:39.734914 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="dfb8b1ec-79f0-4db8-871e-fbd82b1623e6" containerName="istio-proxy" Sep 10 23:52:39.738624 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:39.738600 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-b88386-openshift-default-6dd749cdb4-cwf4b" Sep 10 23:52:39.741482 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:39.741459 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-b88386-openshift-default-dockercfg-9mpcv\"" Sep 10 23:52:39.755694 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:39.755668 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-b88386-openshift-default-6dd749cdb4-cwf4b"] Sep 10 23:52:39.823793 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:39.823213 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpp22\" (UniqueName: \"kubernetes.io/projected/c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8-kube-api-access-bpp22\") pod \"e2e-explicit-b88386-openshift-default-6dd749cdb4-cwf4b\" (UID: \"c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8\") " pod="openshift-ingress/e2e-explicit-b88386-openshift-default-6dd749cdb4-cwf4b" Sep 10 23:52:39.823793 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:39.823314 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8-istio-token\") pod \"e2e-explicit-b88386-openshift-default-6dd749cdb4-cwf4b\" (UID: \"c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8\") " pod="openshift-ingress/e2e-explicit-b88386-openshift-default-6dd749cdb4-cwf4b" Sep 10 23:52:39.823793 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:39.823345 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8-istio-podinfo\") pod \"e2e-explicit-b88386-openshift-default-6dd749cdb4-cwf4b\" (UID: \"c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8\") " pod="openshift-ingress/e2e-explicit-b88386-openshift-default-6dd749cdb4-cwf4b" Sep 10 23:52:39.823793 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:39.823373 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8-istiod-ca-cert\") pod \"e2e-explicit-b88386-openshift-default-6dd749cdb4-cwf4b\" (UID: \"c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8\") " pod="openshift-ingress/e2e-explicit-b88386-openshift-default-6dd749cdb4-cwf4b" Sep 10 23:52:39.823793 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:39.823411 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8-credential-socket\") pod \"e2e-explicit-b88386-openshift-default-6dd749cdb4-cwf4b\" (UID: \"c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8\") " pod="openshift-ingress/e2e-explicit-b88386-openshift-default-6dd749cdb4-cwf4b" Sep 10 23:52:39.823793 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:39.823443 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8-istio-envoy\") pod \"e2e-explicit-b88386-openshift-default-6dd749cdb4-cwf4b\" (UID: \"c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8\") " pod="openshift-ingress/e2e-explicit-b88386-openshift-default-6dd749cdb4-cwf4b" Sep 10 23:52:39.823793 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:39.823488 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8-workload-certs\") pod \"e2e-explicit-b88386-openshift-default-6dd749cdb4-cwf4b\" (UID: \"c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8\") " pod="openshift-ingress/e2e-explicit-b88386-openshift-default-6dd749cdb4-cwf4b" Sep 10 23:52:39.823793 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:39.823603 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8-workload-socket\") pod \"e2e-explicit-b88386-openshift-default-6dd749cdb4-cwf4b\" (UID: \"c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8\") " pod="openshift-ingress/e2e-explicit-b88386-openshift-default-6dd749cdb4-cwf4b" Sep 10 23:52:39.823793 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:39.823628 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8-istio-data\") pod \"e2e-explicit-b88386-openshift-default-6dd749cdb4-cwf4b\" (UID: \"c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8\") " pod="openshift-ingress/e2e-explicit-b88386-openshift-default-6dd749cdb4-cwf4b" Sep 10 23:52:39.924316 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:39.924279 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8-workload-socket\") pod \"e2e-explicit-b88386-openshift-default-6dd749cdb4-cwf4b\" (UID: \"c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8\") " pod="openshift-ingress/e2e-explicit-b88386-openshift-default-6dd749cdb4-cwf4b" Sep 10 23:52:39.924316 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:39.924317 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8-istio-data\") pod \"e2e-explicit-b88386-openshift-default-6dd749cdb4-cwf4b\" (UID: \"c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8\") " pod="openshift-ingress/e2e-explicit-b88386-openshift-default-6dd749cdb4-cwf4b" Sep 10 23:52:39.924532 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:39.924453 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpp22\" (UniqueName: \"kubernetes.io/projected/c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8-kube-api-access-bpp22\") pod \"e2e-explicit-b88386-openshift-default-6dd749cdb4-cwf4b\" (UID: \"c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8\") " pod="openshift-ingress/e2e-explicit-b88386-openshift-default-6dd749cdb4-cwf4b" Sep 10 23:52:39.924580 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:39.924561 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8-istio-token\") pod \"e2e-explicit-b88386-openshift-default-6dd749cdb4-cwf4b\" (UID: \"c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8\") " pod="openshift-ingress/e2e-explicit-b88386-openshift-default-6dd749cdb4-cwf4b" Sep 10 23:52:39.924617 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:39.924600 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8-istio-podinfo\") pod \"e2e-explicit-b88386-openshift-default-6dd749cdb4-cwf4b\" (UID: \"c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8\") " pod="openshift-ingress/e2e-explicit-b88386-openshift-default-6dd749cdb4-cwf4b" Sep 10 23:52:39.924682 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:39.924629 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8-istiod-ca-cert\") pod \"e2e-explicit-b88386-openshift-default-6dd749cdb4-cwf4b\" (UID: \"c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8\") " pod="openshift-ingress/e2e-explicit-b88386-openshift-default-6dd749cdb4-cwf4b" Sep 10 23:52:39.924734 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:39.924700 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8-credential-socket\") pod \"e2e-explicit-b88386-openshift-default-6dd749cdb4-cwf4b\" (UID: \"c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8\") " pod="openshift-ingress/e2e-explicit-b88386-openshift-default-6dd749cdb4-cwf4b" Sep 10 23:52:39.924786 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:39.924726 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8-istio-data\") pod \"e2e-explicit-b88386-openshift-default-6dd749cdb4-cwf4b\" (UID: \"c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8\") " pod="openshift-ingress/e2e-explicit-b88386-openshift-default-6dd749cdb4-cwf4b" Sep 10 23:52:39.924786 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:39.924731 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8-istio-envoy\") pod \"e2e-explicit-b88386-openshift-default-6dd749cdb4-cwf4b\" (UID: \"c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8\") " pod="openshift-ingress/e2e-explicit-b88386-openshift-default-6dd749cdb4-cwf4b" Sep 10 23:52:39.924886 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:39.924781 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8-workload-socket\") pod \"e2e-explicit-b88386-openshift-default-6dd749cdb4-cwf4b\" (UID: \"c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8\") " pod="openshift-ingress/e2e-explicit-b88386-openshift-default-6dd749cdb4-cwf4b" Sep 10 23:52:39.924886 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:39.924803 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8-workload-certs\") pod \"e2e-explicit-b88386-openshift-default-6dd749cdb4-cwf4b\" (UID: \"c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8\") " pod="openshift-ingress/e2e-explicit-b88386-openshift-default-6dd749cdb4-cwf4b" Sep 10 23:52:39.925073 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:39.925055 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8-credential-socket\") pod \"e2e-explicit-b88386-openshift-default-6dd749cdb4-cwf4b\" (UID: \"c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8\") " pod="openshift-ingress/e2e-explicit-b88386-openshift-default-6dd749cdb4-cwf4b" Sep 10 23:52:39.925160 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:39.925138 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8-workload-certs\") pod \"e2e-explicit-b88386-openshift-default-6dd749cdb4-cwf4b\" (UID: \"c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8\") " pod="openshift-ingress/e2e-explicit-b88386-openshift-default-6dd749cdb4-cwf4b" Sep 10 23:52:39.925424 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:39.925405 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8-istiod-ca-cert\") pod \"e2e-explicit-b88386-openshift-default-6dd749cdb4-cwf4b\" (UID: \"c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8\") " pod="openshift-ingress/e2e-explicit-b88386-openshift-default-6dd749cdb4-cwf4b" Sep 10 23:52:39.926939 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:39.926918 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8-istio-envoy\") pod \"e2e-explicit-b88386-openshift-default-6dd749cdb4-cwf4b\" (UID: \"c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8\") " pod="openshift-ingress/e2e-explicit-b88386-openshift-default-6dd749cdb4-cwf4b" Sep 10 23:52:39.927223 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:39.927205 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8-istio-podinfo\") pod \"e2e-explicit-b88386-openshift-default-6dd749cdb4-cwf4b\" (UID: \"c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8\") " pod="openshift-ingress/e2e-explicit-b88386-openshift-default-6dd749cdb4-cwf4b" Sep 10 23:52:39.946248 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:39.946222 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8-istio-token\") pod \"e2e-explicit-b88386-openshift-default-6dd749cdb4-cwf4b\" (UID: \"c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8\") " pod="openshift-ingress/e2e-explicit-b88386-openshift-default-6dd749cdb4-cwf4b" Sep 10 23:52:39.946356 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:39.946262 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpp22\" (UniqueName: \"kubernetes.io/projected/c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8-kube-api-access-bpp22\") pod \"e2e-explicit-b88386-openshift-default-6dd749cdb4-cwf4b\" (UID: \"c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8\") " pod="openshift-ingress/e2e-explicit-b88386-openshift-default-6dd749cdb4-cwf4b" Sep 10 23:52:40.049810 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:40.049726 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-b88386-openshift-default-6dd749cdb4-cwf4b" Sep 10 23:52:40.329854 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:40.329823 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-b88386-openshift-default-6dd749cdb4-cwf4b"] Sep 10 23:52:40.330764 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:52:40.330741 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3f247d4_5f36_4d93_a3c8_7bb54e6ba2f8.slice/crio-816e84122f5c2e6469e4d6f68f788016ba6f6523e0c7213c8668df7b43dd6387 WatchSource:0}: Error finding container 816e84122f5c2e6469e4d6f68f788016ba6f6523e0c7213c8668df7b43dd6387: Status 404 returned error can't find the container with id 816e84122f5c2e6469e4d6f68f788016ba6f6523e0c7213c8668df7b43dd6387 Sep 10 23:52:40.332950 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:40.332915 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 23:52:40.333048 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:40.332983 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 23:52:40.333048 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:40.333010 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 23:52:40.703379 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:40.703335 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-b88386-openshift-default-6dd749cdb4-cwf4b" event={"ID":"c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8","Type":"ContainerStarted","Data":"fd8e8469b9f8ff48441987332ba4c97f2b3a025c3d45d9ae0b89a41e1a5de1c8"} Sep 10 23:52:40.703592 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:40.703406 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-b88386-openshift-default-6dd749cdb4-cwf4b" event={"ID":"c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8","Type":"ContainerStarted","Data":"816e84122f5c2e6469e4d6f68f788016ba6f6523e0c7213c8668df7b43dd6387"} Sep 10 23:52:40.729572 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:40.729520 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-b88386-openshift-default-6dd749cdb4-cwf4b" podStartSLOduration=1.72950482 podStartE2EDuration="1.72950482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:52:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:52:40.726174022 +0000 UTC m=+952.113776780" watchObservedRunningTime="2026-09-10 23:52:40.72950482 +0000 UTC m=+952.117107579" Sep 10 23:52:40.843481 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:40.843434 2574 patch_prober.go:28] interesting pod/e2e-autores-74f1c1-openshift-default-5545549c98-z628g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 10 23:52:40.843687 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:40.843520 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-74f1c1-openshift-default-5545549c98-z628g" podUID="5970ef76-234b-4f98-9f7b-22db89263e2a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 10 23:52:41.050070 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:41.049980 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-b88386-openshift-default-6dd749cdb4-cwf4b" Sep 10 23:52:41.054634 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:41.054609 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-b88386-openshift-default-6dd749cdb4-cwf4b" Sep 10 23:52:41.709233 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:41.709202 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-b88386-openshift-default-6dd749cdb4-cwf4b" Sep 10 23:52:41.710104 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:41.710085 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-b88386-openshift-default-6dd749cdb4-cwf4b" Sep 10 23:52:44.275973 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:44.275946 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-74f1c1-openshift-default-5545549c98-z628g" Sep 10 23:52:44.370689 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:44.370597 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5970ef76-234b-4f98-9f7b-22db89263e2a-workload-socket\") pod \"5970ef76-234b-4f98-9f7b-22db89263e2a\" (UID: \"5970ef76-234b-4f98-9f7b-22db89263e2a\") " Sep 10 23:52:44.370689 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:44.370634 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5970ef76-234b-4f98-9f7b-22db89263e2a-istio-token\") pod \"5970ef76-234b-4f98-9f7b-22db89263e2a\" (UID: \"5970ef76-234b-4f98-9f7b-22db89263e2a\") " Sep 10 23:52:44.370911 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:44.370703 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4jw9d\" (UniqueName: \"kubernetes.io/projected/5970ef76-234b-4f98-9f7b-22db89263e2a-kube-api-access-4jw9d\") pod \"5970ef76-234b-4f98-9f7b-22db89263e2a\" (UID: \"5970ef76-234b-4f98-9f7b-22db89263e2a\") " Sep 10 23:52:44.370911 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:44.370737 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5970ef76-234b-4f98-9f7b-22db89263e2a-istio-data\") pod \"5970ef76-234b-4f98-9f7b-22db89263e2a\" (UID: \"5970ef76-234b-4f98-9f7b-22db89263e2a\") " Sep 10 23:52:44.370911 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:44.370795 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5970ef76-234b-4f98-9f7b-22db89263e2a-istiod-ca-cert\") pod \"5970ef76-234b-4f98-9f7b-22db89263e2a\" (UID: \"5970ef76-234b-4f98-9f7b-22db89263e2a\") " Sep 10 23:52:44.370911 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:44.370829 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5970ef76-234b-4f98-9f7b-22db89263e2a-workload-certs\") pod \"5970ef76-234b-4f98-9f7b-22db89263e2a\" (UID: \"5970ef76-234b-4f98-9f7b-22db89263e2a\") " Sep 10 23:52:44.370911 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:44.370896 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5970ef76-234b-4f98-9f7b-22db89263e2a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5970ef76-234b-4f98-9f7b-22db89263e2a" (UID: "5970ef76-234b-4f98-9f7b-22db89263e2a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:52:44.371159 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:44.370914 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5970ef76-234b-4f98-9f7b-22db89263e2a-istio-podinfo\") pod \"5970ef76-234b-4f98-9f7b-22db89263e2a\" (UID: \"5970ef76-234b-4f98-9f7b-22db89263e2a\") " Sep 10 23:52:44.371159 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:44.370994 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5970ef76-234b-4f98-9f7b-22db89263e2a-istio-envoy\") pod \"5970ef76-234b-4f98-9f7b-22db89263e2a\" (UID: \"5970ef76-234b-4f98-9f7b-22db89263e2a\") " Sep 10 23:52:44.371159 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:44.371039 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5970ef76-234b-4f98-9f7b-22db89263e2a-credential-socket\") pod \"5970ef76-234b-4f98-9f7b-22db89263e2a\" (UID: \"5970ef76-234b-4f98-9f7b-22db89263e2a\") " Sep 10 23:52:44.371307 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:44.371262 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5970ef76-234b-4f98-9f7b-22db89263e2a-istio-data" (OuterVolumeSpecName: "istio-data") pod "5970ef76-234b-4f98-9f7b-22db89263e2a" (UID: "5970ef76-234b-4f98-9f7b-22db89263e2a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:52:44.371588 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:44.371456 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5970ef76-234b-4f98-9f7b-22db89263e2a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5970ef76-234b-4f98-9f7b-22db89263e2a" (UID: "5970ef76-234b-4f98-9f7b-22db89263e2a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:52:44.371588 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:44.371496 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5970ef76-234b-4f98-9f7b-22db89263e2a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5970ef76-234b-4f98-9f7b-22db89263e2a" (UID: "5970ef76-234b-4f98-9f7b-22db89263e2a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 23:52:44.371588 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:44.371526 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5970ef76-234b-4f98-9f7b-22db89263e2a-workload-socket\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:52:44.371588 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:44.371549 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5970ef76-234b-4f98-9f7b-22db89263e2a-istio-data\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:52:44.371909 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:44.371698 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5970ef76-234b-4f98-9f7b-22db89263e2a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5970ef76-234b-4f98-9f7b-22db89263e2a" (UID: "5970ef76-234b-4f98-9f7b-22db89263e2a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:52:44.373382 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:44.373357 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5970ef76-234b-4f98-9f7b-22db89263e2a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5970ef76-234b-4f98-9f7b-22db89263e2a" (UID: "5970ef76-234b-4f98-9f7b-22db89263e2a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 23:52:44.374027 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:44.373991 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5970ef76-234b-4f98-9f7b-22db89263e2a-istio-token" (OuterVolumeSpecName: "istio-token") pod "5970ef76-234b-4f98-9f7b-22db89263e2a" (UID: "5970ef76-234b-4f98-9f7b-22db89263e2a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 23:52:44.374135 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:44.374026 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5970ef76-234b-4f98-9f7b-22db89263e2a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5970ef76-234b-4f98-9f7b-22db89263e2a" (UID: "5970ef76-234b-4f98-9f7b-22db89263e2a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:52:44.374135 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:44.374045 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5970ef76-234b-4f98-9f7b-22db89263e2a-kube-api-access-4jw9d" (OuterVolumeSpecName: "kube-api-access-4jw9d") pod "5970ef76-234b-4f98-9f7b-22db89263e2a" (UID: "5970ef76-234b-4f98-9f7b-22db89263e2a"). InnerVolumeSpecName "kube-api-access-4jw9d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 23:52:44.472253 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:44.472218 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5970ef76-234b-4f98-9f7b-22db89263e2a-istio-podinfo\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:52:44.472253 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:44.472247 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5970ef76-234b-4f98-9f7b-22db89263e2a-istio-envoy\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:52:44.472450 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:44.472262 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5970ef76-234b-4f98-9f7b-22db89263e2a-credential-socket\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:52:44.472450 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:44.472274 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5970ef76-234b-4f98-9f7b-22db89263e2a-istio-token\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:52:44.472450 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:44.472283 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jw9d\" (UniqueName: \"kubernetes.io/projected/5970ef76-234b-4f98-9f7b-22db89263e2a-kube-api-access-4jw9d\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:52:44.472450 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:44.472291 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5970ef76-234b-4f98-9f7b-22db89263e2a-istiod-ca-cert\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:52:44.472450 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:44.472299 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5970ef76-234b-4f98-9f7b-22db89263e2a-workload-certs\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:52:44.729541 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:44.729505 2574 generic.go:356] "Generic (PLEG): container finished" podID="5970ef76-234b-4f98-9f7b-22db89263e2a" containerID="3aa30afa9c76003c052aea2703b56618e1f7189d3d1f4b7969b9ea55f35011f0" exitCode=0 Sep 10 23:52:44.729765 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:44.729553 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-74f1c1-openshift-default-5545549c98-z628g" Sep 10 23:52:44.729765 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:44.729582 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-74f1c1-openshift-default-5545549c98-z628g" event={"ID":"5970ef76-234b-4f98-9f7b-22db89263e2a","Type":"ContainerDied","Data":"3aa30afa9c76003c052aea2703b56618e1f7189d3d1f4b7969b9ea55f35011f0"} Sep 10 23:52:44.729765 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:44.729634 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-74f1c1-openshift-default-5545549c98-z628g" event={"ID":"5970ef76-234b-4f98-9f7b-22db89263e2a","Type":"ContainerDied","Data":"0aa2f2fa4ef7de17b0b5b2182502fa3fc5fbd7d7bc130851d71af09db8a5bbf3"} Sep 10 23:52:44.729765 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:44.729670 2574 scope.go:117] "RemoveContainer" containerID="3aa30afa9c76003c052aea2703b56618e1f7189d3d1f4b7969b9ea55f35011f0" Sep 10 23:52:44.740394 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:44.740369 2574 scope.go:117] "RemoveContainer" containerID="8f0a0304ff1487247e7b71b8006745327b1e9f45b9e3513bda575e0ab97c66db" Sep 10 23:52:44.748914 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:44.748898 2574 scope.go:117] "RemoveContainer" containerID="3aa30afa9c76003c052aea2703b56618e1f7189d3d1f4b7969b9ea55f35011f0" Sep 10 23:52:44.749196 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:52:44.749177 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3aa30afa9c76003c052aea2703b56618e1f7189d3d1f4b7969b9ea55f35011f0\": container with ID starting with 3aa30afa9c76003c052aea2703b56618e1f7189d3d1f4b7969b9ea55f35011f0 not found: ID does not exist" containerID="3aa30afa9c76003c052aea2703b56618e1f7189d3d1f4b7969b9ea55f35011f0" Sep 10 23:52:44.749249 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:44.749206 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3aa30afa9c76003c052aea2703b56618e1f7189d3d1f4b7969b9ea55f35011f0"} err="failed to get container status \"3aa30afa9c76003c052aea2703b56618e1f7189d3d1f4b7969b9ea55f35011f0\": rpc error: code = NotFound desc = could not find container \"3aa30afa9c76003c052aea2703b56618e1f7189d3d1f4b7969b9ea55f35011f0\": container with ID starting with 3aa30afa9c76003c052aea2703b56618e1f7189d3d1f4b7969b9ea55f35011f0 not found: ID does not exist" Sep 10 23:52:44.749249 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:44.749223 2574 scope.go:117] "RemoveContainer" containerID="8f0a0304ff1487247e7b71b8006745327b1e9f45b9e3513bda575e0ab97c66db" Sep 10 23:52:44.749449 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:52:44.749435 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f0a0304ff1487247e7b71b8006745327b1e9f45b9e3513bda575e0ab97c66db\": container with ID starting with 8f0a0304ff1487247e7b71b8006745327b1e9f45b9e3513bda575e0ab97c66db not found: ID does not exist" containerID="8f0a0304ff1487247e7b71b8006745327b1e9f45b9e3513bda575e0ab97c66db" Sep 10 23:52:44.749525 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:44.749452 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f0a0304ff1487247e7b71b8006745327b1e9f45b9e3513bda575e0ab97c66db"} err="failed to get container status \"8f0a0304ff1487247e7b71b8006745327b1e9f45b9e3513bda575e0ab97c66db\": rpc error: code = NotFound desc = could not find container \"8f0a0304ff1487247e7b71b8006745327b1e9f45b9e3513bda575e0ab97c66db\": container with ID starting with 8f0a0304ff1487247e7b71b8006745327b1e9f45b9e3513bda575e0ab97c66db not found: ID does not exist" Sep 10 23:52:44.755580 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:44.755559 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-74f1c1-openshift-default-5545549c98-z628g"] Sep 10 23:52:44.760771 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:44.760752 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-74f1c1-openshift-default-5545549c98-z628g"] Sep 10 23:52:45.102870 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:45.102778 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5970ef76-234b-4f98-9f7b-22db89263e2a" path="/var/lib/kubelet/pods/5970ef76-234b-4f98-9f7b-22db89263e2a/volumes" Sep 10 23:52:50.893871 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:52:50.893834 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-56r7l"] Sep 10 23:53:01.842379 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:53:01.842342 2574 generic.go:356] "Generic (PLEG): container finished" podID="c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8" containerID="fd8e8469b9f8ff48441987332ba4c97f2b3a025c3d45d9ae0b89a41e1a5de1c8" exitCode=0 Sep 10 23:53:01.842816 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:53:01.842416 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-b88386-openshift-default-6dd749cdb4-cwf4b" event={"ID":"c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8","Type":"ContainerDied","Data":"fd8e8469b9f8ff48441987332ba4c97f2b3a025c3d45d9ae0b89a41e1a5de1c8"} Sep 10 23:53:01.842865 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:53:01.842832 2574 scope.go:117] "RemoveContainer" containerID="fd8e8469b9f8ff48441987332ba4c97f2b3a025c3d45d9ae0b89a41e1a5de1c8" Sep 10 23:53:01.843967 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:53:01.843939 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 23:53:01.844034 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:53:01.844011 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 23:53:01.844071 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:53:01.844043 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 23:53:02.850035 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:53:02.849988 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-b88386-openshift-default-6dd749cdb4-cwf4b" event={"ID":"c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8","Type":"ContainerStarted","Data":"e5f651854b909a2204e11f1dfd8e5f296bcb1963bac853377623e1faabcc4fb6"} Sep 10 23:53:03.050722 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:53:03.050687 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-b88386-openshift-default-6dd749cdb4-cwf4b" Sep 10 23:53:03.050996 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:53:03.050978 2574 patch_prober.go:28] interesting pod/e2e-explicit-b88386-openshift-default-6dd749cdb4-cwf4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Sep 10 23:53:03.051048 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:53:03.051022 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-b88386-openshift-default-6dd749cdb4-cwf4b" podUID="c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Sep 10 23:53:04.051014 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:53:04.050977 2574 patch_prober.go:28] interesting pod/e2e-explicit-b88386-openshift-default-6dd749cdb4-cwf4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Sep 10 23:53:04.051398 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:53:04.051034 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-b88386-openshift-default-6dd749cdb4-cwf4b" podUID="c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Sep 10 23:53:05.050840 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:53:05.050806 2574 patch_prober.go:28] interesting pod/e2e-explicit-b88386-openshift-default-6dd749cdb4-cwf4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Sep 10 23:53:05.051019 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:53:05.050860 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-b88386-openshift-default-6dd749cdb4-cwf4b" podUID="c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Sep 10 23:53:06.051063 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:53:06.051025 2574 patch_prober.go:28] interesting pod/e2e-explicit-b88386-openshift-default-6dd749cdb4-cwf4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Sep 10 23:53:06.051455 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:53:06.051087 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-b88386-openshift-default-6dd749cdb4-cwf4b" podUID="c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Sep 10 23:53:06.394997 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:53:06.394909 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-56r7l"] Sep 10 23:53:07.050799 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:53:07.050762 2574 patch_prober.go:28] interesting pod/e2e-explicit-b88386-openshift-default-6dd749cdb4-cwf4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Sep 10 23:53:07.050977 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:53:07.050814 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-b88386-openshift-default-6dd749cdb4-cwf4b" podUID="c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Sep 10 23:53:08.054899 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:53:08.054871 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-b88386-openshift-default-6dd749cdb4-cwf4b" Sep 10 23:53:08.055297 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:53:08.055272 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-b88386-openshift-default-6dd749cdb4-cwf4b" Sep 10 23:53:08.055877 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:53:08.055856 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-b88386-openshift-default-6dd749cdb4-cwf4b" Sep 10 23:53:19.976232 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:53:19.976154 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-56r7l"] Sep 10 23:53:34.502813 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:53:34.502779 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-56r7l"] Sep 10 23:53:37.807090 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:53:37.807059 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-24749012-openshift-default-548b8dc884-twrkz"] Sep 10 23:53:37.807511 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:53:37.807498 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d86ed847-7911-4c26-a728-4e87c31e756d" containerName="istio-proxy" Sep 10 23:53:37.807556 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:53:37.807513 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="d86ed847-7911-4c26-a728-4e87c31e756d" containerName="istio-proxy" Sep 10 23:53:37.807556 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:53:37.807524 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5970ef76-234b-4f98-9f7b-22db89263e2a" containerName="istio-proxy" Sep 10 23:53:37.807556 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:53:37.807529 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="5970ef76-234b-4f98-9f7b-22db89263e2a" containerName="istio-proxy" Sep 10 23:53:37.807556 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:53:37.807548 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5970ef76-234b-4f98-9f7b-22db89263e2a" containerName="istio-proxy" Sep 10 23:53:37.807705 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:53:37.807557 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="5970ef76-234b-4f98-9f7b-22db89263e2a" containerName="istio-proxy" Sep 10 23:53:37.807705 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:53:37.807622 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="5970ef76-234b-4f98-9f7b-22db89263e2a" containerName="istio-proxy" Sep 10 23:53:37.807705 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:53:37.807630 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="5970ef76-234b-4f98-9f7b-22db89263e2a" containerName="istio-proxy" Sep 10 23:53:37.807705 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:53:37.807652 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="f754a97f-d4e6-4276-a1e4-71592585df75" containerName="istio-proxy" Sep 10 23:53:37.811164 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:53:37.811141 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-24749012-openshift-default-548b8dc884-twrkz" Sep 10 23:53:37.814966 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:53:37.814947 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-24749012-openshift-default-dockercfg-9bs7f\"" Sep 10 23:53:37.823706 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:53:37.823678 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-24749012-openshift-default-548b8dc884-twrkz"] Sep 10 23:53:37.931352 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:53:37.931309 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00cee90e-ff1f-4f6f-8196-accf688e9135-workload-socket\") pod \"e2e-mt-24749012-openshift-default-548b8dc884-twrkz\" (UID: \"00cee90e-ff1f-4f6f-8196-accf688e9135\") " pod="openshift-ingress/e2e-mt-24749012-openshift-default-548b8dc884-twrkz" Sep 10 23:53:37.931694 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:53:37.931669 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00cee90e-ff1f-4f6f-8196-accf688e9135-istio-podinfo\") pod \"e2e-mt-24749012-openshift-default-548b8dc884-twrkz\" (UID: \"00cee90e-ff1f-4f6f-8196-accf688e9135\") " pod="openshift-ingress/e2e-mt-24749012-openshift-default-548b8dc884-twrkz" Sep 10 23:53:37.931842 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:53:37.931819 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00cee90e-ff1f-4f6f-8196-accf688e9135-istiod-ca-cert\") pod \"e2e-mt-24749012-openshift-default-548b8dc884-twrkz\" (UID: \"00cee90e-ff1f-4f6f-8196-accf688e9135\") " pod="openshift-ingress/e2e-mt-24749012-openshift-default-548b8dc884-twrkz" Sep 10 23:53:37.931978 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:53:37.931857 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00cee90e-ff1f-4f6f-8196-accf688e9135-istio-envoy\") pod \"e2e-mt-24749012-openshift-default-548b8dc884-twrkz\" (UID: \"00cee90e-ff1f-4f6f-8196-accf688e9135\") " pod="openshift-ingress/e2e-mt-24749012-openshift-default-548b8dc884-twrkz" Sep 10 23:53:37.931978 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:53:37.931938 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00cee90e-ff1f-4f6f-8196-accf688e9135-credential-socket\") pod \"e2e-mt-24749012-openshift-default-548b8dc884-twrkz\" (UID: \"00cee90e-ff1f-4f6f-8196-accf688e9135\") " pod="openshift-ingress/e2e-mt-24749012-openshift-default-548b8dc884-twrkz" Sep 10 23:53:37.932086 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:53:37.931998 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00cee90e-ff1f-4f6f-8196-accf688e9135-istio-token\") pod \"e2e-mt-24749012-openshift-default-548b8dc884-twrkz\" (UID: \"00cee90e-ff1f-4f6f-8196-accf688e9135\") " pod="openshift-ingress/e2e-mt-24749012-openshift-default-548b8dc884-twrkz" Sep 10 23:53:37.932086 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:53:37.932053 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00cee90e-ff1f-4f6f-8196-accf688e9135-istio-data\") pod \"e2e-mt-24749012-openshift-default-548b8dc884-twrkz\" (UID: \"00cee90e-ff1f-4f6f-8196-accf688e9135\") " pod="openshift-ingress/e2e-mt-24749012-openshift-default-548b8dc884-twrkz" Sep 10 23:53:37.932187 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:53:37.932088 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkxbl\" (UniqueName: \"kubernetes.io/projected/00cee90e-ff1f-4f6f-8196-accf688e9135-kube-api-access-tkxbl\") pod \"e2e-mt-24749012-openshift-default-548b8dc884-twrkz\" (UID: \"00cee90e-ff1f-4f6f-8196-accf688e9135\") " pod="openshift-ingress/e2e-mt-24749012-openshift-default-548b8dc884-twrkz" Sep 10 23:53:37.932187 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:53:37.932130 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00cee90e-ff1f-4f6f-8196-accf688e9135-workload-certs\") pod \"e2e-mt-24749012-openshift-default-548b8dc884-twrkz\" (UID: \"00cee90e-ff1f-4f6f-8196-accf688e9135\") " pod="openshift-ingress/e2e-mt-24749012-openshift-default-548b8dc884-twrkz" Sep 10 23:53:38.032822 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:53:38.032791 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00cee90e-ff1f-4f6f-8196-accf688e9135-istio-envoy\") pod \"e2e-mt-24749012-openshift-default-548b8dc884-twrkz\" (UID: \"00cee90e-ff1f-4f6f-8196-accf688e9135\") " pod="openshift-ingress/e2e-mt-24749012-openshift-default-548b8dc884-twrkz" Sep 10 23:53:38.032992 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:53:38.032848 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00cee90e-ff1f-4f6f-8196-accf688e9135-credential-socket\") pod \"e2e-mt-24749012-openshift-default-548b8dc884-twrkz\" (UID: \"00cee90e-ff1f-4f6f-8196-accf688e9135\") " pod="openshift-ingress/e2e-mt-24749012-openshift-default-548b8dc884-twrkz" Sep 10 23:53:38.032992 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:53:38.032884 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00cee90e-ff1f-4f6f-8196-accf688e9135-istio-token\") pod \"e2e-mt-24749012-openshift-default-548b8dc884-twrkz\" (UID: \"00cee90e-ff1f-4f6f-8196-accf688e9135\") " pod="openshift-ingress/e2e-mt-24749012-openshift-default-548b8dc884-twrkz" Sep 10 23:53:38.032992 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:53:38.032916 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00cee90e-ff1f-4f6f-8196-accf688e9135-istio-data\") pod \"e2e-mt-24749012-openshift-default-548b8dc884-twrkz\" (UID: \"00cee90e-ff1f-4f6f-8196-accf688e9135\") " pod="openshift-ingress/e2e-mt-24749012-openshift-default-548b8dc884-twrkz" Sep 10 23:53:38.032992 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:53:38.032940 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkxbl\" (UniqueName: \"kubernetes.io/projected/00cee90e-ff1f-4f6f-8196-accf688e9135-kube-api-access-tkxbl\") pod \"e2e-mt-24749012-openshift-default-548b8dc884-twrkz\" (UID: \"00cee90e-ff1f-4f6f-8196-accf688e9135\") " pod="openshift-ingress/e2e-mt-24749012-openshift-default-548b8dc884-twrkz" Sep 10 23:53:38.033231 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:53:38.033167 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00cee90e-ff1f-4f6f-8196-accf688e9135-workload-certs\") pod \"e2e-mt-24749012-openshift-default-548b8dc884-twrkz\" (UID: \"00cee90e-ff1f-4f6f-8196-accf688e9135\") " pod="openshift-ingress/e2e-mt-24749012-openshift-default-548b8dc884-twrkz" Sep 10 23:53:38.033295 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:53:38.033258 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00cee90e-ff1f-4f6f-8196-accf688e9135-workload-socket\") pod \"e2e-mt-24749012-openshift-default-548b8dc884-twrkz\" (UID: \"00cee90e-ff1f-4f6f-8196-accf688e9135\") " pod="openshift-ingress/e2e-mt-24749012-openshift-default-548b8dc884-twrkz" Sep 10 23:53:38.033351 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:53:38.033303 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00cee90e-ff1f-4f6f-8196-accf688e9135-credential-socket\") pod \"e2e-mt-24749012-openshift-default-548b8dc884-twrkz\" (UID: \"00cee90e-ff1f-4f6f-8196-accf688e9135\") " pod="openshift-ingress/e2e-mt-24749012-openshift-default-548b8dc884-twrkz" Sep 10 23:53:38.033351 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:53:38.033329 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00cee90e-ff1f-4f6f-8196-accf688e9135-istio-podinfo\") pod \"e2e-mt-24749012-openshift-default-548b8dc884-twrkz\" (UID: \"00cee90e-ff1f-4f6f-8196-accf688e9135\") " pod="openshift-ingress/e2e-mt-24749012-openshift-default-548b8dc884-twrkz" Sep 10 23:53:38.033351 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:53:38.033344 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00cee90e-ff1f-4f6f-8196-accf688e9135-istio-data\") pod \"e2e-mt-24749012-openshift-default-548b8dc884-twrkz\" (UID: \"00cee90e-ff1f-4f6f-8196-accf688e9135\") " pod="openshift-ingress/e2e-mt-24749012-openshift-default-548b8dc884-twrkz" Sep 10 23:53:38.033517 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:53:38.033357 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00cee90e-ff1f-4f6f-8196-accf688e9135-istiod-ca-cert\") pod \"e2e-mt-24749012-openshift-default-548b8dc884-twrkz\" (UID: \"00cee90e-ff1f-4f6f-8196-accf688e9135\") " pod="openshift-ingress/e2e-mt-24749012-openshift-default-548b8dc884-twrkz" Sep 10 23:53:38.033573 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:53:38.033552 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00cee90e-ff1f-4f6f-8196-accf688e9135-workload-certs\") pod \"e2e-mt-24749012-openshift-default-548b8dc884-twrkz\" (UID: \"00cee90e-ff1f-4f6f-8196-accf688e9135\") " pod="openshift-ingress/e2e-mt-24749012-openshift-default-548b8dc884-twrkz" Sep 10 23:53:38.033630 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:53:38.033609 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00cee90e-ff1f-4f6f-8196-accf688e9135-workload-socket\") pod \"e2e-mt-24749012-openshift-default-548b8dc884-twrkz\" (UID: \"00cee90e-ff1f-4f6f-8196-accf688e9135\") " pod="openshift-ingress/e2e-mt-24749012-openshift-default-548b8dc884-twrkz" Sep 10 23:53:38.034016 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:53:38.033993 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00cee90e-ff1f-4f6f-8196-accf688e9135-istiod-ca-cert\") pod \"e2e-mt-24749012-openshift-default-548b8dc884-twrkz\" (UID: \"00cee90e-ff1f-4f6f-8196-accf688e9135\") " pod="openshift-ingress/e2e-mt-24749012-openshift-default-548b8dc884-twrkz" Sep 10 23:53:38.035154 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:53:38.035134 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00cee90e-ff1f-4f6f-8196-accf688e9135-istio-envoy\") pod \"e2e-mt-24749012-openshift-default-548b8dc884-twrkz\" (UID: \"00cee90e-ff1f-4f6f-8196-accf688e9135\") " pod="openshift-ingress/e2e-mt-24749012-openshift-default-548b8dc884-twrkz" Sep 10 23:53:38.035655 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:53:38.035622 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00cee90e-ff1f-4f6f-8196-accf688e9135-istio-podinfo\") pod \"e2e-mt-24749012-openshift-default-548b8dc884-twrkz\" (UID: \"00cee90e-ff1f-4f6f-8196-accf688e9135\") " pod="openshift-ingress/e2e-mt-24749012-openshift-default-548b8dc884-twrkz" Sep 10 23:53:38.041045 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:53:38.041020 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkxbl\" (UniqueName: \"kubernetes.io/projected/00cee90e-ff1f-4f6f-8196-accf688e9135-kube-api-access-tkxbl\") pod \"e2e-mt-24749012-openshift-default-548b8dc884-twrkz\" (UID: \"00cee90e-ff1f-4f6f-8196-accf688e9135\") " pod="openshift-ingress/e2e-mt-24749012-openshift-default-548b8dc884-twrkz" Sep 10 23:53:38.041408 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:53:38.041390 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00cee90e-ff1f-4f6f-8196-accf688e9135-istio-token\") pod \"e2e-mt-24749012-openshift-default-548b8dc884-twrkz\" (UID: \"00cee90e-ff1f-4f6f-8196-accf688e9135\") " pod="openshift-ingress/e2e-mt-24749012-openshift-default-548b8dc884-twrkz" Sep 10 23:53:38.122809 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:53:38.122729 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-24749012-openshift-default-548b8dc884-twrkz" Sep 10 23:53:38.406953 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:53:38.406927 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-24749012-openshift-default-548b8dc884-twrkz"] Sep 10 23:53:38.408960 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:53:38.408930 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00cee90e_ff1f_4f6f_8196_accf688e9135.slice/crio-aa68e4c6dadfaf3b08bd49ec766fde7014ff3c357a41a1c978621474ab2f6ef0 WatchSource:0}: Error finding container aa68e4c6dadfaf3b08bd49ec766fde7014ff3c357a41a1c978621474ab2f6ef0: Status 404 returned error can't find the container with id aa68e4c6dadfaf3b08bd49ec766fde7014ff3c357a41a1c978621474ab2f6ef0 Sep 10 23:53:38.411278 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:53:38.411240 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 23:53:38.411372 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:53:38.411308 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 23:53:38.411372 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:53:38.411343 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 23:53:39.083883 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:53:39.083846 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-24749012-openshift-default-548b8dc884-twrkz" event={"ID":"00cee90e-ff1f-4f6f-8196-accf688e9135","Type":"ContainerStarted","Data":"0881e6ee650eb8414bbf61c479a8ae218fc69b3c99c9058a47aaf6ecc0fcfb6d"} Sep 10 23:53:39.084231 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:53:39.083896 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-24749012-openshift-default-548b8dc884-twrkz" event={"ID":"00cee90e-ff1f-4f6f-8196-accf688e9135","Type":"ContainerStarted","Data":"aa68e4c6dadfaf3b08bd49ec766fde7014ff3c357a41a1c978621474ab2f6ef0"} Sep 10 23:53:39.106556 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:53:39.106500 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-24749012-openshift-default-548b8dc884-twrkz" podStartSLOduration=2.106484966 podStartE2EDuration="2.106484966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:53:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:53:39.103221086 +0000 UTC m=+1010.490823855" watchObservedRunningTime="2026-09-10 23:53:39.106484966 +0000 UTC m=+1010.494087725" Sep 10 23:53:39.123286 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:53:39.123258 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-24749012-openshift-default-548b8dc884-twrkz" Sep 10 23:53:39.128773 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:53:39.128751 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-24749012-openshift-default-548b8dc884-twrkz" Sep 10 23:53:40.090062 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:53:40.090030 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-24749012-openshift-default-548b8dc884-twrkz" Sep 10 23:53:40.091090 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:53:40.091069 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-24749012-openshift-default-548b8dc884-twrkz" Sep 10 23:53:42.253922 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:53:42.253893 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-24749012-8488b4f96f-f7n59"] Sep 10 23:53:42.257784 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:53:42.257765 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-24749012-8488b4f96f-f7n59" Sep 10 23:53:42.259884 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:53:42.259860 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-24749012-serving-cert\"" Sep 10 23:53:42.268952 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:53:42.268925 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-24749012-8488b4f96f-f7n59"] Sep 10 23:53:42.379899 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:53:42.379861 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c71d0c3b-503f-4db2-b812-47e52b873345-maas-api-tls\") pod \"maas-api-e2e-mt-24749012-8488b4f96f-f7n59\" (UID: \"c71d0c3b-503f-4db2-b812-47e52b873345\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-24749012-8488b4f96f-f7n59" Sep 10 23:53:42.380074 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:53:42.379928 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c71d0c3b-503f-4db2-b812-47e52b873345-metrics-certs\") pod \"maas-api-e2e-mt-24749012-8488b4f96f-f7n59\" (UID: \"c71d0c3b-503f-4db2-b812-47e52b873345\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-24749012-8488b4f96f-f7n59" Sep 10 23:53:42.380074 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:53:42.380024 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4czz8\" (UniqueName: \"kubernetes.io/projected/c71d0c3b-503f-4db2-b812-47e52b873345-kube-api-access-4czz8\") pod \"maas-api-e2e-mt-24749012-8488b4f96f-f7n59\" (UID: \"c71d0c3b-503f-4db2-b812-47e52b873345\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-24749012-8488b4f96f-f7n59" Sep 10 23:53:42.380179 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:53:42.380120 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c71d0c3b-503f-4db2-b812-47e52b873345-openshift-service-ca\") pod \"maas-api-e2e-mt-24749012-8488b4f96f-f7n59\" (UID: \"c71d0c3b-503f-4db2-b812-47e52b873345\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-24749012-8488b4f96f-f7n59" Sep 10 23:53:42.481107 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:53:42.481074 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c71d0c3b-503f-4db2-b812-47e52b873345-metrics-certs\") pod \"maas-api-e2e-mt-24749012-8488b4f96f-f7n59\" (UID: \"c71d0c3b-503f-4db2-b812-47e52b873345\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-24749012-8488b4f96f-f7n59" Sep 10 23:53:42.481268 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:53:42.481138 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4czz8\" (UniqueName: \"kubernetes.io/projected/c71d0c3b-503f-4db2-b812-47e52b873345-kube-api-access-4czz8\") pod \"maas-api-e2e-mt-24749012-8488b4f96f-f7n59\" (UID: \"c71d0c3b-503f-4db2-b812-47e52b873345\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-24749012-8488b4f96f-f7n59" Sep 10 23:53:42.481268 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:53:42.481200 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c71d0c3b-503f-4db2-b812-47e52b873345-openshift-service-ca\") pod \"maas-api-e2e-mt-24749012-8488b4f96f-f7n59\" (UID: \"c71d0c3b-503f-4db2-b812-47e52b873345\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-24749012-8488b4f96f-f7n59" Sep 10 23:53:42.481268 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:53:42.481236 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c71d0c3b-503f-4db2-b812-47e52b873345-maas-api-tls\") pod \"maas-api-e2e-mt-24749012-8488b4f96f-f7n59\" (UID: \"c71d0c3b-503f-4db2-b812-47e52b873345\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-24749012-8488b4f96f-f7n59" Sep 10 23:53:42.481377 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:53:42.481335 2574 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-24749012-serving-cert: secret "maas-api-e2e-mt-24749012-serving-cert" not found Sep 10 23:53:42.481413 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:53:42.481392 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c71d0c3b-503f-4db2-b812-47e52b873345-maas-api-tls podName:c71d0c3b-503f-4db2-b812-47e52b873345 nodeName:}" failed. No retries permitted until 2026-09-10 23:53:42.98137389 +0000 UTC m=+1014.368976631 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c71d0c3b-503f-4db2-b812-47e52b873345-maas-api-tls") pod "maas-api-e2e-mt-24749012-8488b4f96f-f7n59" (UID: "c71d0c3b-503f-4db2-b812-47e52b873345") : secret "maas-api-e2e-mt-24749012-serving-cert" not found Sep 10 23:53:42.481877 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:53:42.481851 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c71d0c3b-503f-4db2-b812-47e52b873345-openshift-service-ca\") pod \"maas-api-e2e-mt-24749012-8488b4f96f-f7n59\" (UID: \"c71d0c3b-503f-4db2-b812-47e52b873345\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-24749012-8488b4f96f-f7n59" Sep 10 23:53:42.483717 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:53:42.483693 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c71d0c3b-503f-4db2-b812-47e52b873345-metrics-certs\") pod \"maas-api-e2e-mt-24749012-8488b4f96f-f7n59\" (UID: \"c71d0c3b-503f-4db2-b812-47e52b873345\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-24749012-8488b4f96f-f7n59" Sep 10 23:53:42.491818 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:53:42.491791 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4czz8\" (UniqueName: \"kubernetes.io/projected/c71d0c3b-503f-4db2-b812-47e52b873345-kube-api-access-4czz8\") pod \"maas-api-e2e-mt-24749012-8488b4f96f-f7n59\" (UID: \"c71d0c3b-503f-4db2-b812-47e52b873345\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-24749012-8488b4f96f-f7n59" Sep 10 23:53:42.988558 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:53:42.988514 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c71d0c3b-503f-4db2-b812-47e52b873345-maas-api-tls\") pod \"maas-api-e2e-mt-24749012-8488b4f96f-f7n59\" (UID: \"c71d0c3b-503f-4db2-b812-47e52b873345\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-24749012-8488b4f96f-f7n59" Sep 10 23:53:42.994190 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:53:42.994155 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c71d0c3b-503f-4db2-b812-47e52b873345-maas-api-tls\") pod \"maas-api-e2e-mt-24749012-8488b4f96f-f7n59\" (UID: \"c71d0c3b-503f-4db2-b812-47e52b873345\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-24749012-8488b4f96f-f7n59" Sep 10 23:53:43.170593 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:53:43.170551 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-24749012-8488b4f96f-f7n59" Sep 10 23:53:43.452370 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:53:43.452338 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-24749012-8488b4f96f-f7n59"] Sep 10 23:53:43.453567 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:53:43.453543 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc71d0c3b_503f_4db2_b812_47e52b873345.slice/crio-d9bab2c9d8da1de6a6471e96b9abce3ce300d8e5300345540a4968fd2b49d402 WatchSource:0}: Error finding container d9bab2c9d8da1de6a6471e96b9abce3ce300d8e5300345540a4968fd2b49d402: Status 404 returned error can't find the container with id d9bab2c9d8da1de6a6471e96b9abce3ce300d8e5300345540a4968fd2b49d402 Sep 10 23:53:44.115179 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:53:44.115089 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-24749012-8488b4f96f-f7n59" event={"ID":"c71d0c3b-503f-4db2-b812-47e52b873345","Type":"ContainerStarted","Data":"fcce35666113bdb9cb3042af4738ec653c61dac6fabe329243e6d170c656d136"} Sep 10 23:53:44.115179 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:53:44.115141 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-24749012-8488b4f96f-f7n59" event={"ID":"c71d0c3b-503f-4db2-b812-47e52b873345","Type":"ContainerStarted","Data":"d9bab2c9d8da1de6a6471e96b9abce3ce300d8e5300345540a4968fd2b49d402"} Sep 10 23:53:44.115452 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:53:44.115223 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-24749012-8488b4f96f-f7n59" Sep 10 23:53:44.131173 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:53:44.131124 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-24749012-8488b4f96f-f7n59" podStartSLOduration=1.7721170979999998 podStartE2EDuration="2.13110634s" totalImagesPullingTime="358.989242ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:53:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:53:44.130144778 +0000 UTC m=+1015.517747528" watchObservedRunningTime="2026-09-10 23:53:44.13110634 +0000 UTC m=+1015.518709113" Sep 10 23:53:50.128453 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:53:50.128418 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-24749012-8488b4f96f-f7n59" Sep 10 23:53:53.695374 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:53:53.695334 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-56r7l"] Sep 10 23:54:04.253343 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:04.253311 2574 generic.go:356] "Generic (PLEG): container finished" podID="00cee90e-ff1f-4f6f-8196-accf688e9135" containerID="0881e6ee650eb8414bbf61c479a8ae218fc69b3c99c9058a47aaf6ecc0fcfb6d" exitCode=0 Sep 10 23:54:04.253767 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:04.253370 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-24749012-openshift-default-548b8dc884-twrkz" event={"ID":"00cee90e-ff1f-4f6f-8196-accf688e9135","Type":"ContainerDied","Data":"0881e6ee650eb8414bbf61c479a8ae218fc69b3c99c9058a47aaf6ecc0fcfb6d"} Sep 10 23:54:04.253813 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:04.253770 2574 scope.go:117] "RemoveContainer" containerID="0881e6ee650eb8414bbf61c479a8ae218fc69b3c99c9058a47aaf6ecc0fcfb6d" Sep 10 23:54:04.254964 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:04.254936 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 23:54:04.255044 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:04.255007 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 23:54:04.255044 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:04.255037 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 23:54:05.261825 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:05.261794 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-24749012-openshift-default-548b8dc884-twrkz" event={"ID":"00cee90e-ff1f-4f6f-8196-accf688e9135","Type":"ContainerStarted","Data":"a518c7702d5fa6b438dca137870cc0755ac97245c65e3f305a89e9786b386246"} Sep 10 23:54:06.123701 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:06.123666 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-24749012-openshift-default-548b8dc884-twrkz" Sep 10 23:54:06.124615 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:06.124590 2574 patch_prober.go:28] interesting pod/e2e-mt-24749012-openshift-default-548b8dc884-twrkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Sep 10 23:54:06.124727 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:06.124654 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-24749012-openshift-default-548b8dc884-twrkz" podUID="00cee90e-ff1f-4f6f-8196-accf688e9135" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Sep 10 23:54:06.692766 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:06.692726 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-b88386-openshift-default-6dd749cdb4-cwf4b"] Sep 10 23:54:06.693564 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:06.693528 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-b88386-openshift-default-6dd749cdb4-cwf4b" podUID="c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8" containerName="istio-proxy" containerID="cri-o://e5f651854b909a2204e11f1dfd8e5f296bcb1963bac853377623e1faabcc4fb6" gracePeriod=30 Sep 10 23:54:07.124007 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:07.123973 2574 patch_prober.go:28] interesting pod/e2e-mt-24749012-openshift-default-548b8dc884-twrkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Sep 10 23:54:07.124172 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:07.124027 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-24749012-openshift-default-548b8dc884-twrkz" podUID="00cee90e-ff1f-4f6f-8196-accf688e9135" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Sep 10 23:54:08.057311 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:08.057268 2574 patch_prober.go:28] interesting pod/e2e-explicit-b88386-openshift-default-6dd749cdb4-cwf4b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 10 23:54:08.057715 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:08.057333 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-explicit-b88386-openshift-default-6dd749cdb4-cwf4b" podUID="c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 10 23:54:08.124028 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:08.123990 2574 patch_prober.go:28] interesting pod/e2e-mt-24749012-openshift-default-548b8dc884-twrkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Sep 10 23:54:08.124207 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:08.124047 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-24749012-openshift-default-548b8dc884-twrkz" podUID="00cee90e-ff1f-4f6f-8196-accf688e9135" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Sep 10 23:54:09.123695 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:09.123630 2574 patch_prober.go:28] interesting pod/e2e-mt-24749012-openshift-default-548b8dc884-twrkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Sep 10 23:54:09.124144 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:09.123715 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-24749012-openshift-default-548b8dc884-twrkz" podUID="00cee90e-ff1f-4f6f-8196-accf688e9135" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Sep 10 23:54:10.090944 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:10.090900 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-24749012-openshift-default-548b8dc884-twrkz" Sep 10 23:54:10.123710 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:10.123680 2574 patch_prober.go:28] interesting pod/e2e-mt-24749012-openshift-default-548b8dc884-twrkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Sep 10 23:54:10.124126 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:10.124097 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-24749012-openshift-default-548b8dc884-twrkz" podUID="00cee90e-ff1f-4f6f-8196-accf688e9135" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Sep 10 23:54:11.127985 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:11.127949 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-24749012-openshift-default-548b8dc884-twrkz" Sep 10 23:54:11.128955 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:11.128936 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-24749012-openshift-default-548b8dc884-twrkz" Sep 10 23:54:11.949773 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:11.949750 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-b88386-openshift-default-6dd749cdb4-cwf4b" Sep 10 23:54:12.087917 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:12.087827 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bpp22\" (UniqueName: \"kubernetes.io/projected/c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8-kube-api-access-bpp22\") pod \"c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8\" (UID: \"c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8\") " Sep 10 23:54:12.087917 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:12.087884 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8-istio-token\") pod \"c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8\" (UID: \"c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8\") " Sep 10 23:54:12.088149 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:12.087921 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8-istiod-ca-cert\") pod \"c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8\" (UID: \"c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8\") " Sep 10 23:54:12.088149 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:12.087947 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8-istio-podinfo\") pod \"c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8\" (UID: \"c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8\") " Sep 10 23:54:12.088149 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:12.087969 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8-istio-envoy\") pod \"c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8\" (UID: \"c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8\") " Sep 10 23:54:12.088149 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:12.088004 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8-credential-socket\") pod \"c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8\" (UID: \"c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8\") " Sep 10 23:54:12.088149 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:12.088044 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8-workload-certs\") pod \"c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8\" (UID: \"c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8\") " Sep 10 23:54:12.088149 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:12.088088 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8-istio-data\") pod \"c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8\" (UID: \"c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8\") " Sep 10 23:54:12.088149 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:12.088151 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8-workload-socket\") pod \"c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8\" (UID: \"c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8\") " Sep 10 23:54:12.088526 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:12.088297 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8" (UID: "c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 23:54:12.088526 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:12.088319 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8" (UID: "c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:54:12.088526 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:12.088454 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8" (UID: "c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:54:12.088526 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:12.088484 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8" (UID: "c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:54:12.088776 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:12.088538 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8-istio-data" (OuterVolumeSpecName: "istio-data") pod "c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8" (UID: "c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:54:12.088776 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:12.088632 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8-istiod-ca-cert\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:54:12.088776 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:12.088681 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8-credential-socket\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:54:12.088776 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:12.088696 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8-workload-certs\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:54:12.088776 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:12.088708 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8-workload-socket\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:54:12.090242 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:12.090214 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8" (UID: "c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 23:54:12.090242 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:12.090221 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8" (UID: "c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:54:12.090563 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:12.090545 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8-istio-token" (OuterVolumeSpecName: "istio-token") pod "c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8" (UID: "c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 23:54:12.090663 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:12.090572 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8-kube-api-access-bpp22" (OuterVolumeSpecName: "kube-api-access-bpp22") pod "c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8" (UID: "c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8"). InnerVolumeSpecName "kube-api-access-bpp22". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 23:54:12.190276 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:12.190243 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8-istio-token\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:54:12.190276 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:12.190271 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8-istio-podinfo\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:54:12.190276 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:12.190280 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8-istio-envoy\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:54:12.190711 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:12.190292 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8-istio-data\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:54:12.190711 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:12.190301 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpp22\" (UniqueName: \"kubernetes.io/projected/c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8-kube-api-access-bpp22\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:54:12.278944 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:12.278910 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-56r7l"] Sep 10 23:54:12.324576 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:12.324544 2574 generic.go:356] "Generic (PLEG): container finished" podID="c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8" containerID="e5f651854b909a2204e11f1dfd8e5f296bcb1963bac853377623e1faabcc4fb6" exitCode=0 Sep 10 23:54:12.324754 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:12.324590 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-b88386-openshift-default-6dd749cdb4-cwf4b" Sep 10 23:54:12.324754 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:12.324628 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-b88386-openshift-default-6dd749cdb4-cwf4b" event={"ID":"c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8","Type":"ContainerDied","Data":"e5f651854b909a2204e11f1dfd8e5f296bcb1963bac853377623e1faabcc4fb6"} Sep 10 23:54:12.324754 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:12.324698 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-b88386-openshift-default-6dd749cdb4-cwf4b" event={"ID":"c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8","Type":"ContainerDied","Data":"816e84122f5c2e6469e4d6f68f788016ba6f6523e0c7213c8668df7b43dd6387"} Sep 10 23:54:12.324754 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:12.324726 2574 scope.go:117] "RemoveContainer" containerID="e5f651854b909a2204e11f1dfd8e5f296bcb1963bac853377623e1faabcc4fb6" Sep 10 23:54:12.335399 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:12.335380 2574 scope.go:117] "RemoveContainer" containerID="fd8e8469b9f8ff48441987332ba4c97f2b3a025c3d45d9ae0b89a41e1a5de1c8" Sep 10 23:54:12.345044 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:12.345027 2574 scope.go:117] "RemoveContainer" containerID="e5f651854b909a2204e11f1dfd8e5f296bcb1963bac853377623e1faabcc4fb6" Sep 10 23:54:12.345316 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:54:12.345297 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5f651854b909a2204e11f1dfd8e5f296bcb1963bac853377623e1faabcc4fb6\": container with ID starting with e5f651854b909a2204e11f1dfd8e5f296bcb1963bac853377623e1faabcc4fb6 not found: ID does not exist" containerID="e5f651854b909a2204e11f1dfd8e5f296bcb1963bac853377623e1faabcc4fb6" Sep 10 23:54:12.345384 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:12.345324 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5f651854b909a2204e11f1dfd8e5f296bcb1963bac853377623e1faabcc4fb6"} err="failed to get container status \"e5f651854b909a2204e11f1dfd8e5f296bcb1963bac853377623e1faabcc4fb6\": rpc error: code = NotFound desc = could not find container \"e5f651854b909a2204e11f1dfd8e5f296bcb1963bac853377623e1faabcc4fb6\": container with ID starting with e5f651854b909a2204e11f1dfd8e5f296bcb1963bac853377623e1faabcc4fb6 not found: ID does not exist" Sep 10 23:54:12.345384 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:12.345340 2574 scope.go:117] "RemoveContainer" containerID="fd8e8469b9f8ff48441987332ba4c97f2b3a025c3d45d9ae0b89a41e1a5de1c8" Sep 10 23:54:12.345624 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:54:12.345601 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd8e8469b9f8ff48441987332ba4c97f2b3a025c3d45d9ae0b89a41e1a5de1c8\": container with ID starting with fd8e8469b9f8ff48441987332ba4c97f2b3a025c3d45d9ae0b89a41e1a5de1c8 not found: ID does not exist" containerID="fd8e8469b9f8ff48441987332ba4c97f2b3a025c3d45d9ae0b89a41e1a5de1c8" Sep 10 23:54:12.345702 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:12.345660 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd8e8469b9f8ff48441987332ba4c97f2b3a025c3d45d9ae0b89a41e1a5de1c8"} err="failed to get container status \"fd8e8469b9f8ff48441987332ba4c97f2b3a025c3d45d9ae0b89a41e1a5de1c8\": rpc error: code = NotFound desc = could not find container \"fd8e8469b9f8ff48441987332ba4c97f2b3a025c3d45d9ae0b89a41e1a5de1c8\": container with ID starting with fd8e8469b9f8ff48441987332ba4c97f2b3a025c3d45d9ae0b89a41e1a5de1c8 not found: ID does not exist" Sep 10 23:54:12.349298 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:12.349273 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-b88386-openshift-default-6dd749cdb4-cwf4b"] Sep 10 23:54:12.353370 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:12.353349 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-b88386-openshift-default-6dd749cdb4-cwf4b"] Sep 10 23:54:13.094503 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:13.094469 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8" path="/var/lib/kubelet/pods/c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8/volumes" Sep 10 23:54:23.947928 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:23.947891 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-24749012-8488b4f96f-f7n59"] Sep 10 23:54:23.948286 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:23.948194 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-24749012-8488b4f96f-f7n59" podUID="c71d0c3b-503f-4db2-b812-47e52b873345" containerName="maas-api" containerID="cri-o://fcce35666113bdb9cb3042af4738ec653c61dac6fabe329243e6d170c656d136" gracePeriod=30 Sep 10 23:54:24.205602 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:24.205543 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-24749012-8488b4f96f-f7n59" Sep 10 23:54:24.309725 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:24.309694 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4czz8\" (UniqueName: \"kubernetes.io/projected/c71d0c3b-503f-4db2-b812-47e52b873345-kube-api-access-4czz8\") pod \"c71d0c3b-503f-4db2-b812-47e52b873345\" (UID: \"c71d0c3b-503f-4db2-b812-47e52b873345\") " Sep 10 23:54:24.309901 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:24.309802 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c71d0c3b-503f-4db2-b812-47e52b873345-metrics-certs\") pod \"c71d0c3b-503f-4db2-b812-47e52b873345\" (UID: \"c71d0c3b-503f-4db2-b812-47e52b873345\") " Sep 10 23:54:24.309901 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:24.309853 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c71d0c3b-503f-4db2-b812-47e52b873345-openshift-service-ca\") pod \"c71d0c3b-503f-4db2-b812-47e52b873345\" (UID: \"c71d0c3b-503f-4db2-b812-47e52b873345\") " Sep 10 23:54:24.310044 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:24.309903 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c71d0c3b-503f-4db2-b812-47e52b873345-maas-api-tls\") pod \"c71d0c3b-503f-4db2-b812-47e52b873345\" (UID: \"c71d0c3b-503f-4db2-b812-47e52b873345\") " Sep 10 23:54:24.310270 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:24.310241 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c71d0c3b-503f-4db2-b812-47e52b873345-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c71d0c3b-503f-4db2-b812-47e52b873345" (UID: "c71d0c3b-503f-4db2-b812-47e52b873345"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 23:54:24.312021 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:24.311998 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c71d0c3b-503f-4db2-b812-47e52b873345-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c71d0c3b-503f-4db2-b812-47e52b873345" (UID: "c71d0c3b-503f-4db2-b812-47e52b873345"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 23:54:24.312106 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:24.312047 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c71d0c3b-503f-4db2-b812-47e52b873345-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c71d0c3b-503f-4db2-b812-47e52b873345" (UID: "c71d0c3b-503f-4db2-b812-47e52b873345"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 23:54:24.312106 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:24.312076 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c71d0c3b-503f-4db2-b812-47e52b873345-kube-api-access-4czz8" (OuterVolumeSpecName: "kube-api-access-4czz8") pod "c71d0c3b-503f-4db2-b812-47e52b873345" (UID: "c71d0c3b-503f-4db2-b812-47e52b873345"). InnerVolumeSpecName "kube-api-access-4czz8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 23:54:24.407036 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:24.406999 2574 generic.go:356] "Generic (PLEG): container finished" podID="c71d0c3b-503f-4db2-b812-47e52b873345" containerID="fcce35666113bdb9cb3042af4738ec653c61dac6fabe329243e6d170c656d136" exitCode=0 Sep 10 23:54:24.407278 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:24.407110 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-24749012-8488b4f96f-f7n59" event={"ID":"c71d0c3b-503f-4db2-b812-47e52b873345","Type":"ContainerDied","Data":"fcce35666113bdb9cb3042af4738ec653c61dac6fabe329243e6d170c656d136"} Sep 10 23:54:24.407278 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:24.407165 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-24749012-8488b4f96f-f7n59" event={"ID":"c71d0c3b-503f-4db2-b812-47e52b873345","Type":"ContainerDied","Data":"d9bab2c9d8da1de6a6471e96b9abce3ce300d8e5300345540a4968fd2b49d402"} Sep 10 23:54:24.407278 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:24.407195 2574 scope.go:117] "RemoveContainer" containerID="fcce35666113bdb9cb3042af4738ec653c61dac6fabe329243e6d170c656d136" Sep 10 23:54:24.407706 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:24.407363 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-24749012-8488b4f96f-f7n59" Sep 10 23:54:24.411256 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:24.411172 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4czz8\" (UniqueName: \"kubernetes.io/projected/c71d0c3b-503f-4db2-b812-47e52b873345-kube-api-access-4czz8\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:54:24.411256 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:24.411206 2574 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c71d0c3b-503f-4db2-b812-47e52b873345-metrics-certs\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:54:24.411256 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:24.411221 2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c71d0c3b-503f-4db2-b812-47e52b873345-openshift-service-ca\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:54:24.411256 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:24.411234 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c71d0c3b-503f-4db2-b812-47e52b873345-maas-api-tls\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:54:24.420478 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:24.420459 2574 scope.go:117] "RemoveContainer" containerID="fcce35666113bdb9cb3042af4738ec653c61dac6fabe329243e6d170c656d136" Sep 10 23:54:24.420811 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:54:24.420790 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcce35666113bdb9cb3042af4738ec653c61dac6fabe329243e6d170c656d136\": container with ID starting with fcce35666113bdb9cb3042af4738ec653c61dac6fabe329243e6d170c656d136 not found: ID does not exist" containerID="fcce35666113bdb9cb3042af4738ec653c61dac6fabe329243e6d170c656d136" Sep 10 23:54:24.420877 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:24.420822 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcce35666113bdb9cb3042af4738ec653c61dac6fabe329243e6d170c656d136"} err="failed to get container status \"fcce35666113bdb9cb3042af4738ec653c61dac6fabe329243e6d170c656d136\": rpc error: code = NotFound desc = could not find container \"fcce35666113bdb9cb3042af4738ec653c61dac6fabe329243e6d170c656d136\": container with ID starting with fcce35666113bdb9cb3042af4738ec653c61dac6fabe329243e6d170c656d136 not found: ID does not exist" Sep 10 23:54:24.433297 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:24.433266 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-24749012-8488b4f96f-f7n59"] Sep 10 23:54:24.437585 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:24.437560 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-24749012-8488b4f96f-f7n59"] Sep 10 23:54:25.084858 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:25.084824 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-56r7l"] Sep 10 23:54:25.094522 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:25.094494 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c71d0c3b-503f-4db2-b812-47e52b873345" path="/var/lib/kubelet/pods/c71d0c3b-503f-4db2-b812-47e52b873345/volumes" Sep 10 23:54:31.447225 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:31.447193 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-24749012-openshift-default-548b8dc884-twrkz"] Sep 10 23:54:31.447717 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:31.447388 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-24749012-openshift-default-548b8dc884-twrkz" podUID="00cee90e-ff1f-4f6f-8196-accf688e9135" containerName="istio-proxy" containerID="cri-o://a518c7702d5fa6b438dca137870cc0755ac97245c65e3f305a89e9786b386246" gracePeriod=30 Sep 10 23:54:36.700402 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:36.700374 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-24749012-openshift-default-548b8dc884-twrkz" Sep 10 23:54:36.731002 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:36.730934 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00cee90e-ff1f-4f6f-8196-accf688e9135-workload-certs\") pod \"00cee90e-ff1f-4f6f-8196-accf688e9135\" (UID: \"00cee90e-ff1f-4f6f-8196-accf688e9135\") " Sep 10 23:54:36.731131 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:36.731030 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00cee90e-ff1f-4f6f-8196-accf688e9135-istio-podinfo\") pod \"00cee90e-ff1f-4f6f-8196-accf688e9135\" (UID: \"00cee90e-ff1f-4f6f-8196-accf688e9135\") " Sep 10 23:54:36.731131 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:36.731067 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00cee90e-ff1f-4f6f-8196-accf688e9135-credential-socket\") pod \"00cee90e-ff1f-4f6f-8196-accf688e9135\" (UID: \"00cee90e-ff1f-4f6f-8196-accf688e9135\") " Sep 10 23:54:36.731131 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:36.731117 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00cee90e-ff1f-4f6f-8196-accf688e9135-workload-socket\") pod \"00cee90e-ff1f-4f6f-8196-accf688e9135\" (UID: \"00cee90e-ff1f-4f6f-8196-accf688e9135\") " Sep 10 23:54:36.731304 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:36.731143 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00cee90e-ff1f-4f6f-8196-accf688e9135-istio-data\") pod \"00cee90e-ff1f-4f6f-8196-accf688e9135\" (UID: \"00cee90e-ff1f-4f6f-8196-accf688e9135\") " Sep 10 23:54:36.731304 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:36.731184 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00cee90e-ff1f-4f6f-8196-accf688e9135-istiod-ca-cert\") pod \"00cee90e-ff1f-4f6f-8196-accf688e9135\" (UID: \"00cee90e-ff1f-4f6f-8196-accf688e9135\") " Sep 10 23:54:36.731304 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:36.731181 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00cee90e-ff1f-4f6f-8196-accf688e9135-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "00cee90e-ff1f-4f6f-8196-accf688e9135" (UID: "00cee90e-ff1f-4f6f-8196-accf688e9135"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:54:36.731304 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:36.731213 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tkxbl\" (UniqueName: \"kubernetes.io/projected/00cee90e-ff1f-4f6f-8196-accf688e9135-kube-api-access-tkxbl\") pod \"00cee90e-ff1f-4f6f-8196-accf688e9135\" (UID: \"00cee90e-ff1f-4f6f-8196-accf688e9135\") " Sep 10 23:54:36.731304 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:36.731279 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00cee90e-ff1f-4f6f-8196-accf688e9135-istio-token\") pod \"00cee90e-ff1f-4f6f-8196-accf688e9135\" (UID: \"00cee90e-ff1f-4f6f-8196-accf688e9135\") " Sep 10 23:54:36.731565 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:36.731312 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00cee90e-ff1f-4f6f-8196-accf688e9135-istio-envoy\") pod \"00cee90e-ff1f-4f6f-8196-accf688e9135\" (UID: \"00cee90e-ff1f-4f6f-8196-accf688e9135\") " Sep 10 23:54:36.731565 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:36.731404 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00cee90e-ff1f-4f6f-8196-accf688e9135-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "00cee90e-ff1f-4f6f-8196-accf688e9135" (UID: "00cee90e-ff1f-4f6f-8196-accf688e9135"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:54:36.731767 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:36.731743 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00cee90e-ff1f-4f6f-8196-accf688e9135-istio-data" (OuterVolumeSpecName: "istio-data") pod "00cee90e-ff1f-4f6f-8196-accf688e9135" (UID: "00cee90e-ff1f-4f6f-8196-accf688e9135"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:54:36.731840 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:36.731763 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00cee90e-ff1f-4f6f-8196-accf688e9135-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "00cee90e-ff1f-4f6f-8196-accf688e9135" (UID: "00cee90e-ff1f-4f6f-8196-accf688e9135"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 23:54:36.731840 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:36.731773 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00cee90e-ff1f-4f6f-8196-accf688e9135-workload-socket\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:54:36.731840 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:36.731817 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00cee90e-ff1f-4f6f-8196-accf688e9135-workload-certs\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:54:36.732015 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:36.731938 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00cee90e-ff1f-4f6f-8196-accf688e9135-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "00cee90e-ff1f-4f6f-8196-accf688e9135" (UID: "00cee90e-ff1f-4f6f-8196-accf688e9135"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:54:36.733966 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:36.733929 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/00cee90e-ff1f-4f6f-8196-accf688e9135-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "00cee90e-ff1f-4f6f-8196-accf688e9135" (UID: "00cee90e-ff1f-4f6f-8196-accf688e9135"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 23:54:36.734089 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:36.734015 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00cee90e-ff1f-4f6f-8196-accf688e9135-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "00cee90e-ff1f-4f6f-8196-accf688e9135" (UID: "00cee90e-ff1f-4f6f-8196-accf688e9135"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:54:36.734289 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:36.734266 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00cee90e-ff1f-4f6f-8196-accf688e9135-istio-token" (OuterVolumeSpecName: "istio-token") pod "00cee90e-ff1f-4f6f-8196-accf688e9135" (UID: "00cee90e-ff1f-4f6f-8196-accf688e9135"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 23:54:36.734355 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:36.734298 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00cee90e-ff1f-4f6f-8196-accf688e9135-kube-api-access-tkxbl" (OuterVolumeSpecName: "kube-api-access-tkxbl") pod "00cee90e-ff1f-4f6f-8196-accf688e9135" (UID: "00cee90e-ff1f-4f6f-8196-accf688e9135"). InnerVolumeSpecName "kube-api-access-tkxbl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 23:54:36.832609 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:36.832569 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00cee90e-ff1f-4f6f-8196-accf688e9135-istio-envoy\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:54:36.832609 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:36.832602 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00cee90e-ff1f-4f6f-8196-accf688e9135-istio-podinfo\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:54:36.832609 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:36.832613 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00cee90e-ff1f-4f6f-8196-accf688e9135-credential-socket\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:54:36.832609 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:36.832622 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00cee90e-ff1f-4f6f-8196-accf688e9135-istio-data\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:54:36.832917 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:36.832631 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00cee90e-ff1f-4f6f-8196-accf688e9135-istiod-ca-cert\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:54:36.832917 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:36.832664 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkxbl\" (UniqueName: \"kubernetes.io/projected/00cee90e-ff1f-4f6f-8196-accf688e9135-kube-api-access-tkxbl\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:54:36.832917 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:36.832672 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00cee90e-ff1f-4f6f-8196-accf688e9135-istio-token\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:54:37.504462 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:37.504429 2574 generic.go:356] "Generic (PLEG): container finished" podID="00cee90e-ff1f-4f6f-8196-accf688e9135" containerID="a518c7702d5fa6b438dca137870cc0755ac97245c65e3f305a89e9786b386246" exitCode=0 Sep 10 23:54:37.504698 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:37.504492 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-24749012-openshift-default-548b8dc884-twrkz" Sep 10 23:54:37.504698 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:37.504514 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-24749012-openshift-default-548b8dc884-twrkz" event={"ID":"00cee90e-ff1f-4f6f-8196-accf688e9135","Type":"ContainerDied","Data":"a518c7702d5fa6b438dca137870cc0755ac97245c65e3f305a89e9786b386246"} Sep 10 23:54:37.504698 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:37.504576 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-24749012-openshift-default-548b8dc884-twrkz" event={"ID":"00cee90e-ff1f-4f6f-8196-accf688e9135","Type":"ContainerDied","Data":"aa68e4c6dadfaf3b08bd49ec766fde7014ff3c357a41a1c978621474ab2f6ef0"} Sep 10 23:54:37.504698 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:37.504605 2574 scope.go:117] "RemoveContainer" containerID="a518c7702d5fa6b438dca137870cc0755ac97245c65e3f305a89e9786b386246" Sep 10 23:54:37.515686 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:37.515666 2574 scope.go:117] "RemoveContainer" containerID="0881e6ee650eb8414bbf61c479a8ae218fc69b3c99c9058a47aaf6ecc0fcfb6d" Sep 10 23:54:37.524491 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:37.524461 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-24749012-openshift-default-548b8dc884-twrkz"] Sep 10 23:54:37.526698 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:37.526673 2574 scope.go:117] "RemoveContainer" containerID="a518c7702d5fa6b438dca137870cc0755ac97245c65e3f305a89e9786b386246" Sep 10 23:54:37.527096 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:54:37.527072 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a518c7702d5fa6b438dca137870cc0755ac97245c65e3f305a89e9786b386246\": container with ID starting with a518c7702d5fa6b438dca137870cc0755ac97245c65e3f305a89e9786b386246 not found: ID does not exist" containerID="a518c7702d5fa6b438dca137870cc0755ac97245c65e3f305a89e9786b386246" Sep 10 23:54:37.527170 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:37.527110 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a518c7702d5fa6b438dca137870cc0755ac97245c65e3f305a89e9786b386246"} err="failed to get container status \"a518c7702d5fa6b438dca137870cc0755ac97245c65e3f305a89e9786b386246\": rpc error: code = NotFound desc = could not find container \"a518c7702d5fa6b438dca137870cc0755ac97245c65e3f305a89e9786b386246\": container with ID starting with a518c7702d5fa6b438dca137870cc0755ac97245c65e3f305a89e9786b386246 not found: ID does not exist" Sep 10 23:54:37.527170 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:37.527136 2574 scope.go:117] "RemoveContainer" containerID="0881e6ee650eb8414bbf61c479a8ae218fc69b3c99c9058a47aaf6ecc0fcfb6d" Sep 10 23:54:37.527430 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:54:37.527411 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0881e6ee650eb8414bbf61c479a8ae218fc69b3c99c9058a47aaf6ecc0fcfb6d\": container with ID starting with 0881e6ee650eb8414bbf61c479a8ae218fc69b3c99c9058a47aaf6ecc0fcfb6d not found: ID does not exist" containerID="0881e6ee650eb8414bbf61c479a8ae218fc69b3c99c9058a47aaf6ecc0fcfb6d" Sep 10 23:54:37.527477 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:37.527435 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0881e6ee650eb8414bbf61c479a8ae218fc69b3c99c9058a47aaf6ecc0fcfb6d"} err="failed to get container status \"0881e6ee650eb8414bbf61c479a8ae218fc69b3c99c9058a47aaf6ecc0fcfb6d\": rpc error: code = NotFound desc = could not find container \"0881e6ee650eb8414bbf61c479a8ae218fc69b3c99c9058a47aaf6ecc0fcfb6d\": container with ID starting with 0881e6ee650eb8414bbf61c479a8ae218fc69b3c99c9058a47aaf6ecc0fcfb6d not found: ID does not exist" Sep 10 23:54:37.528047 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:37.528029 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-24749012-openshift-default-548b8dc884-twrkz"] Sep 10 23:54:38.141586 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:38.141546 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-56r7l"] Sep 10 23:54:39.094169 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:39.094139 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="00cee90e-ff1f-4f6f-8196-accf688e9135" path="/var/lib/kubelet/pods/00cee90e-ff1f-4f6f-8196-accf688e9135/volumes" Sep 10 23:54:40.051026 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:40.050990 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1fe290-68d856fdd6-wsm2c"] Sep 10 23:54:40.051491 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:40.051231 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1fe290-68d856fdd6-wsm2c" podUID="cf7b6c3f-2561-4739-afa5-98e98f6b7ebf" containerName="payload-pre-processing" containerID="cri-o://83ab22a388d5ff81363e120c34bbb3560cba324700f3d9b0afc7cf1b3e251b0a" gracePeriod=30 Sep 10 23:54:40.312438 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:40.311414 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1fe290-68d856fdd6-wsm2c" Sep 10 23:54:40.367867 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:40.367837 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf7b6c3f-2561-4739-afa5-98e98f6b7ebf-plugins-config-volume\") pod \"cf7b6c3f-2561-4739-afa5-98e98f6b7ebf\" (UID: \"cf7b6c3f-2561-4739-afa5-98e98f6b7ebf\") " Sep 10 23:54:40.368047 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:40.367933 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p64sl\" (UniqueName: \"kubernetes.io/projected/cf7b6c3f-2561-4739-afa5-98e98f6b7ebf-kube-api-access-p64sl\") pod \"cf7b6c3f-2561-4739-afa5-98e98f6b7ebf\" (UID: \"cf7b6c3f-2561-4739-afa5-98e98f6b7ebf\") " Sep 10 23:54:40.368333 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:40.368302 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf7b6c3f-2561-4739-afa5-98e98f6b7ebf-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "cf7b6c3f-2561-4739-afa5-98e98f6b7ebf" (UID: "cf7b6c3f-2561-4739-afa5-98e98f6b7ebf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 23:54:40.370099 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:40.370061 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf7b6c3f-2561-4739-afa5-98e98f6b7ebf-kube-api-access-p64sl" (OuterVolumeSpecName: "kube-api-access-p64sl") pod "cf7b6c3f-2561-4739-afa5-98e98f6b7ebf" (UID: "cf7b6c3f-2561-4739-afa5-98e98f6b7ebf"). InnerVolumeSpecName "kube-api-access-p64sl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 23:54:40.469384 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:40.469355 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p64sl\" (UniqueName: \"kubernetes.io/projected/cf7b6c3f-2561-4739-afa5-98e98f6b7ebf-kube-api-access-p64sl\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:54:40.469384 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:40.469383 2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf7b6c3f-2561-4739-afa5-98e98f6b7ebf-plugins-config-volume\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:54:40.525913 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:40.525879 2574 generic.go:356] "Generic (PLEG): container finished" podID="cf7b6c3f-2561-4739-afa5-98e98f6b7ebf" containerID="83ab22a388d5ff81363e120c34bbb3560cba324700f3d9b0afc7cf1b3e251b0a" exitCode=0 Sep 10 23:54:40.526094 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:40.525931 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1fe290-68d856fdd6-wsm2c" Sep 10 23:54:40.526094 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:40.525971 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1fe290-68d856fdd6-wsm2c" event={"ID":"cf7b6c3f-2561-4739-afa5-98e98f6b7ebf","Type":"ContainerDied","Data":"83ab22a388d5ff81363e120c34bbb3560cba324700f3d9b0afc7cf1b3e251b0a"} Sep 10 23:54:40.526094 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:40.526032 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1fe290-68d856fdd6-wsm2c" event={"ID":"cf7b6c3f-2561-4739-afa5-98e98f6b7ebf","Type":"ContainerDied","Data":"7ac6c893c8d7b57871423ea06d000b02ba2146312c0544110130b400eac64f4f"} Sep 10 23:54:40.526094 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:40.526055 2574 scope.go:117] "RemoveContainer" containerID="83ab22a388d5ff81363e120c34bbb3560cba324700f3d9b0afc7cf1b3e251b0a" Sep 10 23:54:40.535934 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:40.535911 2574 scope.go:117] "RemoveContainer" containerID="83ab22a388d5ff81363e120c34bbb3560cba324700f3d9b0afc7cf1b3e251b0a" Sep 10 23:54:40.536191 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:54:40.536174 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83ab22a388d5ff81363e120c34bbb3560cba324700f3d9b0afc7cf1b3e251b0a\": container with ID starting with 83ab22a388d5ff81363e120c34bbb3560cba324700f3d9b0afc7cf1b3e251b0a not found: ID does not exist" containerID="83ab22a388d5ff81363e120c34bbb3560cba324700f3d9b0afc7cf1b3e251b0a" Sep 10 23:54:40.536265 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:40.536199 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83ab22a388d5ff81363e120c34bbb3560cba324700f3d9b0afc7cf1b3e251b0a"} err="failed to get container status \"83ab22a388d5ff81363e120c34bbb3560cba324700f3d9b0afc7cf1b3e251b0a\": rpc error: code = NotFound desc = could not find container \"83ab22a388d5ff81363e120c34bbb3560cba324700f3d9b0afc7cf1b3e251b0a\": container with ID starting with 83ab22a388d5ff81363e120c34bbb3560cba324700f3d9b0afc7cf1b3e251b0a not found: ID does not exist" Sep 10 23:54:40.548056 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:40.548035 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1fe290-68d856fdd6-wsm2c"] Sep 10 23:54:40.557282 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:40.554519 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1fe290-68d856fdd6-wsm2c"] Sep 10 23:54:41.093971 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:41.093938 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cf7b6c3f-2561-4739-afa5-98e98f6b7ebf" path="/var/lib/kubelet/pods/cf7b6c3f-2561-4739-afa5-98e98f6b7ebf/volumes" Sep 10 23:54:46.560145 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:46.560073 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-b19d25-openshift-default-774f4b768-9dmml"] Sep 10 23:54:46.560749 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:46.560731 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf7b6c3f-2561-4739-afa5-98e98f6b7ebf" containerName="payload-pre-processing" Sep 10 23:54:46.560878 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:46.560751 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf7b6c3f-2561-4739-afa5-98e98f6b7ebf" containerName="payload-pre-processing" Sep 10 23:54:46.560878 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:46.560762 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8" containerName="istio-proxy" Sep 10 23:54:46.560878 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:46.560768 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8" containerName="istio-proxy" Sep 10 23:54:46.560878 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:46.560783 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c71d0c3b-503f-4db2-b812-47e52b873345" containerName="maas-api" Sep 10 23:54:46.560878 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:46.560788 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="c71d0c3b-503f-4db2-b812-47e52b873345" containerName="maas-api" Sep 10 23:54:46.560878 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:46.560797 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00cee90e-ff1f-4f6f-8196-accf688e9135" containerName="istio-proxy" Sep 10 23:54:46.560878 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:46.560803 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="00cee90e-ff1f-4f6f-8196-accf688e9135" containerName="istio-proxy" Sep 10 23:54:46.560878 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:46.560814 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8" containerName="istio-proxy" Sep 10 23:54:46.560878 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:46.560819 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8" containerName="istio-proxy" Sep 10 23:54:46.560878 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:46.560828 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00cee90e-ff1f-4f6f-8196-accf688e9135" containerName="istio-proxy" Sep 10 23:54:46.560878 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:46.560833 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="00cee90e-ff1f-4f6f-8196-accf688e9135" containerName="istio-proxy" Sep 10 23:54:46.561225 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:46.560895 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="c71d0c3b-503f-4db2-b812-47e52b873345" containerName="maas-api" Sep 10 23:54:46.561225 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:46.560903 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="00cee90e-ff1f-4f6f-8196-accf688e9135" containerName="istio-proxy" Sep 10 23:54:46.561225 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:46.560909 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8" containerName="istio-proxy" Sep 10 23:54:46.561225 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:46.560918 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf7b6c3f-2561-4739-afa5-98e98f6b7ebf" containerName="payload-pre-processing" Sep 10 23:54:46.561225 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:46.560924 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="00cee90e-ff1f-4f6f-8196-accf688e9135" containerName="istio-proxy" Sep 10 23:54:46.561225 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:46.560929 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3f247d4-5f36-4d93-a3c8-7bb54e6ba2f8" containerName="istio-proxy" Sep 10 23:54:46.565288 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:46.565268 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-b19d25-openshift-default-774f4b768-9dmml" Sep 10 23:54:46.567461 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:46.567433 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-b19d25-openshift-default-dockercfg-z8rhh\"" Sep 10 23:54:46.576274 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:46.576250 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-b19d25-openshift-default-774f4b768-9dmml"] Sep 10 23:54:46.631038 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:46.631003 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac20aa7e-7b99-4b5f-b498-8aeb713566d9-istio-token\") pod \"e2e-shared-a-w2-b19d25-openshift-default-774f4b768-9dmml\" (UID: \"ac20aa7e-7b99-4b5f-b498-8aeb713566d9\") " pod="openshift-ingress/e2e-shared-a-w2-b19d25-openshift-default-774f4b768-9dmml" Sep 10 23:54:46.631038 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:46.631042 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac20aa7e-7b99-4b5f-b498-8aeb713566d9-istio-podinfo\") pod \"e2e-shared-a-w2-b19d25-openshift-default-774f4b768-9dmml\" (UID: \"ac20aa7e-7b99-4b5f-b498-8aeb713566d9\") " pod="openshift-ingress/e2e-shared-a-w2-b19d25-openshift-default-774f4b768-9dmml" Sep 10 23:54:46.631249 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:46.631059 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxh5k\" (UniqueName: \"kubernetes.io/projected/ac20aa7e-7b99-4b5f-b498-8aeb713566d9-kube-api-access-lxh5k\") pod \"e2e-shared-a-w2-b19d25-openshift-default-774f4b768-9dmml\" (UID: \"ac20aa7e-7b99-4b5f-b498-8aeb713566d9\") " pod="openshift-ingress/e2e-shared-a-w2-b19d25-openshift-default-774f4b768-9dmml" Sep 10 23:54:46.631249 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:46.631089 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac20aa7e-7b99-4b5f-b498-8aeb713566d9-workload-socket\") pod \"e2e-shared-a-w2-b19d25-openshift-default-774f4b768-9dmml\" (UID: \"ac20aa7e-7b99-4b5f-b498-8aeb713566d9\") " pod="openshift-ingress/e2e-shared-a-w2-b19d25-openshift-default-774f4b768-9dmml" Sep 10 23:54:46.631249 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:46.631129 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac20aa7e-7b99-4b5f-b498-8aeb713566d9-workload-certs\") pod \"e2e-shared-a-w2-b19d25-openshift-default-774f4b768-9dmml\" (UID: \"ac20aa7e-7b99-4b5f-b498-8aeb713566d9\") " pod="openshift-ingress/e2e-shared-a-w2-b19d25-openshift-default-774f4b768-9dmml" Sep 10 23:54:46.631249 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:46.631167 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac20aa7e-7b99-4b5f-b498-8aeb713566d9-istio-envoy\") pod \"e2e-shared-a-w2-b19d25-openshift-default-774f4b768-9dmml\" (UID: \"ac20aa7e-7b99-4b5f-b498-8aeb713566d9\") " pod="openshift-ingress/e2e-shared-a-w2-b19d25-openshift-default-774f4b768-9dmml" Sep 10 23:54:46.631249 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:46.631188 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac20aa7e-7b99-4b5f-b498-8aeb713566d9-istio-data\") pod \"e2e-shared-a-w2-b19d25-openshift-default-774f4b768-9dmml\" (UID: \"ac20aa7e-7b99-4b5f-b498-8aeb713566d9\") " pod="openshift-ingress/e2e-shared-a-w2-b19d25-openshift-default-774f4b768-9dmml" Sep 10 23:54:46.631422 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:46.631273 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac20aa7e-7b99-4b5f-b498-8aeb713566d9-istiod-ca-cert\") pod \"e2e-shared-a-w2-b19d25-openshift-default-774f4b768-9dmml\" (UID: \"ac20aa7e-7b99-4b5f-b498-8aeb713566d9\") " pod="openshift-ingress/e2e-shared-a-w2-b19d25-openshift-default-774f4b768-9dmml" Sep 10 23:54:46.631422 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:46.631310 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac20aa7e-7b99-4b5f-b498-8aeb713566d9-credential-socket\") pod \"e2e-shared-a-w2-b19d25-openshift-default-774f4b768-9dmml\" (UID: \"ac20aa7e-7b99-4b5f-b498-8aeb713566d9\") " pod="openshift-ingress/e2e-shared-a-w2-b19d25-openshift-default-774f4b768-9dmml" Sep 10 23:54:46.732443 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:46.732403 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac20aa7e-7b99-4b5f-b498-8aeb713566d9-istio-envoy\") pod \"e2e-shared-a-w2-b19d25-openshift-default-774f4b768-9dmml\" (UID: \"ac20aa7e-7b99-4b5f-b498-8aeb713566d9\") " pod="openshift-ingress/e2e-shared-a-w2-b19d25-openshift-default-774f4b768-9dmml" Sep 10 23:54:46.732688 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:46.732455 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac20aa7e-7b99-4b5f-b498-8aeb713566d9-istio-data\") pod \"e2e-shared-a-w2-b19d25-openshift-default-774f4b768-9dmml\" (UID: \"ac20aa7e-7b99-4b5f-b498-8aeb713566d9\") " pod="openshift-ingress/e2e-shared-a-w2-b19d25-openshift-default-774f4b768-9dmml" Sep 10 23:54:46.732688 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:46.732533 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac20aa7e-7b99-4b5f-b498-8aeb713566d9-istiod-ca-cert\") pod \"e2e-shared-a-w2-b19d25-openshift-default-774f4b768-9dmml\" (UID: \"ac20aa7e-7b99-4b5f-b498-8aeb713566d9\") " pod="openshift-ingress/e2e-shared-a-w2-b19d25-openshift-default-774f4b768-9dmml" Sep 10 23:54:46.732688 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:46.732570 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac20aa7e-7b99-4b5f-b498-8aeb713566d9-credential-socket\") pod \"e2e-shared-a-w2-b19d25-openshift-default-774f4b768-9dmml\" (UID: \"ac20aa7e-7b99-4b5f-b498-8aeb713566d9\") " pod="openshift-ingress/e2e-shared-a-w2-b19d25-openshift-default-774f4b768-9dmml" Sep 10 23:54:46.732688 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:46.732620 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac20aa7e-7b99-4b5f-b498-8aeb713566d9-istio-token\") pod \"e2e-shared-a-w2-b19d25-openshift-default-774f4b768-9dmml\" (UID: \"ac20aa7e-7b99-4b5f-b498-8aeb713566d9\") " pod="openshift-ingress/e2e-shared-a-w2-b19d25-openshift-default-774f4b768-9dmml" Sep 10 23:54:46.732920 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:46.732698 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac20aa7e-7b99-4b5f-b498-8aeb713566d9-istio-podinfo\") pod \"e2e-shared-a-w2-b19d25-openshift-default-774f4b768-9dmml\" (UID: \"ac20aa7e-7b99-4b5f-b498-8aeb713566d9\") " pod="openshift-ingress/e2e-shared-a-w2-b19d25-openshift-default-774f4b768-9dmml" Sep 10 23:54:46.732920 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:46.732738 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxh5k\" (UniqueName: \"kubernetes.io/projected/ac20aa7e-7b99-4b5f-b498-8aeb713566d9-kube-api-access-lxh5k\") pod \"e2e-shared-a-w2-b19d25-openshift-default-774f4b768-9dmml\" (UID: \"ac20aa7e-7b99-4b5f-b498-8aeb713566d9\") " pod="openshift-ingress/e2e-shared-a-w2-b19d25-openshift-default-774f4b768-9dmml" Sep 10 23:54:46.732920 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:46.732774 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac20aa7e-7b99-4b5f-b498-8aeb713566d9-workload-socket\") pod \"e2e-shared-a-w2-b19d25-openshift-default-774f4b768-9dmml\" (UID: \"ac20aa7e-7b99-4b5f-b498-8aeb713566d9\") " pod="openshift-ingress/e2e-shared-a-w2-b19d25-openshift-default-774f4b768-9dmml" Sep 10 23:54:46.732920 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:46.732818 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac20aa7e-7b99-4b5f-b498-8aeb713566d9-workload-certs\") pod \"e2e-shared-a-w2-b19d25-openshift-default-774f4b768-9dmml\" (UID: \"ac20aa7e-7b99-4b5f-b498-8aeb713566d9\") " pod="openshift-ingress/e2e-shared-a-w2-b19d25-openshift-default-774f4b768-9dmml" Sep 10 23:54:46.733108 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:46.732938 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac20aa7e-7b99-4b5f-b498-8aeb713566d9-istio-data\") pod \"e2e-shared-a-w2-b19d25-openshift-default-774f4b768-9dmml\" (UID: \"ac20aa7e-7b99-4b5f-b498-8aeb713566d9\") " pod="openshift-ingress/e2e-shared-a-w2-b19d25-openshift-default-774f4b768-9dmml" Sep 10 23:54:46.733108 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:46.733068 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac20aa7e-7b99-4b5f-b498-8aeb713566d9-credential-socket\") pod \"e2e-shared-a-w2-b19d25-openshift-default-774f4b768-9dmml\" (UID: \"ac20aa7e-7b99-4b5f-b498-8aeb713566d9\") " pod="openshift-ingress/e2e-shared-a-w2-b19d25-openshift-default-774f4b768-9dmml" Sep 10 23:54:46.733189 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:46.733152 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac20aa7e-7b99-4b5f-b498-8aeb713566d9-workload-certs\") pod \"e2e-shared-a-w2-b19d25-openshift-default-774f4b768-9dmml\" (UID: \"ac20aa7e-7b99-4b5f-b498-8aeb713566d9\") " pod="openshift-ingress/e2e-shared-a-w2-b19d25-openshift-default-774f4b768-9dmml" Sep 10 23:54:46.733249 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:46.733186 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac20aa7e-7b99-4b5f-b498-8aeb713566d9-workload-socket\") pod \"e2e-shared-a-w2-b19d25-openshift-default-774f4b768-9dmml\" (UID: \"ac20aa7e-7b99-4b5f-b498-8aeb713566d9\") " pod="openshift-ingress/e2e-shared-a-w2-b19d25-openshift-default-774f4b768-9dmml" Sep 10 23:54:46.733426 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:46.733404 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac20aa7e-7b99-4b5f-b498-8aeb713566d9-istiod-ca-cert\") pod \"e2e-shared-a-w2-b19d25-openshift-default-774f4b768-9dmml\" (UID: \"ac20aa7e-7b99-4b5f-b498-8aeb713566d9\") " pod="openshift-ingress/e2e-shared-a-w2-b19d25-openshift-default-774f4b768-9dmml" Sep 10 23:54:46.734785 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:46.734764 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac20aa7e-7b99-4b5f-b498-8aeb713566d9-istio-envoy\") pod \"e2e-shared-a-w2-b19d25-openshift-default-774f4b768-9dmml\" (UID: \"ac20aa7e-7b99-4b5f-b498-8aeb713566d9\") " pod="openshift-ingress/e2e-shared-a-w2-b19d25-openshift-default-774f4b768-9dmml" Sep 10 23:54:46.735370 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:46.735350 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac20aa7e-7b99-4b5f-b498-8aeb713566d9-istio-podinfo\") pod \"e2e-shared-a-w2-b19d25-openshift-default-774f4b768-9dmml\" (UID: \"ac20aa7e-7b99-4b5f-b498-8aeb713566d9\") " pod="openshift-ingress/e2e-shared-a-w2-b19d25-openshift-default-774f4b768-9dmml" Sep 10 23:54:46.740159 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:46.740137 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac20aa7e-7b99-4b5f-b498-8aeb713566d9-istio-token\") pod \"e2e-shared-a-w2-b19d25-openshift-default-774f4b768-9dmml\" (UID: \"ac20aa7e-7b99-4b5f-b498-8aeb713566d9\") " pod="openshift-ingress/e2e-shared-a-w2-b19d25-openshift-default-774f4b768-9dmml" Sep 10 23:54:46.740569 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:46.740548 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxh5k\" (UniqueName: \"kubernetes.io/projected/ac20aa7e-7b99-4b5f-b498-8aeb713566d9-kube-api-access-lxh5k\") pod \"e2e-shared-a-w2-b19d25-openshift-default-774f4b768-9dmml\" (UID: \"ac20aa7e-7b99-4b5f-b498-8aeb713566d9\") " pod="openshift-ingress/e2e-shared-a-w2-b19d25-openshift-default-774f4b768-9dmml" Sep 10 23:54:46.876981 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:46.876897 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-b19d25-openshift-default-774f4b768-9dmml" Sep 10 23:54:47.209720 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:47.209682 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-b19d25-openshift-default-774f4b768-9dmml"] Sep 10 23:54:47.210694 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:54:47.210668 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac20aa7e_7b99_4b5f_b498_8aeb713566d9.slice/crio-af87f9d07c849c27b969a14de2ab980c7f4261b7ab84900db82a0a4527b8f8b9 WatchSource:0}: Error finding container af87f9d07c849c27b969a14de2ab980c7f4261b7ab84900db82a0a4527b8f8b9: Status 404 returned error can't find the container with id af87f9d07c849c27b969a14de2ab980c7f4261b7ab84900db82a0a4527b8f8b9 Sep 10 23:54:47.212941 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:47.212906 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 23:54:47.213020 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:47.212971 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 23:54:47.213020 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:47.213003 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 23:54:47.304179 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:47.304153 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-1fe290-openshift-default-644c884748-msx97"] Sep 10 23:54:47.304410 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:47.304365 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-1fe290-openshift-default-644c884748-msx97" podUID="4cee31c3-1da1-4483-a73c-1d0e23fb8063" containerName="istio-proxy" containerID="cri-o://e1f6c5a8c0b05204295429ca0cbf03540112c45bb23e2aca8f002070d7217e71" gracePeriod=30 Sep 10 23:54:47.571305 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:47.571207 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-b19d25-openshift-default-774f4b768-9dmml" event={"ID":"ac20aa7e-7b99-4b5f-b498-8aeb713566d9","Type":"ContainerStarted","Data":"11da6f34441d7f840febb86536326a8d23dee7ce4d8a957674a334bcfae635c2"} Sep 10 23:54:47.571305 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:47.571265 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-b19d25-openshift-default-774f4b768-9dmml" event={"ID":"ac20aa7e-7b99-4b5f-b498-8aeb713566d9","Type":"ContainerStarted","Data":"af87f9d07c849c27b969a14de2ab980c7f4261b7ab84900db82a0a4527b8f8b9"} Sep 10 23:54:47.600778 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:47.600726 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-b19d25-openshift-default-774f4b768-9dmml" podStartSLOduration=1.600707128 podStartE2EDuration="1.600707128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:54:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:54:47.598127545 +0000 UTC m=+1078.985730331" watchObservedRunningTime="2026-09-10 23:54:47.600707128 +0000 UTC m=+1078.988309989" Sep 10 23:54:47.877760 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:47.877681 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-b19d25-openshift-default-774f4b768-9dmml" Sep 10 23:54:48.882573 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:48.882544 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-b19d25-openshift-default-774f4b768-9dmml" Sep 10 23:54:49.096604 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:49.096575 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-56r7l"] Sep 10 23:54:49.237157 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:49.237127 2574 patch_prober.go:28] interesting pod/e2e-inf-a-1fe290-openshift-default-644c884748-msx97 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 10 23:54:49.237250 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:49.237191 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-1fe290-openshift-default-644c884748-msx97" podUID="4cee31c3-1da1-4483-a73c-1d0e23fb8063" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 10 23:54:49.583928 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:49.583891 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-b19d25-openshift-default-774f4b768-9dmml" Sep 10 23:54:49.584728 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:49.584708 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-b19d25-openshift-default-774f4b768-9dmml" Sep 10 23:54:52.560313 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:52.560285 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1fe290-openshift-default-644c884748-msx97" Sep 10 23:54:52.603373 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:52.603280 2574 generic.go:356] "Generic (PLEG): container finished" podID="4cee31c3-1da1-4483-a73c-1d0e23fb8063" containerID="e1f6c5a8c0b05204295429ca0cbf03540112c45bb23e2aca8f002070d7217e71" exitCode=0 Sep 10 23:54:52.603373 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:52.603329 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1fe290-openshift-default-644c884748-msx97" Sep 10 23:54:52.603565 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:52.603406 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1fe290-openshift-default-644c884748-msx97" event={"ID":"4cee31c3-1da1-4483-a73c-1d0e23fb8063","Type":"ContainerDied","Data":"e1f6c5a8c0b05204295429ca0cbf03540112c45bb23e2aca8f002070d7217e71"} Sep 10 23:54:52.603565 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:52.603471 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1fe290-openshift-default-644c884748-msx97" event={"ID":"4cee31c3-1da1-4483-a73c-1d0e23fb8063","Type":"ContainerDied","Data":"caddce181bb5ce5a03dd5d72c27e9cdaf688716fb6fc14c20c4bf42ce8ddbee8"} Sep 10 23:54:52.603565 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:52.603506 2574 scope.go:117] "RemoveContainer" containerID="e1f6c5a8c0b05204295429ca0cbf03540112c45bb23e2aca8f002070d7217e71" Sep 10 23:54:52.614738 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:52.614719 2574 scope.go:117] "RemoveContainer" containerID="f26d88deff51be711fcb129535330522310814d64fe4bd8fb0933ee0ff9f16c1" Sep 10 23:54:52.622994 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:52.622975 2574 scope.go:117] "RemoveContainer" containerID="e1f6c5a8c0b05204295429ca0cbf03540112c45bb23e2aca8f002070d7217e71" Sep 10 23:54:52.623275 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:54:52.623254 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1f6c5a8c0b05204295429ca0cbf03540112c45bb23e2aca8f002070d7217e71\": container with ID starting with e1f6c5a8c0b05204295429ca0cbf03540112c45bb23e2aca8f002070d7217e71 not found: ID does not exist" containerID="e1f6c5a8c0b05204295429ca0cbf03540112c45bb23e2aca8f002070d7217e71" Sep 10 23:54:52.623351 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:52.623290 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1f6c5a8c0b05204295429ca0cbf03540112c45bb23e2aca8f002070d7217e71"} err="failed to get container status \"e1f6c5a8c0b05204295429ca0cbf03540112c45bb23e2aca8f002070d7217e71\": rpc error: code = NotFound desc = could not find container \"e1f6c5a8c0b05204295429ca0cbf03540112c45bb23e2aca8f002070d7217e71\": container with ID starting with e1f6c5a8c0b05204295429ca0cbf03540112c45bb23e2aca8f002070d7217e71 not found: ID does not exist" Sep 10 23:54:52.623351 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:52.623316 2574 scope.go:117] "RemoveContainer" containerID="f26d88deff51be711fcb129535330522310814d64fe4bd8fb0933ee0ff9f16c1" Sep 10 23:54:52.623579 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:54:52.623560 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f26d88deff51be711fcb129535330522310814d64fe4bd8fb0933ee0ff9f16c1\": container with ID starting with f26d88deff51be711fcb129535330522310814d64fe4bd8fb0933ee0ff9f16c1 not found: ID does not exist" containerID="f26d88deff51be711fcb129535330522310814d64fe4bd8fb0933ee0ff9f16c1" Sep 10 23:54:52.623617 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:52.623585 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f26d88deff51be711fcb129535330522310814d64fe4bd8fb0933ee0ff9f16c1"} err="failed to get container status \"f26d88deff51be711fcb129535330522310814d64fe4bd8fb0933ee0ff9f16c1\": rpc error: code = NotFound desc = could not find container \"f26d88deff51be711fcb129535330522310814d64fe4bd8fb0933ee0ff9f16c1\": container with ID starting with f26d88deff51be711fcb129535330522310814d64fe4bd8fb0933ee0ff9f16c1 not found: ID does not exist" Sep 10 23:54:52.691254 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:52.691226 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4cee31c3-1da1-4483-a73c-1d0e23fb8063-istio-podinfo\") pod \"4cee31c3-1da1-4483-a73c-1d0e23fb8063\" (UID: \"4cee31c3-1da1-4483-a73c-1d0e23fb8063\") " Sep 10 23:54:52.691405 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:52.691272 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v67w4\" (UniqueName: \"kubernetes.io/projected/4cee31c3-1da1-4483-a73c-1d0e23fb8063-kube-api-access-v67w4\") pod \"4cee31c3-1da1-4483-a73c-1d0e23fb8063\" (UID: \"4cee31c3-1da1-4483-a73c-1d0e23fb8063\") " Sep 10 23:54:52.691405 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:52.691292 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4cee31c3-1da1-4483-a73c-1d0e23fb8063-workload-certs\") pod \"4cee31c3-1da1-4483-a73c-1d0e23fb8063\" (UID: \"4cee31c3-1da1-4483-a73c-1d0e23fb8063\") " Sep 10 23:54:52.691405 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:52.691353 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4cee31c3-1da1-4483-a73c-1d0e23fb8063-istio-envoy\") pod \"4cee31c3-1da1-4483-a73c-1d0e23fb8063\" (UID: \"4cee31c3-1da1-4483-a73c-1d0e23fb8063\") " Sep 10 23:54:52.691405 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:52.691374 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4cee31c3-1da1-4483-a73c-1d0e23fb8063-istiod-ca-cert\") pod \"4cee31c3-1da1-4483-a73c-1d0e23fb8063\" (UID: \"4cee31c3-1da1-4483-a73c-1d0e23fb8063\") " Sep 10 23:54:52.691405 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:52.691390 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cee31c3-1da1-4483-a73c-1d0e23fb8063-workload-socket\") pod \"4cee31c3-1da1-4483-a73c-1d0e23fb8063\" (UID: \"4cee31c3-1da1-4483-a73c-1d0e23fb8063\") " Sep 10 23:54:52.691745 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:52.691437 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4cee31c3-1da1-4483-a73c-1d0e23fb8063-istio-data\") pod \"4cee31c3-1da1-4483-a73c-1d0e23fb8063\" (UID: \"4cee31c3-1da1-4483-a73c-1d0e23fb8063\") " Sep 10 23:54:52.691745 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:52.691477 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cee31c3-1da1-4483-a73c-1d0e23fb8063-credential-socket\") pod \"4cee31c3-1da1-4483-a73c-1d0e23fb8063\" (UID: \"4cee31c3-1da1-4483-a73c-1d0e23fb8063\") " Sep 10 23:54:52.691745 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:52.691522 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4cee31c3-1da1-4483-a73c-1d0e23fb8063-istio-token\") pod \"4cee31c3-1da1-4483-a73c-1d0e23fb8063\" (UID: \"4cee31c3-1da1-4483-a73c-1d0e23fb8063\") " Sep 10 23:54:52.691745 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:52.691687 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cee31c3-1da1-4483-a73c-1d0e23fb8063-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4cee31c3-1da1-4483-a73c-1d0e23fb8063" (UID: "4cee31c3-1da1-4483-a73c-1d0e23fb8063"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:54:52.691745 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:52.691697 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cee31c3-1da1-4483-a73c-1d0e23fb8063-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4cee31c3-1da1-4483-a73c-1d0e23fb8063" (UID: "4cee31c3-1da1-4483-a73c-1d0e23fb8063"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:54:52.692004 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:52.691823 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cee31c3-1da1-4483-a73c-1d0e23fb8063-istio-data" (OuterVolumeSpecName: "istio-data") pod "4cee31c3-1da1-4483-a73c-1d0e23fb8063" (UID: "4cee31c3-1da1-4483-a73c-1d0e23fb8063"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:54:52.692004 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:52.691906 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cee31c3-1da1-4483-a73c-1d0e23fb8063-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4cee31c3-1da1-4483-a73c-1d0e23fb8063" (UID: "4cee31c3-1da1-4483-a73c-1d0e23fb8063"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:54:52.692004 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:52.691949 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cee31c3-1da1-4483-a73c-1d0e23fb8063-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4cee31c3-1da1-4483-a73c-1d0e23fb8063" (UID: "4cee31c3-1da1-4483-a73c-1d0e23fb8063"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 23:54:52.692115 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:52.692008 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cee31c3-1da1-4483-a73c-1d0e23fb8063-workload-socket\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:54:52.692115 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:52.692032 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4cee31c3-1da1-4483-a73c-1d0e23fb8063-istio-data\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:54:52.692115 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:52.692045 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cee31c3-1da1-4483-a73c-1d0e23fb8063-credential-socket\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:54:52.692115 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:52.692059 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4cee31c3-1da1-4483-a73c-1d0e23fb8063-workload-certs\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:54:52.693606 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:52.693575 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cee31c3-1da1-4483-a73c-1d0e23fb8063-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4cee31c3-1da1-4483-a73c-1d0e23fb8063" (UID: "4cee31c3-1da1-4483-a73c-1d0e23fb8063"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:54:52.693783 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:52.693762 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cee31c3-1da1-4483-a73c-1d0e23fb8063-istio-token" (OuterVolumeSpecName: "istio-token") pod "4cee31c3-1da1-4483-a73c-1d0e23fb8063" (UID: "4cee31c3-1da1-4483-a73c-1d0e23fb8063"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 23:54:52.693845 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:52.693796 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cee31c3-1da1-4483-a73c-1d0e23fb8063-kube-api-access-v67w4" (OuterVolumeSpecName: "kube-api-access-v67w4") pod "4cee31c3-1da1-4483-a73c-1d0e23fb8063" (UID: "4cee31c3-1da1-4483-a73c-1d0e23fb8063"). InnerVolumeSpecName "kube-api-access-v67w4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 23:54:52.693894 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:52.693851 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4cee31c3-1da1-4483-a73c-1d0e23fb8063-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4cee31c3-1da1-4483-a73c-1d0e23fb8063" (UID: "4cee31c3-1da1-4483-a73c-1d0e23fb8063"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 23:54:52.793097 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:52.793063 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4cee31c3-1da1-4483-a73c-1d0e23fb8063-istio-token\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:54:52.793097 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:52.793091 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4cee31c3-1da1-4483-a73c-1d0e23fb8063-istio-podinfo\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:54:52.793097 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:52.793101 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v67w4\" (UniqueName: \"kubernetes.io/projected/4cee31c3-1da1-4483-a73c-1d0e23fb8063-kube-api-access-v67w4\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:54:52.793325 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:52.793110 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4cee31c3-1da1-4483-a73c-1d0e23fb8063-istio-envoy\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:54:52.793325 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:52.793119 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4cee31c3-1da1-4483-a73c-1d0e23fb8063-istiod-ca-cert\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:54:52.925820 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:52.925790 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-1fe290-openshift-default-644c884748-msx97"] Sep 10 23:54:52.931188 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:52.931159 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-1fe290-openshift-default-644c884748-msx97"] Sep 10 23:54:53.094190 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:53.094158 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4cee31c3-1da1-4483-a73c-1d0e23fb8063" path="/var/lib/kubelet/pods/4cee31c3-1da1-4483-a73c-1d0e23fb8063/volumes" Sep 10 23:54:58.432813 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:58.432774 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-7f598e-5cb99c745c-6n4tm"] Sep 10 23:54:58.433397 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:58.433041 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7f598e-5cb99c745c-6n4tm" podUID="1305be23-ac85-4b42-a83f-2b0b8fa9d435" containerName="maas-api" containerID="cri-o://09532062b76c06f36ee005b9a5820e00f4b19cf559d02da6789796a1dadaf059" gracePeriod=30 Sep 10 23:54:58.644159 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:58.644082 2574 generic.go:356] "Generic (PLEG): container finished" podID="1305be23-ac85-4b42-a83f-2b0b8fa9d435" containerID="09532062b76c06f36ee005b9a5820e00f4b19cf559d02da6789796a1dadaf059" exitCode=0 Sep 10 23:54:58.644321 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:58.644183 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7f598e-5cb99c745c-6n4tm" event={"ID":"1305be23-ac85-4b42-a83f-2b0b8fa9d435","Type":"ContainerDied","Data":"09532062b76c06f36ee005b9a5820e00f4b19cf559d02da6789796a1dadaf059"} Sep 10 23:54:58.723039 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:58.723008 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7f598e-5cb99c745c-6n4tm" Sep 10 23:54:58.850131 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:58.850100 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1305be23-ac85-4b42-a83f-2b0b8fa9d435-maas-api-tls\") pod \"1305be23-ac85-4b42-a83f-2b0b8fa9d435\" (UID: \"1305be23-ac85-4b42-a83f-2b0b8fa9d435\") " Sep 10 23:54:58.850281 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:58.850169 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p44r6\" (UniqueName: \"kubernetes.io/projected/1305be23-ac85-4b42-a83f-2b0b8fa9d435-kube-api-access-p44r6\") pod \"1305be23-ac85-4b42-a83f-2b0b8fa9d435\" (UID: \"1305be23-ac85-4b42-a83f-2b0b8fa9d435\") " Sep 10 23:54:58.850281 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:58.850188 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1305be23-ac85-4b42-a83f-2b0b8fa9d435-metrics-certs\") pod \"1305be23-ac85-4b42-a83f-2b0b8fa9d435\" (UID: \"1305be23-ac85-4b42-a83f-2b0b8fa9d435\") " Sep 10 23:54:58.850281 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:58.850260 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1305be23-ac85-4b42-a83f-2b0b8fa9d435-openshift-service-ca\") pod \"1305be23-ac85-4b42-a83f-2b0b8fa9d435\" (UID: \"1305be23-ac85-4b42-a83f-2b0b8fa9d435\") " Sep 10 23:54:58.850829 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:58.850795 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1305be23-ac85-4b42-a83f-2b0b8fa9d435-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1305be23-ac85-4b42-a83f-2b0b8fa9d435" (UID: "1305be23-ac85-4b42-a83f-2b0b8fa9d435"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 23:54:58.855462 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:58.855426 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1305be23-ac85-4b42-a83f-2b0b8fa9d435-kube-api-access-p44r6" (OuterVolumeSpecName: "kube-api-access-p44r6") pod "1305be23-ac85-4b42-a83f-2b0b8fa9d435" (UID: "1305be23-ac85-4b42-a83f-2b0b8fa9d435"). InnerVolumeSpecName "kube-api-access-p44r6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 23:54:58.855591 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:58.855569 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1305be23-ac85-4b42-a83f-2b0b8fa9d435-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1305be23-ac85-4b42-a83f-2b0b8fa9d435" (UID: "1305be23-ac85-4b42-a83f-2b0b8fa9d435"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 23:54:58.856368 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:58.856335 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1305be23-ac85-4b42-a83f-2b0b8fa9d435-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "1305be23-ac85-4b42-a83f-2b0b8fa9d435" (UID: "1305be23-ac85-4b42-a83f-2b0b8fa9d435"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 23:54:58.951395 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:58.951315 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p44r6\" (UniqueName: \"kubernetes.io/projected/1305be23-ac85-4b42-a83f-2b0b8fa9d435-kube-api-access-p44r6\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:54:58.951395 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:58.951352 2574 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1305be23-ac85-4b42-a83f-2b0b8fa9d435-metrics-certs\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:54:58.951395 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:58.951370 2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1305be23-ac85-4b42-a83f-2b0b8fa9d435-openshift-service-ca\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:54:58.951395 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:58.951384 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1305be23-ac85-4b42-a83f-2b0b8fa9d435-maas-api-tls\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:54:59.292822 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:59.292712 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-56r7l"] Sep 10 23:54:59.651577 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:59.651483 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7f598e-5cb99c745c-6n4tm" event={"ID":"1305be23-ac85-4b42-a83f-2b0b8fa9d435","Type":"ContainerDied","Data":"05444731b115192da28878d037e433350965c6d8b6b5ba479dfa25d0e07657b8"} Sep 10 23:54:59.651577 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:59.651508 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7f598e-5cb99c745c-6n4tm" Sep 10 23:54:59.651577 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:59.651556 2574 scope.go:117] "RemoveContainer" containerID="09532062b76c06f36ee005b9a5820e00f4b19cf559d02da6789796a1dadaf059" Sep 10 23:54:59.674473 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:59.674440 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-7f598e-5cb99c745c-6n4tm"] Sep 10 23:54:59.681797 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:54:59.681769 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-7f598e-5cb99c745c-6n4tm"] Sep 10 23:55:01.095195 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:01.095164 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1305be23-ac85-4b42-a83f-2b0b8fa9d435" path="/var/lib/kubelet/pods/1305be23-ac85-4b42-a83f-2b0b8fa9d435/volumes" Sep 10 23:55:03.684440 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:03.684349 2574 generic.go:356] "Generic (PLEG): container finished" podID="ac20aa7e-7b99-4b5f-b498-8aeb713566d9" containerID="11da6f34441d7f840febb86536326a8d23dee7ce4d8a957674a334bcfae635c2" exitCode=0 Sep 10 23:55:03.684440 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:03.684420 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-b19d25-openshift-default-774f4b768-9dmml" event={"ID":"ac20aa7e-7b99-4b5f-b498-8aeb713566d9","Type":"ContainerDied","Data":"11da6f34441d7f840febb86536326a8d23dee7ce4d8a957674a334bcfae635c2"} Sep 10 23:55:03.684960 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:03.684848 2574 scope.go:117] "RemoveContainer" containerID="11da6f34441d7f840febb86536326a8d23dee7ce4d8a957674a334bcfae635c2" Sep 10 23:55:03.685973 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:03.685941 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 23:55:03.686051 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:03.686032 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 23:55:03.686099 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:03.686085 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 23:55:04.584356 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:04.584312 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-b19d25-openshift-default-774f4b768-9dmml" Sep 10 23:55:04.694771 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:04.694725 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-b19d25-openshift-default-774f4b768-9dmml" event={"ID":"ac20aa7e-7b99-4b5f-b498-8aeb713566d9","Type":"ContainerStarted","Data":"10d05b53b583659629607bea4de588c05b4177ded91dc486244bcaca3abaa8b5"} Sep 10 23:55:04.879580 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:04.879468 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-b19d25-openshift-default-774f4b768-9dmml" Sep 10 23:55:04.879788 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:04.879685 2574 patch_prober.go:28] interesting pod/e2e-shared-a-w2-b19d25-openshift-default-774f4b768-9dmml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Sep 10 23:55:04.879788 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:04.879740 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-b19d25-openshift-default-774f4b768-9dmml" podUID="ac20aa7e-7b99-4b5f-b498-8aeb713566d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Sep 10 23:55:05.303484 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:05.303444 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-7f598e-openshift-default-97747f75b-tz94v"] Sep 10 23:55:05.304016 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:05.303963 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-7f598e-openshift-default-97747f75b-tz94v" podUID="f7f0cc66-80ee-4f43-8c01-e38ffb30ec32" containerName="istio-proxy" containerID="cri-o://801fe559f65712d9cad406723bb140bf555e8b4482e281bae1f9cba25b583f80" gracePeriod=30 Sep 10 23:55:05.877464 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:05.877430 2574 patch_prober.go:28] interesting pod/e2e-shared-a-w2-b19d25-openshift-default-774f4b768-9dmml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Sep 10 23:55:05.878052 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:05.878017 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-b19d25-openshift-default-774f4b768-9dmml" podUID="ac20aa7e-7b99-4b5f-b498-8aeb713566d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Sep 10 23:55:06.124140 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:06.124108 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-d7a539-openshift-default-7c74c59b78-q7s7n"] Sep 10 23:55:06.124533 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:06.124519 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4cee31c3-1da1-4483-a73c-1d0e23fb8063" containerName="istio-proxy" Sep 10 23:55:06.124533 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:06.124533 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="4cee31c3-1da1-4483-a73c-1d0e23fb8063" containerName="istio-proxy" Sep 10 23:55:06.124700 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:06.124546 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4cee31c3-1da1-4483-a73c-1d0e23fb8063" containerName="istio-proxy" Sep 10 23:55:06.124700 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:06.124551 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="4cee31c3-1da1-4483-a73c-1d0e23fb8063" containerName="istio-proxy" Sep 10 23:55:06.124700 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:06.124571 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1305be23-ac85-4b42-a83f-2b0b8fa9d435" containerName="maas-api" Sep 10 23:55:06.124700 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:06.124577 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="1305be23-ac85-4b42-a83f-2b0b8fa9d435" containerName="maas-api" Sep 10 23:55:06.124700 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:06.124651 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="4cee31c3-1da1-4483-a73c-1d0e23fb8063" containerName="istio-proxy" Sep 10 23:55:06.124700 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:06.124659 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="4cee31c3-1da1-4483-a73c-1d0e23fb8063" containerName="istio-proxy" Sep 10 23:55:06.124700 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:06.124669 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="1305be23-ac85-4b42-a83f-2b0b8fa9d435" containerName="maas-api" Sep 10 23:55:06.127730 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:06.127677 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-d7a539-openshift-default-7c74c59b78-q7s7n" Sep 10 23:55:06.131877 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:06.131851 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-d7a539-openshift-default-dockercfg-n7ffs\"" Sep 10 23:55:06.148001 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:06.147975 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-d7a539-openshift-default-7c74c59b78-q7s7n"] Sep 10 23:55:06.225321 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:06.225279 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f58d38e-bf62-44f6-9522-2ada85451d15-istio-token\") pod \"e2e-shared-a-w3-d7a539-openshift-default-7c74c59b78-q7s7n\" (UID: \"1f58d38e-bf62-44f6-9522-2ada85451d15\") " pod="openshift-ingress/e2e-shared-a-w3-d7a539-openshift-default-7c74c59b78-q7s7n" Sep 10 23:55:06.225550 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:06.225346 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkpvw\" (UniqueName: \"kubernetes.io/projected/1f58d38e-bf62-44f6-9522-2ada85451d15-kube-api-access-bkpvw\") pod \"e2e-shared-a-w3-d7a539-openshift-default-7c74c59b78-q7s7n\" (UID: \"1f58d38e-bf62-44f6-9522-2ada85451d15\") " pod="openshift-ingress/e2e-shared-a-w3-d7a539-openshift-default-7c74c59b78-q7s7n" Sep 10 23:55:06.225550 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:06.225403 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f58d38e-bf62-44f6-9522-2ada85451d15-istio-envoy\") pod \"e2e-shared-a-w3-d7a539-openshift-default-7c74c59b78-q7s7n\" (UID: \"1f58d38e-bf62-44f6-9522-2ada85451d15\") " pod="openshift-ingress/e2e-shared-a-w3-d7a539-openshift-default-7c74c59b78-q7s7n" Sep 10 23:55:06.225550 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:06.225499 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f58d38e-bf62-44f6-9522-2ada85451d15-istio-data\") pod \"e2e-shared-a-w3-d7a539-openshift-default-7c74c59b78-q7s7n\" (UID: \"1f58d38e-bf62-44f6-9522-2ada85451d15\") " pod="openshift-ingress/e2e-shared-a-w3-d7a539-openshift-default-7c74c59b78-q7s7n" Sep 10 23:55:06.225550 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:06.225527 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f58d38e-bf62-44f6-9522-2ada85451d15-workload-socket\") pod \"e2e-shared-a-w3-d7a539-openshift-default-7c74c59b78-q7s7n\" (UID: \"1f58d38e-bf62-44f6-9522-2ada85451d15\") " pod="openshift-ingress/e2e-shared-a-w3-d7a539-openshift-default-7c74c59b78-q7s7n" Sep 10 23:55:06.225902 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:06.225552 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f58d38e-bf62-44f6-9522-2ada85451d15-workload-certs\") pod \"e2e-shared-a-w3-d7a539-openshift-default-7c74c59b78-q7s7n\" (UID: \"1f58d38e-bf62-44f6-9522-2ada85451d15\") " pod="openshift-ingress/e2e-shared-a-w3-d7a539-openshift-default-7c74c59b78-q7s7n" Sep 10 23:55:06.225902 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:06.225593 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f58d38e-bf62-44f6-9522-2ada85451d15-istiod-ca-cert\") pod \"e2e-shared-a-w3-d7a539-openshift-default-7c74c59b78-q7s7n\" (UID: \"1f58d38e-bf62-44f6-9522-2ada85451d15\") " pod="openshift-ingress/e2e-shared-a-w3-d7a539-openshift-default-7c74c59b78-q7s7n" Sep 10 23:55:06.225902 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:06.225714 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f58d38e-bf62-44f6-9522-2ada85451d15-credential-socket\") pod \"e2e-shared-a-w3-d7a539-openshift-default-7c74c59b78-q7s7n\" (UID: \"1f58d38e-bf62-44f6-9522-2ada85451d15\") " pod="openshift-ingress/e2e-shared-a-w3-d7a539-openshift-default-7c74c59b78-q7s7n" Sep 10 23:55:06.225902 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:06.225754 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f58d38e-bf62-44f6-9522-2ada85451d15-istio-podinfo\") pod \"e2e-shared-a-w3-d7a539-openshift-default-7c74c59b78-q7s7n\" (UID: \"1f58d38e-bf62-44f6-9522-2ada85451d15\") " pod="openshift-ingress/e2e-shared-a-w3-d7a539-openshift-default-7c74c59b78-q7s7n" Sep 10 23:55:06.326311 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:06.326276 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f58d38e-bf62-44f6-9522-2ada85451d15-istio-data\") pod \"e2e-shared-a-w3-d7a539-openshift-default-7c74c59b78-q7s7n\" (UID: \"1f58d38e-bf62-44f6-9522-2ada85451d15\") " pod="openshift-ingress/e2e-shared-a-w3-d7a539-openshift-default-7c74c59b78-q7s7n" Sep 10 23:55:06.326311 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:06.326312 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f58d38e-bf62-44f6-9522-2ada85451d15-workload-socket\") pod \"e2e-shared-a-w3-d7a539-openshift-default-7c74c59b78-q7s7n\" (UID: \"1f58d38e-bf62-44f6-9522-2ada85451d15\") " pod="openshift-ingress/e2e-shared-a-w3-d7a539-openshift-default-7c74c59b78-q7s7n" Sep 10 23:55:06.326558 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:06.326338 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f58d38e-bf62-44f6-9522-2ada85451d15-workload-certs\") pod \"e2e-shared-a-w3-d7a539-openshift-default-7c74c59b78-q7s7n\" (UID: \"1f58d38e-bf62-44f6-9522-2ada85451d15\") " pod="openshift-ingress/e2e-shared-a-w3-d7a539-openshift-default-7c74c59b78-q7s7n" Sep 10 23:55:06.326558 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:06.326372 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f58d38e-bf62-44f6-9522-2ada85451d15-istiod-ca-cert\") pod \"e2e-shared-a-w3-d7a539-openshift-default-7c74c59b78-q7s7n\" (UID: \"1f58d38e-bf62-44f6-9522-2ada85451d15\") " pod="openshift-ingress/e2e-shared-a-w3-d7a539-openshift-default-7c74c59b78-q7s7n" Sep 10 23:55:06.326558 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:06.326415 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f58d38e-bf62-44f6-9522-2ada85451d15-credential-socket\") pod \"e2e-shared-a-w3-d7a539-openshift-default-7c74c59b78-q7s7n\" (UID: \"1f58d38e-bf62-44f6-9522-2ada85451d15\") " pod="openshift-ingress/e2e-shared-a-w3-d7a539-openshift-default-7c74c59b78-q7s7n" Sep 10 23:55:06.326558 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:06.326440 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f58d38e-bf62-44f6-9522-2ada85451d15-istio-podinfo\") pod \"e2e-shared-a-w3-d7a539-openshift-default-7c74c59b78-q7s7n\" (UID: \"1f58d38e-bf62-44f6-9522-2ada85451d15\") " pod="openshift-ingress/e2e-shared-a-w3-d7a539-openshift-default-7c74c59b78-q7s7n" Sep 10 23:55:06.326558 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:06.326484 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f58d38e-bf62-44f6-9522-2ada85451d15-istio-token\") pod \"e2e-shared-a-w3-d7a539-openshift-default-7c74c59b78-q7s7n\" (UID: \"1f58d38e-bf62-44f6-9522-2ada85451d15\") " pod="openshift-ingress/e2e-shared-a-w3-d7a539-openshift-default-7c74c59b78-q7s7n" Sep 10 23:55:06.326558 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:06.326523 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkpvw\" (UniqueName: \"kubernetes.io/projected/1f58d38e-bf62-44f6-9522-2ada85451d15-kube-api-access-bkpvw\") pod \"e2e-shared-a-w3-d7a539-openshift-default-7c74c59b78-q7s7n\" (UID: \"1f58d38e-bf62-44f6-9522-2ada85451d15\") " pod="openshift-ingress/e2e-shared-a-w3-d7a539-openshift-default-7c74c59b78-q7s7n" Sep 10 23:55:06.326877 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:06.326576 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f58d38e-bf62-44f6-9522-2ada85451d15-istio-envoy\") pod \"e2e-shared-a-w3-d7a539-openshift-default-7c74c59b78-q7s7n\" (UID: \"1f58d38e-bf62-44f6-9522-2ada85451d15\") " pod="openshift-ingress/e2e-shared-a-w3-d7a539-openshift-default-7c74c59b78-q7s7n" Sep 10 23:55:06.326877 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:06.326701 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f58d38e-bf62-44f6-9522-2ada85451d15-workload-socket\") pod \"e2e-shared-a-w3-d7a539-openshift-default-7c74c59b78-q7s7n\" (UID: \"1f58d38e-bf62-44f6-9522-2ada85451d15\") " pod="openshift-ingress/e2e-shared-a-w3-d7a539-openshift-default-7c74c59b78-q7s7n" Sep 10 23:55:06.326877 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:06.326863 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f58d38e-bf62-44f6-9522-2ada85451d15-workload-certs\") pod \"e2e-shared-a-w3-d7a539-openshift-default-7c74c59b78-q7s7n\" (UID: \"1f58d38e-bf62-44f6-9522-2ada85451d15\") " pod="openshift-ingress/e2e-shared-a-w3-d7a539-openshift-default-7c74c59b78-q7s7n" Sep 10 23:55:06.327053 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:06.327030 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f58d38e-bf62-44f6-9522-2ada85451d15-istio-data\") pod \"e2e-shared-a-w3-d7a539-openshift-default-7c74c59b78-q7s7n\" (UID: \"1f58d38e-bf62-44f6-9522-2ada85451d15\") " pod="openshift-ingress/e2e-shared-a-w3-d7a539-openshift-default-7c74c59b78-q7s7n" Sep 10 23:55:06.327115 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:06.327056 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f58d38e-bf62-44f6-9522-2ada85451d15-credential-socket\") pod \"e2e-shared-a-w3-d7a539-openshift-default-7c74c59b78-q7s7n\" (UID: \"1f58d38e-bf62-44f6-9522-2ada85451d15\") " pod="openshift-ingress/e2e-shared-a-w3-d7a539-openshift-default-7c74c59b78-q7s7n" Sep 10 23:55:06.327369 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:06.327346 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f58d38e-bf62-44f6-9522-2ada85451d15-istiod-ca-cert\") pod \"e2e-shared-a-w3-d7a539-openshift-default-7c74c59b78-q7s7n\" (UID: \"1f58d38e-bf62-44f6-9522-2ada85451d15\") " pod="openshift-ingress/e2e-shared-a-w3-d7a539-openshift-default-7c74c59b78-q7s7n" Sep 10 23:55:06.328786 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:06.328766 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f58d38e-bf62-44f6-9522-2ada85451d15-istio-envoy\") pod \"e2e-shared-a-w3-d7a539-openshift-default-7c74c59b78-q7s7n\" (UID: \"1f58d38e-bf62-44f6-9522-2ada85451d15\") " pod="openshift-ingress/e2e-shared-a-w3-d7a539-openshift-default-7c74c59b78-q7s7n" Sep 10 23:55:06.328922 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:06.328904 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f58d38e-bf62-44f6-9522-2ada85451d15-istio-podinfo\") pod \"e2e-shared-a-w3-d7a539-openshift-default-7c74c59b78-q7s7n\" (UID: \"1f58d38e-bf62-44f6-9522-2ada85451d15\") " pod="openshift-ingress/e2e-shared-a-w3-d7a539-openshift-default-7c74c59b78-q7s7n" Sep 10 23:55:06.334851 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:06.334827 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f58d38e-bf62-44f6-9522-2ada85451d15-istio-token\") pod \"e2e-shared-a-w3-d7a539-openshift-default-7c74c59b78-q7s7n\" (UID: \"1f58d38e-bf62-44f6-9522-2ada85451d15\") " pod="openshift-ingress/e2e-shared-a-w3-d7a539-openshift-default-7c74c59b78-q7s7n" Sep 10 23:55:06.335017 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:06.335002 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkpvw\" (UniqueName: \"kubernetes.io/projected/1f58d38e-bf62-44f6-9522-2ada85451d15-kube-api-access-bkpvw\") pod \"e2e-shared-a-w3-d7a539-openshift-default-7c74c59b78-q7s7n\" (UID: \"1f58d38e-bf62-44f6-9522-2ada85451d15\") " pod="openshift-ingress/e2e-shared-a-w3-d7a539-openshift-default-7c74c59b78-q7s7n" Sep 10 23:55:06.439112 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:06.439033 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-d7a539-openshift-default-7c74c59b78-q7s7n" Sep 10 23:55:06.762383 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:06.762352 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-d7a539-openshift-default-7c74c59b78-q7s7n"] Sep 10 23:55:06.763104 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:55:06.763079 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f58d38e_bf62_44f6_9522_2ada85451d15.slice/crio-006ecb114db21d593d18ff9b9c59b123c339caba50229a9a13667374617c6a3b WatchSource:0}: Error finding container 006ecb114db21d593d18ff9b9c59b123c339caba50229a9a13667374617c6a3b: Status 404 returned error can't find the container with id 006ecb114db21d593d18ff9b9c59b123c339caba50229a9a13667374617c6a3b Sep 10 23:55:06.765334 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:06.765301 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 23:55:06.765408 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:06.765372 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 23:55:06.765408 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:06.765404 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 23:55:06.877826 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:06.877780 2574 patch_prober.go:28] interesting pod/e2e-shared-a-w2-b19d25-openshift-default-774f4b768-9dmml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Sep 10 23:55:06.878310 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:06.877872 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-b19d25-openshift-default-774f4b768-9dmml" podUID="ac20aa7e-7b99-4b5f-b498-8aeb713566d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Sep 10 23:55:07.182488 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:07.182381 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-56r7l"] Sep 10 23:55:07.715204 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:07.715165 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-d7a539-openshift-default-7c74c59b78-q7s7n" event={"ID":"1f58d38e-bf62-44f6-9522-2ada85451d15","Type":"ContainerStarted","Data":"880166dc726b9c67540e69d95a3af0699bcd62e1d21aaf0f71a68df5d9404d7a"} Sep 10 23:55:07.715417 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:07.715222 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-d7a539-openshift-default-7c74c59b78-q7s7n" event={"ID":"1f58d38e-bf62-44f6-9522-2ada85451d15","Type":"ContainerStarted","Data":"006ecb114db21d593d18ff9b9c59b123c339caba50229a9a13667374617c6a3b"} Sep 10 23:55:07.735572 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:07.735523 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-d7a539-openshift-default-7c74c59b78-q7s7n" podStartSLOduration=1.735506645 podStartE2EDuration="1.735506645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:55:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:55:07.733531703 +0000 UTC m=+1099.121134463" watchObservedRunningTime="2026-09-10 23:55:07.735506645 +0000 UTC m=+1099.123109403" Sep 10 23:55:07.877713 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:07.877675 2574 patch_prober.go:28] interesting pod/e2e-shared-a-w2-b19d25-openshift-default-774f4b768-9dmml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Sep 10 23:55:07.877971 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:07.877944 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-b19d25-openshift-default-774f4b768-9dmml" podUID="ac20aa7e-7b99-4b5f-b498-8aeb713566d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Sep 10 23:55:08.439919 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:08.439884 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-d7a539-openshift-default-7c74c59b78-q7s7n" Sep 10 23:55:08.444910 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:08.444883 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-d7a539-openshift-default-7c74c59b78-q7s7n" Sep 10 23:55:08.721366 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:08.721338 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-d7a539-openshift-default-7c74c59b78-q7s7n" Sep 10 23:55:08.722173 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:08.722156 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-d7a539-openshift-default-7c74c59b78-q7s7n" Sep 10 23:55:08.877380 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:08.877345 2574 patch_prober.go:28] interesting pod/e2e-shared-a-w2-b19d25-openshift-default-774f4b768-9dmml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Sep 10 23:55:08.877656 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:08.877606 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-b19d25-openshift-default-774f4b768-9dmml" podUID="ac20aa7e-7b99-4b5f-b498-8aeb713566d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Sep 10 23:55:09.881507 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:09.881480 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-b19d25-openshift-default-774f4b768-9dmml" Sep 10 23:55:09.881897 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:09.881748 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-b19d25-openshift-default-774f4b768-9dmml" Sep 10 23:55:09.882450 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:09.882433 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-b19d25-openshift-default-774f4b768-9dmml" Sep 10 23:55:10.564929 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:10.564907 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7f598e-openshift-default-97747f75b-tz94v" Sep 10 23:55:10.670613 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:10.670542 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7f0cc66-80ee-4f43-8c01-e38ffb30ec32-workload-certs\") pod \"f7f0cc66-80ee-4f43-8c01-e38ffb30ec32\" (UID: \"f7f0cc66-80ee-4f43-8c01-e38ffb30ec32\") " Sep 10 23:55:10.670613 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:10.670571 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7f0cc66-80ee-4f43-8c01-e38ffb30ec32-istio-data\") pod \"f7f0cc66-80ee-4f43-8c01-e38ffb30ec32\" (UID: \"f7f0cc66-80ee-4f43-8c01-e38ffb30ec32\") " Sep 10 23:55:10.670613 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:10.670587 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7f0cc66-80ee-4f43-8c01-e38ffb30ec32-istio-envoy\") pod \"f7f0cc66-80ee-4f43-8c01-e38ffb30ec32\" (UID: \"f7f0cc66-80ee-4f43-8c01-e38ffb30ec32\") " Sep 10 23:55:10.670860 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:10.670618 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7f0cc66-80ee-4f43-8c01-e38ffb30ec32-istiod-ca-cert\") pod \"f7f0cc66-80ee-4f43-8c01-e38ffb30ec32\" (UID: \"f7f0cc66-80ee-4f43-8c01-e38ffb30ec32\") " Sep 10 23:55:10.670860 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:10.670691 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7f0cc66-80ee-4f43-8c01-e38ffb30ec32-credential-socket\") pod \"f7f0cc66-80ee-4f43-8c01-e38ffb30ec32\" (UID: \"f7f0cc66-80ee-4f43-8c01-e38ffb30ec32\") " Sep 10 23:55:10.670860 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:10.670706 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7f0cc66-80ee-4f43-8c01-e38ffb30ec32-workload-socket\") pod \"f7f0cc66-80ee-4f43-8c01-e38ffb30ec32\" (UID: \"f7f0cc66-80ee-4f43-8c01-e38ffb30ec32\") " Sep 10 23:55:10.670860 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:10.670735 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7f0cc66-80ee-4f43-8c01-e38ffb30ec32-istio-podinfo\") pod \"f7f0cc66-80ee-4f43-8c01-e38ffb30ec32\" (UID: \"f7f0cc66-80ee-4f43-8c01-e38ffb30ec32\") " Sep 10 23:55:10.670860 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:10.670765 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8tsjx\" (UniqueName: \"kubernetes.io/projected/f7f0cc66-80ee-4f43-8c01-e38ffb30ec32-kube-api-access-8tsjx\") pod \"f7f0cc66-80ee-4f43-8c01-e38ffb30ec32\" (UID: \"f7f0cc66-80ee-4f43-8c01-e38ffb30ec32\") " Sep 10 23:55:10.670860 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:10.670819 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7f0cc66-80ee-4f43-8c01-e38ffb30ec32-istio-token\") pod \"f7f0cc66-80ee-4f43-8c01-e38ffb30ec32\" (UID: \"f7f0cc66-80ee-4f43-8c01-e38ffb30ec32\") " Sep 10 23:55:10.671164 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:10.670853 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7f0cc66-80ee-4f43-8c01-e38ffb30ec32-istio-data" (OuterVolumeSpecName: "istio-data") pod "f7f0cc66-80ee-4f43-8c01-e38ffb30ec32" (UID: "f7f0cc66-80ee-4f43-8c01-e38ffb30ec32"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:55:10.671164 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:10.670962 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7f0cc66-80ee-4f43-8c01-e38ffb30ec32-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f7f0cc66-80ee-4f43-8c01-e38ffb30ec32" (UID: "f7f0cc66-80ee-4f43-8c01-e38ffb30ec32"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:55:10.671164 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:10.671052 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7f0cc66-80ee-4f43-8c01-e38ffb30ec32-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f7f0cc66-80ee-4f43-8c01-e38ffb30ec32" (UID: "f7f0cc66-80ee-4f43-8c01-e38ffb30ec32"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:55:10.671164 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:10.671122 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7f0cc66-80ee-4f43-8c01-e38ffb30ec32-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f7f0cc66-80ee-4f43-8c01-e38ffb30ec32" (UID: "f7f0cc66-80ee-4f43-8c01-e38ffb30ec32"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:55:10.671341 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:10.671269 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7f0cc66-80ee-4f43-8c01-e38ffb30ec32-workload-certs\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:55:10.671341 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:10.671288 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7f0cc66-80ee-4f43-8c01-e38ffb30ec32-istio-data\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:55:10.671341 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:10.671302 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7f0cc66-80ee-4f43-8c01-e38ffb30ec32-credential-socket\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:55:10.671341 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:10.671317 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7f0cc66-80ee-4f43-8c01-e38ffb30ec32-workload-socket\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:55:10.671341 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:10.671266 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7f0cc66-80ee-4f43-8c01-e38ffb30ec32-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f7f0cc66-80ee-4f43-8c01-e38ffb30ec32" (UID: "f7f0cc66-80ee-4f43-8c01-e38ffb30ec32"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 23:55:10.672936 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:10.672903 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7f0cc66-80ee-4f43-8c01-e38ffb30ec32-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f7f0cc66-80ee-4f43-8c01-e38ffb30ec32" (UID: "f7f0cc66-80ee-4f43-8c01-e38ffb30ec32"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:55:10.673044 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:10.672940 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f7f0cc66-80ee-4f43-8c01-e38ffb30ec32-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f7f0cc66-80ee-4f43-8c01-e38ffb30ec32" (UID: "f7f0cc66-80ee-4f43-8c01-e38ffb30ec32"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 23:55:10.673044 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:10.672965 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7f0cc66-80ee-4f43-8c01-e38ffb30ec32-kube-api-access-8tsjx" (OuterVolumeSpecName: "kube-api-access-8tsjx") pod "f7f0cc66-80ee-4f43-8c01-e38ffb30ec32" (UID: "f7f0cc66-80ee-4f43-8c01-e38ffb30ec32"). InnerVolumeSpecName "kube-api-access-8tsjx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 23:55:10.673146 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:10.673130 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7f0cc66-80ee-4f43-8c01-e38ffb30ec32-istio-token" (OuterVolumeSpecName: "istio-token") pod "f7f0cc66-80ee-4f43-8c01-e38ffb30ec32" (UID: "f7f0cc66-80ee-4f43-8c01-e38ffb30ec32"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 23:55:10.733422 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:10.733386 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d7a539-fb5d4d5c6-nn6gl"] Sep 10 23:55:10.734072 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:10.734049 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7f0cc66-80ee-4f43-8c01-e38ffb30ec32" containerName="istio-proxy" Sep 10 23:55:10.734181 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:10.734074 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7f0cc66-80ee-4f43-8c01-e38ffb30ec32" containerName="istio-proxy" Sep 10 23:55:10.734236 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:10.734218 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7f0cc66-80ee-4f43-8c01-e38ffb30ec32" containerName="istio-proxy" Sep 10 23:55:10.737895 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:10.737874 2574 generic.go:356] "Generic (PLEG): container finished" podID="f7f0cc66-80ee-4f43-8c01-e38ffb30ec32" containerID="801fe559f65712d9cad406723bb140bf555e8b4482e281bae1f9cba25b583f80" exitCode=0 Sep 10 23:55:10.738278 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:10.738247 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7f598e-openshift-default-97747f75b-tz94v" Sep 10 23:55:10.738832 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:10.738807 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7f598e-openshift-default-97747f75b-tz94v" event={"ID":"f7f0cc66-80ee-4f43-8c01-e38ffb30ec32","Type":"ContainerDied","Data":"801fe559f65712d9cad406723bb140bf555e8b4482e281bae1f9cba25b583f80"} Sep 10 23:55:10.738919 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:10.738864 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7f598e-openshift-default-97747f75b-tz94v" event={"ID":"f7f0cc66-80ee-4f43-8c01-e38ffb30ec32","Type":"ContainerDied","Data":"37f2ea978d0d55cfcc5d306780844f7fbf8aa85192f1732dff0e7b4009f6bbbb"} Sep 10 23:55:10.738919 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:10.738896 2574 scope.go:117] "RemoveContainer" containerID="801fe559f65712d9cad406723bb140bf555e8b4482e281bae1f9cba25b583f80" Sep 10 23:55:10.738919 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:10.738819 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d7a539-fb5d4d5c6-nn6gl" Sep 10 23:55:10.740910 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:10.740894 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-d7a539-serving-cert\"" Sep 10 23:55:10.741071 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:10.741055 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-9tsnw\"" Sep 10 23:55:10.741170 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:10.741076 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 23:55:10.741372 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:10.741354 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 10 23:55:10.741481 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:10.741358 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 10 23:55:10.748673 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:10.748625 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d7a539-fb5d4d5c6-nn6gl"] Sep 10 23:55:10.755175 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:10.755158 2574 scope.go:117] "RemoveContainer" containerID="9344c7ba58e4214c7d1716bd492314f72bf2bb398eb77fd28c2d9b4d0891ec5a" Sep 10 23:55:10.763916 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:10.763896 2574 scope.go:117] "RemoveContainer" containerID="801fe559f65712d9cad406723bb140bf555e8b4482e281bae1f9cba25b583f80" Sep 10 23:55:10.764215 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:55:10.764197 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"801fe559f65712d9cad406723bb140bf555e8b4482e281bae1f9cba25b583f80\": container with ID starting with 801fe559f65712d9cad406723bb140bf555e8b4482e281bae1f9cba25b583f80 not found: ID does not exist" containerID="801fe559f65712d9cad406723bb140bf555e8b4482e281bae1f9cba25b583f80" Sep 10 23:55:10.764289 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:10.764227 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"801fe559f65712d9cad406723bb140bf555e8b4482e281bae1f9cba25b583f80"} err="failed to get container status \"801fe559f65712d9cad406723bb140bf555e8b4482e281bae1f9cba25b583f80\": rpc error: code = NotFound desc = could not find container \"801fe559f65712d9cad406723bb140bf555e8b4482e281bae1f9cba25b583f80\": container with ID starting with 801fe559f65712d9cad406723bb140bf555e8b4482e281bae1f9cba25b583f80 not found: ID does not exist" Sep 10 23:55:10.764289 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:10.764252 2574 scope.go:117] "RemoveContainer" containerID="9344c7ba58e4214c7d1716bd492314f72bf2bb398eb77fd28c2d9b4d0891ec5a" Sep 10 23:55:10.764567 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:55:10.764538 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9344c7ba58e4214c7d1716bd492314f72bf2bb398eb77fd28c2d9b4d0891ec5a\": container with ID starting with 9344c7ba58e4214c7d1716bd492314f72bf2bb398eb77fd28c2d9b4d0891ec5a not found: ID does not exist" containerID="9344c7ba58e4214c7d1716bd492314f72bf2bb398eb77fd28c2d9b4d0891ec5a" Sep 10 23:55:10.764681 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:10.764568 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9344c7ba58e4214c7d1716bd492314f72bf2bb398eb77fd28c2d9b4d0891ec5a"} err="failed to get container status \"9344c7ba58e4214c7d1716bd492314f72bf2bb398eb77fd28c2d9b4d0891ec5a\": rpc error: code = NotFound desc = could not find container \"9344c7ba58e4214c7d1716bd492314f72bf2bb398eb77fd28c2d9b4d0891ec5a\": container with ID starting with 9344c7ba58e4214c7d1716bd492314f72bf2bb398eb77fd28c2d9b4d0891ec5a not found: ID does not exist" Sep 10 23:55:10.771961 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:10.771932 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7f0cc66-80ee-4f43-8c01-e38ffb30ec32-istiod-ca-cert\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:55:10.772041 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:10.771964 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7f0cc66-80ee-4f43-8c01-e38ffb30ec32-istio-podinfo\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:55:10.772041 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:10.771980 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tsjx\" (UniqueName: \"kubernetes.io/projected/f7f0cc66-80ee-4f43-8c01-e38ffb30ec32-kube-api-access-8tsjx\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:55:10.772041 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:10.771994 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7f0cc66-80ee-4f43-8c01-e38ffb30ec32-istio-token\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:55:10.772041 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:10.772008 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7f0cc66-80ee-4f43-8c01-e38ffb30ec32-istio-envoy\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:55:10.786451 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:10.786429 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-7f598e-openshift-default-97747f75b-tz94v"] Sep 10 23:55:10.789962 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:10.789942 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-7f598e-openshift-default-97747f75b-tz94v"] Sep 10 23:55:10.873437 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:10.873402 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d8b9058-927e-4b3c-aec9-c94e233d7253-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-d7a539-fb5d4d5c6-nn6gl\" (UID: \"9d8b9058-927e-4b3c-aec9-c94e233d7253\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d7a539-fb5d4d5c6-nn6gl" Sep 10 23:55:10.873623 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:10.873551 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whvbg\" (UniqueName: \"kubernetes.io/projected/9d8b9058-927e-4b3c-aec9-c94e233d7253-kube-api-access-whvbg\") pod \"maas-api-e2e-shared-a-w3-d7a539-fb5d4d5c6-nn6gl\" (UID: \"9d8b9058-927e-4b3c-aec9-c94e233d7253\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d7a539-fb5d4d5c6-nn6gl" Sep 10 23:55:10.873623 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:10.873607 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d8b9058-927e-4b3c-aec9-c94e233d7253-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-d7a539-fb5d4d5c6-nn6gl\" (UID: \"9d8b9058-927e-4b3c-aec9-c94e233d7253\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d7a539-fb5d4d5c6-nn6gl" Sep 10 23:55:10.873858 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:10.873837 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d8b9058-927e-4b3c-aec9-c94e233d7253-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-d7a539-fb5d4d5c6-nn6gl\" (UID: \"9d8b9058-927e-4b3c-aec9-c94e233d7253\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d7a539-fb5d4d5c6-nn6gl" Sep 10 23:55:10.975070 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:10.975041 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d8b9058-927e-4b3c-aec9-c94e233d7253-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-d7a539-fb5d4d5c6-nn6gl\" (UID: \"9d8b9058-927e-4b3c-aec9-c94e233d7253\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d7a539-fb5d4d5c6-nn6gl" Sep 10 23:55:10.975400 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:10.975102 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d8b9058-927e-4b3c-aec9-c94e233d7253-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-d7a539-fb5d4d5c6-nn6gl\" (UID: \"9d8b9058-927e-4b3c-aec9-c94e233d7253\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d7a539-fb5d4d5c6-nn6gl" Sep 10 23:55:10.975400 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:10.975149 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whvbg\" (UniqueName: \"kubernetes.io/projected/9d8b9058-927e-4b3c-aec9-c94e233d7253-kube-api-access-whvbg\") pod \"maas-api-e2e-shared-a-w3-d7a539-fb5d4d5c6-nn6gl\" (UID: \"9d8b9058-927e-4b3c-aec9-c94e233d7253\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d7a539-fb5d4d5c6-nn6gl" Sep 10 23:55:10.975400 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:10.975172 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d8b9058-927e-4b3c-aec9-c94e233d7253-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-d7a539-fb5d4d5c6-nn6gl\" (UID: \"9d8b9058-927e-4b3c-aec9-c94e233d7253\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d7a539-fb5d4d5c6-nn6gl" Sep 10 23:55:10.975400 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:55:10.975266 2574 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d7a539-serving-cert: secret "maas-api-e2e-shared-a-w3-d7a539-serving-cert" not found Sep 10 23:55:10.975400 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:55:10.975345 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d8b9058-927e-4b3c-aec9-c94e233d7253-maas-api-tls podName:9d8b9058-927e-4b3c-aec9-c94e233d7253 nodeName:}" failed. No retries permitted until 2026-09-10 23:55:11.475324953 +0000 UTC m=+1102.862927694 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9d8b9058-927e-4b3c-aec9-c94e233d7253-maas-api-tls") pod "maas-api-e2e-shared-a-w3-d7a539-fb5d4d5c6-nn6gl" (UID: "9d8b9058-927e-4b3c-aec9-c94e233d7253") : secret "maas-api-e2e-shared-a-w3-d7a539-serving-cert" not found Sep 10 23:55:10.975816 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:10.975798 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d8b9058-927e-4b3c-aec9-c94e233d7253-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-d7a539-fb5d4d5c6-nn6gl\" (UID: \"9d8b9058-927e-4b3c-aec9-c94e233d7253\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d7a539-fb5d4d5c6-nn6gl" Sep 10 23:55:10.978281 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:10.978259 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d8b9058-927e-4b3c-aec9-c94e233d7253-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-d7a539-fb5d4d5c6-nn6gl\" (UID: \"9d8b9058-927e-4b3c-aec9-c94e233d7253\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d7a539-fb5d4d5c6-nn6gl" Sep 10 23:55:10.985960 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:10.985935 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-whvbg\" (UniqueName: \"kubernetes.io/projected/9d8b9058-927e-4b3c-aec9-c94e233d7253-kube-api-access-whvbg\") pod \"maas-api-e2e-shared-a-w3-d7a539-fb5d4d5c6-nn6gl\" (UID: \"9d8b9058-927e-4b3c-aec9-c94e233d7253\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d7a539-fb5d4d5c6-nn6gl" Sep 10 23:55:11.094329 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:11.094305 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7f0cc66-80ee-4f43-8c01-e38ffb30ec32" path="/var/lib/kubelet/pods/f7f0cc66-80ee-4f43-8c01-e38ffb30ec32/volumes" Sep 10 23:55:11.480456 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:11.480417 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d8b9058-927e-4b3c-aec9-c94e233d7253-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-d7a539-fb5d4d5c6-nn6gl\" (UID: \"9d8b9058-927e-4b3c-aec9-c94e233d7253\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d7a539-fb5d4d5c6-nn6gl" Sep 10 23:55:11.482767 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:11.482740 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d8b9058-927e-4b3c-aec9-c94e233d7253-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-d7a539-fb5d4d5c6-nn6gl\" (UID: \"9d8b9058-927e-4b3c-aec9-c94e233d7253\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d7a539-fb5d4d5c6-nn6gl" Sep 10 23:55:11.655190 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:11.655148 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d7a539-fb5d4d5c6-nn6gl" Sep 10 23:55:11.940745 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:11.940715 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d7a539-fb5d4d5c6-nn6gl"] Sep 10 23:55:11.942527 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:55:11.942499 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d8b9058_927e_4b3c_aec9_c94e233d7253.slice/crio-3a8298ebb9e73abfd5d0cdcfaa45753e1f2298cfca07a43d301f8a9cab8335d4 WatchSource:0}: Error finding container 3a8298ebb9e73abfd5d0cdcfaa45753e1f2298cfca07a43d301f8a9cab8335d4: Status 404 returned error can't find the container with id 3a8298ebb9e73abfd5d0cdcfaa45753e1f2298cfca07a43d301f8a9cab8335d4 Sep 10 23:55:11.944123 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:11.944102 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 23:55:12.752559 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:12.752527 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d7a539-fb5d4d5c6-nn6gl" event={"ID":"9d8b9058-927e-4b3c-aec9-c94e233d7253","Type":"ContainerStarted","Data":"06217fcc15fbe098f591cce5ecbf875de15263f89866a709a444b6059996918b"} Sep 10 23:55:12.752958 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:12.752579 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d7a539-fb5d4d5c6-nn6gl" event={"ID":"9d8b9058-927e-4b3c-aec9-c94e233d7253","Type":"ContainerStarted","Data":"3a8298ebb9e73abfd5d0cdcfaa45753e1f2298cfca07a43d301f8a9cab8335d4"} Sep 10 23:55:12.752958 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:12.752656 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d7a539-fb5d4d5c6-nn6gl" Sep 10 23:55:12.774933 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:12.774883 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d7a539-fb5d4d5c6-nn6gl" podStartSLOduration=2.442313962 podStartE2EDuration="2.774868194s" totalImagesPullingTime="332.554232ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:55:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:55:12.774009158 +0000 UTC m=+1104.161611916" watchObservedRunningTime="2026-09-10 23:55:12.774868194 +0000 UTC m=+1104.162470954" Sep 10 23:55:15.023450 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:15.023344 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-a78403-openshift-default-6b9d4896d6-cs5dw"] Sep 10 23:55:15.024229 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:15.024200 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7f0cc66-80ee-4f43-8c01-e38ffb30ec32" containerName="istio-proxy" Sep 10 23:55:15.024229 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:15.024225 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7f0cc66-80ee-4f43-8c01-e38ffb30ec32" containerName="istio-proxy" Sep 10 23:55:15.024399 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:15.024368 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7f0cc66-80ee-4f43-8c01-e38ffb30ec32" containerName="istio-proxy" Sep 10 23:55:15.027602 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:15.027583 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-a78403-openshift-default-6b9d4896d6-cs5dw" Sep 10 23:55:15.029748 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:15.029726 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-a78403-openshift-default-dockercfg-nlnf8\"" Sep 10 23:55:15.039973 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:15.039951 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-a78403-openshift-default-6b9d4896d6-cs5dw"] Sep 10 23:55:15.118153 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:15.118121 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c52960ab-0f37-4e17-a6e0-bbc346e0aa38-istio-data\") pod \"e2e-shared-b-w2-a78403-openshift-default-6b9d4896d6-cs5dw\" (UID: \"c52960ab-0f37-4e17-a6e0-bbc346e0aa38\") " pod="openshift-ingress/e2e-shared-b-w2-a78403-openshift-default-6b9d4896d6-cs5dw" Sep 10 23:55:15.118153 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:15.118155 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c52960ab-0f37-4e17-a6e0-bbc346e0aa38-istio-token\") pod \"e2e-shared-b-w2-a78403-openshift-default-6b9d4896d6-cs5dw\" (UID: \"c52960ab-0f37-4e17-a6e0-bbc346e0aa38\") " pod="openshift-ingress/e2e-shared-b-w2-a78403-openshift-default-6b9d4896d6-cs5dw" Sep 10 23:55:15.118361 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:15.118198 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c52960ab-0f37-4e17-a6e0-bbc346e0aa38-workload-socket\") pod \"e2e-shared-b-w2-a78403-openshift-default-6b9d4896d6-cs5dw\" (UID: \"c52960ab-0f37-4e17-a6e0-bbc346e0aa38\") " pod="openshift-ingress/e2e-shared-b-w2-a78403-openshift-default-6b9d4896d6-cs5dw" Sep 10 23:55:15.118361 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:15.118263 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c52960ab-0f37-4e17-a6e0-bbc346e0aa38-istiod-ca-cert\") pod \"e2e-shared-b-w2-a78403-openshift-default-6b9d4896d6-cs5dw\" (UID: \"c52960ab-0f37-4e17-a6e0-bbc346e0aa38\") " pod="openshift-ingress/e2e-shared-b-w2-a78403-openshift-default-6b9d4896d6-cs5dw" Sep 10 23:55:15.118361 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:15.118329 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c52960ab-0f37-4e17-a6e0-bbc346e0aa38-workload-certs\") pod \"e2e-shared-b-w2-a78403-openshift-default-6b9d4896d6-cs5dw\" (UID: \"c52960ab-0f37-4e17-a6e0-bbc346e0aa38\") " pod="openshift-ingress/e2e-shared-b-w2-a78403-openshift-default-6b9d4896d6-cs5dw" Sep 10 23:55:15.118477 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:15.118442 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c52960ab-0f37-4e17-a6e0-bbc346e0aa38-credential-socket\") pod \"e2e-shared-b-w2-a78403-openshift-default-6b9d4896d6-cs5dw\" (UID: \"c52960ab-0f37-4e17-a6e0-bbc346e0aa38\") " pod="openshift-ingress/e2e-shared-b-w2-a78403-openshift-default-6b9d4896d6-cs5dw" Sep 10 23:55:15.118533 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:15.118477 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2txbs\" (UniqueName: \"kubernetes.io/projected/c52960ab-0f37-4e17-a6e0-bbc346e0aa38-kube-api-access-2txbs\") pod \"e2e-shared-b-w2-a78403-openshift-default-6b9d4896d6-cs5dw\" (UID: \"c52960ab-0f37-4e17-a6e0-bbc346e0aa38\") " pod="openshift-ingress/e2e-shared-b-w2-a78403-openshift-default-6b9d4896d6-cs5dw" Sep 10 23:55:15.118533 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:15.118513 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c52960ab-0f37-4e17-a6e0-bbc346e0aa38-istio-envoy\") pod \"e2e-shared-b-w2-a78403-openshift-default-6b9d4896d6-cs5dw\" (UID: \"c52960ab-0f37-4e17-a6e0-bbc346e0aa38\") " pod="openshift-ingress/e2e-shared-b-w2-a78403-openshift-default-6b9d4896d6-cs5dw" Sep 10 23:55:15.118672 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:15.118537 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c52960ab-0f37-4e17-a6e0-bbc346e0aa38-istio-podinfo\") pod \"e2e-shared-b-w2-a78403-openshift-default-6b9d4896d6-cs5dw\" (UID: \"c52960ab-0f37-4e17-a6e0-bbc346e0aa38\") " pod="openshift-ingress/e2e-shared-b-w2-a78403-openshift-default-6b9d4896d6-cs5dw" Sep 10 23:55:15.219725 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:15.219674 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c52960ab-0f37-4e17-a6e0-bbc346e0aa38-credential-socket\") pod \"e2e-shared-b-w2-a78403-openshift-default-6b9d4896d6-cs5dw\" (UID: \"c52960ab-0f37-4e17-a6e0-bbc346e0aa38\") " pod="openshift-ingress/e2e-shared-b-w2-a78403-openshift-default-6b9d4896d6-cs5dw" Sep 10 23:55:15.219891 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:15.219741 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2txbs\" (UniqueName: \"kubernetes.io/projected/c52960ab-0f37-4e17-a6e0-bbc346e0aa38-kube-api-access-2txbs\") pod \"e2e-shared-b-w2-a78403-openshift-default-6b9d4896d6-cs5dw\" (UID: \"c52960ab-0f37-4e17-a6e0-bbc346e0aa38\") " pod="openshift-ingress/e2e-shared-b-w2-a78403-openshift-default-6b9d4896d6-cs5dw" Sep 10 23:55:15.219891 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:15.219778 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c52960ab-0f37-4e17-a6e0-bbc346e0aa38-istio-envoy\") pod \"e2e-shared-b-w2-a78403-openshift-default-6b9d4896d6-cs5dw\" (UID: \"c52960ab-0f37-4e17-a6e0-bbc346e0aa38\") " pod="openshift-ingress/e2e-shared-b-w2-a78403-openshift-default-6b9d4896d6-cs5dw" Sep 10 23:55:15.219891 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:15.219810 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c52960ab-0f37-4e17-a6e0-bbc346e0aa38-istio-podinfo\") pod \"e2e-shared-b-w2-a78403-openshift-default-6b9d4896d6-cs5dw\" (UID: \"c52960ab-0f37-4e17-a6e0-bbc346e0aa38\") " pod="openshift-ingress/e2e-shared-b-w2-a78403-openshift-default-6b9d4896d6-cs5dw" Sep 10 23:55:15.219891 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:15.219861 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c52960ab-0f37-4e17-a6e0-bbc346e0aa38-istio-data\") pod \"e2e-shared-b-w2-a78403-openshift-default-6b9d4896d6-cs5dw\" (UID: \"c52960ab-0f37-4e17-a6e0-bbc346e0aa38\") " pod="openshift-ingress/e2e-shared-b-w2-a78403-openshift-default-6b9d4896d6-cs5dw" Sep 10 23:55:15.219891 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:15.219882 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c52960ab-0f37-4e17-a6e0-bbc346e0aa38-istio-token\") pod \"e2e-shared-b-w2-a78403-openshift-default-6b9d4896d6-cs5dw\" (UID: \"c52960ab-0f37-4e17-a6e0-bbc346e0aa38\") " pod="openshift-ingress/e2e-shared-b-w2-a78403-openshift-default-6b9d4896d6-cs5dw" Sep 10 23:55:15.220173 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:15.219932 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c52960ab-0f37-4e17-a6e0-bbc346e0aa38-workload-socket\") pod \"e2e-shared-b-w2-a78403-openshift-default-6b9d4896d6-cs5dw\" (UID: \"c52960ab-0f37-4e17-a6e0-bbc346e0aa38\") " pod="openshift-ingress/e2e-shared-b-w2-a78403-openshift-default-6b9d4896d6-cs5dw" Sep 10 23:55:15.220173 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:15.219961 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c52960ab-0f37-4e17-a6e0-bbc346e0aa38-istiod-ca-cert\") pod \"e2e-shared-b-w2-a78403-openshift-default-6b9d4896d6-cs5dw\" (UID: \"c52960ab-0f37-4e17-a6e0-bbc346e0aa38\") " pod="openshift-ingress/e2e-shared-b-w2-a78403-openshift-default-6b9d4896d6-cs5dw" Sep 10 23:55:15.220173 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:15.220014 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c52960ab-0f37-4e17-a6e0-bbc346e0aa38-workload-certs\") pod \"e2e-shared-b-w2-a78403-openshift-default-6b9d4896d6-cs5dw\" (UID: \"c52960ab-0f37-4e17-a6e0-bbc346e0aa38\") " pod="openshift-ingress/e2e-shared-b-w2-a78403-openshift-default-6b9d4896d6-cs5dw" Sep 10 23:55:15.220332 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:15.220256 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c52960ab-0f37-4e17-a6e0-bbc346e0aa38-istio-data\") pod \"e2e-shared-b-w2-a78403-openshift-default-6b9d4896d6-cs5dw\" (UID: \"c52960ab-0f37-4e17-a6e0-bbc346e0aa38\") " pod="openshift-ingress/e2e-shared-b-w2-a78403-openshift-default-6b9d4896d6-cs5dw" Sep 10 23:55:15.220390 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:15.220343 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c52960ab-0f37-4e17-a6e0-bbc346e0aa38-workload-certs\") pod \"e2e-shared-b-w2-a78403-openshift-default-6b9d4896d6-cs5dw\" (UID: \"c52960ab-0f37-4e17-a6e0-bbc346e0aa38\") " pod="openshift-ingress/e2e-shared-b-w2-a78403-openshift-default-6b9d4896d6-cs5dw" Sep 10 23:55:15.220437 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:15.220401 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c52960ab-0f37-4e17-a6e0-bbc346e0aa38-workload-socket\") pod \"e2e-shared-b-w2-a78403-openshift-default-6b9d4896d6-cs5dw\" (UID: \"c52960ab-0f37-4e17-a6e0-bbc346e0aa38\") " pod="openshift-ingress/e2e-shared-b-w2-a78403-openshift-default-6b9d4896d6-cs5dw" Sep 10 23:55:15.220522 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:15.220503 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c52960ab-0f37-4e17-a6e0-bbc346e0aa38-credential-socket\") pod \"e2e-shared-b-w2-a78403-openshift-default-6b9d4896d6-cs5dw\" (UID: \"c52960ab-0f37-4e17-a6e0-bbc346e0aa38\") " pod="openshift-ingress/e2e-shared-b-w2-a78403-openshift-default-6b9d4896d6-cs5dw" Sep 10 23:55:15.220700 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:15.220683 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c52960ab-0f37-4e17-a6e0-bbc346e0aa38-istiod-ca-cert\") pod \"e2e-shared-b-w2-a78403-openshift-default-6b9d4896d6-cs5dw\" (UID: \"c52960ab-0f37-4e17-a6e0-bbc346e0aa38\") " pod="openshift-ingress/e2e-shared-b-w2-a78403-openshift-default-6b9d4896d6-cs5dw" Sep 10 23:55:15.222548 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:15.222521 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c52960ab-0f37-4e17-a6e0-bbc346e0aa38-istio-envoy\") pod \"e2e-shared-b-w2-a78403-openshift-default-6b9d4896d6-cs5dw\" (UID: \"c52960ab-0f37-4e17-a6e0-bbc346e0aa38\") " pod="openshift-ingress/e2e-shared-b-w2-a78403-openshift-default-6b9d4896d6-cs5dw" Sep 10 23:55:15.222697 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:15.222681 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c52960ab-0f37-4e17-a6e0-bbc346e0aa38-istio-podinfo\") pod \"e2e-shared-b-w2-a78403-openshift-default-6b9d4896d6-cs5dw\" (UID: \"c52960ab-0f37-4e17-a6e0-bbc346e0aa38\") " pod="openshift-ingress/e2e-shared-b-w2-a78403-openshift-default-6b9d4896d6-cs5dw" Sep 10 23:55:15.228312 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:15.228291 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2txbs\" (UniqueName: \"kubernetes.io/projected/c52960ab-0f37-4e17-a6e0-bbc346e0aa38-kube-api-access-2txbs\") pod \"e2e-shared-b-w2-a78403-openshift-default-6b9d4896d6-cs5dw\" (UID: \"c52960ab-0f37-4e17-a6e0-bbc346e0aa38\") " pod="openshift-ingress/e2e-shared-b-w2-a78403-openshift-default-6b9d4896d6-cs5dw" Sep 10 23:55:15.228421 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:15.228327 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c52960ab-0f37-4e17-a6e0-bbc346e0aa38-istio-token\") pod \"e2e-shared-b-w2-a78403-openshift-default-6b9d4896d6-cs5dw\" (UID: \"c52960ab-0f37-4e17-a6e0-bbc346e0aa38\") " pod="openshift-ingress/e2e-shared-b-w2-a78403-openshift-default-6b9d4896d6-cs5dw" Sep 10 23:55:15.338089 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:15.338014 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-a78403-openshift-default-6b9d4896d6-cs5dw" Sep 10 23:55:15.643083 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:15.643054 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-a78403-openshift-default-6b9d4896d6-cs5dw"] Sep 10 23:55:15.644249 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:55:15.644222 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc52960ab_0f37_4e17_a6e0_bbc346e0aa38.slice/crio-5ebafcb983daf8038e04df3228bf7d8d35d89bb7990759fc3ad156683d002c9f WatchSource:0}: Error finding container 5ebafcb983daf8038e04df3228bf7d8d35d89bb7990759fc3ad156683d002c9f: Status 404 returned error can't find the container with id 5ebafcb983daf8038e04df3228bf7d8d35d89bb7990759fc3ad156683d002c9f Sep 10 23:55:15.646697 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:15.646630 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 23:55:15.646792 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:15.646745 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 23:55:15.646842 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:15.646792 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 23:55:15.785665 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:15.782291 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-a78403-openshift-default-6b9d4896d6-cs5dw" event={"ID":"c52960ab-0f37-4e17-a6e0-bbc346e0aa38","Type":"ContainerStarted","Data":"2b4f71e50e35396a6bd685ad9e0011b3d726c10ebb764307ae0d43b2e2e71c66"} Sep 10 23:55:15.785665 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:15.782360 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-a78403-openshift-default-6b9d4896d6-cs5dw" event={"ID":"c52960ab-0f37-4e17-a6e0-bbc346e0aa38","Type":"ContainerStarted","Data":"5ebafcb983daf8038e04df3228bf7d8d35d89bb7990759fc3ad156683d002c9f"} Sep 10 23:55:15.806743 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:15.806668 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-a78403-openshift-default-6b9d4896d6-cs5dw" podStartSLOduration=0.80662998 podStartE2EDuration="806.62998ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:55:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:55:15.80348813 +0000 UTC m=+1107.191090891" watchObservedRunningTime="2026-09-10 23:55:15.80662998 +0000 UTC m=+1107.194232739" Sep 10 23:55:16.339075 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:16.339048 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-a78403-openshift-default-6b9d4896d6-cs5dw" Sep 10 23:55:16.344022 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:16.343998 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-a78403-openshift-default-6b9d4896d6-cs5dw" Sep 10 23:55:16.791652 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:16.791607 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-a78403-openshift-default-6b9d4896d6-cs5dw" Sep 10 23:55:16.792741 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:16.792711 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-a78403-openshift-default-6b9d4896d6-cs5dw" Sep 10 23:55:18.192415 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:18.192367 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-56r7l"] Sep 10 23:55:18.766287 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:18.766257 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d7a539-fb5d4d5c6-nn6gl" Sep 10 23:55:18.914043 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:18.914011 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a78403-6694469658-kvj7z"] Sep 10 23:55:18.917803 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:18.917782 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a78403-6694469658-kvj7z" Sep 10 23:55:18.920102 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:18.920080 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-a78403-serving-cert\"" Sep 10 23:55:18.931631 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:18.931606 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a78403-6694469658-kvj7z"] Sep 10 23:55:18.954013 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:18.953979 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51ad6c59-aeb5-4c42-8ee6-f4c889ca454a-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-a78403-6694469658-kvj7z\" (UID: \"51ad6c59-aeb5-4c42-8ee6-f4c889ca454a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a78403-6694469658-kvj7z" Sep 10 23:55:18.954154 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:18.954036 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51ad6c59-aeb5-4c42-8ee6-f4c889ca454a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-a78403-6694469658-kvj7z\" (UID: \"51ad6c59-aeb5-4c42-8ee6-f4c889ca454a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a78403-6694469658-kvj7z" Sep 10 23:55:18.954154 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:18.954105 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q58kl\" (UniqueName: \"kubernetes.io/projected/51ad6c59-aeb5-4c42-8ee6-f4c889ca454a-kube-api-access-q58kl\") pod \"maas-api-e2e-shared-b-w2-a78403-6694469658-kvj7z\" (UID: \"51ad6c59-aeb5-4c42-8ee6-f4c889ca454a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a78403-6694469658-kvj7z" Sep 10 23:55:18.954154 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:18.954132 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51ad6c59-aeb5-4c42-8ee6-f4c889ca454a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-a78403-6694469658-kvj7z\" (UID: \"51ad6c59-aeb5-4c42-8ee6-f4c889ca454a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a78403-6694469658-kvj7z" Sep 10 23:55:19.055799 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:19.055702 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51ad6c59-aeb5-4c42-8ee6-f4c889ca454a-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-a78403-6694469658-kvj7z\" (UID: \"51ad6c59-aeb5-4c42-8ee6-f4c889ca454a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a78403-6694469658-kvj7z" Sep 10 23:55:19.056585 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:19.056559 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51ad6c59-aeb5-4c42-8ee6-f4c889ca454a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-a78403-6694469658-kvj7z\" (UID: \"51ad6c59-aeb5-4c42-8ee6-f4c889ca454a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a78403-6694469658-kvj7z" Sep 10 23:55:19.056877 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:19.056855 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q58kl\" (UniqueName: \"kubernetes.io/projected/51ad6c59-aeb5-4c42-8ee6-f4c889ca454a-kube-api-access-q58kl\") pod \"maas-api-e2e-shared-b-w2-a78403-6694469658-kvj7z\" (UID: \"51ad6c59-aeb5-4c42-8ee6-f4c889ca454a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a78403-6694469658-kvj7z" Sep 10 23:55:19.057001 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:19.056986 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51ad6c59-aeb5-4c42-8ee6-f4c889ca454a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-a78403-6694469658-kvj7z\" (UID: \"51ad6c59-aeb5-4c42-8ee6-f4c889ca454a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a78403-6694469658-kvj7z" Sep 10 23:55:19.057187 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:55:19.056728 2574 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a78403-serving-cert: secret "maas-api-e2e-shared-b-w2-a78403-serving-cert" not found Sep 10 23:55:19.057381 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:55:19.057366 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51ad6c59-aeb5-4c42-8ee6-f4c889ca454a-maas-api-tls podName:51ad6c59-aeb5-4c42-8ee6-f4c889ca454a nodeName:}" failed. No retries permitted until 2026-09-10 23:55:19.557342347 +0000 UTC m=+1110.944945110 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/51ad6c59-aeb5-4c42-8ee6-f4c889ca454a-maas-api-tls") pod "maas-api-e2e-shared-b-w2-a78403-6694469658-kvj7z" (UID: "51ad6c59-aeb5-4c42-8ee6-f4c889ca454a") : secret "maas-api-e2e-shared-b-w2-a78403-serving-cert" not found Sep 10 23:55:19.057965 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:19.057944 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51ad6c59-aeb5-4c42-8ee6-f4c889ca454a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-a78403-6694469658-kvj7z\" (UID: \"51ad6c59-aeb5-4c42-8ee6-f4c889ca454a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a78403-6694469658-kvj7z" Sep 10 23:55:19.059497 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:19.059472 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51ad6c59-aeb5-4c42-8ee6-f4c889ca454a-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-a78403-6694469658-kvj7z\" (UID: \"51ad6c59-aeb5-4c42-8ee6-f4c889ca454a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a78403-6694469658-kvj7z" Sep 10 23:55:19.067201 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:19.067178 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q58kl\" (UniqueName: \"kubernetes.io/projected/51ad6c59-aeb5-4c42-8ee6-f4c889ca454a-kube-api-access-q58kl\") pod \"maas-api-e2e-shared-b-w2-a78403-6694469658-kvj7z\" (UID: \"51ad6c59-aeb5-4c42-8ee6-f4c889ca454a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a78403-6694469658-kvj7z" Sep 10 23:55:19.563620 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:19.563578 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51ad6c59-aeb5-4c42-8ee6-f4c889ca454a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-a78403-6694469658-kvj7z\" (UID: \"51ad6c59-aeb5-4c42-8ee6-f4c889ca454a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a78403-6694469658-kvj7z" Sep 10 23:55:19.566511 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:19.566243 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51ad6c59-aeb5-4c42-8ee6-f4c889ca454a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-a78403-6694469658-kvj7z\" (UID: \"51ad6c59-aeb5-4c42-8ee6-f4c889ca454a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a78403-6694469658-kvj7z" Sep 10 23:55:19.830069 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:19.830034 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a78403-6694469658-kvj7z" Sep 10 23:55:20.108179 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:20.108150 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a78403-6694469658-kvj7z"] Sep 10 23:55:20.110242 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:55:20.110202 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51ad6c59_aeb5_4c42_8ee6_f4c889ca454a.slice/crio-38476388961a87959906620558ab3c82d3475be4eac2bc792633af6573a3b7d3 WatchSource:0}: Error finding container 38476388961a87959906620558ab3c82d3475be4eac2bc792633af6573a3b7d3: Status 404 returned error can't find the container with id 38476388961a87959906620558ab3c82d3475be4eac2bc792633af6573a3b7d3 Sep 10 23:55:20.818948 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:20.818904 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a78403-6694469658-kvj7z" event={"ID":"51ad6c59-aeb5-4c42-8ee6-f4c889ca454a","Type":"ContainerStarted","Data":"0655b931b5996df30f6da4a94af3ea044e19c64854f5f536e09e6322cb7f0b46"} Sep 10 23:55:20.819339 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:20.818971 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a78403-6694469658-kvj7z" event={"ID":"51ad6c59-aeb5-4c42-8ee6-f4c889ca454a","Type":"ContainerStarted","Data":"38476388961a87959906620558ab3c82d3475be4eac2bc792633af6573a3b7d3"} Sep 10 23:55:20.819339 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:20.819033 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a78403-6694469658-kvj7z" Sep 10 23:55:20.838726 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:20.838679 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a78403-6694469658-kvj7z" podStartSLOduration=2.534749012 podStartE2EDuration="2.838663917s" totalImagesPullingTime="303.914905ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:55:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:55:20.835777614 +0000 UTC m=+1112.223380395" watchObservedRunningTime="2026-09-10 23:55:20.838663917 +0000 UTC m=+1112.226266675" Sep 10 23:55:22.832363 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:22.832320 2574 generic.go:356] "Generic (PLEG): container finished" podID="1f58d38e-bf62-44f6-9522-2ada85451d15" containerID="880166dc726b9c67540e69d95a3af0699bcd62e1d21aaf0f71a68df5d9404d7a" exitCode=0 Sep 10 23:55:22.832831 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:22.832395 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-d7a539-openshift-default-7c74c59b78-q7s7n" event={"ID":"1f58d38e-bf62-44f6-9522-2ada85451d15","Type":"ContainerDied","Data":"880166dc726b9c67540e69d95a3af0699bcd62e1d21aaf0f71a68df5d9404d7a"} Sep 10 23:55:22.832877 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:22.832840 2574 scope.go:117] "RemoveContainer" containerID="880166dc726b9c67540e69d95a3af0699bcd62e1d21aaf0f71a68df5d9404d7a" Sep 10 23:55:22.834043 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:22.834017 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 23:55:22.834113 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:22.834087 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 23:55:22.834151 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:22.834121 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 23:55:23.721351 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:23.721322 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-d7a539-openshift-default-7c74c59b78-q7s7n" Sep 10 23:55:23.841212 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:23.841173 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-d7a539-openshift-default-7c74c59b78-q7s7n" event={"ID":"1f58d38e-bf62-44f6-9522-2ada85451d15","Type":"ContainerStarted","Data":"393310600ba877303bbbb18665b2078d73065ba1658618e1611fe8db21354889"} Sep 10 23:55:24.439699 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:24.439664 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-d7a539-openshift-default-7c74c59b78-q7s7n" Sep 10 23:55:24.440804 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:24.440778 2574 patch_prober.go:28] interesting pod/e2e-shared-a-w3-d7a539-openshift-default-7c74c59b78-q7s7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Sep 10 23:55:24.440940 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:24.440828 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-d7a539-openshift-default-7c74c59b78-q7s7n" podUID="1f58d38e-bf62-44f6-9522-2ada85451d15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Sep 10 23:55:25.439594 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:25.439554 2574 patch_prober.go:28] interesting pod/e2e-shared-a-w3-d7a539-openshift-default-7c74c59b78-q7s7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Sep 10 23:55:25.440030 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:25.439616 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-d7a539-openshift-default-7c74c59b78-q7s7n" podUID="1f58d38e-bf62-44f6-9522-2ada85451d15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Sep 10 23:55:26.439713 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:26.439670 2574 patch_prober.go:28] interesting pod/e2e-shared-a-w3-d7a539-openshift-default-7c74c59b78-q7s7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Sep 10 23:55:26.440147 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:26.439733 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-d7a539-openshift-default-7c74c59b78-q7s7n" podUID="1f58d38e-bf62-44f6-9522-2ada85451d15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Sep 10 23:55:26.830238 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:26.830205 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a78403-6694469658-kvj7z" Sep 10 23:55:27.439577 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:27.439539 2574 patch_prober.go:28] interesting pod/e2e-shared-a-w3-d7a539-openshift-default-7c74c59b78-q7s7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Sep 10 23:55:27.439789 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:27.439600 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-d7a539-openshift-default-7c74c59b78-q7s7n" podUID="1f58d38e-bf62-44f6-9522-2ada85451d15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Sep 10 23:55:28.285364 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:28.285314 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-56r7l"] Sep 10 23:55:28.440132 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:28.440097 2574 patch_prober.go:28] interesting pod/e2e-shared-a-w3-d7a539-openshift-default-7c74c59b78-q7s7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Sep 10 23:55:28.440584 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:28.440162 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-d7a539-openshift-default-7c74c59b78-q7s7n" podUID="1f58d38e-bf62-44f6-9522-2ada85451d15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Sep 10 23:55:29.494283 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:29.494241 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-d7a539-openshift-default-7c74c59b78-q7s7n" Sep 10 23:55:29.495020 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:29.494992 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-d7a539-openshift-default-7c74c59b78-q7s7n" Sep 10 23:55:29.496434 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:29.496406 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-d7a539-openshift-default-7c74c59b78-q7s7n" Sep 10 23:55:32.834782 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:32.834732 2574 patch_prober.go:28] interesting pod/e2e-shared-b-w2-a78403-openshift-default-6b9d4896d6-cs5dw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 10 23:55:32.835156 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:32.834804 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-a78403-openshift-default-6b9d4896d6-cs5dw" podUID="c52960ab-0f37-4e17-a6e0-bbc346e0aa38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 10 23:55:34.920091 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:34.920055 2574 generic.go:356] "Generic (PLEG): container finished" podID="c52960ab-0f37-4e17-a6e0-bbc346e0aa38" containerID="2b4f71e50e35396a6bd685ad9e0011b3d726c10ebb764307ae0d43b2e2e71c66" exitCode=0 Sep 10 23:55:34.920501 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:34.920125 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-a78403-openshift-default-6b9d4896d6-cs5dw" event={"ID":"c52960ab-0f37-4e17-a6e0-bbc346e0aa38","Type":"ContainerDied","Data":"2b4f71e50e35396a6bd685ad9e0011b3d726c10ebb764307ae0d43b2e2e71c66"} Sep 10 23:55:34.920544 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:34.920516 2574 scope.go:117] "RemoveContainer" containerID="2b4f71e50e35396a6bd685ad9e0011b3d726c10ebb764307ae0d43b2e2e71c66" Sep 10 23:55:34.921725 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:34.921695 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 23:55:34.921789 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:34.921766 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 23:55:34.921832 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:34.921801 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 23:55:35.219438 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:35.219405 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-4094ce-openshift-default-7558d5c9fb-vp9l5"] Sep 10 23:55:35.222444 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:35.222419 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-4094ce-openshift-default-7558d5c9fb-vp9l5" Sep 10 23:55:35.225156 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:35.225135 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-4094ce-openshift-default-dockercfg-xdw6s\"" Sep 10 23:55:35.242253 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:35.242228 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-4094ce-openshift-default-7558d5c9fb-vp9l5"] Sep 10 23:55:35.342150 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:35.342117 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e85f9c50-8a5c-4191-9441-9b30ca611ffb-workload-socket\") pod \"e2e-shared-b-w3-4094ce-openshift-default-7558d5c9fb-vp9l5\" (UID: \"e85f9c50-8a5c-4191-9441-9b30ca611ffb\") " pod="openshift-ingress/e2e-shared-b-w3-4094ce-openshift-default-7558d5c9fb-vp9l5" Sep 10 23:55:35.342150 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:35.342154 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e85f9c50-8a5c-4191-9441-9b30ca611ffb-istio-data\") pod \"e2e-shared-b-w3-4094ce-openshift-default-7558d5c9fb-vp9l5\" (UID: \"e85f9c50-8a5c-4191-9441-9b30ca611ffb\") " pod="openshift-ingress/e2e-shared-b-w3-4094ce-openshift-default-7558d5c9fb-vp9l5" Sep 10 23:55:35.342425 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:35.342175 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e85f9c50-8a5c-4191-9441-9b30ca611ffb-workload-certs\") pod \"e2e-shared-b-w3-4094ce-openshift-default-7558d5c9fb-vp9l5\" (UID: \"e85f9c50-8a5c-4191-9441-9b30ca611ffb\") " pod="openshift-ingress/e2e-shared-b-w3-4094ce-openshift-default-7558d5c9fb-vp9l5" Sep 10 23:55:35.342425 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:35.342195 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e85f9c50-8a5c-4191-9441-9b30ca611ffb-credential-socket\") pod \"e2e-shared-b-w3-4094ce-openshift-default-7558d5c9fb-vp9l5\" (UID: \"e85f9c50-8a5c-4191-9441-9b30ca611ffb\") " pod="openshift-ingress/e2e-shared-b-w3-4094ce-openshift-default-7558d5c9fb-vp9l5" Sep 10 23:55:35.342425 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:35.342254 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e85f9c50-8a5c-4191-9441-9b30ca611ffb-istio-token\") pod \"e2e-shared-b-w3-4094ce-openshift-default-7558d5c9fb-vp9l5\" (UID: \"e85f9c50-8a5c-4191-9441-9b30ca611ffb\") " pod="openshift-ingress/e2e-shared-b-w3-4094ce-openshift-default-7558d5c9fb-vp9l5" Sep 10 23:55:35.342425 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:35.342389 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e85f9c50-8a5c-4191-9441-9b30ca611ffb-istio-envoy\") pod \"e2e-shared-b-w3-4094ce-openshift-default-7558d5c9fb-vp9l5\" (UID: \"e85f9c50-8a5c-4191-9441-9b30ca611ffb\") " pod="openshift-ingress/e2e-shared-b-w3-4094ce-openshift-default-7558d5c9fb-vp9l5" Sep 10 23:55:35.342425 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:35.342414 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e85f9c50-8a5c-4191-9441-9b30ca611ffb-istiod-ca-cert\") pod \"e2e-shared-b-w3-4094ce-openshift-default-7558d5c9fb-vp9l5\" (UID: \"e85f9c50-8a5c-4191-9441-9b30ca611ffb\") " pod="openshift-ingress/e2e-shared-b-w3-4094ce-openshift-default-7558d5c9fb-vp9l5" Sep 10 23:55:35.342617 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:35.342439 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e85f9c50-8a5c-4191-9441-9b30ca611ffb-istio-podinfo\") pod \"e2e-shared-b-w3-4094ce-openshift-default-7558d5c9fb-vp9l5\" (UID: \"e85f9c50-8a5c-4191-9441-9b30ca611ffb\") " pod="openshift-ingress/e2e-shared-b-w3-4094ce-openshift-default-7558d5c9fb-vp9l5" Sep 10 23:55:35.342617 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:35.342515 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dcvn\" (UniqueName: \"kubernetes.io/projected/e85f9c50-8a5c-4191-9441-9b30ca611ffb-kube-api-access-2dcvn\") pod \"e2e-shared-b-w3-4094ce-openshift-default-7558d5c9fb-vp9l5\" (UID: \"e85f9c50-8a5c-4191-9441-9b30ca611ffb\") " pod="openshift-ingress/e2e-shared-b-w3-4094ce-openshift-default-7558d5c9fb-vp9l5" Sep 10 23:55:35.443963 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:35.443926 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e85f9c50-8a5c-4191-9441-9b30ca611ffb-workload-socket\") pod \"e2e-shared-b-w3-4094ce-openshift-default-7558d5c9fb-vp9l5\" (UID: \"e85f9c50-8a5c-4191-9441-9b30ca611ffb\") " pod="openshift-ingress/e2e-shared-b-w3-4094ce-openshift-default-7558d5c9fb-vp9l5" Sep 10 23:55:35.443963 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:35.443963 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e85f9c50-8a5c-4191-9441-9b30ca611ffb-istio-data\") pod \"e2e-shared-b-w3-4094ce-openshift-default-7558d5c9fb-vp9l5\" (UID: \"e85f9c50-8a5c-4191-9441-9b30ca611ffb\") " pod="openshift-ingress/e2e-shared-b-w3-4094ce-openshift-default-7558d5c9fb-vp9l5" Sep 10 23:55:35.444234 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:35.443982 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e85f9c50-8a5c-4191-9441-9b30ca611ffb-workload-certs\") pod \"e2e-shared-b-w3-4094ce-openshift-default-7558d5c9fb-vp9l5\" (UID: \"e85f9c50-8a5c-4191-9441-9b30ca611ffb\") " pod="openshift-ingress/e2e-shared-b-w3-4094ce-openshift-default-7558d5c9fb-vp9l5" Sep 10 23:55:35.444234 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:35.444001 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e85f9c50-8a5c-4191-9441-9b30ca611ffb-credential-socket\") pod \"e2e-shared-b-w3-4094ce-openshift-default-7558d5c9fb-vp9l5\" (UID: \"e85f9c50-8a5c-4191-9441-9b30ca611ffb\") " pod="openshift-ingress/e2e-shared-b-w3-4094ce-openshift-default-7558d5c9fb-vp9l5" Sep 10 23:55:35.444234 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:35.444019 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e85f9c50-8a5c-4191-9441-9b30ca611ffb-istio-token\") pod \"e2e-shared-b-w3-4094ce-openshift-default-7558d5c9fb-vp9l5\" (UID: \"e85f9c50-8a5c-4191-9441-9b30ca611ffb\") " pod="openshift-ingress/e2e-shared-b-w3-4094ce-openshift-default-7558d5c9fb-vp9l5" Sep 10 23:55:35.444234 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:35.444106 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e85f9c50-8a5c-4191-9441-9b30ca611ffb-istio-envoy\") pod \"e2e-shared-b-w3-4094ce-openshift-default-7558d5c9fb-vp9l5\" (UID: \"e85f9c50-8a5c-4191-9441-9b30ca611ffb\") " pod="openshift-ingress/e2e-shared-b-w3-4094ce-openshift-default-7558d5c9fb-vp9l5" Sep 10 23:55:35.444479 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:35.444253 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e85f9c50-8a5c-4191-9441-9b30ca611ffb-istiod-ca-cert\") pod \"e2e-shared-b-w3-4094ce-openshift-default-7558d5c9fb-vp9l5\" (UID: \"e85f9c50-8a5c-4191-9441-9b30ca611ffb\") " pod="openshift-ingress/e2e-shared-b-w3-4094ce-openshift-default-7558d5c9fb-vp9l5" Sep 10 23:55:35.444479 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:35.444313 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e85f9c50-8a5c-4191-9441-9b30ca611ffb-istio-podinfo\") pod \"e2e-shared-b-w3-4094ce-openshift-default-7558d5c9fb-vp9l5\" (UID: \"e85f9c50-8a5c-4191-9441-9b30ca611ffb\") " pod="openshift-ingress/e2e-shared-b-w3-4094ce-openshift-default-7558d5c9fb-vp9l5" Sep 10 23:55:35.444479 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:35.444390 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e85f9c50-8a5c-4191-9441-9b30ca611ffb-workload-certs\") pod \"e2e-shared-b-w3-4094ce-openshift-default-7558d5c9fb-vp9l5\" (UID: \"e85f9c50-8a5c-4191-9441-9b30ca611ffb\") " pod="openshift-ingress/e2e-shared-b-w3-4094ce-openshift-default-7558d5c9fb-vp9l5" Sep 10 23:55:35.444479 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:35.444400 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dcvn\" (UniqueName: \"kubernetes.io/projected/e85f9c50-8a5c-4191-9441-9b30ca611ffb-kube-api-access-2dcvn\") pod \"e2e-shared-b-w3-4094ce-openshift-default-7558d5c9fb-vp9l5\" (UID: \"e85f9c50-8a5c-4191-9441-9b30ca611ffb\") " pod="openshift-ingress/e2e-shared-b-w3-4094ce-openshift-default-7558d5c9fb-vp9l5" Sep 10 23:55:35.444757 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:35.444733 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e85f9c50-8a5c-4191-9441-9b30ca611ffb-credential-socket\") pod \"e2e-shared-b-w3-4094ce-openshift-default-7558d5c9fb-vp9l5\" (UID: \"e85f9c50-8a5c-4191-9441-9b30ca611ffb\") " pod="openshift-ingress/e2e-shared-b-w3-4094ce-openshift-default-7558d5c9fb-vp9l5" Sep 10 23:55:35.444824 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:35.444783 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e85f9c50-8a5c-4191-9441-9b30ca611ffb-workload-socket\") pod \"e2e-shared-b-w3-4094ce-openshift-default-7558d5c9fb-vp9l5\" (UID: \"e85f9c50-8a5c-4191-9441-9b30ca611ffb\") " pod="openshift-ingress/e2e-shared-b-w3-4094ce-openshift-default-7558d5c9fb-vp9l5" Sep 10 23:55:35.444869 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:35.444841 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e85f9c50-8a5c-4191-9441-9b30ca611ffb-istio-data\") pod \"e2e-shared-b-w3-4094ce-openshift-default-7558d5c9fb-vp9l5\" (UID: \"e85f9c50-8a5c-4191-9441-9b30ca611ffb\") " pod="openshift-ingress/e2e-shared-b-w3-4094ce-openshift-default-7558d5c9fb-vp9l5" Sep 10 23:55:35.445194 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:35.445172 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e85f9c50-8a5c-4191-9441-9b30ca611ffb-istiod-ca-cert\") pod \"e2e-shared-b-w3-4094ce-openshift-default-7558d5c9fb-vp9l5\" (UID: \"e85f9c50-8a5c-4191-9441-9b30ca611ffb\") " pod="openshift-ingress/e2e-shared-b-w3-4094ce-openshift-default-7558d5c9fb-vp9l5" Sep 10 23:55:35.446399 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:35.446375 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e85f9c50-8a5c-4191-9441-9b30ca611ffb-istio-envoy\") pod \"e2e-shared-b-w3-4094ce-openshift-default-7558d5c9fb-vp9l5\" (UID: \"e85f9c50-8a5c-4191-9441-9b30ca611ffb\") " pod="openshift-ingress/e2e-shared-b-w3-4094ce-openshift-default-7558d5c9fb-vp9l5" Sep 10 23:55:35.447018 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:35.446995 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e85f9c50-8a5c-4191-9441-9b30ca611ffb-istio-podinfo\") pod \"e2e-shared-b-w3-4094ce-openshift-default-7558d5c9fb-vp9l5\" (UID: \"e85f9c50-8a5c-4191-9441-9b30ca611ffb\") " pod="openshift-ingress/e2e-shared-b-w3-4094ce-openshift-default-7558d5c9fb-vp9l5" Sep 10 23:55:35.452266 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:35.452238 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e85f9c50-8a5c-4191-9441-9b30ca611ffb-istio-token\") pod \"e2e-shared-b-w3-4094ce-openshift-default-7558d5c9fb-vp9l5\" (UID: \"e85f9c50-8a5c-4191-9441-9b30ca611ffb\") " pod="openshift-ingress/e2e-shared-b-w3-4094ce-openshift-default-7558d5c9fb-vp9l5" Sep 10 23:55:35.452379 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:35.452362 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dcvn\" (UniqueName: \"kubernetes.io/projected/e85f9c50-8a5c-4191-9441-9b30ca611ffb-kube-api-access-2dcvn\") pod \"e2e-shared-b-w3-4094ce-openshift-default-7558d5c9fb-vp9l5\" (UID: \"e85f9c50-8a5c-4191-9441-9b30ca611ffb\") " pod="openshift-ingress/e2e-shared-b-w3-4094ce-openshift-default-7558d5c9fb-vp9l5" Sep 10 23:55:35.534799 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:35.534715 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-4094ce-openshift-default-7558d5c9fb-vp9l5" Sep 10 23:55:35.857903 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:35.857872 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-4094ce-openshift-default-7558d5c9fb-vp9l5"] Sep 10 23:55:35.859417 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:55:35.859392 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode85f9c50_8a5c_4191_9441_9b30ca611ffb.slice/crio-0958aeef5a3070629862d7c6bb3c008591c4433aeec837a2543d92c1d9c246ca WatchSource:0}: Error finding container 0958aeef5a3070629862d7c6bb3c008591c4433aeec837a2543d92c1d9c246ca: Status 404 returned error can't find the container with id 0958aeef5a3070629862d7c6bb3c008591c4433aeec837a2543d92c1d9c246ca Sep 10 23:55:35.861864 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:35.861833 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 23:55:35.861940 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:35.861908 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 23:55:35.861978 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:35.861943 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 23:55:35.931273 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:35.931247 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-a78403-openshift-default-6b9d4896d6-cs5dw" event={"ID":"c52960ab-0f37-4e17-a6e0-bbc346e0aa38","Type":"ContainerStarted","Data":"8c84ebe08796d9536ffe4c56d504d0fba3852642376fc1582c084fbc4609615a"} Sep 10 23:55:35.932550 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:35.932527 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-4094ce-openshift-default-7558d5c9fb-vp9l5" event={"ID":"e85f9c50-8a5c-4191-9441-9b30ca611ffb","Type":"ContainerStarted","Data":"0958aeef5a3070629862d7c6bb3c008591c4433aeec837a2543d92c1d9c246ca"} Sep 10 23:55:36.338380 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:36.338345 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-a78403-openshift-default-6b9d4896d6-cs5dw" Sep 10 23:55:36.338681 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:36.338529 2574 patch_prober.go:28] interesting pod/e2e-shared-b-w2-a78403-openshift-default-6b9d4896d6-cs5dw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Sep 10 23:55:36.338681 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:36.338587 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-a78403-openshift-default-6b9d4896d6-cs5dw" podUID="c52960ab-0f37-4e17-a6e0-bbc346e0aa38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Sep 10 23:55:36.940768 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:36.940708 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-4094ce-openshift-default-7558d5c9fb-vp9l5" event={"ID":"e85f9c50-8a5c-4191-9441-9b30ca611ffb","Type":"ContainerStarted","Data":"86d66a2854991aac386fda6ac27492cd24fbbe529fba777c342907572eee3fd6"} Sep 10 23:55:36.963559 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:36.963500 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-4094ce-openshift-default-7558d5c9fb-vp9l5" podStartSLOduration=1.9634841509999998 podStartE2EDuration="1.963484151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:55:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:55:36.960311257 +0000 UTC m=+1128.347914020" watchObservedRunningTime="2026-09-10 23:55:36.963484151 +0000 UTC m=+1128.351086975" Sep 10 23:55:37.338883 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:37.338852 2574 patch_prober.go:28] interesting pod/e2e-shared-b-w2-a78403-openshift-default-6b9d4896d6-cs5dw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Sep 10 23:55:37.339053 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:37.338916 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-a78403-openshift-default-6b9d4896d6-cs5dw" podUID="c52960ab-0f37-4e17-a6e0-bbc346e0aa38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Sep 10 23:55:37.535085 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:37.535051 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-4094ce-openshift-default-7558d5c9fb-vp9l5" Sep 10 23:55:37.540080 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:37.540057 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-4094ce-openshift-default-7558d5c9fb-vp9l5" Sep 10 23:55:37.947023 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:37.946987 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-4094ce-openshift-default-7558d5c9fb-vp9l5" Sep 10 23:55:37.947918 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:37.947903 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-4094ce-openshift-default-7558d5c9fb-vp9l5" Sep 10 23:55:38.339305 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:38.339223 2574 patch_prober.go:28] interesting pod/e2e-shared-b-w2-a78403-openshift-default-6b9d4896d6-cs5dw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Sep 10 23:55:38.339456 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:38.339304 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-a78403-openshift-default-6b9d4896d6-cs5dw" podUID="c52960ab-0f37-4e17-a6e0-bbc346e0aa38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Sep 10 23:55:39.339395 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:39.339356 2574 patch_prober.go:28] interesting pod/e2e-shared-b-w2-a78403-openshift-default-6b9d4896d6-cs5dw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Sep 10 23:55:39.339805 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:39.339436 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-a78403-openshift-default-6b9d4896d6-cs5dw" podUID="c52960ab-0f37-4e17-a6e0-bbc346e0aa38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Sep 10 23:55:40.339120 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:40.339083 2574 patch_prober.go:28] interesting pod/e2e-shared-b-w2-a78403-openshift-default-6b9d4896d6-cs5dw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Sep 10 23:55:40.339334 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:40.339156 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-a78403-openshift-default-6b9d4896d6-cs5dw" podUID="c52960ab-0f37-4e17-a6e0-bbc346e0aa38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Sep 10 23:55:41.342924 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:41.342892 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-a78403-openshift-default-6b9d4896d6-cs5dw" Sep 10 23:55:41.343325 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:41.343171 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-a78403-openshift-default-6b9d4896d6-cs5dw" Sep 10 23:55:41.343941 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:41.343907 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-a78403-openshift-default-6b9d4896d6-cs5dw" Sep 10 23:55:45.984277 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:45.984246 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-56r7l"] Sep 10 23:55:51.573950 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:51.573904 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-e1f72544-openshift-default-86c84fd5b-khbgn"] Sep 10 23:55:51.599188 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:51.599160 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e1f72544-openshift-default-86c84fd5b-khbgn" Sep 10 23:55:51.599470 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:51.599431 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e1f72544-openshift-default-86c84fd5b-khbgn"] Sep 10 23:55:51.601571 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:51.601548 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-e1f72544-openshift-default-dockercfg-hjxdd\"" Sep 10 23:55:51.639780 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:51.639751 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09db6ab7-c732-41e1-a76f-ca403f1eb53f-istio-envoy\") pod \"e2e-mt-e1f72544-openshift-default-86c84fd5b-khbgn\" (UID: \"09db6ab7-c732-41e1-a76f-ca403f1eb53f\") " pod="openshift-ingress/e2e-mt-e1f72544-openshift-default-86c84fd5b-khbgn" Sep 10 23:55:51.639918 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:51.639804 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09db6ab7-c732-41e1-a76f-ca403f1eb53f-istiod-ca-cert\") pod \"e2e-mt-e1f72544-openshift-default-86c84fd5b-khbgn\" (UID: \"09db6ab7-c732-41e1-a76f-ca403f1eb53f\") " pod="openshift-ingress/e2e-mt-e1f72544-openshift-default-86c84fd5b-khbgn" Sep 10 23:55:51.639918 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:51.639832 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09db6ab7-c732-41e1-a76f-ca403f1eb53f-istio-token\") pod \"e2e-mt-e1f72544-openshift-default-86c84fd5b-khbgn\" (UID: \"09db6ab7-c732-41e1-a76f-ca403f1eb53f\") " pod="openshift-ingress/e2e-mt-e1f72544-openshift-default-86c84fd5b-khbgn" Sep 10 23:55:51.639918 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:51.639857 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09db6ab7-c732-41e1-a76f-ca403f1eb53f-workload-certs\") pod \"e2e-mt-e1f72544-openshift-default-86c84fd5b-khbgn\" (UID: \"09db6ab7-c732-41e1-a76f-ca403f1eb53f\") " pod="openshift-ingress/e2e-mt-e1f72544-openshift-default-86c84fd5b-khbgn" Sep 10 23:55:51.640037 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:51.639930 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jbvn\" (UniqueName: \"kubernetes.io/projected/09db6ab7-c732-41e1-a76f-ca403f1eb53f-kube-api-access-9jbvn\") pod \"e2e-mt-e1f72544-openshift-default-86c84fd5b-khbgn\" (UID: \"09db6ab7-c732-41e1-a76f-ca403f1eb53f\") " pod="openshift-ingress/e2e-mt-e1f72544-openshift-default-86c84fd5b-khbgn" Sep 10 23:55:51.640037 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:51.639976 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09db6ab7-c732-41e1-a76f-ca403f1eb53f-istio-data\") pod \"e2e-mt-e1f72544-openshift-default-86c84fd5b-khbgn\" (UID: \"09db6ab7-c732-41e1-a76f-ca403f1eb53f\") " pod="openshift-ingress/e2e-mt-e1f72544-openshift-default-86c84fd5b-khbgn" Sep 10 23:55:51.640110 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:51.640041 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09db6ab7-c732-41e1-a76f-ca403f1eb53f-workload-socket\") pod \"e2e-mt-e1f72544-openshift-default-86c84fd5b-khbgn\" (UID: \"09db6ab7-c732-41e1-a76f-ca403f1eb53f\") " pod="openshift-ingress/e2e-mt-e1f72544-openshift-default-86c84fd5b-khbgn" Sep 10 23:55:51.640110 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:51.640063 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09db6ab7-c732-41e1-a76f-ca403f1eb53f-credential-socket\") pod \"e2e-mt-e1f72544-openshift-default-86c84fd5b-khbgn\" (UID: \"09db6ab7-c732-41e1-a76f-ca403f1eb53f\") " pod="openshift-ingress/e2e-mt-e1f72544-openshift-default-86c84fd5b-khbgn" Sep 10 23:55:51.640110 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:51.640086 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09db6ab7-c732-41e1-a76f-ca403f1eb53f-istio-podinfo\") pod \"e2e-mt-e1f72544-openshift-default-86c84fd5b-khbgn\" (UID: \"09db6ab7-c732-41e1-a76f-ca403f1eb53f\") " pod="openshift-ingress/e2e-mt-e1f72544-openshift-default-86c84fd5b-khbgn" Sep 10 23:55:51.741397 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:51.741362 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jbvn\" (UniqueName: \"kubernetes.io/projected/09db6ab7-c732-41e1-a76f-ca403f1eb53f-kube-api-access-9jbvn\") pod \"e2e-mt-e1f72544-openshift-default-86c84fd5b-khbgn\" (UID: \"09db6ab7-c732-41e1-a76f-ca403f1eb53f\") " pod="openshift-ingress/e2e-mt-e1f72544-openshift-default-86c84fd5b-khbgn" Sep 10 23:55:51.741595 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:51.741413 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09db6ab7-c732-41e1-a76f-ca403f1eb53f-istio-data\") pod \"e2e-mt-e1f72544-openshift-default-86c84fd5b-khbgn\" (UID: \"09db6ab7-c732-41e1-a76f-ca403f1eb53f\") " pod="openshift-ingress/e2e-mt-e1f72544-openshift-default-86c84fd5b-khbgn" Sep 10 23:55:51.741595 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:51.741483 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09db6ab7-c732-41e1-a76f-ca403f1eb53f-workload-socket\") pod \"e2e-mt-e1f72544-openshift-default-86c84fd5b-khbgn\" (UID: \"09db6ab7-c732-41e1-a76f-ca403f1eb53f\") " pod="openshift-ingress/e2e-mt-e1f72544-openshift-default-86c84fd5b-khbgn" Sep 10 23:55:51.741595 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:51.741509 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09db6ab7-c732-41e1-a76f-ca403f1eb53f-credential-socket\") pod \"e2e-mt-e1f72544-openshift-default-86c84fd5b-khbgn\" (UID: \"09db6ab7-c732-41e1-a76f-ca403f1eb53f\") " pod="openshift-ingress/e2e-mt-e1f72544-openshift-default-86c84fd5b-khbgn" Sep 10 23:55:51.741595 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:51.741543 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09db6ab7-c732-41e1-a76f-ca403f1eb53f-istio-podinfo\") pod \"e2e-mt-e1f72544-openshift-default-86c84fd5b-khbgn\" (UID: \"09db6ab7-c732-41e1-a76f-ca403f1eb53f\") " pod="openshift-ingress/e2e-mt-e1f72544-openshift-default-86c84fd5b-khbgn" Sep 10 23:55:51.741595 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:51.741584 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09db6ab7-c732-41e1-a76f-ca403f1eb53f-istio-envoy\") pod \"e2e-mt-e1f72544-openshift-default-86c84fd5b-khbgn\" (UID: \"09db6ab7-c732-41e1-a76f-ca403f1eb53f\") " pod="openshift-ingress/e2e-mt-e1f72544-openshift-default-86c84fd5b-khbgn" Sep 10 23:55:51.741877 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:51.741676 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09db6ab7-c732-41e1-a76f-ca403f1eb53f-istiod-ca-cert\") pod \"e2e-mt-e1f72544-openshift-default-86c84fd5b-khbgn\" (UID: \"09db6ab7-c732-41e1-a76f-ca403f1eb53f\") " pod="openshift-ingress/e2e-mt-e1f72544-openshift-default-86c84fd5b-khbgn" Sep 10 23:55:51.741877 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:51.741714 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09db6ab7-c732-41e1-a76f-ca403f1eb53f-istio-token\") pod \"e2e-mt-e1f72544-openshift-default-86c84fd5b-khbgn\" (UID: \"09db6ab7-c732-41e1-a76f-ca403f1eb53f\") " pod="openshift-ingress/e2e-mt-e1f72544-openshift-default-86c84fd5b-khbgn" Sep 10 23:55:51.741877 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:51.741754 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09db6ab7-c732-41e1-a76f-ca403f1eb53f-workload-certs\") pod \"e2e-mt-e1f72544-openshift-default-86c84fd5b-khbgn\" (UID: \"09db6ab7-c732-41e1-a76f-ca403f1eb53f\") " pod="openshift-ingress/e2e-mt-e1f72544-openshift-default-86c84fd5b-khbgn" Sep 10 23:55:51.742052 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:51.741948 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09db6ab7-c732-41e1-a76f-ca403f1eb53f-istio-data\") pod \"e2e-mt-e1f72544-openshift-default-86c84fd5b-khbgn\" (UID: \"09db6ab7-c732-41e1-a76f-ca403f1eb53f\") " pod="openshift-ingress/e2e-mt-e1f72544-openshift-default-86c84fd5b-khbgn" Sep 10 23:55:51.742114 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:51.742059 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09db6ab7-c732-41e1-a76f-ca403f1eb53f-credential-socket\") pod \"e2e-mt-e1f72544-openshift-default-86c84fd5b-khbgn\" (UID: \"09db6ab7-c732-41e1-a76f-ca403f1eb53f\") " pod="openshift-ingress/e2e-mt-e1f72544-openshift-default-86c84fd5b-khbgn" Sep 10 23:55:51.742157 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:51.742110 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09db6ab7-c732-41e1-a76f-ca403f1eb53f-workload-certs\") pod \"e2e-mt-e1f72544-openshift-default-86c84fd5b-khbgn\" (UID: \"09db6ab7-c732-41e1-a76f-ca403f1eb53f\") " pod="openshift-ingress/e2e-mt-e1f72544-openshift-default-86c84fd5b-khbgn" Sep 10 23:55:51.742222 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:51.742204 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09db6ab7-c732-41e1-a76f-ca403f1eb53f-workload-socket\") pod \"e2e-mt-e1f72544-openshift-default-86c84fd5b-khbgn\" (UID: \"09db6ab7-c732-41e1-a76f-ca403f1eb53f\") " pod="openshift-ingress/e2e-mt-e1f72544-openshift-default-86c84fd5b-khbgn" Sep 10 23:55:51.742466 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:51.742438 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09db6ab7-c732-41e1-a76f-ca403f1eb53f-istiod-ca-cert\") pod \"e2e-mt-e1f72544-openshift-default-86c84fd5b-khbgn\" (UID: \"09db6ab7-c732-41e1-a76f-ca403f1eb53f\") " pod="openshift-ingress/e2e-mt-e1f72544-openshift-default-86c84fd5b-khbgn" Sep 10 23:55:51.743992 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:51.743967 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09db6ab7-c732-41e1-a76f-ca403f1eb53f-istio-envoy\") pod \"e2e-mt-e1f72544-openshift-default-86c84fd5b-khbgn\" (UID: \"09db6ab7-c732-41e1-a76f-ca403f1eb53f\") " pod="openshift-ingress/e2e-mt-e1f72544-openshift-default-86c84fd5b-khbgn" Sep 10 23:55:51.744230 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:51.744198 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09db6ab7-c732-41e1-a76f-ca403f1eb53f-istio-podinfo\") pod \"e2e-mt-e1f72544-openshift-default-86c84fd5b-khbgn\" (UID: \"09db6ab7-c732-41e1-a76f-ca403f1eb53f\") " pod="openshift-ingress/e2e-mt-e1f72544-openshift-default-86c84fd5b-khbgn" Sep 10 23:55:51.749501 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:51.749474 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jbvn\" (UniqueName: \"kubernetes.io/projected/09db6ab7-c732-41e1-a76f-ca403f1eb53f-kube-api-access-9jbvn\") pod \"e2e-mt-e1f72544-openshift-default-86c84fd5b-khbgn\" (UID: \"09db6ab7-c732-41e1-a76f-ca403f1eb53f\") " pod="openshift-ingress/e2e-mt-e1f72544-openshift-default-86c84fd5b-khbgn" Sep 10 23:55:51.749765 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:51.749747 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09db6ab7-c732-41e1-a76f-ca403f1eb53f-istio-token\") pod \"e2e-mt-e1f72544-openshift-default-86c84fd5b-khbgn\" (UID: \"09db6ab7-c732-41e1-a76f-ca403f1eb53f\") " pod="openshift-ingress/e2e-mt-e1f72544-openshift-default-86c84fd5b-khbgn" Sep 10 23:55:51.911129 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:51.911039 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e1f72544-openshift-default-86c84fd5b-khbgn" Sep 10 23:55:52.191281 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:55:52.191240 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09db6ab7_c732_41e1_a76f_ca403f1eb53f.slice/crio-a2575f3c39012d21e7b030628129c031e398dc4675707700014d99712bd4ea26 WatchSource:0}: Error finding container a2575f3c39012d21e7b030628129c031e398dc4675707700014d99712bd4ea26: Status 404 returned error can't find the container with id a2575f3c39012d21e7b030628129c031e398dc4675707700014d99712bd4ea26 Sep 10 23:55:52.192030 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:52.191995 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e1f72544-openshift-default-86c84fd5b-khbgn"] Sep 10 23:55:52.193923 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:52.193887 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 23:55:52.194037 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:52.193955 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 23:55:52.194037 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:52.193985 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 23:55:53.046112 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:53.046076 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e1f72544-openshift-default-86c84fd5b-khbgn" event={"ID":"09db6ab7-c732-41e1-a76f-ca403f1eb53f","Type":"ContainerStarted","Data":"6abf676fe98125e2abad96bede7490fc15d473f6e5cef420f71fcad0387ec67e"} Sep 10 23:55:53.046536 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:53.046171 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e1f72544-openshift-default-86c84fd5b-khbgn" event={"ID":"09db6ab7-c732-41e1-a76f-ca403f1eb53f","Type":"ContainerStarted","Data":"a2575f3c39012d21e7b030628129c031e398dc4675707700014d99712bd4ea26"} Sep 10 23:55:53.068389 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:53.068332 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-e1f72544-openshift-default-86c84fd5b-khbgn" podStartSLOduration=2.068315559 podStartE2EDuration="2.068315559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:55:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:55:53.066125301 +0000 UTC m=+1144.453728065" watchObservedRunningTime="2026-09-10 23:55:53.068315559 +0000 UTC m=+1144.455918318" Sep 10 23:55:53.911576 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:53.911528 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-e1f72544-openshift-default-86c84fd5b-khbgn" Sep 10 23:55:53.917447 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:53.917419 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-e1f72544-openshift-default-86c84fd5b-khbgn" Sep 10 23:55:53.988896 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:53.988854 2574 patch_prober.go:28] interesting pod/e2e-shared-b-w3-4094ce-openshift-default-7558d5c9fb-vp9l5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 10 23:55:53.989056 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:53.988911 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-4094ce-openshift-default-7558d5c9fb-vp9l5" podUID="e85f9c50-8a5c-4191-9441-9b30ca611ffb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 10 23:55:54.053120 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:54.053084 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e1f72544-openshift-default-86c84fd5b-khbgn" Sep 10 23:55:54.054027 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:54.054005 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-e1f72544-openshift-default-86c84fd5b-khbgn" Sep 10 23:55:56.066427 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:56.066396 2574 generic.go:356] "Generic (PLEG): container finished" podID="e85f9c50-8a5c-4191-9441-9b30ca611ffb" containerID="86d66a2854991aac386fda6ac27492cd24fbbe529fba777c342907572eee3fd6" exitCode=0 Sep 10 23:55:56.066934 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:56.066467 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-4094ce-openshift-default-7558d5c9fb-vp9l5" event={"ID":"e85f9c50-8a5c-4191-9441-9b30ca611ffb","Type":"ContainerDied","Data":"86d66a2854991aac386fda6ac27492cd24fbbe529fba777c342907572eee3fd6"} Sep 10 23:55:56.066990 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:56.066960 2574 scope.go:117] "RemoveContainer" containerID="86d66a2854991aac386fda6ac27492cd24fbbe529fba777c342907572eee3fd6" Sep 10 23:55:56.068143 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:56.068112 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 23:55:56.068243 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:56.068185 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 23:55:56.068243 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:56.068217 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 23:55:57.076261 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:57.076225 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-4094ce-openshift-default-7558d5c9fb-vp9l5" event={"ID":"e85f9c50-8a5c-4191-9441-9b30ca611ffb","Type":"ContainerStarted","Data":"221fd91fca6256fbabd3db5d34ad3c7b8bcb0b9dd81ae92a05d5145c578f58af"} Sep 10 23:55:57.535434 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:57.535400 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-4094ce-openshift-default-7558d5c9fb-vp9l5" Sep 10 23:55:57.535664 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:57.535632 2574 patch_prober.go:28] interesting pod/e2e-shared-b-w3-4094ce-openshift-default-7558d5c9fb-vp9l5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Sep 10 23:55:57.535720 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:57.535692 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-4094ce-openshift-default-7558d5c9fb-vp9l5" podUID="e85f9c50-8a5c-4191-9441-9b30ca611ffb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Sep 10 23:55:58.535485 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:58.535442 2574 patch_prober.go:28] interesting pod/e2e-shared-b-w3-4094ce-openshift-default-7558d5c9fb-vp9l5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Sep 10 23:55:58.535899 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:58.535512 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-4094ce-openshift-default-7558d5c9fb-vp9l5" podUID="e85f9c50-8a5c-4191-9441-9b30ca611ffb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Sep 10 23:55:59.536013 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:59.535982 2574 patch_prober.go:28] interesting pod/e2e-shared-b-w3-4094ce-openshift-default-7558d5c9fb-vp9l5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Sep 10 23:55:59.536400 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:55:59.536034 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-4094ce-openshift-default-7558d5c9fb-vp9l5" podUID="e85f9c50-8a5c-4191-9441-9b30ca611ffb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Sep 10 23:56:00.535505 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:56:00.535465 2574 patch_prober.go:28] interesting pod/e2e-shared-b-w3-4094ce-openshift-default-7558d5c9fb-vp9l5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Sep 10 23:56:00.535807 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:56:00.535529 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-4094ce-openshift-default-7558d5c9fb-vp9l5" podUID="e85f9c50-8a5c-4191-9441-9b30ca611ffb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Sep 10 23:56:01.535593 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:56:01.535555 2574 patch_prober.go:28] interesting pod/e2e-shared-b-w3-4094ce-openshift-default-7558d5c9fb-vp9l5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Sep 10 23:56:01.536006 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:56:01.535629 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-4094ce-openshift-default-7558d5c9fb-vp9l5" podUID="e85f9c50-8a5c-4191-9441-9b30ca611ffb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Sep 10 23:56:02.386349 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:56:02.386294 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-56r7l"] Sep 10 23:56:02.535743 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:56:02.535705 2574 patch_prober.go:28] interesting pod/e2e-shared-b-w3-4094ce-openshift-default-7558d5c9fb-vp9l5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Sep 10 23:56:02.536118 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:56:02.535768 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-4094ce-openshift-default-7558d5c9fb-vp9l5" podUID="e85f9c50-8a5c-4191-9441-9b30ca611ffb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Sep 10 23:56:03.539816 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:56:03.539779 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-4094ce-openshift-default-7558d5c9fb-vp9l5" Sep 10 23:56:03.540195 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:56:03.540056 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-4094ce-openshift-default-7558d5c9fb-vp9l5" Sep 10 23:56:03.540799 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:56:03.540779 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-4094ce-openshift-default-7558d5c9fb-vp9l5" Sep 10 23:56:13.183702 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:56:13.183614 2574 generic.go:356] "Generic (PLEG): container finished" podID="09db6ab7-c732-41e1-a76f-ca403f1eb53f" containerID="6abf676fe98125e2abad96bede7490fc15d473f6e5cef420f71fcad0387ec67e" exitCode=0 Sep 10 23:56:13.184128 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:56:13.183696 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e1f72544-openshift-default-86c84fd5b-khbgn" event={"ID":"09db6ab7-c732-41e1-a76f-ca403f1eb53f","Type":"ContainerDied","Data":"6abf676fe98125e2abad96bede7490fc15d473f6e5cef420f71fcad0387ec67e"} Sep 10 23:56:13.184200 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:56:13.184129 2574 scope.go:117] "RemoveContainer" containerID="6abf676fe98125e2abad96bede7490fc15d473f6e5cef420f71fcad0387ec67e" Sep 10 23:56:13.185264 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:56:13.185235 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 23:56:13.185316 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:56:13.185305 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 23:56:13.185395 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:56:13.185341 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 23:56:14.197509 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:56:14.197476 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e1f72544-openshift-default-86c84fd5b-khbgn" event={"ID":"09db6ab7-c732-41e1-a76f-ca403f1eb53f","Type":"ContainerStarted","Data":"94d40f759bccba7bb3096ec64a4340e52d8df68547021ef3c8c586c6bfd67940"} Sep 10 23:56:14.912197 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:56:14.912157 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-e1f72544-openshift-default-86c84fd5b-khbgn" Sep 10 23:56:14.912395 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:56:14.912373 2574 patch_prober.go:28] interesting pod/e2e-mt-e1f72544-openshift-default-86c84fd5b-khbgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Sep 10 23:56:14.912445 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:56:14.912422 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e1f72544-openshift-default-86c84fd5b-khbgn" podUID="09db6ab7-c732-41e1-a76f-ca403f1eb53f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Sep 10 23:56:15.912513 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:56:15.912477 2574 patch_prober.go:28] interesting pod/e2e-mt-e1f72544-openshift-default-86c84fd5b-khbgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Sep 10 23:56:15.912913 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:56:15.912534 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e1f72544-openshift-default-86c84fd5b-khbgn" podUID="09db6ab7-c732-41e1-a76f-ca403f1eb53f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Sep 10 23:56:16.911586 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:56:16.911550 2574 patch_prober.go:28] interesting pod/e2e-mt-e1f72544-openshift-default-86c84fd5b-khbgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Sep 10 23:56:16.911787 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:56:16.911610 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e1f72544-openshift-default-86c84fd5b-khbgn" podUID="09db6ab7-c732-41e1-a76f-ca403f1eb53f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Sep 10 23:56:17.912327 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:56:17.912293 2574 patch_prober.go:28] interesting pod/e2e-mt-e1f72544-openshift-default-86c84fd5b-khbgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Sep 10 23:56:17.912767 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:56:17.912351 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e1f72544-openshift-default-86c84fd5b-khbgn" podUID="09db6ab7-c732-41e1-a76f-ca403f1eb53f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Sep 10 23:56:18.787402 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:56:18.787365 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-56r7l"] Sep 10 23:56:18.912498 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:56:18.912461 2574 patch_prober.go:28] interesting pod/e2e-mt-e1f72544-openshift-default-86c84fd5b-khbgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Sep 10 23:56:18.912928 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:56:18.912520 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e1f72544-openshift-default-86c84fd5b-khbgn" podUID="09db6ab7-c732-41e1-a76f-ca403f1eb53f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Sep 10 23:56:19.916152 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:56:19.916122 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-e1f72544-openshift-default-86c84fd5b-khbgn" Sep 10 23:56:19.916531 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:56:19.916519 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e1f72544-openshift-default-86c84fd5b-khbgn" Sep 10 23:56:19.917104 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:56:19.917082 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-e1f72544-openshift-default-86c84fd5b-khbgn" Sep 10 23:56:35.886100 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:56:35.886061 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-56r7l"] Sep 10 23:56:50.275148 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:56:50.275115 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-56r7l"] Sep 10 23:57:06.483227 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:57:06.483190 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-56r7l"] Sep 10 23:57:24.573343 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:57:24.573302 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-56r7l"] Sep 10 23:57:39.074779 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:57:39.074743 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-56r7l"] Sep 10 23:57:47.375824 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:57:47.375740 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e1f72544-openshift-default-86c84fd5b-khbgn"] Sep 10 23:57:47.376205 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:57:47.375976 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-e1f72544-openshift-default-86c84fd5b-khbgn" podUID="09db6ab7-c732-41e1-a76f-ca403f1eb53f" containerName="istio-proxy" containerID="cri-o://94d40f759bccba7bb3096ec64a4340e52d8df68547021ef3c8c586c6bfd67940" gracePeriod=30 Sep 10 23:57:49.919040 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:57:49.919006 2574 patch_prober.go:28] interesting pod/e2e-mt-e1f72544-openshift-default-86c84fd5b-khbgn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 10 23:57:49.919410 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:57:49.919073 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-e1f72544-openshift-default-86c84fd5b-khbgn" podUID="09db6ab7-c732-41e1-a76f-ca403f1eb53f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 10 23:57:52.633268 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:57:52.633231 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e1f72544-openshift-default-86c84fd5b-khbgn" Sep 10 23:57:52.704361 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:57:52.704336 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09db6ab7-c732-41e1-a76f-ca403f1eb53f-workload-socket\") pod \"09db6ab7-c732-41e1-a76f-ca403f1eb53f\" (UID: \"09db6ab7-c732-41e1-a76f-ca403f1eb53f\") " Sep 10 23:57:52.704531 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:57:52.704390 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09db6ab7-c732-41e1-a76f-ca403f1eb53f-credential-socket\") pod \"09db6ab7-c732-41e1-a76f-ca403f1eb53f\" (UID: \"09db6ab7-c732-41e1-a76f-ca403f1eb53f\") " Sep 10 23:57:52.704531 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:57:52.704427 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09db6ab7-c732-41e1-a76f-ca403f1eb53f-istio-podinfo\") pod \"09db6ab7-c732-41e1-a76f-ca403f1eb53f\" (UID: \"09db6ab7-c732-41e1-a76f-ca403f1eb53f\") " Sep 10 23:57:52.704531 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:57:52.704450 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09db6ab7-c732-41e1-a76f-ca403f1eb53f-workload-certs\") pod \"09db6ab7-c732-41e1-a76f-ca403f1eb53f\" (UID: \"09db6ab7-c732-41e1-a76f-ca403f1eb53f\") " Sep 10 23:57:52.704531 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:57:52.704483 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09db6ab7-c732-41e1-a76f-ca403f1eb53f-istiod-ca-cert\") pod \"09db6ab7-c732-41e1-a76f-ca403f1eb53f\" (UID: \"09db6ab7-c732-41e1-a76f-ca403f1eb53f\") " Sep 10 23:57:52.704531 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:57:52.704507 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09db6ab7-c732-41e1-a76f-ca403f1eb53f-istio-envoy\") pod \"09db6ab7-c732-41e1-a76f-ca403f1eb53f\" (UID: \"09db6ab7-c732-41e1-a76f-ca403f1eb53f\") " Sep 10 23:57:52.704847 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:57:52.704593 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9jbvn\" (UniqueName: \"kubernetes.io/projected/09db6ab7-c732-41e1-a76f-ca403f1eb53f-kube-api-access-9jbvn\") pod \"09db6ab7-c732-41e1-a76f-ca403f1eb53f\" (UID: \"09db6ab7-c732-41e1-a76f-ca403f1eb53f\") " Sep 10 23:57:52.704847 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:57:52.704675 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09db6ab7-c732-41e1-a76f-ca403f1eb53f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "09db6ab7-c732-41e1-a76f-ca403f1eb53f" (UID: "09db6ab7-c732-41e1-a76f-ca403f1eb53f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:57:52.704847 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:57:52.704735 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09db6ab7-c732-41e1-a76f-ca403f1eb53f-istio-data\") pod \"09db6ab7-c732-41e1-a76f-ca403f1eb53f\" (UID: \"09db6ab7-c732-41e1-a76f-ca403f1eb53f\") " Sep 10 23:57:52.704847 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:57:52.704765 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09db6ab7-c732-41e1-a76f-ca403f1eb53f-istio-token\") pod \"09db6ab7-c732-41e1-a76f-ca403f1eb53f\" (UID: \"09db6ab7-c732-41e1-a76f-ca403f1eb53f\") " Sep 10 23:57:52.704847 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:57:52.704786 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09db6ab7-c732-41e1-a76f-ca403f1eb53f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "09db6ab7-c732-41e1-a76f-ca403f1eb53f" (UID: "09db6ab7-c732-41e1-a76f-ca403f1eb53f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:57:52.704847 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:57:52.704801 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09db6ab7-c732-41e1-a76f-ca403f1eb53f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "09db6ab7-c732-41e1-a76f-ca403f1eb53f" (UID: "09db6ab7-c732-41e1-a76f-ca403f1eb53f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:57:52.705160 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:57:52.704989 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09db6ab7-c732-41e1-a76f-ca403f1eb53f-istio-data" (OuterVolumeSpecName: "istio-data") pod "09db6ab7-c732-41e1-a76f-ca403f1eb53f" (UID: "09db6ab7-c732-41e1-a76f-ca403f1eb53f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:57:52.705160 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:57:52.705105 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09db6ab7-c732-41e1-a76f-ca403f1eb53f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "09db6ab7-c732-41e1-a76f-ca403f1eb53f" (UID: "09db6ab7-c732-41e1-a76f-ca403f1eb53f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 23:57:52.705251 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:57:52.705212 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09db6ab7-c732-41e1-a76f-ca403f1eb53f-istiod-ca-cert\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:57:52.705251 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:57:52.705234 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09db6ab7-c732-41e1-a76f-ca403f1eb53f-istio-data\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:57:52.705251 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:57:52.705246 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09db6ab7-c732-41e1-a76f-ca403f1eb53f-workload-socket\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:57:52.705357 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:57:52.705262 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09db6ab7-c732-41e1-a76f-ca403f1eb53f-credential-socket\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:57:52.705357 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:57:52.705277 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09db6ab7-c732-41e1-a76f-ca403f1eb53f-workload-certs\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:57:52.706855 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:57:52.706820 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09db6ab7-c732-41e1-a76f-ca403f1eb53f-kube-api-access-9jbvn" (OuterVolumeSpecName: "kube-api-access-9jbvn") pod "09db6ab7-c732-41e1-a76f-ca403f1eb53f" (UID: "09db6ab7-c732-41e1-a76f-ca403f1eb53f"). InnerVolumeSpecName "kube-api-access-9jbvn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 23:57:52.706961 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:57:52.706874 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/09db6ab7-c732-41e1-a76f-ca403f1eb53f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "09db6ab7-c732-41e1-a76f-ca403f1eb53f" (UID: "09db6ab7-c732-41e1-a76f-ca403f1eb53f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 23:57:52.707087 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:57:52.707066 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09db6ab7-c732-41e1-a76f-ca403f1eb53f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "09db6ab7-c732-41e1-a76f-ca403f1eb53f" (UID: "09db6ab7-c732-41e1-a76f-ca403f1eb53f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 23:57:52.707144 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:57:52.707098 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09db6ab7-c732-41e1-a76f-ca403f1eb53f-istio-token" (OuterVolumeSpecName: "istio-token") pod "09db6ab7-c732-41e1-a76f-ca403f1eb53f" (UID: "09db6ab7-c732-41e1-a76f-ca403f1eb53f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 23:57:52.806358 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:57:52.806316 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09db6ab7-c732-41e1-a76f-ca403f1eb53f-istio-envoy\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:57:52.806358 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:57:52.806354 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jbvn\" (UniqueName: \"kubernetes.io/projected/09db6ab7-c732-41e1-a76f-ca403f1eb53f-kube-api-access-9jbvn\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:57:52.806358 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:57:52.806364 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09db6ab7-c732-41e1-a76f-ca403f1eb53f-istio-token\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:57:52.806358 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:57:52.806373 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09db6ab7-c732-41e1-a76f-ca403f1eb53f-istio-podinfo\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 10 23:57:52.829021 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:57:52.828988 2574 generic.go:356] "Generic (PLEG): container finished" podID="09db6ab7-c732-41e1-a76f-ca403f1eb53f" containerID="94d40f759bccba7bb3096ec64a4340e52d8df68547021ef3c8c586c6bfd67940" exitCode=0 Sep 10 23:57:52.829158 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:57:52.829035 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e1f72544-openshift-default-86c84fd5b-khbgn" Sep 10 23:57:52.829158 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:57:52.829044 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e1f72544-openshift-default-86c84fd5b-khbgn" event={"ID":"09db6ab7-c732-41e1-a76f-ca403f1eb53f","Type":"ContainerDied","Data":"94d40f759bccba7bb3096ec64a4340e52d8df68547021ef3c8c586c6bfd67940"} Sep 10 23:57:52.829158 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:57:52.829095 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e1f72544-openshift-default-86c84fd5b-khbgn" event={"ID":"09db6ab7-c732-41e1-a76f-ca403f1eb53f","Type":"ContainerDied","Data":"a2575f3c39012d21e7b030628129c031e398dc4675707700014d99712bd4ea26"} Sep 10 23:57:52.829158 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:57:52.829119 2574 scope.go:117] "RemoveContainer" containerID="94d40f759bccba7bb3096ec64a4340e52d8df68547021ef3c8c586c6bfd67940" Sep 10 23:57:52.839847 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:57:52.839826 2574 scope.go:117] "RemoveContainer" containerID="6abf676fe98125e2abad96bede7490fc15d473f6e5cef420f71fcad0387ec67e" Sep 10 23:57:52.848629 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:57:52.848600 2574 scope.go:117] "RemoveContainer" containerID="94d40f759bccba7bb3096ec64a4340e52d8df68547021ef3c8c586c6bfd67940" Sep 10 23:57:52.848909 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:57:52.848887 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94d40f759bccba7bb3096ec64a4340e52d8df68547021ef3c8c586c6bfd67940\": container with ID starting with 94d40f759bccba7bb3096ec64a4340e52d8df68547021ef3c8c586c6bfd67940 not found: ID does not exist" containerID="94d40f759bccba7bb3096ec64a4340e52d8df68547021ef3c8c586c6bfd67940" Sep 10 23:57:52.848995 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:57:52.848917 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94d40f759bccba7bb3096ec64a4340e52d8df68547021ef3c8c586c6bfd67940"} err="failed to get container status \"94d40f759bccba7bb3096ec64a4340e52d8df68547021ef3c8c586c6bfd67940\": rpc error: code = NotFound desc = could not find container \"94d40f759bccba7bb3096ec64a4340e52d8df68547021ef3c8c586c6bfd67940\": container with ID starting with 94d40f759bccba7bb3096ec64a4340e52d8df68547021ef3c8c586c6bfd67940 not found: ID does not exist" Sep 10 23:57:52.848995 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:57:52.848934 2574 scope.go:117] "RemoveContainer" containerID="6abf676fe98125e2abad96bede7490fc15d473f6e5cef420f71fcad0387ec67e" Sep 10 23:57:52.849198 ip-10-0-138-128 kubenswrapper[2574]: E0910 23:57:52.849178 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6abf676fe98125e2abad96bede7490fc15d473f6e5cef420f71fcad0387ec67e\": container with ID starting with 6abf676fe98125e2abad96bede7490fc15d473f6e5cef420f71fcad0387ec67e not found: ID does not exist" containerID="6abf676fe98125e2abad96bede7490fc15d473f6e5cef420f71fcad0387ec67e" Sep 10 23:57:52.849258 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:57:52.849207 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6abf676fe98125e2abad96bede7490fc15d473f6e5cef420f71fcad0387ec67e"} err="failed to get container status \"6abf676fe98125e2abad96bede7490fc15d473f6e5cef420f71fcad0387ec67e\": rpc error: code = NotFound desc = could not find container \"6abf676fe98125e2abad96bede7490fc15d473f6e5cef420f71fcad0387ec67e\": container with ID starting with 6abf676fe98125e2abad96bede7490fc15d473f6e5cef420f71fcad0387ec67e not found: ID does not exist" Sep 10 23:57:52.851515 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:57:52.851489 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e1f72544-openshift-default-86c84fd5b-khbgn"] Sep 10 23:57:52.855964 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:57:52.855943 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-e1f72544-openshift-default-86c84fd5b-khbgn"] Sep 10 23:57:53.094450 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:57:53.094366 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="09db6ab7-c732-41e1-a76f-ca403f1eb53f" path="/var/lib/kubelet/pods/09db6ab7-c732-41e1-a76f-ca403f1eb53f/volumes" Sep 10 23:57:55.577574 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:57:55.577536 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-56r7l"] Sep 10 23:58:09.173901 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:58:09.173864 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-56r7l"] Sep 10 23:58:23.272788 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:58:23.272752 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-56r7l"] Sep 10 23:58:39.574740 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:58:39.574701 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-56r7l"] Sep 10 23:58:41.444909 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:58:41.444877 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-8b47cf-openshift-default-85854f7c9b-mtcrx"] Sep 10 23:58:41.445360 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:58:41.445347 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09db6ab7-c732-41e1-a76f-ca403f1eb53f" containerName="istio-proxy" Sep 10 23:58:41.445411 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:58:41.445362 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="09db6ab7-c732-41e1-a76f-ca403f1eb53f" containerName="istio-proxy" Sep 10 23:58:41.445446 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:58:41.445437 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="09db6ab7-c732-41e1-a76f-ca403f1eb53f" containerName="istio-proxy" Sep 10 23:58:41.445481 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:58:41.445448 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="09db6ab7-c732-41e1-a76f-ca403f1eb53f" containerName="istio-proxy" Sep 10 23:58:41.448826 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:58:41.448800 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-8b47cf-openshift-default-85854f7c9b-mtcrx" Sep 10 23:58:41.451001 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:58:41.450979 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-8b47cf-openshift-default-dockercfg-xl58p\"" Sep 10 23:58:41.460916 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:58:41.460891 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-8b47cf-openshift-default-85854f7c9b-mtcrx"] Sep 10 23:58:41.493144 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:58:41.493114 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eef2f53c-651e-41b0-a742-2d509e67bf4b-istio-token\") pod \"e2e-api-a-8b47cf-openshift-default-85854f7c9b-mtcrx\" (UID: \"eef2f53c-651e-41b0-a742-2d509e67bf4b\") " pod="openshift-ingress/e2e-api-a-8b47cf-openshift-default-85854f7c9b-mtcrx" Sep 10 23:58:41.493318 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:58:41.493152 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eef2f53c-651e-41b0-a742-2d509e67bf4b-istio-data\") pod \"e2e-api-a-8b47cf-openshift-default-85854f7c9b-mtcrx\" (UID: \"eef2f53c-651e-41b0-a742-2d509e67bf4b\") " pod="openshift-ingress/e2e-api-a-8b47cf-openshift-default-85854f7c9b-mtcrx" Sep 10 23:58:41.493318 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:58:41.493177 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eef2f53c-651e-41b0-a742-2d509e67bf4b-istio-envoy\") pod \"e2e-api-a-8b47cf-openshift-default-85854f7c9b-mtcrx\" (UID: \"eef2f53c-651e-41b0-a742-2d509e67bf4b\") " pod="openshift-ingress/e2e-api-a-8b47cf-openshift-default-85854f7c9b-mtcrx" Sep 10 23:58:41.493318 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:58:41.493210 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eef2f53c-651e-41b0-a742-2d509e67bf4b-istiod-ca-cert\") pod \"e2e-api-a-8b47cf-openshift-default-85854f7c9b-mtcrx\" (UID: \"eef2f53c-651e-41b0-a742-2d509e67bf4b\") " pod="openshift-ingress/e2e-api-a-8b47cf-openshift-default-85854f7c9b-mtcrx" Sep 10 23:58:41.493318 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:58:41.493241 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eef2f53c-651e-41b0-a742-2d509e67bf4b-workload-certs\") pod \"e2e-api-a-8b47cf-openshift-default-85854f7c9b-mtcrx\" (UID: \"eef2f53c-651e-41b0-a742-2d509e67bf4b\") " pod="openshift-ingress/e2e-api-a-8b47cf-openshift-default-85854f7c9b-mtcrx" Sep 10 23:58:41.493318 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:58:41.493259 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7ntb\" (UniqueName: \"kubernetes.io/projected/eef2f53c-651e-41b0-a742-2d509e67bf4b-kube-api-access-q7ntb\") pod \"e2e-api-a-8b47cf-openshift-default-85854f7c9b-mtcrx\" (UID: \"eef2f53c-651e-41b0-a742-2d509e67bf4b\") " pod="openshift-ingress/e2e-api-a-8b47cf-openshift-default-85854f7c9b-mtcrx" Sep 10 23:58:41.493604 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:58:41.493336 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eef2f53c-651e-41b0-a742-2d509e67bf4b-workload-socket\") pod \"e2e-api-a-8b47cf-openshift-default-85854f7c9b-mtcrx\" (UID: \"eef2f53c-651e-41b0-a742-2d509e67bf4b\") " pod="openshift-ingress/e2e-api-a-8b47cf-openshift-default-85854f7c9b-mtcrx" Sep 10 23:58:41.493604 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:58:41.493380 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eef2f53c-651e-41b0-a742-2d509e67bf4b-istio-podinfo\") pod \"e2e-api-a-8b47cf-openshift-default-85854f7c9b-mtcrx\" (UID: \"eef2f53c-651e-41b0-a742-2d509e67bf4b\") " pod="openshift-ingress/e2e-api-a-8b47cf-openshift-default-85854f7c9b-mtcrx" Sep 10 23:58:41.493604 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:58:41.493442 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eef2f53c-651e-41b0-a742-2d509e67bf4b-credential-socket\") pod \"e2e-api-a-8b47cf-openshift-default-85854f7c9b-mtcrx\" (UID: \"eef2f53c-651e-41b0-a742-2d509e67bf4b\") " pod="openshift-ingress/e2e-api-a-8b47cf-openshift-default-85854f7c9b-mtcrx" Sep 10 23:58:41.594435 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:58:41.594400 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eef2f53c-651e-41b0-a742-2d509e67bf4b-workload-socket\") pod \"e2e-api-a-8b47cf-openshift-default-85854f7c9b-mtcrx\" (UID: \"eef2f53c-651e-41b0-a742-2d509e67bf4b\") " pod="openshift-ingress/e2e-api-a-8b47cf-openshift-default-85854f7c9b-mtcrx" Sep 10 23:58:41.594608 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:58:41.594474 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eef2f53c-651e-41b0-a742-2d509e67bf4b-istio-podinfo\") pod \"e2e-api-a-8b47cf-openshift-default-85854f7c9b-mtcrx\" (UID: \"eef2f53c-651e-41b0-a742-2d509e67bf4b\") " pod="openshift-ingress/e2e-api-a-8b47cf-openshift-default-85854f7c9b-mtcrx" Sep 10 23:58:41.594608 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:58:41.594542 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eef2f53c-651e-41b0-a742-2d509e67bf4b-credential-socket\") pod \"e2e-api-a-8b47cf-openshift-default-85854f7c9b-mtcrx\" (UID: \"eef2f53c-651e-41b0-a742-2d509e67bf4b\") " pod="openshift-ingress/e2e-api-a-8b47cf-openshift-default-85854f7c9b-mtcrx" Sep 10 23:58:41.594608 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:58:41.594593 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eef2f53c-651e-41b0-a742-2d509e67bf4b-istio-token\") pod \"e2e-api-a-8b47cf-openshift-default-85854f7c9b-mtcrx\" (UID: \"eef2f53c-651e-41b0-a742-2d509e67bf4b\") " pod="openshift-ingress/e2e-api-a-8b47cf-openshift-default-85854f7c9b-mtcrx" Sep 10 23:58:41.594809 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:58:41.594751 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eef2f53c-651e-41b0-a742-2d509e67bf4b-istio-data\") pod \"e2e-api-a-8b47cf-openshift-default-85854f7c9b-mtcrx\" (UID: \"eef2f53c-651e-41b0-a742-2d509e67bf4b\") " pod="openshift-ingress/e2e-api-a-8b47cf-openshift-default-85854f7c9b-mtcrx" Sep 10 23:58:41.594913 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:58:41.594886 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eef2f53c-651e-41b0-a742-2d509e67bf4b-workload-socket\") pod \"e2e-api-a-8b47cf-openshift-default-85854f7c9b-mtcrx\" (UID: \"eef2f53c-651e-41b0-a742-2d509e67bf4b\") " pod="openshift-ingress/e2e-api-a-8b47cf-openshift-default-85854f7c9b-mtcrx" Sep 10 23:58:41.595082 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:58:41.594916 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eef2f53c-651e-41b0-a742-2d509e67bf4b-istio-envoy\") pod \"e2e-api-a-8b47cf-openshift-default-85854f7c9b-mtcrx\" (UID: \"eef2f53c-651e-41b0-a742-2d509e67bf4b\") " pod="openshift-ingress/e2e-api-a-8b47cf-openshift-default-85854f7c9b-mtcrx" Sep 10 23:58:41.595196 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:58:41.595076 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eef2f53c-651e-41b0-a742-2d509e67bf4b-istio-data\") pod \"e2e-api-a-8b47cf-openshift-default-85854f7c9b-mtcrx\" (UID: \"eef2f53c-651e-41b0-a742-2d509e67bf4b\") " pod="openshift-ingress/e2e-api-a-8b47cf-openshift-default-85854f7c9b-mtcrx" Sep 10 23:58:41.595196 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:58:41.595098 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eef2f53c-651e-41b0-a742-2d509e67bf4b-istiod-ca-cert\") pod \"e2e-api-a-8b47cf-openshift-default-85854f7c9b-mtcrx\" (UID: \"eef2f53c-651e-41b0-a742-2d509e67bf4b\") " pod="openshift-ingress/e2e-api-a-8b47cf-openshift-default-85854f7c9b-mtcrx" Sep 10 23:58:41.595196 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:58:41.594921 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eef2f53c-651e-41b0-a742-2d509e67bf4b-credential-socket\") pod \"e2e-api-a-8b47cf-openshift-default-85854f7c9b-mtcrx\" (UID: \"eef2f53c-651e-41b0-a742-2d509e67bf4b\") " pod="openshift-ingress/e2e-api-a-8b47cf-openshift-default-85854f7c9b-mtcrx" Sep 10 23:58:41.595196 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:58:41.595121 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eef2f53c-651e-41b0-a742-2d509e67bf4b-workload-certs\") pod \"e2e-api-a-8b47cf-openshift-default-85854f7c9b-mtcrx\" (UID: \"eef2f53c-651e-41b0-a742-2d509e67bf4b\") " pod="openshift-ingress/e2e-api-a-8b47cf-openshift-default-85854f7c9b-mtcrx" Sep 10 23:58:41.595196 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:58:41.595144 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7ntb\" (UniqueName: \"kubernetes.io/projected/eef2f53c-651e-41b0-a742-2d509e67bf4b-kube-api-access-q7ntb\") pod \"e2e-api-a-8b47cf-openshift-default-85854f7c9b-mtcrx\" (UID: \"eef2f53c-651e-41b0-a742-2d509e67bf4b\") " pod="openshift-ingress/e2e-api-a-8b47cf-openshift-default-85854f7c9b-mtcrx" Sep 10 23:58:41.595478 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:58:41.595422 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eef2f53c-651e-41b0-a742-2d509e67bf4b-workload-certs\") pod \"e2e-api-a-8b47cf-openshift-default-85854f7c9b-mtcrx\" (UID: \"eef2f53c-651e-41b0-a742-2d509e67bf4b\") " pod="openshift-ingress/e2e-api-a-8b47cf-openshift-default-85854f7c9b-mtcrx" Sep 10 23:58:41.595725 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:58:41.595707 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eef2f53c-651e-41b0-a742-2d509e67bf4b-istiod-ca-cert\") pod \"e2e-api-a-8b47cf-openshift-default-85854f7c9b-mtcrx\" (UID: \"eef2f53c-651e-41b0-a742-2d509e67bf4b\") " pod="openshift-ingress/e2e-api-a-8b47cf-openshift-default-85854f7c9b-mtcrx" Sep 10 23:58:41.597001 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:58:41.596982 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eef2f53c-651e-41b0-a742-2d509e67bf4b-istio-envoy\") pod \"e2e-api-a-8b47cf-openshift-default-85854f7c9b-mtcrx\" (UID: \"eef2f53c-651e-41b0-a742-2d509e67bf4b\") " pod="openshift-ingress/e2e-api-a-8b47cf-openshift-default-85854f7c9b-mtcrx" Sep 10 23:58:41.597088 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:58:41.597027 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eef2f53c-651e-41b0-a742-2d509e67bf4b-istio-podinfo\") pod \"e2e-api-a-8b47cf-openshift-default-85854f7c9b-mtcrx\" (UID: \"eef2f53c-651e-41b0-a742-2d509e67bf4b\") " pod="openshift-ingress/e2e-api-a-8b47cf-openshift-default-85854f7c9b-mtcrx" Sep 10 23:58:41.605288 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:58:41.605262 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eef2f53c-651e-41b0-a742-2d509e67bf4b-istio-token\") pod \"e2e-api-a-8b47cf-openshift-default-85854f7c9b-mtcrx\" (UID: \"eef2f53c-651e-41b0-a742-2d509e67bf4b\") " pod="openshift-ingress/e2e-api-a-8b47cf-openshift-default-85854f7c9b-mtcrx" Sep 10 23:58:41.605396 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:58:41.605322 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7ntb\" (UniqueName: \"kubernetes.io/projected/eef2f53c-651e-41b0-a742-2d509e67bf4b-kube-api-access-q7ntb\") pod \"e2e-api-a-8b47cf-openshift-default-85854f7c9b-mtcrx\" (UID: \"eef2f53c-651e-41b0-a742-2d509e67bf4b\") " pod="openshift-ingress/e2e-api-a-8b47cf-openshift-default-85854f7c9b-mtcrx" Sep 10 23:58:41.760777 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:58:41.760746 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-8b47cf-openshift-default-85854f7c9b-mtcrx" Sep 10 23:58:42.038916 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:58:42.038890 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-8b47cf-openshift-default-85854f7c9b-mtcrx"] Sep 10 23:58:42.039614 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:58:42.039586 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeef2f53c_651e_41b0_a742_2d509e67bf4b.slice/crio-1e2a553dffb979033fbbc19c4d47aa04166885d9cdbe1d5cf91e918adaae1cc1 WatchSource:0}: Error finding container 1e2a553dffb979033fbbc19c4d47aa04166885d9cdbe1d5cf91e918adaae1cc1: Status 404 returned error can't find the container with id 1e2a553dffb979033fbbc19c4d47aa04166885d9cdbe1d5cf91e918adaae1cc1 Sep 10 23:58:42.042104 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:58:42.042068 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 23:58:42.042185 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:58:42.042157 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 23:58:42.042225 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:58:42.042185 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 23:58:42.132774 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:58:42.132743 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8b47cf-openshift-default-85854f7c9b-mtcrx" event={"ID":"eef2f53c-651e-41b0-a742-2d509e67bf4b","Type":"ContainerStarted","Data":"1ca3f87921b5823c0252a4b2295c422cbc1b1f6a35ff381b84c10a0c76f42128"} Sep 10 23:58:42.132928 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:58:42.132803 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8b47cf-openshift-default-85854f7c9b-mtcrx" event={"ID":"eef2f53c-651e-41b0-a742-2d509e67bf4b","Type":"ContainerStarted","Data":"1e2a553dffb979033fbbc19c4d47aa04166885d9cdbe1d5cf91e918adaae1cc1"} Sep 10 23:58:42.152344 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:58:42.151711 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-8b47cf-openshift-default-85854f7c9b-mtcrx" podStartSLOduration=1.151689929 podStartE2EDuration="1.151689929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:58:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:58:42.151304027 +0000 UTC m=+1313.538906791" watchObservedRunningTime="2026-09-10 23:58:42.151689929 +0000 UTC m=+1313.539292688" Sep 10 23:58:42.761476 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:58:42.761441 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-8b47cf-openshift-default-85854f7c9b-mtcrx" Sep 10 23:58:43.766551 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:58:43.766514 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-8b47cf-openshift-default-85854f7c9b-mtcrx" Sep 10 23:58:44.145496 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:58:44.145415 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-8b47cf-openshift-default-85854f7c9b-mtcrx" Sep 10 23:58:44.146368 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:58:44.146349 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-8b47cf-openshift-default-85854f7c9b-mtcrx" Sep 10 23:58:56.674377 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:58:56.674335 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-56r7l"] Sep 10 23:58:59.147279 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:58:59.147241 2574 patch_prober.go:28] interesting pod/e2e-api-a-8b47cf-openshift-default-85854f7c9b-mtcrx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Sep 10 23:58:59.147795 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:58:59.147303 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-8b47cf-openshift-default-85854f7c9b-mtcrx" podUID="eef2f53c-651e-41b0-a742-2d509e67bf4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Sep 10 23:58:59.251045 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:58:59.251013 2574 generic.go:356] "Generic (PLEG): container finished" podID="eef2f53c-651e-41b0-a742-2d509e67bf4b" containerID="1ca3f87921b5823c0252a4b2295c422cbc1b1f6a35ff381b84c10a0c76f42128" exitCode=0 Sep 10 23:58:59.251209 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:58:59.251087 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8b47cf-openshift-default-85854f7c9b-mtcrx" event={"ID":"eef2f53c-651e-41b0-a742-2d509e67bf4b","Type":"ContainerDied","Data":"1ca3f87921b5823c0252a4b2295c422cbc1b1f6a35ff381b84c10a0c76f42128"} Sep 10 23:58:59.251518 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:58:59.251505 2574 scope.go:117] "RemoveContainer" containerID="1ca3f87921b5823c0252a4b2295c422cbc1b1f6a35ff381b84c10a0c76f42128" Sep 10 23:58:59.252704 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:58:59.252673 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 23:58:59.252755 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:58:59.252742 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 23:58:59.252790 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:58:59.252772 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 23:59:00.259214 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:59:00.259176 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8b47cf-openshift-default-85854f7c9b-mtcrx" event={"ID":"eef2f53c-651e-41b0-a742-2d509e67bf4b","Type":"ContainerStarted","Data":"aad883c84d45feea1da8dc1eb616580ab39816fd92091eec41c4f69290889ca0"} Sep 10 23:59:00.761531 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:59:00.761492 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-8b47cf-openshift-default-85854f7c9b-mtcrx" Sep 10 23:59:00.761754 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:59:00.761734 2574 patch_prober.go:28] interesting pod/e2e-api-a-8b47cf-openshift-default-85854f7c9b-mtcrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Sep 10 23:59:00.761813 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:59:00.761784 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-8b47cf-openshift-default-85854f7c9b-mtcrx" podUID="eef2f53c-651e-41b0-a742-2d509e67bf4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Sep 10 23:59:01.761408 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:59:01.761368 2574 patch_prober.go:28] interesting pod/e2e-api-a-8b47cf-openshift-default-85854f7c9b-mtcrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Sep 10 23:59:01.761813 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:59:01.761442 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-8b47cf-openshift-default-85854f7c9b-mtcrx" podUID="eef2f53c-651e-41b0-a742-2d509e67bf4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Sep 10 23:59:02.761721 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:59:02.761680 2574 patch_prober.go:28] interesting pod/e2e-api-a-8b47cf-openshift-default-85854f7c9b-mtcrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Sep 10 23:59:02.762127 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:59:02.761752 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-8b47cf-openshift-default-85854f7c9b-mtcrx" podUID="eef2f53c-651e-41b0-a742-2d509e67bf4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Sep 10 23:59:03.761046 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:59:03.761010 2574 patch_prober.go:28] interesting pod/e2e-api-a-8b47cf-openshift-default-85854f7c9b-mtcrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Sep 10 23:59:03.761334 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:59:03.761298 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-8b47cf-openshift-default-85854f7c9b-mtcrx" podUID="eef2f53c-651e-41b0-a742-2d509e67bf4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Sep 10 23:59:04.761415 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:59:04.761377 2574 patch_prober.go:28] interesting pod/e2e-api-a-8b47cf-openshift-default-85854f7c9b-mtcrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Sep 10 23:59:04.761850 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:59:04.761432 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-8b47cf-openshift-default-85854f7c9b-mtcrx" podUID="eef2f53c-651e-41b0-a742-2d509e67bf4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Sep 10 23:59:05.765123 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:59:05.765091 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-8b47cf-openshift-default-85854f7c9b-mtcrx" Sep 10 23:59:05.765494 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:59:05.765473 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-8b47cf-openshift-default-85854f7c9b-mtcrx" Sep 10 23:59:05.766213 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:59:05.766188 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-8b47cf-openshift-default-85854f7c9b-mtcrx" Sep 10 23:59:09.698404 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:59:09.698367 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-4ec4c8-openshift-default-56bf45968-xpv9q"] Sep 10 23:59:09.699448 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:59:09.699423 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09db6ab7-c732-41e1-a76f-ca403f1eb53f" containerName="istio-proxy" Sep 10 23:59:09.699448 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:59:09.699447 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="09db6ab7-c732-41e1-a76f-ca403f1eb53f" containerName="istio-proxy" Sep 10 23:59:09.703825 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:59:09.703802 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4ec4c8-openshift-default-56bf45968-xpv9q" Sep 10 23:59:09.705855 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:59:09.705834 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-4ec4c8-openshift-default-dockercfg-z2lrz\"" Sep 10 23:59:09.714317 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:59:09.714290 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-4ec4c8-openshift-default-56bf45968-xpv9q"] Sep 10 23:59:09.786987 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:59:09.786951 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2prgl\" (UniqueName: \"kubernetes.io/projected/7e485538-3e4e-40da-bd0a-ff61aa91455d-kube-api-access-2prgl\") pod \"e2e-api-b-4ec4c8-openshift-default-56bf45968-xpv9q\" (UID: \"7e485538-3e4e-40da-bd0a-ff61aa91455d\") " pod="openshift-ingress/e2e-api-b-4ec4c8-openshift-default-56bf45968-xpv9q" Sep 10 23:59:09.787157 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:59:09.787017 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e485538-3e4e-40da-bd0a-ff61aa91455d-istio-podinfo\") pod \"e2e-api-b-4ec4c8-openshift-default-56bf45968-xpv9q\" (UID: \"7e485538-3e4e-40da-bd0a-ff61aa91455d\") " pod="openshift-ingress/e2e-api-b-4ec4c8-openshift-default-56bf45968-xpv9q" Sep 10 23:59:09.787157 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:59:09.787082 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e485538-3e4e-40da-bd0a-ff61aa91455d-workload-socket\") pod \"e2e-api-b-4ec4c8-openshift-default-56bf45968-xpv9q\" (UID: \"7e485538-3e4e-40da-bd0a-ff61aa91455d\") " pod="openshift-ingress/e2e-api-b-4ec4c8-openshift-default-56bf45968-xpv9q" Sep 10 23:59:09.787157 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:59:09.787153 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e485538-3e4e-40da-bd0a-ff61aa91455d-istio-data\") pod \"e2e-api-b-4ec4c8-openshift-default-56bf45968-xpv9q\" (UID: \"7e485538-3e4e-40da-bd0a-ff61aa91455d\") " pod="openshift-ingress/e2e-api-b-4ec4c8-openshift-default-56bf45968-xpv9q" Sep 10 23:59:09.787335 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:59:09.787188 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e485538-3e4e-40da-bd0a-ff61aa91455d-istio-token\") pod \"e2e-api-b-4ec4c8-openshift-default-56bf45968-xpv9q\" (UID: \"7e485538-3e4e-40da-bd0a-ff61aa91455d\") " pod="openshift-ingress/e2e-api-b-4ec4c8-openshift-default-56bf45968-xpv9q" Sep 10 23:59:09.787335 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:59:09.787226 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e485538-3e4e-40da-bd0a-ff61aa91455d-istio-envoy\") pod \"e2e-api-b-4ec4c8-openshift-default-56bf45968-xpv9q\" (UID: \"7e485538-3e4e-40da-bd0a-ff61aa91455d\") " pod="openshift-ingress/e2e-api-b-4ec4c8-openshift-default-56bf45968-xpv9q" Sep 10 23:59:09.787335 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:59:09.787260 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e485538-3e4e-40da-bd0a-ff61aa91455d-credential-socket\") pod \"e2e-api-b-4ec4c8-openshift-default-56bf45968-xpv9q\" (UID: \"7e485538-3e4e-40da-bd0a-ff61aa91455d\") " pod="openshift-ingress/e2e-api-b-4ec4c8-openshift-default-56bf45968-xpv9q" Sep 10 23:59:09.787335 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:59:09.787285 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e485538-3e4e-40da-bd0a-ff61aa91455d-istiod-ca-cert\") pod \"e2e-api-b-4ec4c8-openshift-default-56bf45968-xpv9q\" (UID: \"7e485538-3e4e-40da-bd0a-ff61aa91455d\") " pod="openshift-ingress/e2e-api-b-4ec4c8-openshift-default-56bf45968-xpv9q" Sep 10 23:59:09.787335 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:59:09.787314 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e485538-3e4e-40da-bd0a-ff61aa91455d-workload-certs\") pod \"e2e-api-b-4ec4c8-openshift-default-56bf45968-xpv9q\" (UID: \"7e485538-3e4e-40da-bd0a-ff61aa91455d\") " pod="openshift-ingress/e2e-api-b-4ec4c8-openshift-default-56bf45968-xpv9q" Sep 10 23:59:09.888389 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:59:09.888362 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2prgl\" (UniqueName: \"kubernetes.io/projected/7e485538-3e4e-40da-bd0a-ff61aa91455d-kube-api-access-2prgl\") pod \"e2e-api-b-4ec4c8-openshift-default-56bf45968-xpv9q\" (UID: \"7e485538-3e4e-40da-bd0a-ff61aa91455d\") " pod="openshift-ingress/e2e-api-b-4ec4c8-openshift-default-56bf45968-xpv9q" Sep 10 23:59:09.888580 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:59:09.888415 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e485538-3e4e-40da-bd0a-ff61aa91455d-istio-podinfo\") pod \"e2e-api-b-4ec4c8-openshift-default-56bf45968-xpv9q\" (UID: \"7e485538-3e4e-40da-bd0a-ff61aa91455d\") " pod="openshift-ingress/e2e-api-b-4ec4c8-openshift-default-56bf45968-xpv9q" Sep 10 23:59:09.888580 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:59:09.888453 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e485538-3e4e-40da-bd0a-ff61aa91455d-workload-socket\") pod \"e2e-api-b-4ec4c8-openshift-default-56bf45968-xpv9q\" (UID: \"7e485538-3e4e-40da-bd0a-ff61aa91455d\") " pod="openshift-ingress/e2e-api-b-4ec4c8-openshift-default-56bf45968-xpv9q" Sep 10 23:59:09.888580 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:59:09.888494 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e485538-3e4e-40da-bd0a-ff61aa91455d-istio-data\") pod \"e2e-api-b-4ec4c8-openshift-default-56bf45968-xpv9q\" (UID: \"7e485538-3e4e-40da-bd0a-ff61aa91455d\") " pod="openshift-ingress/e2e-api-b-4ec4c8-openshift-default-56bf45968-xpv9q" Sep 10 23:59:09.888580 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:59:09.888521 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e485538-3e4e-40da-bd0a-ff61aa91455d-istio-token\") pod \"e2e-api-b-4ec4c8-openshift-default-56bf45968-xpv9q\" (UID: \"7e485538-3e4e-40da-bd0a-ff61aa91455d\") " pod="openshift-ingress/e2e-api-b-4ec4c8-openshift-default-56bf45968-xpv9q" Sep 10 23:59:09.888580 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:59:09.888561 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e485538-3e4e-40da-bd0a-ff61aa91455d-istio-envoy\") pod \"e2e-api-b-4ec4c8-openshift-default-56bf45968-xpv9q\" (UID: \"7e485538-3e4e-40da-bd0a-ff61aa91455d\") " pod="openshift-ingress/e2e-api-b-4ec4c8-openshift-default-56bf45968-xpv9q" Sep 10 23:59:09.888898 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:59:09.888593 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e485538-3e4e-40da-bd0a-ff61aa91455d-credential-socket\") pod \"e2e-api-b-4ec4c8-openshift-default-56bf45968-xpv9q\" (UID: \"7e485538-3e4e-40da-bd0a-ff61aa91455d\") " pod="openshift-ingress/e2e-api-b-4ec4c8-openshift-default-56bf45968-xpv9q" Sep 10 23:59:09.888898 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:59:09.888618 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e485538-3e4e-40da-bd0a-ff61aa91455d-istiod-ca-cert\") pod \"e2e-api-b-4ec4c8-openshift-default-56bf45968-xpv9q\" (UID: \"7e485538-3e4e-40da-bd0a-ff61aa91455d\") " pod="openshift-ingress/e2e-api-b-4ec4c8-openshift-default-56bf45968-xpv9q" Sep 10 23:59:09.888898 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:59:09.888751 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e485538-3e4e-40da-bd0a-ff61aa91455d-workload-certs\") pod \"e2e-api-b-4ec4c8-openshift-default-56bf45968-xpv9q\" (UID: \"7e485538-3e4e-40da-bd0a-ff61aa91455d\") " pod="openshift-ingress/e2e-api-b-4ec4c8-openshift-default-56bf45968-xpv9q" Sep 10 23:59:09.889048 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:59:09.888923 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e485538-3e4e-40da-bd0a-ff61aa91455d-workload-socket\") pod \"e2e-api-b-4ec4c8-openshift-default-56bf45968-xpv9q\" (UID: \"7e485538-3e4e-40da-bd0a-ff61aa91455d\") " pod="openshift-ingress/e2e-api-b-4ec4c8-openshift-default-56bf45968-xpv9q" Sep 10 23:59:09.889048 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:59:09.888972 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e485538-3e4e-40da-bd0a-ff61aa91455d-istio-data\") pod \"e2e-api-b-4ec4c8-openshift-default-56bf45968-xpv9q\" (UID: \"7e485538-3e4e-40da-bd0a-ff61aa91455d\") " pod="openshift-ingress/e2e-api-b-4ec4c8-openshift-default-56bf45968-xpv9q" Sep 10 23:59:09.889200 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:59:09.889180 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e485538-3e4e-40da-bd0a-ff61aa91455d-workload-certs\") pod \"e2e-api-b-4ec4c8-openshift-default-56bf45968-xpv9q\" (UID: \"7e485538-3e4e-40da-bd0a-ff61aa91455d\") " pod="openshift-ingress/e2e-api-b-4ec4c8-openshift-default-56bf45968-xpv9q" Sep 10 23:59:09.889264 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:59:09.889220 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e485538-3e4e-40da-bd0a-ff61aa91455d-credential-socket\") pod \"e2e-api-b-4ec4c8-openshift-default-56bf45968-xpv9q\" (UID: \"7e485538-3e4e-40da-bd0a-ff61aa91455d\") " pod="openshift-ingress/e2e-api-b-4ec4c8-openshift-default-56bf45968-xpv9q" Sep 10 23:59:09.889339 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:59:09.889319 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e485538-3e4e-40da-bd0a-ff61aa91455d-istiod-ca-cert\") pod \"e2e-api-b-4ec4c8-openshift-default-56bf45968-xpv9q\" (UID: \"7e485538-3e4e-40da-bd0a-ff61aa91455d\") " pod="openshift-ingress/e2e-api-b-4ec4c8-openshift-default-56bf45968-xpv9q" Sep 10 23:59:09.891027 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:59:09.890993 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e485538-3e4e-40da-bd0a-ff61aa91455d-istio-envoy\") pod \"e2e-api-b-4ec4c8-openshift-default-56bf45968-xpv9q\" (UID: \"7e485538-3e4e-40da-bd0a-ff61aa91455d\") " pod="openshift-ingress/e2e-api-b-4ec4c8-openshift-default-56bf45968-xpv9q" Sep 10 23:59:09.891169 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:59:09.891154 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e485538-3e4e-40da-bd0a-ff61aa91455d-istio-podinfo\") pod \"e2e-api-b-4ec4c8-openshift-default-56bf45968-xpv9q\" (UID: \"7e485538-3e4e-40da-bd0a-ff61aa91455d\") " pod="openshift-ingress/e2e-api-b-4ec4c8-openshift-default-56bf45968-xpv9q" Sep 10 23:59:09.895757 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:59:09.895734 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2prgl\" (UniqueName: \"kubernetes.io/projected/7e485538-3e4e-40da-bd0a-ff61aa91455d-kube-api-access-2prgl\") pod \"e2e-api-b-4ec4c8-openshift-default-56bf45968-xpv9q\" (UID: \"7e485538-3e4e-40da-bd0a-ff61aa91455d\") " pod="openshift-ingress/e2e-api-b-4ec4c8-openshift-default-56bf45968-xpv9q" Sep 10 23:59:09.896204 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:59:09.896183 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e485538-3e4e-40da-bd0a-ff61aa91455d-istio-token\") pod \"e2e-api-b-4ec4c8-openshift-default-56bf45968-xpv9q\" (UID: \"7e485538-3e4e-40da-bd0a-ff61aa91455d\") " pod="openshift-ingress/e2e-api-b-4ec4c8-openshift-default-56bf45968-xpv9q" Sep 10 23:59:10.016898 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:59:10.016808 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4ec4c8-openshift-default-56bf45968-xpv9q" Sep 10 23:59:10.292931 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:59:10.292900 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-4ec4c8-openshift-default-56bf45968-xpv9q"] Sep 10 23:59:10.294060 ip-10-0-138-128 kubenswrapper[2574]: W0910 23:59:10.294037 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e485538_3e4e_40da_bd0a_ff61aa91455d.slice/crio-cbff45a0aac81bbb6e95126d9e992ea607ea3f441f91b1710f199c26802a2c5e WatchSource:0}: Error finding container cbff45a0aac81bbb6e95126d9e992ea607ea3f441f91b1710f199c26802a2c5e: Status 404 returned error can't find the container with id cbff45a0aac81bbb6e95126d9e992ea607ea3f441f91b1710f199c26802a2c5e Sep 10 23:59:10.297039 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:59:10.297000 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 23:59:10.297265 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:59:10.297240 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 23:59:10.297436 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:59:10.297397 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 23:59:10.333582 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:59:10.333552 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4ec4c8-openshift-default-56bf45968-xpv9q" event={"ID":"7e485538-3e4e-40da-bd0a-ff61aa91455d","Type":"ContainerStarted","Data":"cbff45a0aac81bbb6e95126d9e992ea607ea3f441f91b1710f199c26802a2c5e"} Sep 10 23:59:10.382984 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:59:10.382949 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-56r7l"] Sep 10 23:59:11.341043 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:59:11.341002 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4ec4c8-openshift-default-56bf45968-xpv9q" event={"ID":"7e485538-3e4e-40da-bd0a-ff61aa91455d","Type":"ContainerStarted","Data":"f046b4487bb7de419f8379e5565528f5aa4c6aac191209447a7fa8d859756867"} Sep 10 23:59:11.361459 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:59:11.361410 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-4ec4c8-openshift-default-56bf45968-xpv9q" podStartSLOduration=2.361395074 podStartE2EDuration="2.361395074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:59:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:59:11.357871566 +0000 UTC m=+1342.745474351" watchObservedRunningTime="2026-09-10 23:59:11.361395074 +0000 UTC m=+1342.748997899" Sep 10 23:59:12.017198 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:59:12.017159 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-4ec4c8-openshift-default-56bf45968-xpv9q" Sep 10 23:59:12.022355 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:59:12.022326 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-4ec4c8-openshift-default-56bf45968-xpv9q" Sep 10 23:59:12.346962 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:59:12.346881 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-4ec4c8-openshift-default-56bf45968-xpv9q" Sep 10 23:59:12.347982 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:59:12.347964 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-4ec4c8-openshift-default-56bf45968-xpv9q" Sep 10 23:59:22.988807 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:59:22.988770 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-56r7l"] Sep 10 23:59:33.484149 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:59:33.484066 2574 generic.go:356] "Generic (PLEG): container finished" podID="7e485538-3e4e-40da-bd0a-ff61aa91455d" containerID="f046b4487bb7de419f8379e5565528f5aa4c6aac191209447a7fa8d859756867" exitCode=0 Sep 10 23:59:33.484556 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:59:33.484139 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4ec4c8-openshift-default-56bf45968-xpv9q" event={"ID":"7e485538-3e4e-40da-bd0a-ff61aa91455d","Type":"ContainerDied","Data":"f046b4487bb7de419f8379e5565528f5aa4c6aac191209447a7fa8d859756867"} Sep 10 23:59:33.484602 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:59:33.484559 2574 scope.go:117] "RemoveContainer" containerID="f046b4487bb7de419f8379e5565528f5aa4c6aac191209447a7fa8d859756867" Sep 10 23:59:33.485789 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:59:33.485761 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 23:59:33.485854 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:59:33.485828 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 23:59:33.485893 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:59:33.485861 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 10 23:59:34.496396 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:59:34.496364 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4ec4c8-openshift-default-56bf45968-xpv9q" event={"ID":"7e485538-3e4e-40da-bd0a-ff61aa91455d","Type":"ContainerStarted","Data":"3d79f4132e9a5aa8125ef7e063aa5108e05a2d82b2da63f1c58dd3d799a5ce42"} Sep 10 23:59:35.017882 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:59:35.017842 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-4ec4c8-openshift-default-56bf45968-xpv9q" Sep 10 23:59:35.018113 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:59:35.018087 2574 patch_prober.go:28] interesting pod/e2e-api-b-4ec4c8-openshift-default-56bf45968-xpv9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Sep 10 23:59:35.018207 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:59:35.018145 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4ec4c8-openshift-default-56bf45968-xpv9q" podUID="7e485538-3e4e-40da-bd0a-ff61aa91455d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Sep 10 23:59:36.017817 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:59:36.017776 2574 patch_prober.go:28] interesting pod/e2e-api-b-4ec4c8-openshift-default-56bf45968-xpv9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Sep 10 23:59:36.018222 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:59:36.017852 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4ec4c8-openshift-default-56bf45968-xpv9q" podUID="7e485538-3e4e-40da-bd0a-ff61aa91455d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Sep 10 23:59:37.018094 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:59:37.018050 2574 patch_prober.go:28] interesting pod/e2e-api-b-4ec4c8-openshift-default-56bf45968-xpv9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Sep 10 23:59:37.018482 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:59:37.018129 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4ec4c8-openshift-default-56bf45968-xpv9q" podUID="7e485538-3e4e-40da-bd0a-ff61aa91455d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Sep 10 23:59:38.017903 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:59:38.017863 2574 patch_prober.go:28] interesting pod/e2e-api-b-4ec4c8-openshift-default-56bf45968-xpv9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Sep 10 23:59:38.018110 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:59:38.017938 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4ec4c8-openshift-default-56bf45968-xpv9q" podUID="7e485538-3e4e-40da-bd0a-ff61aa91455d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Sep 10 23:59:39.017406 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:59:39.017370 2574 patch_prober.go:28] interesting pod/e2e-api-b-4ec4c8-openshift-default-56bf45968-xpv9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Sep 10 23:59:39.017613 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:59:39.017431 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4ec4c8-openshift-default-56bf45968-xpv9q" podUID="7e485538-3e4e-40da-bd0a-ff61aa91455d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Sep 10 23:59:40.022083 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:59:40.022049 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-4ec4c8-openshift-default-56bf45968-xpv9q" Sep 10 23:59:40.022534 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:59:40.022442 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-4ec4c8-openshift-default-56bf45968-xpv9q" Sep 10 23:59:40.023060 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:59:40.023042 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-4ec4c8-openshift-default-56bf45968-xpv9q" Sep 10 23:59:42.492210 ip-10-0-138-128 kubenswrapper[2574]: I0910 23:59:42.492165 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-56r7l"] Sep 11 00:00:00.172653 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:00.172615 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29818080-6xrvk"] Sep 11 00:00:00.177100 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:00.177068 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d7a539-29818080-kjzfg"] Sep 11 00:00:00.177259 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:00.177172 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818080-6xrvk" Sep 11 00:00:00.180770 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:00.180680 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d7a539-29818080-kjzfg" Sep 11 00:00:00.181011 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:00.180994 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zrcdn\"" Sep 11 00:00:00.197802 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:00.197778 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29818080-6xrvk"] Sep 11 00:00:00.203143 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:00.203115 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d7a539-29818080-kjzfg"] Sep 11 00:00:00.214079 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:00.214057 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818080-6xrvk" Sep 11 00:00:00.221759 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:00.221728 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d7a539-29818080-kjzfg" Sep 11 00:00:00.513432 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:00.513404 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29818080-6xrvk"] Sep 11 00:00:00.516058 ip-10-0-138-128 kubenswrapper[2574]: W0911 00:00:00.516022 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod198384a6_ba0b_4d39_956d_f33800f5b531.slice/crio-7c91fc631b91256da255f3de37de56e1e78c4e629d62cf74bedd506e9a5da632 WatchSource:0}: Error finding container 7c91fc631b91256da255f3de37de56e1e78c4e629d62cf74bedd506e9a5da632: Status 404 returned error can't find the container with id 7c91fc631b91256da255f3de37de56e1e78c4e629d62cf74bedd506e9a5da632 Sep 11 00:00:00.531840 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:00.531812 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d7a539-29818080-kjzfg"] Sep 11 00:00:00.532802 ip-10-0-138-128 kubenswrapper[2574]: W0911 00:00:00.532777 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3394f36c_47c5_4ea7_a9a8_264bf006cb45.slice/crio-3476ca3f17ccaa1767eecb5e3c9d2a1f7a5e40a258917262cda5859e8c638cb6 WatchSource:0}: Error finding container 3476ca3f17ccaa1767eecb5e3c9d2a1f7a5e40a258917262cda5859e8c638cb6: Status 404 returned error can't find the container with id 3476ca3f17ccaa1767eecb5e3c9d2a1f7a5e40a258917262cda5859e8c638cb6 Sep 11 00:00:00.668986 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:00.668948 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d7a539-29818080-kjzfg" event={"ID":"3394f36c-47c5-4ea7-a9a8-264bf006cb45","Type":"ContainerStarted","Data":"3476ca3f17ccaa1767eecb5e3c9d2a1f7a5e40a258917262cda5859e8c638cb6"} Sep 11 00:00:00.670417 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:00.670393 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818080-6xrvk" event={"ID":"198384a6-ba0b-4d39-956d-f33800f5b531","Type":"ContainerStarted","Data":"7c91fc631b91256da255f3de37de56e1e78c4e629d62cf74bedd506e9a5da632"} Sep 11 00:00:00.747634 ip-10-0-138-128 kubenswrapper[2574]: E0911 00:00:00.747599 2574 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/system.slice/logrotate.service\": RecentStats: unable to find data in memory cache]" Sep 11 00:00:02.480879 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:02.480845 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-56r7l"] Sep 11 00:00:03.502492 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:03.502456 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-8b47cf-openshift-default-85854f7c9b-mtcrx"] Sep 11 00:00:03.502912 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:03.502738 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-8b47cf-openshift-default-85854f7c9b-mtcrx" podUID="eef2f53c-651e-41b0-a742-2d509e67bf4b" containerName="istio-proxy" containerID="cri-o://aad883c84d45feea1da8dc1eb616580ab39816fd92091eec41c4f69290889ca0" gracePeriod=30 Sep 11 00:00:03.693795 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:03.693683 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d7a539-29818080-kjzfg" event={"ID":"3394f36c-47c5-4ea7-a9a8-264bf006cb45","Type":"ContainerStarted","Data":"abefd27950f53a51cebaa336b23b234e74d81d79bc426d30870f6b1e148995b5"} Sep 11 00:00:03.695400 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:03.695375 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818080-6xrvk" event={"ID":"198384a6-ba0b-4d39-956d-f33800f5b531","Type":"ContainerStarted","Data":"7ba6ab217c5ae45081ee9bc9573c67725c64ee89e40afd69a18c32805dfa9ef2"} Sep 11 00:00:03.712146 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:03.712092 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d7a539-29818080-kjzfg" podStartSLOduration=1.182300645 podStartE2EDuration="3.712077997s" totalImagesPullingTime="2.529777352s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 00:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:00:03.711441362 +0000 UTC m=+1395.099044121" watchObservedRunningTime="2026-09-11 00:00:03.712077997 +0000 UTC m=+1395.099680755" Sep 11 00:00:03.730754 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:03.730706 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818080-6xrvk" podStartSLOduration=1.187966789 podStartE2EDuration="3.730691835s" totalImagesPullingTime="2.542725046s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 00:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:00:03.728673548 +0000 UTC m=+1395.116276301" watchObservedRunningTime="2026-09-11 00:00:03.730691835 +0000 UTC m=+1395.118294594" Sep 11 00:00:04.702940 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:04.702902 2574 generic.go:356] "Generic (PLEG): container finished" podID="198384a6-ba0b-4d39-956d-f33800f5b531" containerID="7ba6ab217c5ae45081ee9bc9573c67725c64ee89e40afd69a18c32805dfa9ef2" exitCode=0 Sep 11 00:00:04.703368 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:04.702995 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818080-6xrvk" event={"ID":"198384a6-ba0b-4d39-956d-f33800f5b531","Type":"ContainerDied","Data":"7ba6ab217c5ae45081ee9bc9573c67725c64ee89e40afd69a18c32805dfa9ef2"} Sep 11 00:00:04.704690 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:04.704669 2574 generic.go:356] "Generic (PLEG): container finished" podID="3394f36c-47c5-4ea7-a9a8-264bf006cb45" containerID="abefd27950f53a51cebaa336b23b234e74d81d79bc426d30870f6b1e148995b5" exitCode=0 Sep 11 00:00:04.704817 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:04.704719 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d7a539-29818080-kjzfg" event={"ID":"3394f36c-47c5-4ea7-a9a8-264bf006cb45","Type":"ContainerDied","Data":"abefd27950f53a51cebaa336b23b234e74d81d79bc426d30870f6b1e148995b5"} Sep 11 00:00:05.768433 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:05.768336 2574 patch_prober.go:28] interesting pod/e2e-api-a-8b47cf-openshift-default-85854f7c9b-mtcrx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 11 00:00:05.768916 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:05.768431 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-8b47cf-openshift-default-85854f7c9b-mtcrx" podUID="eef2f53c-651e-41b0-a742-2d509e67bf4b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 11 00:00:05.854565 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:05.854532 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d7a539-29818080-kjzfg" Sep 11 00:00:05.900895 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:05.900863 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818080-6xrvk" Sep 11 00:00:06.724608 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:06.724582 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818080-6xrvk" Sep 11 00:00:06.724813 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:06.724671 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29818080-6xrvk" event={"ID":"198384a6-ba0b-4d39-956d-f33800f5b531","Type":"ContainerDied","Data":"7c91fc631b91256da255f3de37de56e1e78c4e629d62cf74bedd506e9a5da632"} Sep 11 00:00:06.724813 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:06.724725 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7c91fc631b91256da255f3de37de56e1e78c4e629d62cf74bedd506e9a5da632" Sep 11 00:00:06.725963 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:06.725938 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d7a539-29818080-kjzfg" event={"ID":"3394f36c-47c5-4ea7-a9a8-264bf006cb45","Type":"ContainerDied","Data":"3476ca3f17ccaa1767eecb5e3c9d2a1f7a5e40a258917262cda5859e8c638cb6"} Sep 11 00:00:06.726086 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:06.725978 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3476ca3f17ccaa1767eecb5e3c9d2a1f7a5e40a258917262cda5859e8c638cb6" Sep 11 00:00:06.726086 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:06.726018 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d7a539-29818080-kjzfg" Sep 11 00:00:08.740513 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:08.740480 2574 generic.go:356] "Generic (PLEG): container finished" podID="eef2f53c-651e-41b0-a742-2d509e67bf4b" containerID="aad883c84d45feea1da8dc1eb616580ab39816fd92091eec41c4f69290889ca0" exitCode=0 Sep 11 00:00:08.740980 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:08.740566 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8b47cf-openshift-default-85854f7c9b-mtcrx" event={"ID":"eef2f53c-651e-41b0-a742-2d509e67bf4b","Type":"ContainerDied","Data":"aad883c84d45feea1da8dc1eb616580ab39816fd92091eec41c4f69290889ca0"} Sep 11 00:00:08.740980 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:08.740629 2574 scope.go:117] "RemoveContainer" containerID="1ca3f87921b5823c0252a4b2295c422cbc1b1f6a35ff381b84c10a0c76f42128" Sep 11 00:00:08.768753 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:08.768731 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-8b47cf-openshift-default-85854f7c9b-mtcrx" Sep 11 00:00:08.830137 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:08.830093 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eef2f53c-651e-41b0-a742-2d509e67bf4b-istio-data\") pod \"eef2f53c-651e-41b0-a742-2d509e67bf4b\" (UID: \"eef2f53c-651e-41b0-a742-2d509e67bf4b\") " Sep 11 00:00:08.830137 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:08.830146 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eef2f53c-651e-41b0-a742-2d509e67bf4b-credential-socket\") pod \"eef2f53c-651e-41b0-a742-2d509e67bf4b\" (UID: \"eef2f53c-651e-41b0-a742-2d509e67bf4b\") " Sep 11 00:00:08.830391 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:08.830175 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eef2f53c-651e-41b0-a742-2d509e67bf4b-workload-certs\") pod \"eef2f53c-651e-41b0-a742-2d509e67bf4b\" (UID: \"eef2f53c-651e-41b0-a742-2d509e67bf4b\") " Sep 11 00:00:08.830391 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:08.830196 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eef2f53c-651e-41b0-a742-2d509e67bf4b-istiod-ca-cert\") pod \"eef2f53c-651e-41b0-a742-2d509e67bf4b\" (UID: \"eef2f53c-651e-41b0-a742-2d509e67bf4b\") " Sep 11 00:00:08.830391 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:08.830228 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eef2f53c-651e-41b0-a742-2d509e67bf4b-istio-envoy\") pod \"eef2f53c-651e-41b0-a742-2d509e67bf4b\" (UID: \"eef2f53c-651e-41b0-a742-2d509e67bf4b\") " Sep 11 00:00:08.830391 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:08.830266 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eef2f53c-651e-41b0-a742-2d509e67bf4b-istio-token\") pod \"eef2f53c-651e-41b0-a742-2d509e67bf4b\" (UID: \"eef2f53c-651e-41b0-a742-2d509e67bf4b\") " Sep 11 00:00:08.830391 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:08.830312 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q7ntb\" (UniqueName: \"kubernetes.io/projected/eef2f53c-651e-41b0-a742-2d509e67bf4b-kube-api-access-q7ntb\") pod \"eef2f53c-651e-41b0-a742-2d509e67bf4b\" (UID: \"eef2f53c-651e-41b0-a742-2d509e67bf4b\") " Sep 11 00:00:08.830391 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:08.830360 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eef2f53c-651e-41b0-a742-2d509e67bf4b-workload-socket\") pod \"eef2f53c-651e-41b0-a742-2d509e67bf4b\" (UID: \"eef2f53c-651e-41b0-a742-2d509e67bf4b\") " Sep 11 00:00:08.830391 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:08.830390 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eef2f53c-651e-41b0-a742-2d509e67bf4b-istio-podinfo\") pod \"eef2f53c-651e-41b0-a742-2d509e67bf4b\" (UID: \"eef2f53c-651e-41b0-a742-2d509e67bf4b\") " Sep 11 00:00:08.830834 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:08.830404 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eef2f53c-651e-41b0-a742-2d509e67bf4b-istio-data" (OuterVolumeSpecName: "istio-data") pod "eef2f53c-651e-41b0-a742-2d509e67bf4b" (UID: "eef2f53c-651e-41b0-a742-2d509e67bf4b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 00:00:08.830834 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:08.830464 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eef2f53c-651e-41b0-a742-2d509e67bf4b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "eef2f53c-651e-41b0-a742-2d509e67bf4b" (UID: "eef2f53c-651e-41b0-a742-2d509e67bf4b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 00:00:08.830834 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:08.830514 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eef2f53c-651e-41b0-a742-2d509e67bf4b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "eef2f53c-651e-41b0-a742-2d509e67bf4b" (UID: "eef2f53c-651e-41b0-a742-2d509e67bf4b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 00:00:08.830834 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:08.830714 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eef2f53c-651e-41b0-a742-2d509e67bf4b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "eef2f53c-651e-41b0-a742-2d509e67bf4b" (UID: "eef2f53c-651e-41b0-a742-2d509e67bf4b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 00:00:08.830834 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:08.830620 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eef2f53c-651e-41b0-a742-2d509e67bf4b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "eef2f53c-651e-41b0-a742-2d509e67bf4b" (UID: "eef2f53c-651e-41b0-a742-2d509e67bf4b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 00:00:08.831105 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:08.830849 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eef2f53c-651e-41b0-a742-2d509e67bf4b-istio-data\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 00:00:08.831105 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:08.830866 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eef2f53c-651e-41b0-a742-2d509e67bf4b-credential-socket\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 00:00:08.831105 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:08.830879 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eef2f53c-651e-41b0-a742-2d509e67bf4b-workload-certs\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 00:00:08.831105 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:08.830891 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eef2f53c-651e-41b0-a742-2d509e67bf4b-istiod-ca-cert\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 00:00:08.831105 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:08.830898 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eef2f53c-651e-41b0-a742-2d509e67bf4b-workload-socket\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 00:00:08.832551 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:08.832527 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eef2f53c-651e-41b0-a742-2d509e67bf4b-istio-token" (OuterVolumeSpecName: "istio-token") pod "eef2f53c-651e-41b0-a742-2d509e67bf4b" (UID: "eef2f53c-651e-41b0-a742-2d509e67bf4b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 00:00:08.832551 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:08.832536 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eef2f53c-651e-41b0-a742-2d509e67bf4b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "eef2f53c-651e-41b0-a742-2d509e67bf4b" (UID: "eef2f53c-651e-41b0-a742-2d509e67bf4b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 00:00:08.832733 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:08.832579 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eef2f53c-651e-41b0-a742-2d509e67bf4b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "eef2f53c-651e-41b0-a742-2d509e67bf4b" (UID: "eef2f53c-651e-41b0-a742-2d509e67bf4b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 00:00:08.832733 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:08.832720 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eef2f53c-651e-41b0-a742-2d509e67bf4b-kube-api-access-q7ntb" (OuterVolumeSpecName: "kube-api-access-q7ntb") pod "eef2f53c-651e-41b0-a742-2d509e67bf4b" (UID: "eef2f53c-651e-41b0-a742-2d509e67bf4b"). InnerVolumeSpecName "kube-api-access-q7ntb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 00:00:08.931951 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:08.931917 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eef2f53c-651e-41b0-a742-2d509e67bf4b-istio-envoy\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 00:00:08.931951 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:08.931945 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eef2f53c-651e-41b0-a742-2d509e67bf4b-istio-token\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 00:00:08.931951 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:08.931955 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7ntb\" (UniqueName: \"kubernetes.io/projected/eef2f53c-651e-41b0-a742-2d509e67bf4b-kube-api-access-q7ntb\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 00:00:08.932157 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:08.931963 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eef2f53c-651e-41b0-a742-2d509e67bf4b-istio-podinfo\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 00:00:09.747823 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:09.747796 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-8b47cf-openshift-default-85854f7c9b-mtcrx" Sep 11 00:00:09.747823 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:09.747812 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8b47cf-openshift-default-85854f7c9b-mtcrx" event={"ID":"eef2f53c-651e-41b0-a742-2d509e67bf4b","Type":"ContainerDied","Data":"1e2a553dffb979033fbbc19c4d47aa04166885d9cdbe1d5cf91e918adaae1cc1"} Sep 11 00:00:09.748406 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:09.747873 2574 scope.go:117] "RemoveContainer" containerID="aad883c84d45feea1da8dc1eb616580ab39816fd92091eec41c4f69290889ca0" Sep 11 00:00:09.778015 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:09.777980 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-8b47cf-openshift-default-85854f7c9b-mtcrx"] Sep 11 00:00:09.784423 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:09.784399 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-8b47cf-openshift-default-85854f7c9b-mtcrx"] Sep 11 00:00:11.094714 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:11.094674 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eef2f53c-651e-41b0-a742-2d509e67bf4b" path="/var/lib/kubelet/pods/eef2f53c-651e-41b0-a742-2d509e67bf4b/volumes" Sep 11 00:00:18.484228 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:18.484191 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-56r7l"] Sep 11 00:00:26.125205 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:26.125168 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-4ec4c8-openshift-default-56bf45968-xpv9q"] Sep 11 00:00:26.125755 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:26.125390 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-4ec4c8-openshift-default-56bf45968-xpv9q" podUID="7e485538-3e4e-40da-bd0a-ff61aa91455d" containerName="istio-proxy" containerID="cri-o://3d79f4132e9a5aa8125ef7e063aa5108e05a2d82b2da63f1c58dd3d799a5ce42" gracePeriod=30 Sep 11 00:00:31.405329 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:31.405297 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4ec4c8-openshift-default-56bf45968-xpv9q" Sep 11 00:00:31.478599 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:31.478565 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e485538-3e4e-40da-bd0a-ff61aa91455d-istio-data\") pod \"7e485538-3e4e-40da-bd0a-ff61aa91455d\" (UID: \"7e485538-3e4e-40da-bd0a-ff61aa91455d\") " Sep 11 00:00:31.478758 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:31.478615 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e485538-3e4e-40da-bd0a-ff61aa91455d-istiod-ca-cert\") pod \"7e485538-3e4e-40da-bd0a-ff61aa91455d\" (UID: \"7e485538-3e4e-40da-bd0a-ff61aa91455d\") " Sep 11 00:00:31.478758 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:31.478667 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2prgl\" (UniqueName: \"kubernetes.io/projected/7e485538-3e4e-40da-bd0a-ff61aa91455d-kube-api-access-2prgl\") pod \"7e485538-3e4e-40da-bd0a-ff61aa91455d\" (UID: \"7e485538-3e4e-40da-bd0a-ff61aa91455d\") " Sep 11 00:00:31.478758 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:31.478735 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e485538-3e4e-40da-bd0a-ff61aa91455d-istio-envoy\") pod \"7e485538-3e4e-40da-bd0a-ff61aa91455d\" (UID: \"7e485538-3e4e-40da-bd0a-ff61aa91455d\") " Sep 11 00:00:31.478902 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:31.478769 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e485538-3e4e-40da-bd0a-ff61aa91455d-credential-socket\") pod \"7e485538-3e4e-40da-bd0a-ff61aa91455d\" (UID: \"7e485538-3e4e-40da-bd0a-ff61aa91455d\") " Sep 11 00:00:31.478902 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:31.478824 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e485538-3e4e-40da-bd0a-ff61aa91455d-workload-certs\") pod \"7e485538-3e4e-40da-bd0a-ff61aa91455d\" (UID: \"7e485538-3e4e-40da-bd0a-ff61aa91455d\") " Sep 11 00:00:31.478902 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:31.478882 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e485538-3e4e-40da-bd0a-ff61aa91455d-istio-podinfo\") pod \"7e485538-3e4e-40da-bd0a-ff61aa91455d\" (UID: \"7e485538-3e4e-40da-bd0a-ff61aa91455d\") " Sep 11 00:00:31.479055 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:31.478911 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e485538-3e4e-40da-bd0a-ff61aa91455d-workload-socket\") pod \"7e485538-3e4e-40da-bd0a-ff61aa91455d\" (UID: \"7e485538-3e4e-40da-bd0a-ff61aa91455d\") " Sep 11 00:00:31.479055 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:31.478932 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e485538-3e4e-40da-bd0a-ff61aa91455d-istio-token\") pod \"7e485538-3e4e-40da-bd0a-ff61aa91455d\" (UID: \"7e485538-3e4e-40da-bd0a-ff61aa91455d\") " Sep 11 00:00:31.479055 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:31.478935 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e485538-3e4e-40da-bd0a-ff61aa91455d-istio-data" (OuterVolumeSpecName: "istio-data") pod "7e485538-3e4e-40da-bd0a-ff61aa91455d" (UID: "7e485538-3e4e-40da-bd0a-ff61aa91455d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 00:00:31.479055 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:31.479015 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e485538-3e4e-40da-bd0a-ff61aa91455d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7e485538-3e4e-40da-bd0a-ff61aa91455d" (UID: "7e485538-3e4e-40da-bd0a-ff61aa91455d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 00:00:31.479262 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:31.479150 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e485538-3e4e-40da-bd0a-ff61aa91455d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7e485538-3e4e-40da-bd0a-ff61aa91455d" (UID: "7e485538-3e4e-40da-bd0a-ff61aa91455d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 00:00:31.479262 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:31.479200 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e485538-3e4e-40da-bd0a-ff61aa91455d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7e485538-3e4e-40da-bd0a-ff61aa91455d" (UID: "7e485538-3e4e-40da-bd0a-ff61aa91455d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 00:00:31.479372 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:31.479259 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e485538-3e4e-40da-bd0a-ff61aa91455d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7e485538-3e4e-40da-bd0a-ff61aa91455d" (UID: "7e485538-3e4e-40da-bd0a-ff61aa91455d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 00:00:31.479372 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:31.479363 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e485538-3e4e-40da-bd0a-ff61aa91455d-workload-certs\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 00:00:31.479534 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:31.479381 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e485538-3e4e-40da-bd0a-ff61aa91455d-workload-socket\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 00:00:31.479534 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:31.479396 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e485538-3e4e-40da-bd0a-ff61aa91455d-istio-data\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 00:00:31.479534 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:31.479410 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e485538-3e4e-40da-bd0a-ff61aa91455d-istiod-ca-cert\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 00:00:31.479534 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:31.479422 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e485538-3e4e-40da-bd0a-ff61aa91455d-credential-socket\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 00:00:31.481108 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:31.481076 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7e485538-3e4e-40da-bd0a-ff61aa91455d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7e485538-3e4e-40da-bd0a-ff61aa91455d" (UID: "7e485538-3e4e-40da-bd0a-ff61aa91455d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 00:00:31.481108 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:31.481077 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e485538-3e4e-40da-bd0a-ff61aa91455d-kube-api-access-2prgl" (OuterVolumeSpecName: "kube-api-access-2prgl") pod "7e485538-3e4e-40da-bd0a-ff61aa91455d" (UID: "7e485538-3e4e-40da-bd0a-ff61aa91455d"). InnerVolumeSpecName "kube-api-access-2prgl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 00:00:31.481299 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:31.481131 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e485538-3e4e-40da-bd0a-ff61aa91455d-istio-token" (OuterVolumeSpecName: "istio-token") pod "7e485538-3e4e-40da-bd0a-ff61aa91455d" (UID: "7e485538-3e4e-40da-bd0a-ff61aa91455d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 00:00:31.481299 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:31.481134 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e485538-3e4e-40da-bd0a-ff61aa91455d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7e485538-3e4e-40da-bd0a-ff61aa91455d" (UID: "7e485538-3e4e-40da-bd0a-ff61aa91455d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 00:00:31.580313 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:31.580274 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2prgl\" (UniqueName: \"kubernetes.io/projected/7e485538-3e4e-40da-bd0a-ff61aa91455d-kube-api-access-2prgl\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 00:00:31.580313 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:31.580307 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e485538-3e4e-40da-bd0a-ff61aa91455d-istio-envoy\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 00:00:31.580313 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:31.580317 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e485538-3e4e-40da-bd0a-ff61aa91455d-istio-podinfo\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 00:00:31.580545 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:31.580325 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e485538-3e4e-40da-bd0a-ff61aa91455d-istio-token\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 00:00:31.891414 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:31.891379 2574 generic.go:356] "Generic (PLEG): container finished" podID="7e485538-3e4e-40da-bd0a-ff61aa91455d" containerID="3d79f4132e9a5aa8125ef7e063aa5108e05a2d82b2da63f1c58dd3d799a5ce42" exitCode=0 Sep 11 00:00:31.891562 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:31.891436 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4ec4c8-openshift-default-56bf45968-xpv9q" Sep 11 00:00:31.891562 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:31.891467 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4ec4c8-openshift-default-56bf45968-xpv9q" event={"ID":"7e485538-3e4e-40da-bd0a-ff61aa91455d","Type":"ContainerDied","Data":"3d79f4132e9a5aa8125ef7e063aa5108e05a2d82b2da63f1c58dd3d799a5ce42"} Sep 11 00:00:31.891562 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:31.891524 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4ec4c8-openshift-default-56bf45968-xpv9q" event={"ID":"7e485538-3e4e-40da-bd0a-ff61aa91455d","Type":"ContainerDied","Data":"cbff45a0aac81bbb6e95126d9e992ea607ea3f441f91b1710f199c26802a2c5e"} Sep 11 00:00:31.891725 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:31.891547 2574 scope.go:117] "RemoveContainer" containerID="3d79f4132e9a5aa8125ef7e063aa5108e05a2d82b2da63f1c58dd3d799a5ce42" Sep 11 00:00:31.901808 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:31.901782 2574 scope.go:117] "RemoveContainer" containerID="f046b4487bb7de419f8379e5565528f5aa4c6aac191209447a7fa8d859756867" Sep 11 00:00:31.911195 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:31.911171 2574 scope.go:117] "RemoveContainer" containerID="3d79f4132e9a5aa8125ef7e063aa5108e05a2d82b2da63f1c58dd3d799a5ce42" Sep 11 00:00:31.911550 ip-10-0-138-128 kubenswrapper[2574]: E0911 00:00:31.911525 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d79f4132e9a5aa8125ef7e063aa5108e05a2d82b2da63f1c58dd3d799a5ce42\": container with ID starting with 3d79f4132e9a5aa8125ef7e063aa5108e05a2d82b2da63f1c58dd3d799a5ce42 not found: ID does not exist" containerID="3d79f4132e9a5aa8125ef7e063aa5108e05a2d82b2da63f1c58dd3d799a5ce42" Sep 11 00:00:31.911677 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:31.911556 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d79f4132e9a5aa8125ef7e063aa5108e05a2d82b2da63f1c58dd3d799a5ce42"} err="failed to get container status \"3d79f4132e9a5aa8125ef7e063aa5108e05a2d82b2da63f1c58dd3d799a5ce42\": rpc error: code = NotFound desc = could not find container \"3d79f4132e9a5aa8125ef7e063aa5108e05a2d82b2da63f1c58dd3d799a5ce42\": container with ID starting with 3d79f4132e9a5aa8125ef7e063aa5108e05a2d82b2da63f1c58dd3d799a5ce42 not found: ID does not exist" Sep 11 00:00:31.911677 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:31.911575 2574 scope.go:117] "RemoveContainer" containerID="f046b4487bb7de419f8379e5565528f5aa4c6aac191209447a7fa8d859756867" Sep 11 00:00:31.911843 ip-10-0-138-128 kubenswrapper[2574]: E0911 00:00:31.911824 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f046b4487bb7de419f8379e5565528f5aa4c6aac191209447a7fa8d859756867\": container with ID starting with f046b4487bb7de419f8379e5565528f5aa4c6aac191209447a7fa8d859756867 not found: ID does not exist" containerID="f046b4487bb7de419f8379e5565528f5aa4c6aac191209447a7fa8d859756867" Sep 11 00:00:31.911883 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:31.911850 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f046b4487bb7de419f8379e5565528f5aa4c6aac191209447a7fa8d859756867"} err="failed to get container status \"f046b4487bb7de419f8379e5565528f5aa4c6aac191209447a7fa8d859756867\": rpc error: code = NotFound desc = could not find container \"f046b4487bb7de419f8379e5565528f5aa4c6aac191209447a7fa8d859756867\": container with ID starting with f046b4487bb7de419f8379e5565528f5aa4c6aac191209447a7fa8d859756867 not found: ID does not exist" Sep 11 00:00:31.915936 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:31.915897 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-4ec4c8-openshift-default-56bf45968-xpv9q"] Sep 11 00:00:31.921463 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:31.921436 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-4ec4c8-openshift-default-56bf45968-xpv9q"] Sep 11 00:00:31.988982 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:31.988941 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-56r7l"] Sep 11 00:00:33.095477 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:33.095446 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e485538-3e4e-40da-bd0a-ff61aa91455d" path="/var/lib/kubelet/pods/7e485538-3e4e-40da-bd0a-ff61aa91455d/volumes" Sep 11 00:00:44.051548 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:44.051469 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-b19d25-openshift-default-774f4b768-9dmml"] Sep 11 00:00:44.052109 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:44.051753 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-b19d25-openshift-default-774f4b768-9dmml" podUID="ac20aa7e-7b99-4b5f-b498-8aeb713566d9" containerName="istio-proxy" containerID="cri-o://10d05b53b583659629607bea4de588c05b4177ded91dc486244bcaca3abaa8b5" gracePeriod=30 Sep 11 00:00:48.599066 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:48.599034 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-56r7l"] Sep 11 00:00:49.312200 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:49.312146 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-b19d25-openshift-default-774f4b768-9dmml" Sep 11 00:00:49.478035 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:49.478003 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac20aa7e-7b99-4b5f-b498-8aeb713566d9-istio-data\") pod \"ac20aa7e-7b99-4b5f-b498-8aeb713566d9\" (UID: \"ac20aa7e-7b99-4b5f-b498-8aeb713566d9\") " Sep 11 00:00:49.478225 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:49.478068 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lxh5k\" (UniqueName: \"kubernetes.io/projected/ac20aa7e-7b99-4b5f-b498-8aeb713566d9-kube-api-access-lxh5k\") pod \"ac20aa7e-7b99-4b5f-b498-8aeb713566d9\" (UID: \"ac20aa7e-7b99-4b5f-b498-8aeb713566d9\") " Sep 11 00:00:49.478225 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:49.478190 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac20aa7e-7b99-4b5f-b498-8aeb713566d9-istio-token\") pod \"ac20aa7e-7b99-4b5f-b498-8aeb713566d9\" (UID: \"ac20aa7e-7b99-4b5f-b498-8aeb713566d9\") " Sep 11 00:00:49.478352 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:49.478239 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac20aa7e-7b99-4b5f-b498-8aeb713566d9-istio-podinfo\") pod \"ac20aa7e-7b99-4b5f-b498-8aeb713566d9\" (UID: \"ac20aa7e-7b99-4b5f-b498-8aeb713566d9\") " Sep 11 00:00:49.478352 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:49.478267 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac20aa7e-7b99-4b5f-b498-8aeb713566d9-istio-envoy\") pod \"ac20aa7e-7b99-4b5f-b498-8aeb713566d9\" (UID: \"ac20aa7e-7b99-4b5f-b498-8aeb713566d9\") " Sep 11 00:00:49.478352 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:49.478288 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac20aa7e-7b99-4b5f-b498-8aeb713566d9-istio-data" (OuterVolumeSpecName: "istio-data") pod "ac20aa7e-7b99-4b5f-b498-8aeb713566d9" (UID: "ac20aa7e-7b99-4b5f-b498-8aeb713566d9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 00:00:49.478352 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:49.478305 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac20aa7e-7b99-4b5f-b498-8aeb713566d9-workload-socket\") pod \"ac20aa7e-7b99-4b5f-b498-8aeb713566d9\" (UID: \"ac20aa7e-7b99-4b5f-b498-8aeb713566d9\") " Sep 11 00:00:49.478563 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:49.478389 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac20aa7e-7b99-4b5f-b498-8aeb713566d9-workload-certs\") pod \"ac20aa7e-7b99-4b5f-b498-8aeb713566d9\" (UID: \"ac20aa7e-7b99-4b5f-b498-8aeb713566d9\") " Sep 11 00:00:49.478563 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:49.478430 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac20aa7e-7b99-4b5f-b498-8aeb713566d9-credential-socket\") pod \"ac20aa7e-7b99-4b5f-b498-8aeb713566d9\" (UID: \"ac20aa7e-7b99-4b5f-b498-8aeb713566d9\") " Sep 11 00:00:49.478563 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:49.478504 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac20aa7e-7b99-4b5f-b498-8aeb713566d9-istiod-ca-cert\") pod \"ac20aa7e-7b99-4b5f-b498-8aeb713566d9\" (UID: \"ac20aa7e-7b99-4b5f-b498-8aeb713566d9\") " Sep 11 00:00:49.478563 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:49.478514 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac20aa7e-7b99-4b5f-b498-8aeb713566d9-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ac20aa7e-7b99-4b5f-b498-8aeb713566d9" (UID: "ac20aa7e-7b99-4b5f-b498-8aeb713566d9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 00:00:49.478810 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:49.478671 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac20aa7e-7b99-4b5f-b498-8aeb713566d9-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ac20aa7e-7b99-4b5f-b498-8aeb713566d9" (UID: "ac20aa7e-7b99-4b5f-b498-8aeb713566d9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 00:00:49.478810 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:49.478766 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac20aa7e-7b99-4b5f-b498-8aeb713566d9-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ac20aa7e-7b99-4b5f-b498-8aeb713566d9" (UID: "ac20aa7e-7b99-4b5f-b498-8aeb713566d9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 00:00:49.478975 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:49.478946 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac20aa7e-7b99-4b5f-b498-8aeb713566d9-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ac20aa7e-7b99-4b5f-b498-8aeb713566d9" (UID: "ac20aa7e-7b99-4b5f-b498-8aeb713566d9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 00:00:49.479033 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:49.479021 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac20aa7e-7b99-4b5f-b498-8aeb713566d9-workload-socket\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 00:00:49.479090 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:49.479042 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac20aa7e-7b99-4b5f-b498-8aeb713566d9-workload-certs\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 00:00:49.479090 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:49.479057 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac20aa7e-7b99-4b5f-b498-8aeb713566d9-credential-socket\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 00:00:49.479090 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:49.479070 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac20aa7e-7b99-4b5f-b498-8aeb713566d9-istio-data\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 00:00:49.480322 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:49.480291 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac20aa7e-7b99-4b5f-b498-8aeb713566d9-kube-api-access-lxh5k" (OuterVolumeSpecName: "kube-api-access-lxh5k") pod "ac20aa7e-7b99-4b5f-b498-8aeb713566d9" (UID: "ac20aa7e-7b99-4b5f-b498-8aeb713566d9"). InnerVolumeSpecName "kube-api-access-lxh5k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 00:00:49.480431 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:49.480371 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac20aa7e-7b99-4b5f-b498-8aeb713566d9-istio-token" (OuterVolumeSpecName: "istio-token") pod "ac20aa7e-7b99-4b5f-b498-8aeb713566d9" (UID: "ac20aa7e-7b99-4b5f-b498-8aeb713566d9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 00:00:49.480664 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:49.480628 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ac20aa7e-7b99-4b5f-b498-8aeb713566d9-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ac20aa7e-7b99-4b5f-b498-8aeb713566d9" (UID: "ac20aa7e-7b99-4b5f-b498-8aeb713566d9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 00:00:49.480786 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:49.480768 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac20aa7e-7b99-4b5f-b498-8aeb713566d9-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ac20aa7e-7b99-4b5f-b498-8aeb713566d9" (UID: "ac20aa7e-7b99-4b5f-b498-8aeb713566d9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 00:00:49.580155 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:49.580066 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac20aa7e-7b99-4b5f-b498-8aeb713566d9-istio-envoy\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 00:00:49.580155 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:49.580097 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac20aa7e-7b99-4b5f-b498-8aeb713566d9-istiod-ca-cert\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 00:00:49.580155 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:49.580107 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxh5k\" (UniqueName: \"kubernetes.io/projected/ac20aa7e-7b99-4b5f-b498-8aeb713566d9-kube-api-access-lxh5k\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 00:00:49.580155 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:49.580118 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac20aa7e-7b99-4b5f-b498-8aeb713566d9-istio-token\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 00:00:49.580155 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:49.580126 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac20aa7e-7b99-4b5f-b498-8aeb713566d9-istio-podinfo\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 00:00:50.003748 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:50.003715 2574 generic.go:356] "Generic (PLEG): container finished" podID="ac20aa7e-7b99-4b5f-b498-8aeb713566d9" containerID="10d05b53b583659629607bea4de588c05b4177ded91dc486244bcaca3abaa8b5" exitCode=0 Sep 11 00:00:50.004165 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:50.003767 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-b19d25-openshift-default-774f4b768-9dmml" Sep 11 00:00:50.004165 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:50.003796 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-b19d25-openshift-default-774f4b768-9dmml" event={"ID":"ac20aa7e-7b99-4b5f-b498-8aeb713566d9","Type":"ContainerDied","Data":"10d05b53b583659629607bea4de588c05b4177ded91dc486244bcaca3abaa8b5"} Sep 11 00:00:50.004165 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:50.003851 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-b19d25-openshift-default-774f4b768-9dmml" event={"ID":"ac20aa7e-7b99-4b5f-b498-8aeb713566d9","Type":"ContainerDied","Data":"af87f9d07c849c27b969a14de2ab980c7f4261b7ab84900db82a0a4527b8f8b9"} Sep 11 00:00:50.004165 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:50.003875 2574 scope.go:117] "RemoveContainer" containerID="10d05b53b583659629607bea4de588c05b4177ded91dc486244bcaca3abaa8b5" Sep 11 00:00:50.015112 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:50.015094 2574 scope.go:117] "RemoveContainer" containerID="11da6f34441d7f840febb86536326a8d23dee7ce4d8a957674a334bcfae635c2" Sep 11 00:00:50.024389 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:50.024075 2574 scope.go:117] "RemoveContainer" containerID="10d05b53b583659629607bea4de588c05b4177ded91dc486244bcaca3abaa8b5" Sep 11 00:00:50.024464 ip-10-0-138-128 kubenswrapper[2574]: E0911 00:00:50.024387 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10d05b53b583659629607bea4de588c05b4177ded91dc486244bcaca3abaa8b5\": container with ID starting with 10d05b53b583659629607bea4de588c05b4177ded91dc486244bcaca3abaa8b5 not found: ID does not exist" containerID="10d05b53b583659629607bea4de588c05b4177ded91dc486244bcaca3abaa8b5" Sep 11 00:00:50.024464 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:50.024415 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10d05b53b583659629607bea4de588c05b4177ded91dc486244bcaca3abaa8b5"} err="failed to get container status \"10d05b53b583659629607bea4de588c05b4177ded91dc486244bcaca3abaa8b5\": rpc error: code = NotFound desc = could not find container \"10d05b53b583659629607bea4de588c05b4177ded91dc486244bcaca3abaa8b5\": container with ID starting with 10d05b53b583659629607bea4de588c05b4177ded91dc486244bcaca3abaa8b5 not found: ID does not exist" Sep 11 00:00:50.024464 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:50.024433 2574 scope.go:117] "RemoveContainer" containerID="11da6f34441d7f840febb86536326a8d23dee7ce4d8a957674a334bcfae635c2" Sep 11 00:00:50.024722 ip-10-0-138-128 kubenswrapper[2574]: E0911 00:00:50.024704 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11da6f34441d7f840febb86536326a8d23dee7ce4d8a957674a334bcfae635c2\": container with ID starting with 11da6f34441d7f840febb86536326a8d23dee7ce4d8a957674a334bcfae635c2 not found: ID does not exist" containerID="11da6f34441d7f840febb86536326a8d23dee7ce4d8a957674a334bcfae635c2" Sep 11 00:00:50.024773 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:50.024729 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11da6f34441d7f840febb86536326a8d23dee7ce4d8a957674a334bcfae635c2"} err="failed to get container status \"11da6f34441d7f840febb86536326a8d23dee7ce4d8a957674a334bcfae635c2\": rpc error: code = NotFound desc = could not find container \"11da6f34441d7f840febb86536326a8d23dee7ce4d8a957674a334bcfae635c2\": container with ID starting with 11da6f34441d7f840febb86536326a8d23dee7ce4d8a957674a334bcfae635c2 not found: ID does not exist" Sep 11 00:00:50.027030 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:50.027011 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-b19d25-openshift-default-774f4b768-9dmml"] Sep 11 00:00:50.032590 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:50.032568 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-b19d25-openshift-default-774f4b768-9dmml"] Sep 11 00:00:51.097840 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:51.097794 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac20aa7e-7b99-4b5f-b498-8aeb713566d9" path="/var/lib/kubelet/pods/ac20aa7e-7b99-4b5f-b498-8aeb713566d9/volumes" Sep 11 00:00:54.801228 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:54.801191 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a78403-6694469658-kvj7z"] Sep 11 00:00:54.801800 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:54.801439 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a78403-6694469658-kvj7z" podUID="51ad6c59-aeb5-4c42-8ee6-f4c889ca454a" containerName="maas-api" containerID="cri-o://0655b931b5996df30f6da4a94af3ea044e19c64854f5f536e09e6322cb7f0b46" gracePeriod=30 Sep 11 00:00:55.036187 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:55.036158 2574 generic.go:356] "Generic (PLEG): container finished" podID="51ad6c59-aeb5-4c42-8ee6-f4c889ca454a" containerID="0655b931b5996df30f6da4a94af3ea044e19c64854f5f536e09e6322cb7f0b46" exitCode=0 Sep 11 00:00:55.036330 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:55.036204 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a78403-6694469658-kvj7z" event={"ID":"51ad6c59-aeb5-4c42-8ee6-f4c889ca454a","Type":"ContainerDied","Data":"0655b931b5996df30f6da4a94af3ea044e19c64854f5f536e09e6322cb7f0b46"} Sep 11 00:00:55.058997 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:55.058912 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a78403-6694469658-kvj7z" Sep 11 00:00:55.138825 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:55.138786 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51ad6c59-aeb5-4c42-8ee6-f4c889ca454a-openshift-service-ca\") pod \"51ad6c59-aeb5-4c42-8ee6-f4c889ca454a\" (UID: \"51ad6c59-aeb5-4c42-8ee6-f4c889ca454a\") " Sep 11 00:00:55.139014 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:55.138863 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q58kl\" (UniqueName: \"kubernetes.io/projected/51ad6c59-aeb5-4c42-8ee6-f4c889ca454a-kube-api-access-q58kl\") pod \"51ad6c59-aeb5-4c42-8ee6-f4c889ca454a\" (UID: \"51ad6c59-aeb5-4c42-8ee6-f4c889ca454a\") " Sep 11 00:00:55.139014 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:55.138948 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51ad6c59-aeb5-4c42-8ee6-f4c889ca454a-metrics-certs\") pod \"51ad6c59-aeb5-4c42-8ee6-f4c889ca454a\" (UID: \"51ad6c59-aeb5-4c42-8ee6-f4c889ca454a\") " Sep 11 00:00:55.139014 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:55.139002 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51ad6c59-aeb5-4c42-8ee6-f4c889ca454a-maas-api-tls\") pod \"51ad6c59-aeb5-4c42-8ee6-f4c889ca454a\" (UID: \"51ad6c59-aeb5-4c42-8ee6-f4c889ca454a\") " Sep 11 00:00:55.139227 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:55.139202 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51ad6c59-aeb5-4c42-8ee6-f4c889ca454a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "51ad6c59-aeb5-4c42-8ee6-f4c889ca454a" (UID: "51ad6c59-aeb5-4c42-8ee6-f4c889ca454a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 00:00:55.139487 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:55.139462 2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51ad6c59-aeb5-4c42-8ee6-f4c889ca454a-openshift-service-ca\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 00:00:55.141159 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:55.141131 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51ad6c59-aeb5-4c42-8ee6-f4c889ca454a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "51ad6c59-aeb5-4c42-8ee6-f4c889ca454a" (UID: "51ad6c59-aeb5-4c42-8ee6-f4c889ca454a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 00:00:55.141286 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:55.141187 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51ad6c59-aeb5-4c42-8ee6-f4c889ca454a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "51ad6c59-aeb5-4c42-8ee6-f4c889ca454a" (UID: "51ad6c59-aeb5-4c42-8ee6-f4c889ca454a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 00:00:55.141286 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:55.141236 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51ad6c59-aeb5-4c42-8ee6-f4c889ca454a-kube-api-access-q58kl" (OuterVolumeSpecName: "kube-api-access-q58kl") pod "51ad6c59-aeb5-4c42-8ee6-f4c889ca454a" (UID: "51ad6c59-aeb5-4c42-8ee6-f4c889ca454a"). InnerVolumeSpecName "kube-api-access-q58kl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 00:00:55.240684 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:55.240628 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q58kl\" (UniqueName: \"kubernetes.io/projected/51ad6c59-aeb5-4c42-8ee6-f4c889ca454a-kube-api-access-q58kl\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 00:00:55.240684 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:55.240680 2574 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51ad6c59-aeb5-4c42-8ee6-f4c889ca454a-metrics-certs\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 00:00:55.240684 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:55.240690 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51ad6c59-aeb5-4c42-8ee6-f4c889ca454a-maas-api-tls\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 00:00:56.044493 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:56.044462 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a78403-6694469658-kvj7z" Sep 11 00:00:56.044969 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:56.044502 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a78403-6694469658-kvj7z" event={"ID":"51ad6c59-aeb5-4c42-8ee6-f4c889ca454a","Type":"ContainerDied","Data":"38476388961a87959906620558ab3c82d3475be4eac2bc792633af6573a3b7d3"} Sep 11 00:00:56.044969 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:56.044565 2574 scope.go:117] "RemoveContainer" containerID="0655b931b5996df30f6da4a94af3ea044e19c64854f5f536e09e6322cb7f0b46" Sep 11 00:00:56.066100 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:56.066073 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a78403-6694469658-kvj7z"] Sep 11 00:00:56.071428 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:56.071389 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a78403-6694469658-kvj7z"] Sep 11 00:00:57.093937 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:57.093906 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51ad6c59-aeb5-4c42-8ee6-f4c889ca454a" path="/var/lib/kubelet/pods/51ad6c59-aeb5-4c42-8ee6-f4c889ca454a/volumes" Sep 11 00:00:59.484327 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:00:59.484290 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-56r7l"] Sep 11 00:01:02.339116 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:02.339075 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-a78403-openshift-default-6b9d4896d6-cs5dw"] Sep 11 00:01:02.339664 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:02.339290 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-a78403-openshift-default-6b9d4896d6-cs5dw" podUID="c52960ab-0f37-4e17-a6e0-bbc346e0aa38" containerName="istio-proxy" containerID="cri-o://8c84ebe08796d9536ffe4c56d504d0fba3852642376fc1582c084fbc4609615a" gracePeriod=30 Sep 11 00:01:07.584634 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:07.584607 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-a78403-openshift-default-6b9d4896d6-cs5dw" Sep 11 00:01:07.663840 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:07.663755 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c52960ab-0f37-4e17-a6e0-bbc346e0aa38-istio-token\") pod \"c52960ab-0f37-4e17-a6e0-bbc346e0aa38\" (UID: \"c52960ab-0f37-4e17-a6e0-bbc346e0aa38\") " Sep 11 00:01:07.663994 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:07.663844 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c52960ab-0f37-4e17-a6e0-bbc346e0aa38-istio-podinfo\") pod \"c52960ab-0f37-4e17-a6e0-bbc346e0aa38\" (UID: \"c52960ab-0f37-4e17-a6e0-bbc346e0aa38\") " Sep 11 00:01:07.663994 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:07.663880 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2txbs\" (UniqueName: \"kubernetes.io/projected/c52960ab-0f37-4e17-a6e0-bbc346e0aa38-kube-api-access-2txbs\") pod \"c52960ab-0f37-4e17-a6e0-bbc346e0aa38\" (UID: \"c52960ab-0f37-4e17-a6e0-bbc346e0aa38\") " Sep 11 00:01:07.663994 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:07.663909 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c52960ab-0f37-4e17-a6e0-bbc346e0aa38-istio-data\") pod \"c52960ab-0f37-4e17-a6e0-bbc346e0aa38\" (UID: \"c52960ab-0f37-4e17-a6e0-bbc346e0aa38\") " Sep 11 00:01:07.663994 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:07.663971 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c52960ab-0f37-4e17-a6e0-bbc346e0aa38-workload-socket\") pod \"c52960ab-0f37-4e17-a6e0-bbc346e0aa38\" (UID: \"c52960ab-0f37-4e17-a6e0-bbc346e0aa38\") " Sep 11 00:01:07.664198 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:07.664026 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c52960ab-0f37-4e17-a6e0-bbc346e0aa38-workload-certs\") pod \"c52960ab-0f37-4e17-a6e0-bbc346e0aa38\" (UID: \"c52960ab-0f37-4e17-a6e0-bbc346e0aa38\") " Sep 11 00:01:07.664198 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:07.664063 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c52960ab-0f37-4e17-a6e0-bbc346e0aa38-istiod-ca-cert\") pod \"c52960ab-0f37-4e17-a6e0-bbc346e0aa38\" (UID: \"c52960ab-0f37-4e17-a6e0-bbc346e0aa38\") " Sep 11 00:01:07.664198 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:07.664099 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c52960ab-0f37-4e17-a6e0-bbc346e0aa38-istio-envoy\") pod \"c52960ab-0f37-4e17-a6e0-bbc346e0aa38\" (UID: \"c52960ab-0f37-4e17-a6e0-bbc346e0aa38\") " Sep 11 00:01:07.664198 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:07.664135 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c52960ab-0f37-4e17-a6e0-bbc346e0aa38-credential-socket\") pod \"c52960ab-0f37-4e17-a6e0-bbc346e0aa38\" (UID: \"c52960ab-0f37-4e17-a6e0-bbc346e0aa38\") " Sep 11 00:01:07.664401 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:07.664229 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c52960ab-0f37-4e17-a6e0-bbc346e0aa38-istio-data" (OuterVolumeSpecName: "istio-data") pod "c52960ab-0f37-4e17-a6e0-bbc346e0aa38" (UID: "c52960ab-0f37-4e17-a6e0-bbc346e0aa38"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 00:01:07.664401 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:07.664293 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c52960ab-0f37-4e17-a6e0-bbc346e0aa38-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c52960ab-0f37-4e17-a6e0-bbc346e0aa38" (UID: "c52960ab-0f37-4e17-a6e0-bbc346e0aa38"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 00:01:07.664502 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:07.664464 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c52960ab-0f37-4e17-a6e0-bbc346e0aa38-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c52960ab-0f37-4e17-a6e0-bbc346e0aa38" (UID: "c52960ab-0f37-4e17-a6e0-bbc346e0aa38"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 00:01:07.664614 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:07.664595 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c52960ab-0f37-4e17-a6e0-bbc346e0aa38-istio-data\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 00:01:07.664821 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:07.664796 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c52960ab-0f37-4e17-a6e0-bbc346e0aa38-workload-socket\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 00:01:07.664941 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:07.664823 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c52960ab-0f37-4e17-a6e0-bbc346e0aa38-credential-socket\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 00:01:07.664941 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:07.664591 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c52960ab-0f37-4e17-a6e0-bbc346e0aa38-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c52960ab-0f37-4e17-a6e0-bbc346e0aa38" (UID: "c52960ab-0f37-4e17-a6e0-bbc346e0aa38"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 00:01:07.664941 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:07.664701 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c52960ab-0f37-4e17-a6e0-bbc346e0aa38-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c52960ab-0f37-4e17-a6e0-bbc346e0aa38" (UID: "c52960ab-0f37-4e17-a6e0-bbc346e0aa38"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 00:01:07.666203 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:07.666160 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c52960ab-0f37-4e17-a6e0-bbc346e0aa38-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c52960ab-0f37-4e17-a6e0-bbc346e0aa38" (UID: "c52960ab-0f37-4e17-a6e0-bbc346e0aa38"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 00:01:07.666289 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:07.666256 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c52960ab-0f37-4e17-a6e0-bbc346e0aa38-kube-api-access-2txbs" (OuterVolumeSpecName: "kube-api-access-2txbs") pod "c52960ab-0f37-4e17-a6e0-bbc346e0aa38" (UID: "c52960ab-0f37-4e17-a6e0-bbc346e0aa38"). InnerVolumeSpecName "kube-api-access-2txbs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 00:01:07.666289 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:07.666259 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c52960ab-0f37-4e17-a6e0-bbc346e0aa38-istio-token" (OuterVolumeSpecName: "istio-token") pod "c52960ab-0f37-4e17-a6e0-bbc346e0aa38" (UID: "c52960ab-0f37-4e17-a6e0-bbc346e0aa38"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 00:01:07.666363 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:07.666313 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c52960ab-0f37-4e17-a6e0-bbc346e0aa38-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c52960ab-0f37-4e17-a6e0-bbc346e0aa38" (UID: "c52960ab-0f37-4e17-a6e0-bbc346e0aa38"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 00:01:07.765892 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:07.765849 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c52960ab-0f37-4e17-a6e0-bbc346e0aa38-workload-certs\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 00:01:07.765892 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:07.765888 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c52960ab-0f37-4e17-a6e0-bbc346e0aa38-istiod-ca-cert\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 00:01:07.765892 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:07.765900 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c52960ab-0f37-4e17-a6e0-bbc346e0aa38-istio-envoy\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 00:01:07.766143 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:07.765911 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c52960ab-0f37-4e17-a6e0-bbc346e0aa38-istio-token\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 00:01:07.766143 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:07.765921 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c52960ab-0f37-4e17-a6e0-bbc346e0aa38-istio-podinfo\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 00:01:07.766143 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:07.765932 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2txbs\" (UniqueName: \"kubernetes.io/projected/c52960ab-0f37-4e17-a6e0-bbc346e0aa38-kube-api-access-2txbs\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 00:01:08.123972 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:08.123937 2574 generic.go:356] "Generic (PLEG): container finished" podID="c52960ab-0f37-4e17-a6e0-bbc346e0aa38" containerID="8c84ebe08796d9536ffe4c56d504d0fba3852642376fc1582c084fbc4609615a" exitCode=0 Sep 11 00:01:08.124175 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:08.123984 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-a78403-openshift-default-6b9d4896d6-cs5dw" Sep 11 00:01:08.124175 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:08.124022 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-a78403-openshift-default-6b9d4896d6-cs5dw" event={"ID":"c52960ab-0f37-4e17-a6e0-bbc346e0aa38","Type":"ContainerDied","Data":"8c84ebe08796d9536ffe4c56d504d0fba3852642376fc1582c084fbc4609615a"} Sep 11 00:01:08.124175 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:08.124069 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-a78403-openshift-default-6b9d4896d6-cs5dw" event={"ID":"c52960ab-0f37-4e17-a6e0-bbc346e0aa38","Type":"ContainerDied","Data":"5ebafcb983daf8038e04df3228bf7d8d35d89bb7990759fc3ad156683d002c9f"} Sep 11 00:01:08.124175 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:08.124081 2574 scope.go:117] "RemoveContainer" containerID="8c84ebe08796d9536ffe4c56d504d0fba3852642376fc1582c084fbc4609615a" Sep 11 00:01:08.133987 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:08.133971 2574 scope.go:117] "RemoveContainer" containerID="2b4f71e50e35396a6bd685ad9e0011b3d726c10ebb764307ae0d43b2e2e71c66" Sep 11 00:01:08.142734 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:08.142711 2574 scope.go:117] "RemoveContainer" containerID="8c84ebe08796d9536ffe4c56d504d0fba3852642376fc1582c084fbc4609615a" Sep 11 00:01:08.143020 ip-10-0-138-128 kubenswrapper[2574]: E0911 00:01:08.143001 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c84ebe08796d9536ffe4c56d504d0fba3852642376fc1582c084fbc4609615a\": container with ID starting with 8c84ebe08796d9536ffe4c56d504d0fba3852642376fc1582c084fbc4609615a not found: ID does not exist" containerID="8c84ebe08796d9536ffe4c56d504d0fba3852642376fc1582c084fbc4609615a" Sep 11 00:01:08.143071 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:08.143051 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c84ebe08796d9536ffe4c56d504d0fba3852642376fc1582c084fbc4609615a"} err="failed to get container status \"8c84ebe08796d9536ffe4c56d504d0fba3852642376fc1582c084fbc4609615a\": rpc error: code = NotFound desc = could not find container \"8c84ebe08796d9536ffe4c56d504d0fba3852642376fc1582c084fbc4609615a\": container with ID starting with 8c84ebe08796d9536ffe4c56d504d0fba3852642376fc1582c084fbc4609615a not found: ID does not exist" Sep 11 00:01:08.143071 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:08.143069 2574 scope.go:117] "RemoveContainer" containerID="2b4f71e50e35396a6bd685ad9e0011b3d726c10ebb764307ae0d43b2e2e71c66" Sep 11 00:01:08.143355 ip-10-0-138-128 kubenswrapper[2574]: E0911 00:01:08.143323 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b4f71e50e35396a6bd685ad9e0011b3d726c10ebb764307ae0d43b2e2e71c66\": container with ID starting with 2b4f71e50e35396a6bd685ad9e0011b3d726c10ebb764307ae0d43b2e2e71c66 not found: ID does not exist" containerID="2b4f71e50e35396a6bd685ad9e0011b3d726c10ebb764307ae0d43b2e2e71c66" Sep 11 00:01:08.143397 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:08.143359 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b4f71e50e35396a6bd685ad9e0011b3d726c10ebb764307ae0d43b2e2e71c66"} err="failed to get container status \"2b4f71e50e35396a6bd685ad9e0011b3d726c10ebb764307ae0d43b2e2e71c66\": rpc error: code = NotFound desc = could not find container \"2b4f71e50e35396a6bd685ad9e0011b3d726c10ebb764307ae0d43b2e2e71c66\": container with ID starting with 2b4f71e50e35396a6bd685ad9e0011b3d726c10ebb764307ae0d43b2e2e71c66 not found: ID does not exist" Sep 11 00:01:08.145054 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:08.145036 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-a78403-openshift-default-6b9d4896d6-cs5dw"] Sep 11 00:01:08.154061 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:08.154036 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-a78403-openshift-default-6b9d4896d6-cs5dw"] Sep 11 00:01:09.102265 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:09.102231 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c52960ab-0f37-4e17-a6e0-bbc346e0aa38" path="/var/lib/kubelet/pods/c52960ab-0f37-4e17-a6e0-bbc346e0aa38/volumes" Sep 11 00:01:12.681232 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:12.681192 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-56r7l"] Sep 11 00:01:26.577966 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:26.577883 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-56r7l"] Sep 11 00:01:36.385510 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:36.385469 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-56r7l"] Sep 11 00:01:47.688681 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:47.688626 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-56r7l"] Sep 11 00:01:53.554201 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:53.554164 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-17664f-openshift-default-69464f8bf4-bvqbl"] Sep 11 00:01:53.554818 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:53.554798 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c52960ab-0f37-4e17-a6e0-bbc346e0aa38" containerName="istio-proxy" Sep 11 00:01:53.554901 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:53.554819 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="c52960ab-0f37-4e17-a6e0-bbc346e0aa38" containerName="istio-proxy" Sep 11 00:01:53.554901 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:53.554828 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51ad6c59-aeb5-4c42-8ee6-f4c889ca454a" containerName="maas-api" Sep 11 00:01:53.554901 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:53.554834 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="51ad6c59-aeb5-4c42-8ee6-f4c889ca454a" containerName="maas-api" Sep 11 00:01:53.554901 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:53.554846 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eef2f53c-651e-41b0-a742-2d509e67bf4b" containerName="istio-proxy" Sep 11 00:01:53.554901 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:53.554855 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="eef2f53c-651e-41b0-a742-2d509e67bf4b" containerName="istio-proxy" Sep 11 00:01:53.554901 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:53.554864 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e485538-3e4e-40da-bd0a-ff61aa91455d" containerName="istio-proxy" Sep 11 00:01:53.554901 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:53.554870 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e485538-3e4e-40da-bd0a-ff61aa91455d" containerName="istio-proxy" Sep 11 00:01:53.554901 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:53.554876 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3394f36c-47c5-4ea7-a9a8-264bf006cb45" containerName="cleanup" Sep 11 00:01:53.554901 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:53.554881 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="3394f36c-47c5-4ea7-a9a8-264bf006cb45" containerName="cleanup" Sep 11 00:01:53.554901 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:53.554893 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eef2f53c-651e-41b0-a742-2d509e67bf4b" containerName="istio-proxy" Sep 11 00:01:53.554901 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:53.554899 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="eef2f53c-651e-41b0-a742-2d509e67bf4b" containerName="istio-proxy" Sep 11 00:01:53.554901 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:53.554905 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e485538-3e4e-40da-bd0a-ff61aa91455d" containerName="istio-proxy" Sep 11 00:01:53.555574 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:53.554910 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e485538-3e4e-40da-bd0a-ff61aa91455d" containerName="istio-proxy" Sep 11 00:01:53.555574 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:53.554916 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="198384a6-ba0b-4d39-956d-f33800f5b531" containerName="cleanup" Sep 11 00:01:53.555574 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:53.554922 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="198384a6-ba0b-4d39-956d-f33800f5b531" containerName="cleanup" Sep 11 00:01:53.555574 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:53.554936 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac20aa7e-7b99-4b5f-b498-8aeb713566d9" containerName="istio-proxy" Sep 11 00:01:53.555574 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:53.554942 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac20aa7e-7b99-4b5f-b498-8aeb713566d9" containerName="istio-proxy" Sep 11 00:01:53.555574 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:53.554950 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac20aa7e-7b99-4b5f-b498-8aeb713566d9" containerName="istio-proxy" Sep 11 00:01:53.555574 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:53.554955 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac20aa7e-7b99-4b5f-b498-8aeb713566d9" containerName="istio-proxy" Sep 11 00:01:53.555574 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:53.555021 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e485538-3e4e-40da-bd0a-ff61aa91455d" containerName="istio-proxy" Sep 11 00:01:53.555574 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:53.555031 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="c52960ab-0f37-4e17-a6e0-bbc346e0aa38" containerName="istio-proxy" Sep 11 00:01:53.555574 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:53.555037 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="eef2f53c-651e-41b0-a742-2d509e67bf4b" containerName="istio-proxy" Sep 11 00:01:53.555574 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:53.555042 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="198384a6-ba0b-4d39-956d-f33800f5b531" containerName="cleanup" Sep 11 00:01:53.555574 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:53.555047 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac20aa7e-7b99-4b5f-b498-8aeb713566d9" containerName="istio-proxy" Sep 11 00:01:53.555574 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:53.555053 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="c52960ab-0f37-4e17-a6e0-bbc346e0aa38" containerName="istio-proxy" Sep 11 00:01:53.555574 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:53.555059 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="eef2f53c-651e-41b0-a742-2d509e67bf4b" containerName="istio-proxy" Sep 11 00:01:53.555574 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:53.555064 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac20aa7e-7b99-4b5f-b498-8aeb713566d9" containerName="istio-proxy" Sep 11 00:01:53.555574 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:53.555069 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="3394f36c-47c5-4ea7-a9a8-264bf006cb45" containerName="cleanup" Sep 11 00:01:53.555574 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:53.555073 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="51ad6c59-aeb5-4c42-8ee6-f4c889ca454a" containerName="maas-api" Sep 11 00:01:53.558294 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:53.558269 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-17664f-openshift-default-69464f8bf4-bvqbl" Sep 11 00:01:53.560596 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:53.560574 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-17664f-openshift-default-dockercfg-nnftq\"" Sep 11 00:01:53.570961 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:53.570938 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-17664f-openshift-default-69464f8bf4-bvqbl"] Sep 11 00:01:53.704823 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:53.704789 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7c5b79b-f87f-4fc7-9734-e52a01fb7023-istio-podinfo\") pod \"e2e-ipp-a-17664f-openshift-default-69464f8bf4-bvqbl\" (UID: \"f7c5b79b-f87f-4fc7-9734-e52a01fb7023\") " pod="openshift-ingress/e2e-ipp-a-17664f-openshift-default-69464f8bf4-bvqbl" Sep 11 00:01:53.704995 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:53.704830 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7c5b79b-f87f-4fc7-9734-e52a01fb7023-workload-certs\") pod \"e2e-ipp-a-17664f-openshift-default-69464f8bf4-bvqbl\" (UID: \"f7c5b79b-f87f-4fc7-9734-e52a01fb7023\") " pod="openshift-ingress/e2e-ipp-a-17664f-openshift-default-69464f8bf4-bvqbl" Sep 11 00:01:53.704995 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:53.704850 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7c5b79b-f87f-4fc7-9734-e52a01fb7023-istio-envoy\") pod \"e2e-ipp-a-17664f-openshift-default-69464f8bf4-bvqbl\" (UID: \"f7c5b79b-f87f-4fc7-9734-e52a01fb7023\") " pod="openshift-ingress/e2e-ipp-a-17664f-openshift-default-69464f8bf4-bvqbl" Sep 11 00:01:53.704995 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:53.704884 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxn7v\" (UniqueName: \"kubernetes.io/projected/f7c5b79b-f87f-4fc7-9734-e52a01fb7023-kube-api-access-cxn7v\") pod \"e2e-ipp-a-17664f-openshift-default-69464f8bf4-bvqbl\" (UID: \"f7c5b79b-f87f-4fc7-9734-e52a01fb7023\") " pod="openshift-ingress/e2e-ipp-a-17664f-openshift-default-69464f8bf4-bvqbl" Sep 11 00:01:53.704995 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:53.704982 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7c5b79b-f87f-4fc7-9734-e52a01fb7023-workload-socket\") pod \"e2e-ipp-a-17664f-openshift-default-69464f8bf4-bvqbl\" (UID: \"f7c5b79b-f87f-4fc7-9734-e52a01fb7023\") " pod="openshift-ingress/e2e-ipp-a-17664f-openshift-default-69464f8bf4-bvqbl" Sep 11 00:01:53.705145 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:53.705017 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7c5b79b-f87f-4fc7-9734-e52a01fb7023-credential-socket\") pod \"e2e-ipp-a-17664f-openshift-default-69464f8bf4-bvqbl\" (UID: \"f7c5b79b-f87f-4fc7-9734-e52a01fb7023\") " pod="openshift-ingress/e2e-ipp-a-17664f-openshift-default-69464f8bf4-bvqbl" Sep 11 00:01:53.705145 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:53.705090 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7c5b79b-f87f-4fc7-9734-e52a01fb7023-istiod-ca-cert\") pod \"e2e-ipp-a-17664f-openshift-default-69464f8bf4-bvqbl\" (UID: \"f7c5b79b-f87f-4fc7-9734-e52a01fb7023\") " pod="openshift-ingress/e2e-ipp-a-17664f-openshift-default-69464f8bf4-bvqbl" Sep 11 00:01:53.705212 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:53.705147 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7c5b79b-f87f-4fc7-9734-e52a01fb7023-istio-token\") pod \"e2e-ipp-a-17664f-openshift-default-69464f8bf4-bvqbl\" (UID: \"f7c5b79b-f87f-4fc7-9734-e52a01fb7023\") " pod="openshift-ingress/e2e-ipp-a-17664f-openshift-default-69464f8bf4-bvqbl" Sep 11 00:01:53.705212 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:53.705181 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7c5b79b-f87f-4fc7-9734-e52a01fb7023-istio-data\") pod \"e2e-ipp-a-17664f-openshift-default-69464f8bf4-bvqbl\" (UID: \"f7c5b79b-f87f-4fc7-9734-e52a01fb7023\") " pod="openshift-ingress/e2e-ipp-a-17664f-openshift-default-69464f8bf4-bvqbl" Sep 11 00:01:53.806064 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:53.805985 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7c5b79b-f87f-4fc7-9734-e52a01fb7023-istio-podinfo\") pod \"e2e-ipp-a-17664f-openshift-default-69464f8bf4-bvqbl\" (UID: \"f7c5b79b-f87f-4fc7-9734-e52a01fb7023\") " pod="openshift-ingress/e2e-ipp-a-17664f-openshift-default-69464f8bf4-bvqbl" Sep 11 00:01:53.806064 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:53.806026 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7c5b79b-f87f-4fc7-9734-e52a01fb7023-workload-certs\") pod \"e2e-ipp-a-17664f-openshift-default-69464f8bf4-bvqbl\" (UID: \"f7c5b79b-f87f-4fc7-9734-e52a01fb7023\") " pod="openshift-ingress/e2e-ipp-a-17664f-openshift-default-69464f8bf4-bvqbl" Sep 11 00:01:53.806064 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:53.806044 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7c5b79b-f87f-4fc7-9734-e52a01fb7023-istio-envoy\") pod \"e2e-ipp-a-17664f-openshift-default-69464f8bf4-bvqbl\" (UID: \"f7c5b79b-f87f-4fc7-9734-e52a01fb7023\") " pod="openshift-ingress/e2e-ipp-a-17664f-openshift-default-69464f8bf4-bvqbl" Sep 11 00:01:53.806314 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:53.806102 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxn7v\" (UniqueName: \"kubernetes.io/projected/f7c5b79b-f87f-4fc7-9734-e52a01fb7023-kube-api-access-cxn7v\") pod \"e2e-ipp-a-17664f-openshift-default-69464f8bf4-bvqbl\" (UID: \"f7c5b79b-f87f-4fc7-9734-e52a01fb7023\") " pod="openshift-ingress/e2e-ipp-a-17664f-openshift-default-69464f8bf4-bvqbl" Sep 11 00:01:53.806314 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:53.806191 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7c5b79b-f87f-4fc7-9734-e52a01fb7023-workload-socket\") pod \"e2e-ipp-a-17664f-openshift-default-69464f8bf4-bvqbl\" (UID: \"f7c5b79b-f87f-4fc7-9734-e52a01fb7023\") " pod="openshift-ingress/e2e-ipp-a-17664f-openshift-default-69464f8bf4-bvqbl" Sep 11 00:01:53.806314 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:53.806222 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7c5b79b-f87f-4fc7-9734-e52a01fb7023-credential-socket\") pod \"e2e-ipp-a-17664f-openshift-default-69464f8bf4-bvqbl\" (UID: \"f7c5b79b-f87f-4fc7-9734-e52a01fb7023\") " pod="openshift-ingress/e2e-ipp-a-17664f-openshift-default-69464f8bf4-bvqbl" Sep 11 00:01:53.806314 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:53.806291 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7c5b79b-f87f-4fc7-9734-e52a01fb7023-istiod-ca-cert\") pod \"e2e-ipp-a-17664f-openshift-default-69464f8bf4-bvqbl\" (UID: \"f7c5b79b-f87f-4fc7-9734-e52a01fb7023\") " pod="openshift-ingress/e2e-ipp-a-17664f-openshift-default-69464f8bf4-bvqbl" Sep 11 00:01:53.806535 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:53.806358 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7c5b79b-f87f-4fc7-9734-e52a01fb7023-istio-token\") pod \"e2e-ipp-a-17664f-openshift-default-69464f8bf4-bvqbl\" (UID: \"f7c5b79b-f87f-4fc7-9734-e52a01fb7023\") " pod="openshift-ingress/e2e-ipp-a-17664f-openshift-default-69464f8bf4-bvqbl" Sep 11 00:01:53.806535 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:53.806519 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7c5b79b-f87f-4fc7-9734-e52a01fb7023-workload-socket\") pod \"e2e-ipp-a-17664f-openshift-default-69464f8bf4-bvqbl\" (UID: \"f7c5b79b-f87f-4fc7-9734-e52a01fb7023\") " pod="openshift-ingress/e2e-ipp-a-17664f-openshift-default-69464f8bf4-bvqbl" Sep 11 00:01:53.806685 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:53.806597 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7c5b79b-f87f-4fc7-9734-e52a01fb7023-credential-socket\") pod \"e2e-ipp-a-17664f-openshift-default-69464f8bf4-bvqbl\" (UID: \"f7c5b79b-f87f-4fc7-9734-e52a01fb7023\") " pod="openshift-ingress/e2e-ipp-a-17664f-openshift-default-69464f8bf4-bvqbl" Sep 11 00:01:53.806749 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:53.806717 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7c5b79b-f87f-4fc7-9734-e52a01fb7023-workload-certs\") pod \"e2e-ipp-a-17664f-openshift-default-69464f8bf4-bvqbl\" (UID: \"f7c5b79b-f87f-4fc7-9734-e52a01fb7023\") " pod="openshift-ingress/e2e-ipp-a-17664f-openshift-default-69464f8bf4-bvqbl" Sep 11 00:01:53.806957 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:53.806934 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7c5b79b-f87f-4fc7-9734-e52a01fb7023-istio-data\") pod \"e2e-ipp-a-17664f-openshift-default-69464f8bf4-bvqbl\" (UID: \"f7c5b79b-f87f-4fc7-9734-e52a01fb7023\") " pod="openshift-ingress/e2e-ipp-a-17664f-openshift-default-69464f8bf4-bvqbl" Sep 11 00:01:53.807012 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:53.806995 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7c5b79b-f87f-4fc7-9734-e52a01fb7023-istiod-ca-cert\") pod \"e2e-ipp-a-17664f-openshift-default-69464f8bf4-bvqbl\" (UID: \"f7c5b79b-f87f-4fc7-9734-e52a01fb7023\") " pod="openshift-ingress/e2e-ipp-a-17664f-openshift-default-69464f8bf4-bvqbl" Sep 11 00:01:53.807188 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:53.807174 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7c5b79b-f87f-4fc7-9734-e52a01fb7023-istio-data\") pod \"e2e-ipp-a-17664f-openshift-default-69464f8bf4-bvqbl\" (UID: \"f7c5b79b-f87f-4fc7-9734-e52a01fb7023\") " pod="openshift-ingress/e2e-ipp-a-17664f-openshift-default-69464f8bf4-bvqbl" Sep 11 00:01:53.808214 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:53.808185 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7c5b79b-f87f-4fc7-9734-e52a01fb7023-istio-envoy\") pod \"e2e-ipp-a-17664f-openshift-default-69464f8bf4-bvqbl\" (UID: \"f7c5b79b-f87f-4fc7-9734-e52a01fb7023\") " pod="openshift-ingress/e2e-ipp-a-17664f-openshift-default-69464f8bf4-bvqbl" Sep 11 00:01:53.808558 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:53.808539 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7c5b79b-f87f-4fc7-9734-e52a01fb7023-istio-podinfo\") pod \"e2e-ipp-a-17664f-openshift-default-69464f8bf4-bvqbl\" (UID: \"f7c5b79b-f87f-4fc7-9734-e52a01fb7023\") " pod="openshift-ingress/e2e-ipp-a-17664f-openshift-default-69464f8bf4-bvqbl" Sep 11 00:01:53.813718 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:53.813695 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7c5b79b-f87f-4fc7-9734-e52a01fb7023-istio-token\") pod \"e2e-ipp-a-17664f-openshift-default-69464f8bf4-bvqbl\" (UID: \"f7c5b79b-f87f-4fc7-9734-e52a01fb7023\") " pod="openshift-ingress/e2e-ipp-a-17664f-openshift-default-69464f8bf4-bvqbl" Sep 11 00:01:53.814404 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:53.814380 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxn7v\" (UniqueName: \"kubernetes.io/projected/f7c5b79b-f87f-4fc7-9734-e52a01fb7023-kube-api-access-cxn7v\") pod \"e2e-ipp-a-17664f-openshift-default-69464f8bf4-bvqbl\" (UID: \"f7c5b79b-f87f-4fc7-9734-e52a01fb7023\") " pod="openshift-ingress/e2e-ipp-a-17664f-openshift-default-69464f8bf4-bvqbl" Sep 11 00:01:53.871204 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:53.871173 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-17664f-openshift-default-69464f8bf4-bvqbl" Sep 11 00:01:54.150660 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:54.150624 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-17664f-openshift-default-69464f8bf4-bvqbl"] Sep 11 00:01:54.152177 ip-10-0-138-128 kubenswrapper[2574]: W0911 00:01:54.152149 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7c5b79b_f87f_4fc7_9734_e52a01fb7023.slice/crio-141c8a0d9f26f8ca712d6b7c0952745cefb7577da6ad64c4374b5f4a4e4b8c4e WatchSource:0}: Error finding container 141c8a0d9f26f8ca712d6b7c0952745cefb7577da6ad64c4374b5f4a4e4b8c4e: Status 404 returned error can't find the container with id 141c8a0d9f26f8ca712d6b7c0952745cefb7577da6ad64c4374b5f4a4e4b8c4e Sep 11 00:01:54.154156 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:54.154119 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 00:01:54.154263 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:54.154196 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 00:01:54.154263 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:54.154238 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 00:01:54.409911 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:54.409819 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-17664f-openshift-default-69464f8bf4-bvqbl" event={"ID":"f7c5b79b-f87f-4fc7-9734-e52a01fb7023","Type":"ContainerStarted","Data":"eb74dc5fea3def5f4d9795cd7cd3b72a7111dcb914c33148f005147e624dc50f"} Sep 11 00:01:54.409911 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:54.409883 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-17664f-openshift-default-69464f8bf4-bvqbl" event={"ID":"f7c5b79b-f87f-4fc7-9734-e52a01fb7023","Type":"ContainerStarted","Data":"141c8a0d9f26f8ca712d6b7c0952745cefb7577da6ad64c4374b5f4a4e4b8c4e"} Sep 11 00:01:54.431752 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:54.431706 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-17664f-openshift-default-69464f8bf4-bvqbl" podStartSLOduration=1.431690643 podStartE2EDuration="1.431690643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 00:01:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:01:54.430806823 +0000 UTC m=+1505.818409610" watchObservedRunningTime="2026-09-11 00:01:54.431690643 +0000 UTC m=+1505.819293402" Sep 11 00:01:54.871947 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:54.871909 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-17664f-openshift-default-69464f8bf4-bvqbl" Sep 11 00:01:55.876539 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:55.876511 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-17664f-openshift-default-69464f8bf4-bvqbl" Sep 11 00:01:55.883923 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:55.883891 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-56r7l"] Sep 11 00:01:56.422261 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:56.422227 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-17664f-openshift-default-69464f8bf4-bvqbl" Sep 11 00:01:56.423345 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:56.423325 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-17664f-openshift-default-69464f8bf4-bvqbl" Sep 11 00:01:57.441118 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:57.441085 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-17664f-5655b4f67d-9kfxn"] Sep 11 00:01:57.441623 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:57.441569 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c52960ab-0f37-4e17-a6e0-bbc346e0aa38" containerName="istio-proxy" Sep 11 00:01:57.441623 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:57.441585 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="c52960ab-0f37-4e17-a6e0-bbc346e0aa38" containerName="istio-proxy" Sep 11 00:01:57.441793 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:57.441679 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e485538-3e4e-40da-bd0a-ff61aa91455d" containerName="istio-proxy" Sep 11 00:01:57.445079 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:57.445059 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-17664f-5655b4f67d-9kfxn" Sep 11 00:01:57.447276 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:57.447258 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-17664f-serving-cert\"" Sep 11 00:01:57.456662 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:57.456618 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-17664f-5655b4f67d-9kfxn"] Sep 11 00:01:57.543787 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:57.543757 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75191a66-c747-4414-8840-c3c7748fe55a-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-17664f-5655b4f67d-9kfxn\" (UID: \"75191a66-c747-4414-8840-c3c7748fe55a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-17664f-5655b4f67d-9kfxn" Sep 11 00:01:57.543969 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:57.543862 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75191a66-c747-4414-8840-c3c7748fe55a-metrics-certs\") pod \"maas-api-e2e-ipp-a-17664f-5655b4f67d-9kfxn\" (UID: \"75191a66-c747-4414-8840-c3c7748fe55a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-17664f-5655b4f67d-9kfxn" Sep 11 00:01:57.543969 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:57.543907 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r47w9\" (UniqueName: \"kubernetes.io/projected/75191a66-c747-4414-8840-c3c7748fe55a-kube-api-access-r47w9\") pod \"maas-api-e2e-ipp-a-17664f-5655b4f67d-9kfxn\" (UID: \"75191a66-c747-4414-8840-c3c7748fe55a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-17664f-5655b4f67d-9kfxn" Sep 11 00:01:57.543969 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:57.543938 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75191a66-c747-4414-8840-c3c7748fe55a-maas-api-tls\") pod \"maas-api-e2e-ipp-a-17664f-5655b4f67d-9kfxn\" (UID: \"75191a66-c747-4414-8840-c3c7748fe55a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-17664f-5655b4f67d-9kfxn" Sep 11 00:01:57.644925 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:57.644885 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75191a66-c747-4414-8840-c3c7748fe55a-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-17664f-5655b4f67d-9kfxn\" (UID: \"75191a66-c747-4414-8840-c3c7748fe55a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-17664f-5655b4f67d-9kfxn" Sep 11 00:01:57.645102 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:57.644987 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75191a66-c747-4414-8840-c3c7748fe55a-metrics-certs\") pod \"maas-api-e2e-ipp-a-17664f-5655b4f67d-9kfxn\" (UID: \"75191a66-c747-4414-8840-c3c7748fe55a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-17664f-5655b4f67d-9kfxn" Sep 11 00:01:57.645102 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:57.645015 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r47w9\" (UniqueName: \"kubernetes.io/projected/75191a66-c747-4414-8840-c3c7748fe55a-kube-api-access-r47w9\") pod \"maas-api-e2e-ipp-a-17664f-5655b4f67d-9kfxn\" (UID: \"75191a66-c747-4414-8840-c3c7748fe55a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-17664f-5655b4f67d-9kfxn" Sep 11 00:01:57.645102 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:57.645036 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75191a66-c747-4414-8840-c3c7748fe55a-maas-api-tls\") pod \"maas-api-e2e-ipp-a-17664f-5655b4f67d-9kfxn\" (UID: \"75191a66-c747-4414-8840-c3c7748fe55a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-17664f-5655b4f67d-9kfxn" Sep 11 00:01:57.645635 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:57.645606 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75191a66-c747-4414-8840-c3c7748fe55a-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-17664f-5655b4f67d-9kfxn\" (UID: \"75191a66-c747-4414-8840-c3c7748fe55a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-17664f-5655b4f67d-9kfxn" Sep 11 00:01:57.647686 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:57.647664 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75191a66-c747-4414-8840-c3c7748fe55a-metrics-certs\") pod \"maas-api-e2e-ipp-a-17664f-5655b4f67d-9kfxn\" (UID: \"75191a66-c747-4414-8840-c3c7748fe55a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-17664f-5655b4f67d-9kfxn" Sep 11 00:01:57.647686 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:57.647676 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75191a66-c747-4414-8840-c3c7748fe55a-maas-api-tls\") pod \"maas-api-e2e-ipp-a-17664f-5655b4f67d-9kfxn\" (UID: \"75191a66-c747-4414-8840-c3c7748fe55a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-17664f-5655b4f67d-9kfxn" Sep 11 00:01:57.653031 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:57.653008 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r47w9\" (UniqueName: \"kubernetes.io/projected/75191a66-c747-4414-8840-c3c7748fe55a-kube-api-access-r47w9\") pod \"maas-api-e2e-ipp-a-17664f-5655b4f67d-9kfxn\" (UID: \"75191a66-c747-4414-8840-c3c7748fe55a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-17664f-5655b4f67d-9kfxn" Sep 11 00:01:57.758127 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:57.758098 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-17664f-5655b4f67d-9kfxn" Sep 11 00:01:58.037840 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:58.037813 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-17664f-5655b4f67d-9kfxn"] Sep 11 00:01:58.039177 ip-10-0-138-128 kubenswrapper[2574]: W0911 00:01:58.039148 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75191a66_c747_4414_8840_c3c7748fe55a.slice/crio-8ad7d53f1817651a8504c014dbf04ec4cc30ab89295e05c899f4a0407546a86d WatchSource:0}: Error finding container 8ad7d53f1817651a8504c014dbf04ec4cc30ab89295e05c899f4a0407546a86d: Status 404 returned error can't find the container with id 8ad7d53f1817651a8504c014dbf04ec4cc30ab89295e05c899f4a0407546a86d Sep 11 00:01:58.040586 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:58.040567 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 00:01:58.437272 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:58.437182 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-17664f-5655b4f67d-9kfxn" event={"ID":"75191a66-c747-4414-8840-c3c7748fe55a","Type":"ContainerStarted","Data":"8ad7d53f1817651a8504c014dbf04ec4cc30ab89295e05c899f4a0407546a86d"} Sep 11 00:01:59.444976 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:59.444936 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-17664f-5655b4f67d-9kfxn" event={"ID":"75191a66-c747-4414-8840-c3c7748fe55a","Type":"ContainerStarted","Data":"e32cdaa8738a103b177c58bb48ab7a8a95b3fac6b79579a9712e6fbe544a956e"} Sep 11 00:01:59.445374 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:59.445063 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-17664f-5655b4f67d-9kfxn" Sep 11 00:01:59.461389 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:01:59.461339 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-17664f-5655b4f67d-9kfxn" podStartSLOduration=2.012116596 podStartE2EDuration="2.461324054s" totalImagesPullingTime="449.207458ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 00:01:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:01:59.458886547 +0000 UTC m=+1510.846489306" watchObservedRunningTime="2026-09-11 00:01:59.461324054 +0000 UTC m=+1510.848926812" Sep 11 00:02:05.456538 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:02:05.456507 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-17664f-5655b4f67d-9kfxn" Sep 11 00:02:05.983035 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:02:05.982975 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-56r7l"] Sep 11 00:02:11.767221 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:02:11.767185 2574 patch_prober.go:28] interesting pod/e2e-ipp-a-17664f-openshift-default-69464f8bf4-bvqbl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": read tcp 10.134.0.2:59168->10.134.0.75:15021: read: connection reset by peer" start-of-body= Sep 11 00:02:11.767620 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:02:11.767254 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-17664f-openshift-default-69464f8bf4-bvqbl" podUID="f7c5b79b-f87f-4fc7-9734-e52a01fb7023" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": read tcp 10.134.0.2:59168->10.134.0.75:15021: read: connection reset by peer" Sep 11 00:02:12.542093 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:02:12.542062 2574 generic.go:356] "Generic (PLEG): container finished" podID="f7c5b79b-f87f-4fc7-9734-e52a01fb7023" containerID="eb74dc5fea3def5f4d9795cd7cd3b72a7111dcb914c33148f005147e624dc50f" exitCode=0 Sep 11 00:02:12.542305 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:02:12.542138 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-17664f-openshift-default-69464f8bf4-bvqbl" event={"ID":"f7c5b79b-f87f-4fc7-9734-e52a01fb7023","Type":"ContainerDied","Data":"eb74dc5fea3def5f4d9795cd7cd3b72a7111dcb914c33148f005147e624dc50f"} Sep 11 00:02:12.542602 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:02:12.542586 2574 scope.go:117] "RemoveContainer" containerID="eb74dc5fea3def5f4d9795cd7cd3b72a7111dcb914c33148f005147e624dc50f" Sep 11 00:02:12.543681 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:02:12.543622 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 00:02:12.543765 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:02:12.543721 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 00:02:12.543765 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:02:12.543754 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 00:02:13.553912 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:02:13.553875 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-17664f-openshift-default-69464f8bf4-bvqbl" event={"ID":"f7c5b79b-f87f-4fc7-9734-e52a01fb7023","Type":"ContainerStarted","Data":"060cd1f4aa197cb2439da42b3c21a6e6bb9af65559e780107dbb2c926ea37ed8"} Sep 11 00:02:13.871712 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:02:13.871601 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-17664f-openshift-default-69464f8bf4-bvqbl" Sep 11 00:02:13.872009 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:02:13.871654 2574 patch_prober.go:28] interesting pod/e2e-ipp-a-17664f-openshift-default-69464f8bf4-bvqbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Sep 11 00:02:13.872235 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:02:13.872181 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-17664f-openshift-default-69464f8bf4-bvqbl" podUID="f7c5b79b-f87f-4fc7-9734-e52a01fb7023" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Sep 11 00:02:14.872232 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:02:14.872188 2574 patch_prober.go:28] interesting pod/e2e-ipp-a-17664f-openshift-default-69464f8bf4-bvqbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Sep 11 00:02:14.872762 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:02:14.872261 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-17664f-openshift-default-69464f8bf4-bvqbl" podUID="f7c5b79b-f87f-4fc7-9734-e52a01fb7023" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Sep 11 00:02:15.872568 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:02:15.872525 2574 patch_prober.go:28] interesting pod/e2e-ipp-a-17664f-openshift-default-69464f8bf4-bvqbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Sep 11 00:02:15.873205 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:02:15.873169 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-17664f-openshift-default-69464f8bf4-bvqbl" podUID="f7c5b79b-f87f-4fc7-9734-e52a01fb7023" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Sep 11 00:02:16.872027 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:02:16.871991 2574 patch_prober.go:28] interesting pod/e2e-ipp-a-17664f-openshift-default-69464f8bf4-bvqbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Sep 11 00:02:16.872329 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:02:16.872299 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-17664f-openshift-default-69464f8bf4-bvqbl" podUID="f7c5b79b-f87f-4fc7-9734-e52a01fb7023" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Sep 11 00:02:17.871818 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:02:17.871778 2574 patch_prober.go:28] interesting pod/e2e-ipp-a-17664f-openshift-default-69464f8bf4-bvqbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Sep 11 00:02:17.872817 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:02:17.872390 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-17664f-openshift-default-69464f8bf4-bvqbl" podUID="f7c5b79b-f87f-4fc7-9734-e52a01fb7023" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Sep 11 00:02:18.875584 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:02:18.875550 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-17664f-openshift-default-69464f8bf4-bvqbl" Sep 11 00:02:18.876006 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:02:18.875910 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-17664f-openshift-default-69464f8bf4-bvqbl" Sep 11 00:02:18.876653 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:02:18.876621 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-17664f-openshift-default-69464f8bf4-bvqbl" Sep 11 00:02:22.247173 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:02:22.247139 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-43f569-openshift-default-5c7f95d694-ccm7g"] Sep 11 00:02:22.249878 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:02:22.249857 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-43f569-openshift-default-5c7f95d694-ccm7g" Sep 11 00:02:22.252014 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:02:22.251992 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-43f569-openshift-default-dockercfg-ffzkb\"" Sep 11 00:02:22.262745 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:02:22.262723 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-43f569-openshift-default-5c7f95d694-ccm7g"] Sep 11 00:02:22.393565 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:02:22.393532 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d738baa4-79b0-4d1d-80d2-7efbd3d91ade-istio-podinfo\") pod \"e2e-ipp-b-43f569-openshift-default-5c7f95d694-ccm7g\" (UID: \"d738baa4-79b0-4d1d-80d2-7efbd3d91ade\") " pod="openshift-ingress/e2e-ipp-b-43f569-openshift-default-5c7f95d694-ccm7g" Sep 11 00:02:22.393749 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:02:22.393633 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d738baa4-79b0-4d1d-80d2-7efbd3d91ade-istio-token\") pod \"e2e-ipp-b-43f569-openshift-default-5c7f95d694-ccm7g\" (UID: \"d738baa4-79b0-4d1d-80d2-7efbd3d91ade\") " pod="openshift-ingress/e2e-ipp-b-43f569-openshift-default-5c7f95d694-ccm7g" Sep 11 00:02:22.393749 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:02:22.393699 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d738baa4-79b0-4d1d-80d2-7efbd3d91ade-istiod-ca-cert\") pod \"e2e-ipp-b-43f569-openshift-default-5c7f95d694-ccm7g\" (UID: \"d738baa4-79b0-4d1d-80d2-7efbd3d91ade\") " pod="openshift-ingress/e2e-ipp-b-43f569-openshift-default-5c7f95d694-ccm7g" Sep 11 00:02:22.393749 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:02:22.393717 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d738baa4-79b0-4d1d-80d2-7efbd3d91ade-workload-socket\") pod \"e2e-ipp-b-43f569-openshift-default-5c7f95d694-ccm7g\" (UID: \"d738baa4-79b0-4d1d-80d2-7efbd3d91ade\") " pod="openshift-ingress/e2e-ipp-b-43f569-openshift-default-5c7f95d694-ccm7g" Sep 11 00:02:22.393861 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:02:22.393758 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d738baa4-79b0-4d1d-80d2-7efbd3d91ade-credential-socket\") pod \"e2e-ipp-b-43f569-openshift-default-5c7f95d694-ccm7g\" (UID: \"d738baa4-79b0-4d1d-80d2-7efbd3d91ade\") " pod="openshift-ingress/e2e-ipp-b-43f569-openshift-default-5c7f95d694-ccm7g" Sep 11 00:02:22.393861 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:02:22.393780 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d738baa4-79b0-4d1d-80d2-7efbd3d91ade-workload-certs\") pod \"e2e-ipp-b-43f569-openshift-default-5c7f95d694-ccm7g\" (UID: \"d738baa4-79b0-4d1d-80d2-7efbd3d91ade\") " pod="openshift-ingress/e2e-ipp-b-43f569-openshift-default-5c7f95d694-ccm7g" Sep 11 00:02:22.393861 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:02:22.393803 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gsct\" (UniqueName: \"kubernetes.io/projected/d738baa4-79b0-4d1d-80d2-7efbd3d91ade-kube-api-access-5gsct\") pod \"e2e-ipp-b-43f569-openshift-default-5c7f95d694-ccm7g\" (UID: \"d738baa4-79b0-4d1d-80d2-7efbd3d91ade\") " pod="openshift-ingress/e2e-ipp-b-43f569-openshift-default-5c7f95d694-ccm7g" Sep 11 00:02:22.393861 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:02:22.393829 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d738baa4-79b0-4d1d-80d2-7efbd3d91ade-istio-data\") pod \"e2e-ipp-b-43f569-openshift-default-5c7f95d694-ccm7g\" (UID: \"d738baa4-79b0-4d1d-80d2-7efbd3d91ade\") " pod="openshift-ingress/e2e-ipp-b-43f569-openshift-default-5c7f95d694-ccm7g" Sep 11 00:02:22.394003 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:02:22.393895 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d738baa4-79b0-4d1d-80d2-7efbd3d91ade-istio-envoy\") pod \"e2e-ipp-b-43f569-openshift-default-5c7f95d694-ccm7g\" (UID: \"d738baa4-79b0-4d1d-80d2-7efbd3d91ade\") " pod="openshift-ingress/e2e-ipp-b-43f569-openshift-default-5c7f95d694-ccm7g" Sep 11 00:02:22.494797 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:02:22.494766 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d738baa4-79b0-4d1d-80d2-7efbd3d91ade-istio-envoy\") pod \"e2e-ipp-b-43f569-openshift-default-5c7f95d694-ccm7g\" (UID: \"d738baa4-79b0-4d1d-80d2-7efbd3d91ade\") " pod="openshift-ingress/e2e-ipp-b-43f569-openshift-default-5c7f95d694-ccm7g" Sep 11 00:02:22.494936 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:02:22.494811 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d738baa4-79b0-4d1d-80d2-7efbd3d91ade-istio-podinfo\") pod \"e2e-ipp-b-43f569-openshift-default-5c7f95d694-ccm7g\" (UID: \"d738baa4-79b0-4d1d-80d2-7efbd3d91ade\") " pod="openshift-ingress/e2e-ipp-b-43f569-openshift-default-5c7f95d694-ccm7g" Sep 11 00:02:22.494936 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:02:22.494856 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d738baa4-79b0-4d1d-80d2-7efbd3d91ade-istio-token\") pod \"e2e-ipp-b-43f569-openshift-default-5c7f95d694-ccm7g\" (UID: \"d738baa4-79b0-4d1d-80d2-7efbd3d91ade\") " pod="openshift-ingress/e2e-ipp-b-43f569-openshift-default-5c7f95d694-ccm7g" Sep 11 00:02:22.494936 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:02:22.494882 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d738baa4-79b0-4d1d-80d2-7efbd3d91ade-istiod-ca-cert\") pod \"e2e-ipp-b-43f569-openshift-default-5c7f95d694-ccm7g\" (UID: \"d738baa4-79b0-4d1d-80d2-7efbd3d91ade\") " pod="openshift-ingress/e2e-ipp-b-43f569-openshift-default-5c7f95d694-ccm7g" Sep 11 00:02:22.494936 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:02:22.494898 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d738baa4-79b0-4d1d-80d2-7efbd3d91ade-workload-socket\") pod \"e2e-ipp-b-43f569-openshift-default-5c7f95d694-ccm7g\" (UID: \"d738baa4-79b0-4d1d-80d2-7efbd3d91ade\") " pod="openshift-ingress/e2e-ipp-b-43f569-openshift-default-5c7f95d694-ccm7g" Sep 11 00:02:22.495135 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:02:22.494937 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d738baa4-79b0-4d1d-80d2-7efbd3d91ade-credential-socket\") pod \"e2e-ipp-b-43f569-openshift-default-5c7f95d694-ccm7g\" (UID: \"d738baa4-79b0-4d1d-80d2-7efbd3d91ade\") " pod="openshift-ingress/e2e-ipp-b-43f569-openshift-default-5c7f95d694-ccm7g" Sep 11 00:02:22.495135 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:02:22.494960 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d738baa4-79b0-4d1d-80d2-7efbd3d91ade-workload-certs\") pod \"e2e-ipp-b-43f569-openshift-default-5c7f95d694-ccm7g\" (UID: \"d738baa4-79b0-4d1d-80d2-7efbd3d91ade\") " pod="openshift-ingress/e2e-ipp-b-43f569-openshift-default-5c7f95d694-ccm7g" Sep 11 00:02:22.495313 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:02:22.495089 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gsct\" (UniqueName: \"kubernetes.io/projected/d738baa4-79b0-4d1d-80d2-7efbd3d91ade-kube-api-access-5gsct\") pod \"e2e-ipp-b-43f569-openshift-default-5c7f95d694-ccm7g\" (UID: \"d738baa4-79b0-4d1d-80d2-7efbd3d91ade\") " pod="openshift-ingress/e2e-ipp-b-43f569-openshift-default-5c7f95d694-ccm7g" Sep 11 00:02:22.495449 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:02:22.495309 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d738baa4-79b0-4d1d-80d2-7efbd3d91ade-workload-certs\") pod \"e2e-ipp-b-43f569-openshift-default-5c7f95d694-ccm7g\" (UID: \"d738baa4-79b0-4d1d-80d2-7efbd3d91ade\") " pod="openshift-ingress/e2e-ipp-b-43f569-openshift-default-5c7f95d694-ccm7g" Sep 11 00:02:22.495449 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:02:22.495326 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d738baa4-79b0-4d1d-80d2-7efbd3d91ade-workload-socket\") pod \"e2e-ipp-b-43f569-openshift-default-5c7f95d694-ccm7g\" (UID: \"d738baa4-79b0-4d1d-80d2-7efbd3d91ade\") " pod="openshift-ingress/e2e-ipp-b-43f569-openshift-default-5c7f95d694-ccm7g" Sep 11 00:02:22.495449 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:02:22.495333 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d738baa4-79b0-4d1d-80d2-7efbd3d91ade-istio-data\") pod \"e2e-ipp-b-43f569-openshift-default-5c7f95d694-ccm7g\" (UID: \"d738baa4-79b0-4d1d-80d2-7efbd3d91ade\") " pod="openshift-ingress/e2e-ipp-b-43f569-openshift-default-5c7f95d694-ccm7g" Sep 11 00:02:22.495449 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:02:22.495415 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d738baa4-79b0-4d1d-80d2-7efbd3d91ade-credential-socket\") pod \"e2e-ipp-b-43f569-openshift-default-5c7f95d694-ccm7g\" (UID: \"d738baa4-79b0-4d1d-80d2-7efbd3d91ade\") " pod="openshift-ingress/e2e-ipp-b-43f569-openshift-default-5c7f95d694-ccm7g" Sep 11 00:02:22.495625 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:02:22.495545 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d738baa4-79b0-4d1d-80d2-7efbd3d91ade-istio-data\") pod \"e2e-ipp-b-43f569-openshift-default-5c7f95d694-ccm7g\" (UID: \"d738baa4-79b0-4d1d-80d2-7efbd3d91ade\") " pod="openshift-ingress/e2e-ipp-b-43f569-openshift-default-5c7f95d694-ccm7g" Sep 11 00:02:22.495768 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:02:22.495746 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d738baa4-79b0-4d1d-80d2-7efbd3d91ade-istiod-ca-cert\") pod \"e2e-ipp-b-43f569-openshift-default-5c7f95d694-ccm7g\" (UID: \"d738baa4-79b0-4d1d-80d2-7efbd3d91ade\") " pod="openshift-ingress/e2e-ipp-b-43f569-openshift-default-5c7f95d694-ccm7g" Sep 11 00:02:22.497282 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:02:22.497232 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d738baa4-79b0-4d1d-80d2-7efbd3d91ade-istio-envoy\") pod \"e2e-ipp-b-43f569-openshift-default-5c7f95d694-ccm7g\" (UID: \"d738baa4-79b0-4d1d-80d2-7efbd3d91ade\") " pod="openshift-ingress/e2e-ipp-b-43f569-openshift-default-5c7f95d694-ccm7g" Sep 11 00:02:22.497395 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:02:22.497378 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d738baa4-79b0-4d1d-80d2-7efbd3d91ade-istio-podinfo\") pod \"e2e-ipp-b-43f569-openshift-default-5c7f95d694-ccm7g\" (UID: \"d738baa4-79b0-4d1d-80d2-7efbd3d91ade\") " pod="openshift-ingress/e2e-ipp-b-43f569-openshift-default-5c7f95d694-ccm7g" Sep 11 00:02:22.502656 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:02:22.502619 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gsct\" (UniqueName: \"kubernetes.io/projected/d738baa4-79b0-4d1d-80d2-7efbd3d91ade-kube-api-access-5gsct\") pod \"e2e-ipp-b-43f569-openshift-default-5c7f95d694-ccm7g\" (UID: \"d738baa4-79b0-4d1d-80d2-7efbd3d91ade\") " pod="openshift-ingress/e2e-ipp-b-43f569-openshift-default-5c7f95d694-ccm7g" Sep 11 00:02:22.503039 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:02:22.503017 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d738baa4-79b0-4d1d-80d2-7efbd3d91ade-istio-token\") pod \"e2e-ipp-b-43f569-openshift-default-5c7f95d694-ccm7g\" (UID: \"d738baa4-79b0-4d1d-80d2-7efbd3d91ade\") " pod="openshift-ingress/e2e-ipp-b-43f569-openshift-default-5c7f95d694-ccm7g" Sep 11 00:02:22.562121 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:02:22.562096 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-43f569-openshift-default-5c7f95d694-ccm7g" Sep 11 00:02:22.842729 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:02:22.842697 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-43f569-openshift-default-5c7f95d694-ccm7g"] Sep 11 00:02:22.843374 ip-10-0-138-128 kubenswrapper[2574]: W0911 00:02:22.843346 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd738baa4_79b0_4d1d_80d2_7efbd3d91ade.slice/crio-76995e235ba3c8a284ff1cbcefa4565da096734c62a383d310e431572f092789 WatchSource:0}: Error finding container 76995e235ba3c8a284ff1cbcefa4565da096734c62a383d310e431572f092789: Status 404 returned error can't find the container with id 76995e235ba3c8a284ff1cbcefa4565da096734c62a383d310e431572f092789 Sep 11 00:02:22.845619 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:02:22.845585 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 00:02:22.845715 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:02:22.845676 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 00:02:22.845715 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:02:22.845707 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 00:02:23.624866 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:02:23.624835 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-43f569-openshift-default-5c7f95d694-ccm7g" event={"ID":"d738baa4-79b0-4d1d-80d2-7efbd3d91ade","Type":"ContainerStarted","Data":"43f25a2779835058242892c1424ca23b40403b94eeb8063ffc32e32b7d258f7a"} Sep 11 00:02:23.625263 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:02:23.624889 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-43f569-openshift-default-5c7f95d694-ccm7g" event={"ID":"d738baa4-79b0-4d1d-80d2-7efbd3d91ade","Type":"ContainerStarted","Data":"76995e235ba3c8a284ff1cbcefa4565da096734c62a383d310e431572f092789"} Sep 11 00:02:23.645593 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:02:23.645548 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-43f569-openshift-default-5c7f95d694-ccm7g" podStartSLOduration=1.645531535 podStartE2EDuration="1.645531535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 00:02:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:02:23.643452582 +0000 UTC m=+1535.031055352" watchObservedRunningTime="2026-09-11 00:02:23.645531535 +0000 UTC m=+1535.033134293" Sep 11 00:02:24.563177 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:02:24.563143 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-43f569-openshift-default-5c7f95d694-ccm7g" Sep 11 00:02:24.568423 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:02:24.568389 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-43f569-openshift-default-5c7f95d694-ccm7g" Sep 11 00:02:24.631413 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:02:24.631390 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-43f569-openshift-default-5c7f95d694-ccm7g" Sep 11 00:02:24.632144 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:02:24.632125 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-43f569-openshift-default-5c7f95d694-ccm7g" Sep 11 00:02:32.378657 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:02:32.378603 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-56r7l"] Sep 11 00:02:40.673814 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:02:40.673773 2574 patch_prober.go:28] interesting pod/e2e-ipp-b-43f569-openshift-default-5c7f95d694-ccm7g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 11 00:02:40.674256 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:02:40.673859 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-43f569-openshift-default-5c7f95d694-ccm7g" podUID="d738baa4-79b0-4d1d-80d2-7efbd3d91ade" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 11 00:02:42.750803 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:02:42.750764 2574 generic.go:356] "Generic (PLEG): container finished" podID="d738baa4-79b0-4d1d-80d2-7efbd3d91ade" containerID="43f25a2779835058242892c1424ca23b40403b94eeb8063ffc32e32b7d258f7a" exitCode=0 Sep 11 00:02:42.751238 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:02:42.750838 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-43f569-openshift-default-5c7f95d694-ccm7g" event={"ID":"d738baa4-79b0-4d1d-80d2-7efbd3d91ade","Type":"ContainerDied","Data":"43f25a2779835058242892c1424ca23b40403b94eeb8063ffc32e32b7d258f7a"} Sep 11 00:02:42.751281 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:02:42.751252 2574 scope.go:117] "RemoveContainer" containerID="43f25a2779835058242892c1424ca23b40403b94eeb8063ffc32e32b7d258f7a" Sep 11 00:02:42.752435 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:02:42.752396 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 00:02:42.752496 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:02:42.752482 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 00:02:42.752533 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:02:42.752515 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 00:02:43.759709 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:02:43.759670 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-43f569-openshift-default-5c7f95d694-ccm7g" event={"ID":"d738baa4-79b0-4d1d-80d2-7efbd3d91ade","Type":"ContainerStarted","Data":"1a686afd4b48cf7efbc6cb2a6fbddf64a81b24ed4b05c347b7f967cc71dbcc49"} Sep 11 00:02:44.562723 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:02:44.562677 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-43f569-openshift-default-5c7f95d694-ccm7g" Sep 11 00:02:44.562954 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:02:44.562925 2574 patch_prober.go:28] interesting pod/e2e-ipp-b-43f569-openshift-default-5c7f95d694-ccm7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Sep 11 00:02:44.563038 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:02:44.562993 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-43f569-openshift-default-5c7f95d694-ccm7g" podUID="d738baa4-79b0-4d1d-80d2-7efbd3d91ade" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Sep 11 00:02:45.562801 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:02:45.562755 2574 patch_prober.go:28] interesting pod/e2e-ipp-b-43f569-openshift-default-5c7f95d694-ccm7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Sep 11 00:02:45.563217 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:02:45.562824 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-43f569-openshift-default-5c7f95d694-ccm7g" podUID="d738baa4-79b0-4d1d-80d2-7efbd3d91ade" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Sep 11 00:02:46.562501 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:02:46.562461 2574 patch_prober.go:28] interesting pod/e2e-ipp-b-43f569-openshift-default-5c7f95d694-ccm7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Sep 11 00:02:46.562707 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:02:46.562525 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-43f569-openshift-default-5c7f95d694-ccm7g" podUID="d738baa4-79b0-4d1d-80d2-7efbd3d91ade" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Sep 11 00:02:47.562945 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:02:47.562905 2574 patch_prober.go:28] interesting pod/e2e-ipp-b-43f569-openshift-default-5c7f95d694-ccm7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Sep 11 00:02:47.563347 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:02:47.562972 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-43f569-openshift-default-5c7f95d694-ccm7g" podUID="d738baa4-79b0-4d1d-80d2-7efbd3d91ade" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Sep 11 00:02:48.562917 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:02:48.562874 2574 patch_prober.go:28] interesting pod/e2e-ipp-b-43f569-openshift-default-5c7f95d694-ccm7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Sep 11 00:02:48.563093 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:02:48.562944 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-43f569-openshift-default-5c7f95d694-ccm7g" podUID="d738baa4-79b0-4d1d-80d2-7efbd3d91ade" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Sep 11 00:02:49.566440 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:02:49.566410 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-43f569-openshift-default-5c7f95d694-ccm7g" Sep 11 00:02:49.566942 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:02:49.566847 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-43f569-openshift-default-5c7f95d694-ccm7g" Sep 11 00:02:49.567721 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:02:49.567697 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-43f569-openshift-default-5c7f95d694-ccm7g" Sep 11 00:03:07.677520 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:03:07.677481 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-56r7l"] Sep 11 00:03:29.978009 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:03:29.977963 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-56r7l"] Sep 11 00:03:41.420621 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:03:41.420583 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ab675d-7c657c88d-c468w"] Sep 11 00:03:41.424777 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:03:41.424750 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ab675d-7c657c88d-c468w" Sep 11 00:03:41.426848 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:03:41.426826 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-ab675d-serving-cert\"" Sep 11 00:03:41.431689 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:03:41.431662 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ab675d-7c657c88d-c468w"] Sep 11 00:03:41.511516 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:03:41.511440 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5311cf89-76b5-4321-a46c-39fafaf1b802-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-ab675d-7c657c88d-c468w\" (UID: \"5311cf89-76b5-4321-a46c-39fafaf1b802\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ab675d-7c657c88d-c468w" Sep 11 00:03:41.511516 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:03:41.511512 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5311cf89-76b5-4321-a46c-39fafaf1b802-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-ab675d-7c657c88d-c468w\" (UID: \"5311cf89-76b5-4321-a46c-39fafaf1b802\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ab675d-7c657c88d-c468w" Sep 11 00:03:41.511763 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:03:41.511531 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qx4f\" (UniqueName: \"kubernetes.io/projected/5311cf89-76b5-4321-a46c-39fafaf1b802-kube-api-access-8qx4f\") pod \"maas-api-e2e-ipp-cleanup-ab675d-7c657c88d-c468w\" (UID: \"5311cf89-76b5-4321-a46c-39fafaf1b802\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ab675d-7c657c88d-c468w" Sep 11 00:03:41.511763 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:03:41.511566 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5311cf89-76b5-4321-a46c-39fafaf1b802-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-ab675d-7c657c88d-c468w\" (UID: \"5311cf89-76b5-4321-a46c-39fafaf1b802\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ab675d-7c657c88d-c468w" Sep 11 00:03:41.612897 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:03:41.612860 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5311cf89-76b5-4321-a46c-39fafaf1b802-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-ab675d-7c657c88d-c468w\" (UID: \"5311cf89-76b5-4321-a46c-39fafaf1b802\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ab675d-7c657c88d-c468w" Sep 11 00:03:41.612897 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:03:41.612897 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qx4f\" (UniqueName: \"kubernetes.io/projected/5311cf89-76b5-4321-a46c-39fafaf1b802-kube-api-access-8qx4f\") pod \"maas-api-e2e-ipp-cleanup-ab675d-7c657c88d-c468w\" (UID: \"5311cf89-76b5-4321-a46c-39fafaf1b802\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ab675d-7c657c88d-c468w" Sep 11 00:03:41.613157 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:03:41.612948 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5311cf89-76b5-4321-a46c-39fafaf1b802-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-ab675d-7c657c88d-c468w\" (UID: \"5311cf89-76b5-4321-a46c-39fafaf1b802\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ab675d-7c657c88d-c468w" Sep 11 00:03:41.613157 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:03:41.613028 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5311cf89-76b5-4321-a46c-39fafaf1b802-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-ab675d-7c657c88d-c468w\" (UID: \"5311cf89-76b5-4321-a46c-39fafaf1b802\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ab675d-7c657c88d-c468w" Sep 11 00:03:41.613585 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:03:41.613560 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5311cf89-76b5-4321-a46c-39fafaf1b802-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-ab675d-7c657c88d-c468w\" (UID: \"5311cf89-76b5-4321-a46c-39fafaf1b802\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ab675d-7c657c88d-c468w" Sep 11 00:03:41.615557 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:03:41.615528 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5311cf89-76b5-4321-a46c-39fafaf1b802-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-ab675d-7c657c88d-c468w\" (UID: \"5311cf89-76b5-4321-a46c-39fafaf1b802\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ab675d-7c657c88d-c468w" Sep 11 00:03:41.615695 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:03:41.615604 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5311cf89-76b5-4321-a46c-39fafaf1b802-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-ab675d-7c657c88d-c468w\" (UID: \"5311cf89-76b5-4321-a46c-39fafaf1b802\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ab675d-7c657c88d-c468w" Sep 11 00:03:41.620892 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:03:41.620863 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qx4f\" (UniqueName: \"kubernetes.io/projected/5311cf89-76b5-4321-a46c-39fafaf1b802-kube-api-access-8qx4f\") pod \"maas-api-e2e-ipp-cleanup-ab675d-7c657c88d-c468w\" (UID: \"5311cf89-76b5-4321-a46c-39fafaf1b802\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ab675d-7c657c88d-c468w" Sep 11 00:03:41.738232 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:03:41.738202 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ab675d-7c657c88d-c468w" Sep 11 00:03:42.019703 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:03:42.019674 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ab675d-7c657c88d-c468w"] Sep 11 00:03:42.021855 ip-10-0-138-128 kubenswrapper[2574]: W0911 00:03:42.021828 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5311cf89_76b5_4321_a46c_39fafaf1b802.slice/crio-d9bb1ac8b35f7fdda3d0720eada6858470b607d22cfa3f18841ecf5099e8601c WatchSource:0}: Error finding container d9bb1ac8b35f7fdda3d0720eada6858470b607d22cfa3f18841ecf5099e8601c: Status 404 returned error can't find the container with id d9bb1ac8b35f7fdda3d0720eada6858470b607d22cfa3f18841ecf5099e8601c Sep 11 00:03:42.148223 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:03:42.148165 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ab675d-7c657c88d-c468w" event={"ID":"5311cf89-76b5-4321-a46c-39fafaf1b802","Type":"ContainerStarted","Data":"d9bb1ac8b35f7fdda3d0720eada6858470b607d22cfa3f18841ecf5099e8601c"} Sep 11 00:03:43.156197 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:03:43.156161 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ab675d-7c657c88d-c468w" event={"ID":"5311cf89-76b5-4321-a46c-39fafaf1b802","Type":"ContainerStarted","Data":"2557da6d6456345a8ad4f74aa56b3f1ff7dccf73dc2468a8dd8e11540aacfe5b"} Sep 11 00:03:43.156582 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:03:43.156293 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ab675d-7c657c88d-c468w" Sep 11 00:03:43.176598 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:03:43.176549 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ab675d-7c657c88d-c468w" podStartSLOduration=1.5676158610000002 podStartE2EDuration="2.176533212s" totalImagesPullingTime="608.917351ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 00:03:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:03:43.173922062 +0000 UTC m=+1614.561524822" watchObservedRunningTime="2026-09-11 00:03:43.176533212 +0000 UTC m=+1614.564135971" Sep 11 00:03:48.891751 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:03:48.891719 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-56r7l"] Sep 11 00:03:49.167658 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:03:49.167568 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ab675d-7c657c88d-c468w" Sep 11 00:04:11.912934 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:04:11.912898 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ab675d-7c657c88d-c468w"] Sep 11 00:04:11.913440 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:04:11.913151 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ab675d-7c657c88d-c468w" podUID="5311cf89-76b5-4321-a46c-39fafaf1b802" containerName="maas-api" containerID="cri-o://2557da6d6456345a8ad4f74aa56b3f1ff7dccf73dc2468a8dd8e11540aacfe5b" gracePeriod=30 Sep 11 00:04:12.167374 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:04:12.167309 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ab675d-7c657c88d-c468w" Sep 11 00:04:12.323167 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:04:12.323125 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8qx4f\" (UniqueName: \"kubernetes.io/projected/5311cf89-76b5-4321-a46c-39fafaf1b802-kube-api-access-8qx4f\") pod \"5311cf89-76b5-4321-a46c-39fafaf1b802\" (UID: \"5311cf89-76b5-4321-a46c-39fafaf1b802\") " Sep 11 00:04:12.323371 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:04:12.323219 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5311cf89-76b5-4321-a46c-39fafaf1b802-metrics-certs\") pod \"5311cf89-76b5-4321-a46c-39fafaf1b802\" (UID: \"5311cf89-76b5-4321-a46c-39fafaf1b802\") " Sep 11 00:04:12.323371 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:04:12.323309 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5311cf89-76b5-4321-a46c-39fafaf1b802-openshift-service-ca\") pod \"5311cf89-76b5-4321-a46c-39fafaf1b802\" (UID: \"5311cf89-76b5-4321-a46c-39fafaf1b802\") " Sep 11 00:04:12.323371 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:04:12.323345 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5311cf89-76b5-4321-a46c-39fafaf1b802-maas-api-tls\") pod \"5311cf89-76b5-4321-a46c-39fafaf1b802\" (UID: \"5311cf89-76b5-4321-a46c-39fafaf1b802\") " Sep 11 00:04:12.324495 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:04:12.324454 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5311cf89-76b5-4321-a46c-39fafaf1b802-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5311cf89-76b5-4321-a46c-39fafaf1b802" (UID: "5311cf89-76b5-4321-a46c-39fafaf1b802"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 00:04:12.326832 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:04:12.326803 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5311cf89-76b5-4321-a46c-39fafaf1b802-kube-api-access-8qx4f" (OuterVolumeSpecName: "kube-api-access-8qx4f") pod "5311cf89-76b5-4321-a46c-39fafaf1b802" (UID: "5311cf89-76b5-4321-a46c-39fafaf1b802"). InnerVolumeSpecName "kube-api-access-8qx4f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 00:04:12.326952 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:04:12.326876 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5311cf89-76b5-4321-a46c-39fafaf1b802-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5311cf89-76b5-4321-a46c-39fafaf1b802" (UID: "5311cf89-76b5-4321-a46c-39fafaf1b802"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 00:04:12.327026 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:04:12.326995 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5311cf89-76b5-4321-a46c-39fafaf1b802-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "5311cf89-76b5-4321-a46c-39fafaf1b802" (UID: "5311cf89-76b5-4321-a46c-39fafaf1b802"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 00:04:12.335396 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:04:12.335370 2574 generic.go:356] "Generic (PLEG): container finished" podID="5311cf89-76b5-4321-a46c-39fafaf1b802" containerID="2557da6d6456345a8ad4f74aa56b3f1ff7dccf73dc2468a8dd8e11540aacfe5b" exitCode=0 Sep 11 00:04:12.335529 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:04:12.335468 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ab675d-7c657c88d-c468w" Sep 11 00:04:12.335529 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:04:12.335458 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ab675d-7c657c88d-c468w" event={"ID":"5311cf89-76b5-4321-a46c-39fafaf1b802","Type":"ContainerDied","Data":"2557da6d6456345a8ad4f74aa56b3f1ff7dccf73dc2468a8dd8e11540aacfe5b"} Sep 11 00:04:12.335671 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:04:12.335560 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ab675d-7c657c88d-c468w" event={"ID":"5311cf89-76b5-4321-a46c-39fafaf1b802","Type":"ContainerDied","Data":"d9bb1ac8b35f7fdda3d0720eada6858470b607d22cfa3f18841ecf5099e8601c"} Sep 11 00:04:12.335671 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:04:12.335585 2574 scope.go:117] "RemoveContainer" containerID="2557da6d6456345a8ad4f74aa56b3f1ff7dccf73dc2468a8dd8e11540aacfe5b" Sep 11 00:04:12.349460 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:04:12.349435 2574 scope.go:117] "RemoveContainer" containerID="2557da6d6456345a8ad4f74aa56b3f1ff7dccf73dc2468a8dd8e11540aacfe5b" Sep 11 00:04:12.349777 ip-10-0-138-128 kubenswrapper[2574]: E0911 00:04:12.349757 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2557da6d6456345a8ad4f74aa56b3f1ff7dccf73dc2468a8dd8e11540aacfe5b\": container with ID starting with 2557da6d6456345a8ad4f74aa56b3f1ff7dccf73dc2468a8dd8e11540aacfe5b not found: ID does not exist" containerID="2557da6d6456345a8ad4f74aa56b3f1ff7dccf73dc2468a8dd8e11540aacfe5b" Sep 11 00:04:12.349856 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:04:12.349782 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2557da6d6456345a8ad4f74aa56b3f1ff7dccf73dc2468a8dd8e11540aacfe5b"} err="failed to get container status \"2557da6d6456345a8ad4f74aa56b3f1ff7dccf73dc2468a8dd8e11540aacfe5b\": rpc error: code = NotFound desc = could not find container \"2557da6d6456345a8ad4f74aa56b3f1ff7dccf73dc2468a8dd8e11540aacfe5b\": container with ID starting with 2557da6d6456345a8ad4f74aa56b3f1ff7dccf73dc2468a8dd8e11540aacfe5b not found: ID does not exist" Sep 11 00:04:12.360555 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:04:12.360533 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ab675d-7c657c88d-c468w"] Sep 11 00:04:12.364680 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:04:12.364658 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ab675d-7c657c88d-c468w"] Sep 11 00:04:12.424309 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:04:12.424253 2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5311cf89-76b5-4321-a46c-39fafaf1b802-openshift-service-ca\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 00:04:12.424309 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:04:12.424275 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5311cf89-76b5-4321-a46c-39fafaf1b802-maas-api-tls\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 00:04:12.424309 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:04:12.424285 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qx4f\" (UniqueName: \"kubernetes.io/projected/5311cf89-76b5-4321-a46c-39fafaf1b802-kube-api-access-8qx4f\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 00:04:12.424309 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:04:12.424293 2574 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5311cf89-76b5-4321-a46c-39fafaf1b802-metrics-certs\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 00:04:13.094981 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:04:13.094952 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5311cf89-76b5-4321-a46c-39fafaf1b802" path="/var/lib/kubelet/pods/5311cf89-76b5-4321-a46c-39fafaf1b802/volumes" Sep 11 00:04:16.377822 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:04:16.377789 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-56r7l"] Sep 11 00:04:34.987705 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:04:34.987668 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-56r7l"] Sep 11 00:04:37.544056 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:04:37.544018 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-17664f-5655b4f67d-9kfxn"] Sep 11 00:04:37.544515 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:04:37.544324 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-17664f-5655b4f67d-9kfxn" podUID="75191a66-c747-4414-8840-c3c7748fe55a" containerName="maas-api" containerID="cri-o://e32cdaa8738a103b177c58bb48ab7a8a95b3fac6b79579a9712e6fbe544a956e" gracePeriod=30 Sep 11 00:04:37.816004 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:04:37.815981 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-17664f-5655b4f67d-9kfxn" Sep 11 00:04:37.887061 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:04:37.887027 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r47w9\" (UniqueName: \"kubernetes.io/projected/75191a66-c747-4414-8840-c3c7748fe55a-kube-api-access-r47w9\") pod \"75191a66-c747-4414-8840-c3c7748fe55a\" (UID: \"75191a66-c747-4414-8840-c3c7748fe55a\") " Sep 11 00:04:37.887061 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:04:37.887087 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75191a66-c747-4414-8840-c3c7748fe55a-metrics-certs\") pod \"75191a66-c747-4414-8840-c3c7748fe55a\" (UID: \"75191a66-c747-4414-8840-c3c7748fe55a\") " Sep 11 00:04:37.887463 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:04:37.887205 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75191a66-c747-4414-8840-c3c7748fe55a-maas-api-tls\") pod \"75191a66-c747-4414-8840-c3c7748fe55a\" (UID: \"75191a66-c747-4414-8840-c3c7748fe55a\") " Sep 11 00:04:37.887463 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:04:37.887240 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75191a66-c747-4414-8840-c3c7748fe55a-openshift-service-ca\") pod \"75191a66-c747-4414-8840-c3c7748fe55a\" (UID: \"75191a66-c747-4414-8840-c3c7748fe55a\") " Sep 11 00:04:37.888606 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:04:37.887812 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75191a66-c747-4414-8840-c3c7748fe55a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "75191a66-c747-4414-8840-c3c7748fe55a" (UID: "75191a66-c747-4414-8840-c3c7748fe55a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 00:04:37.889859 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:04:37.889813 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75191a66-c747-4414-8840-c3c7748fe55a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "75191a66-c747-4414-8840-c3c7748fe55a" (UID: "75191a66-c747-4414-8840-c3c7748fe55a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 00:04:37.890568 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:04:37.890527 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75191a66-c747-4414-8840-c3c7748fe55a-kube-api-access-r47w9" (OuterVolumeSpecName: "kube-api-access-r47w9") pod "75191a66-c747-4414-8840-c3c7748fe55a" (UID: "75191a66-c747-4414-8840-c3c7748fe55a"). InnerVolumeSpecName "kube-api-access-r47w9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 00:04:37.891089 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:04:37.891055 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75191a66-c747-4414-8840-c3c7748fe55a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "75191a66-c747-4414-8840-c3c7748fe55a" (UID: "75191a66-c747-4414-8840-c3c7748fe55a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 00:04:37.988791 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:04:37.988758 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r47w9\" (UniqueName: \"kubernetes.io/projected/75191a66-c747-4414-8840-c3c7748fe55a-kube-api-access-r47w9\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 00:04:37.988791 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:04:37.988786 2574 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75191a66-c747-4414-8840-c3c7748fe55a-metrics-certs\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 00:04:37.988791 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:04:37.988795 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75191a66-c747-4414-8840-c3c7748fe55a-maas-api-tls\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 00:04:37.989008 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:04:37.988803 2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75191a66-c747-4414-8840-c3c7748fe55a-openshift-service-ca\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 00:04:38.496679 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:04:38.496624 2574 generic.go:356] "Generic (PLEG): container finished" podID="75191a66-c747-4414-8840-c3c7748fe55a" containerID="e32cdaa8738a103b177c58bb48ab7a8a95b3fac6b79579a9712e6fbe544a956e" exitCode=0 Sep 11 00:04:38.496886 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:04:38.496683 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-17664f-5655b4f67d-9kfxn" Sep 11 00:04:38.496886 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:04:38.496711 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-17664f-5655b4f67d-9kfxn" event={"ID":"75191a66-c747-4414-8840-c3c7748fe55a","Type":"ContainerDied","Data":"e32cdaa8738a103b177c58bb48ab7a8a95b3fac6b79579a9712e6fbe544a956e"} Sep 11 00:04:38.496886 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:04:38.496770 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-17664f-5655b4f67d-9kfxn" event={"ID":"75191a66-c747-4414-8840-c3c7748fe55a","Type":"ContainerDied","Data":"8ad7d53f1817651a8504c014dbf04ec4cc30ab89295e05c899f4a0407546a86d"} Sep 11 00:04:38.496886 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:04:38.496796 2574 scope.go:117] "RemoveContainer" containerID="e32cdaa8738a103b177c58bb48ab7a8a95b3fac6b79579a9712e6fbe544a956e" Sep 11 00:04:38.507564 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:04:38.507544 2574 scope.go:117] "RemoveContainer" containerID="e32cdaa8738a103b177c58bb48ab7a8a95b3fac6b79579a9712e6fbe544a956e" Sep 11 00:04:38.507852 ip-10-0-138-128 kubenswrapper[2574]: E0911 00:04:38.507831 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e32cdaa8738a103b177c58bb48ab7a8a95b3fac6b79579a9712e6fbe544a956e\": container with ID starting with e32cdaa8738a103b177c58bb48ab7a8a95b3fac6b79579a9712e6fbe544a956e not found: ID does not exist" containerID="e32cdaa8738a103b177c58bb48ab7a8a95b3fac6b79579a9712e6fbe544a956e" Sep 11 00:04:38.507941 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:04:38.507859 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e32cdaa8738a103b177c58bb48ab7a8a95b3fac6b79579a9712e6fbe544a956e"} err="failed to get container status \"e32cdaa8738a103b177c58bb48ab7a8a95b3fac6b79579a9712e6fbe544a956e\": rpc error: code = NotFound desc = could not find container \"e32cdaa8738a103b177c58bb48ab7a8a95b3fac6b79579a9712e6fbe544a956e\": container with ID starting with e32cdaa8738a103b177c58bb48ab7a8a95b3fac6b79579a9712e6fbe544a956e not found: ID does not exist" Sep 11 00:04:38.519339 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:04:38.519317 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-17664f-5655b4f67d-9kfxn"] Sep 11 00:04:38.523748 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:04:38.523727 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-17664f-5655b4f67d-9kfxn"] Sep 11 00:04:39.094592 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:04:39.094556 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="75191a66-c747-4414-8840-c3c7748fe55a" path="/var/lib/kubelet/pods/75191a66-c747-4414-8840-c3c7748fe55a/volumes" Sep 11 00:04:48.083457 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:04:48.083425 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-56r7l"] Sep 11 00:05:26.034550 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:05:26.034460 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-17664f-openshift-default-69464f8bf4-bvqbl"] Sep 11 00:05:26.035114 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:05:26.034726 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-17664f-openshift-default-69464f8bf4-bvqbl" podUID="f7c5b79b-f87f-4fc7-9734-e52a01fb7023" containerName="istio-proxy" containerID="cri-o://060cd1f4aa197cb2439da42b3c21a6e6bb9af65559e780107dbb2c926ea37ed8" gracePeriod=30 Sep 11 00:05:31.293596 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:05:31.293536 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-17664f-openshift-default-69464f8bf4-bvqbl" Sep 11 00:05:31.325297 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:05:31.325270 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7c5b79b-f87f-4fc7-9734-e52a01fb7023-workload-socket\") pod \"f7c5b79b-f87f-4fc7-9734-e52a01fb7023\" (UID: \"f7c5b79b-f87f-4fc7-9734-e52a01fb7023\") " Sep 11 00:05:31.325481 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:05:31.325327 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7c5b79b-f87f-4fc7-9734-e52a01fb7023-istiod-ca-cert\") pod \"f7c5b79b-f87f-4fc7-9734-e52a01fb7023\" (UID: \"f7c5b79b-f87f-4fc7-9734-e52a01fb7023\") " Sep 11 00:05:31.325481 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:05:31.325387 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cxn7v\" (UniqueName: \"kubernetes.io/projected/f7c5b79b-f87f-4fc7-9734-e52a01fb7023-kube-api-access-cxn7v\") pod \"f7c5b79b-f87f-4fc7-9734-e52a01fb7023\" (UID: \"f7c5b79b-f87f-4fc7-9734-e52a01fb7023\") " Sep 11 00:05:31.325481 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:05:31.325438 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7c5b79b-f87f-4fc7-9734-e52a01fb7023-istio-podinfo\") pod \"f7c5b79b-f87f-4fc7-9734-e52a01fb7023\" (UID: \"f7c5b79b-f87f-4fc7-9734-e52a01fb7023\") " Sep 11 00:05:31.325481 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:05:31.325464 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7c5b79b-f87f-4fc7-9734-e52a01fb7023-istio-envoy\") pod \"f7c5b79b-f87f-4fc7-9734-e52a01fb7023\" (UID: \"f7c5b79b-f87f-4fc7-9734-e52a01fb7023\") " Sep 11 00:05:31.325734 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:05:31.325489 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7c5b79b-f87f-4fc7-9734-e52a01fb7023-credential-socket\") pod \"f7c5b79b-f87f-4fc7-9734-e52a01fb7023\" (UID: \"f7c5b79b-f87f-4fc7-9734-e52a01fb7023\") " Sep 11 00:05:31.325734 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:05:31.325504 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7c5b79b-f87f-4fc7-9734-e52a01fb7023-istio-data\") pod \"f7c5b79b-f87f-4fc7-9734-e52a01fb7023\" (UID: \"f7c5b79b-f87f-4fc7-9734-e52a01fb7023\") " Sep 11 00:05:31.325734 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:05:31.325530 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7c5b79b-f87f-4fc7-9734-e52a01fb7023-workload-certs\") pod \"f7c5b79b-f87f-4fc7-9734-e52a01fb7023\" (UID: \"f7c5b79b-f87f-4fc7-9734-e52a01fb7023\") " Sep 11 00:05:31.325734 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:05:31.325566 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7c5b79b-f87f-4fc7-9734-e52a01fb7023-istio-token\") pod \"f7c5b79b-f87f-4fc7-9734-e52a01fb7023\" (UID: \"f7c5b79b-f87f-4fc7-9734-e52a01fb7023\") " Sep 11 00:05:31.325734 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:05:31.325576 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7c5b79b-f87f-4fc7-9734-e52a01fb7023-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f7c5b79b-f87f-4fc7-9734-e52a01fb7023" (UID: "f7c5b79b-f87f-4fc7-9734-e52a01fb7023"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 00:05:31.326004 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:05:31.325870 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7c5b79b-f87f-4fc7-9734-e52a01fb7023-istio-data" (OuterVolumeSpecName: "istio-data") pod "f7c5b79b-f87f-4fc7-9734-e52a01fb7023" (UID: "f7c5b79b-f87f-4fc7-9734-e52a01fb7023"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 00:05:31.326004 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:05:31.325892 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7c5b79b-f87f-4fc7-9734-e52a01fb7023-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f7c5b79b-f87f-4fc7-9734-e52a01fb7023" (UID: "f7c5b79b-f87f-4fc7-9734-e52a01fb7023"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 00:05:31.326004 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:05:31.325969 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7c5b79b-f87f-4fc7-9734-e52a01fb7023-workload-socket\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 00:05:31.326004 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:05:31.325990 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7c5b79b-f87f-4fc7-9734-e52a01fb7023-istiod-ca-cert\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 00:05:31.326004 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:05:31.326004 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7c5b79b-f87f-4fc7-9734-e52a01fb7023-istio-data\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 00:05:31.326284 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:05:31.326053 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7c5b79b-f87f-4fc7-9734-e52a01fb7023-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f7c5b79b-f87f-4fc7-9734-e52a01fb7023" (UID: "f7c5b79b-f87f-4fc7-9734-e52a01fb7023"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 00:05:31.326284 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:05:31.326084 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7c5b79b-f87f-4fc7-9734-e52a01fb7023-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f7c5b79b-f87f-4fc7-9734-e52a01fb7023" (UID: "f7c5b79b-f87f-4fc7-9734-e52a01fb7023"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 00:05:31.328306 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:05:31.328279 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7c5b79b-f87f-4fc7-9734-e52a01fb7023-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f7c5b79b-f87f-4fc7-9734-e52a01fb7023" (UID: "f7c5b79b-f87f-4fc7-9734-e52a01fb7023"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 00:05:31.328439 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:05:31.328319 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7c5b79b-f87f-4fc7-9734-e52a01fb7023-kube-api-access-cxn7v" (OuterVolumeSpecName: "kube-api-access-cxn7v") pod "f7c5b79b-f87f-4fc7-9734-e52a01fb7023" (UID: "f7c5b79b-f87f-4fc7-9734-e52a01fb7023"). InnerVolumeSpecName "kube-api-access-cxn7v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 00:05:31.328439 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:05:31.328391 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7c5b79b-f87f-4fc7-9734-e52a01fb7023-istio-token" (OuterVolumeSpecName: "istio-token") pod "f7c5b79b-f87f-4fc7-9734-e52a01fb7023" (UID: "f7c5b79b-f87f-4fc7-9734-e52a01fb7023"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 00:05:31.328439 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:05:31.328411 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f7c5b79b-f87f-4fc7-9734-e52a01fb7023-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f7c5b79b-f87f-4fc7-9734-e52a01fb7023" (UID: "f7c5b79b-f87f-4fc7-9734-e52a01fb7023"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 00:05:31.427495 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:05:31.427458 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxn7v\" (UniqueName: \"kubernetes.io/projected/f7c5b79b-f87f-4fc7-9734-e52a01fb7023-kube-api-access-cxn7v\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 00:05:31.427495 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:05:31.427490 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7c5b79b-f87f-4fc7-9734-e52a01fb7023-istio-podinfo\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 00:05:31.427495 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:05:31.427499 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7c5b79b-f87f-4fc7-9734-e52a01fb7023-istio-envoy\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 00:05:31.427495 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:05:31.427507 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7c5b79b-f87f-4fc7-9734-e52a01fb7023-credential-socket\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 00:05:31.427804 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:05:31.427516 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7c5b79b-f87f-4fc7-9734-e52a01fb7023-workload-certs\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 00:05:31.427804 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:05:31.427524 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7c5b79b-f87f-4fc7-9734-e52a01fb7023-istio-token\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 00:05:31.823122 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:05:31.823081 2574 generic.go:356] "Generic (PLEG): container finished" podID="f7c5b79b-f87f-4fc7-9734-e52a01fb7023" containerID="060cd1f4aa197cb2439da42b3c21a6e6bb9af65559e780107dbb2c926ea37ed8" exitCode=0 Sep 11 00:05:31.823331 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:05:31.823137 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-17664f-openshift-default-69464f8bf4-bvqbl" Sep 11 00:05:31.823331 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:05:31.823172 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-17664f-openshift-default-69464f8bf4-bvqbl" event={"ID":"f7c5b79b-f87f-4fc7-9734-e52a01fb7023","Type":"ContainerDied","Data":"060cd1f4aa197cb2439da42b3c21a6e6bb9af65559e780107dbb2c926ea37ed8"} Sep 11 00:05:31.823331 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:05:31.823243 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-17664f-openshift-default-69464f8bf4-bvqbl" event={"ID":"f7c5b79b-f87f-4fc7-9734-e52a01fb7023","Type":"ContainerDied","Data":"141c8a0d9f26f8ca712d6b7c0952745cefb7577da6ad64c4374b5f4a4e4b8c4e"} Sep 11 00:05:31.823331 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:05:31.823274 2574 scope.go:117] "RemoveContainer" containerID="060cd1f4aa197cb2439da42b3c21a6e6bb9af65559e780107dbb2c926ea37ed8" Sep 11 00:05:31.835672 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:05:31.835631 2574 scope.go:117] "RemoveContainer" containerID="eb74dc5fea3def5f4d9795cd7cd3b72a7111dcb914c33148f005147e624dc50f" Sep 11 00:05:31.844750 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:05:31.844726 2574 scope.go:117] "RemoveContainer" containerID="060cd1f4aa197cb2439da42b3c21a6e6bb9af65559e780107dbb2c926ea37ed8" Sep 11 00:05:31.845059 ip-10-0-138-128 kubenswrapper[2574]: E0911 00:05:31.845038 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"060cd1f4aa197cb2439da42b3c21a6e6bb9af65559e780107dbb2c926ea37ed8\": container with ID starting with 060cd1f4aa197cb2439da42b3c21a6e6bb9af65559e780107dbb2c926ea37ed8 not found: ID does not exist" containerID="060cd1f4aa197cb2439da42b3c21a6e6bb9af65559e780107dbb2c926ea37ed8" Sep 11 00:05:31.845109 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:05:31.845070 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"060cd1f4aa197cb2439da42b3c21a6e6bb9af65559e780107dbb2c926ea37ed8"} err="failed to get container status \"060cd1f4aa197cb2439da42b3c21a6e6bb9af65559e780107dbb2c926ea37ed8\": rpc error: code = NotFound desc = could not find container \"060cd1f4aa197cb2439da42b3c21a6e6bb9af65559e780107dbb2c926ea37ed8\": container with ID starting with 060cd1f4aa197cb2439da42b3c21a6e6bb9af65559e780107dbb2c926ea37ed8 not found: ID does not exist" Sep 11 00:05:31.845109 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:05:31.845088 2574 scope.go:117] "RemoveContainer" containerID="eb74dc5fea3def5f4d9795cd7cd3b72a7111dcb914c33148f005147e624dc50f" Sep 11 00:05:31.845358 ip-10-0-138-128 kubenswrapper[2574]: E0911 00:05:31.845340 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb74dc5fea3def5f4d9795cd7cd3b72a7111dcb914c33148f005147e624dc50f\": container with ID starting with eb74dc5fea3def5f4d9795cd7cd3b72a7111dcb914c33148f005147e624dc50f not found: ID does not exist" containerID="eb74dc5fea3def5f4d9795cd7cd3b72a7111dcb914c33148f005147e624dc50f" Sep 11 00:05:31.845417 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:05:31.845368 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb74dc5fea3def5f4d9795cd7cd3b72a7111dcb914c33148f005147e624dc50f"} err="failed to get container status \"eb74dc5fea3def5f4d9795cd7cd3b72a7111dcb914c33148f005147e624dc50f\": rpc error: code = NotFound desc = could not find container \"eb74dc5fea3def5f4d9795cd7cd3b72a7111dcb914c33148f005147e624dc50f\": container with ID starting with eb74dc5fea3def5f4d9795cd7cd3b72a7111dcb914c33148f005147e624dc50f not found: ID does not exist" Sep 11 00:05:31.849868 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:05:31.849843 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-17664f-openshift-default-69464f8bf4-bvqbl"] Sep 11 00:05:31.853110 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:05:31.853088 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-17664f-openshift-default-69464f8bf4-bvqbl"] Sep 11 00:05:33.095074 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:05:33.095041 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7c5b79b-f87f-4fc7-9734-e52a01fb7023" path="/var/lib/kubelet/pods/f7c5b79b-f87f-4fc7-9734-e52a01fb7023/volumes" Sep 11 00:05:40.382596 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:05:40.382560 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-56r7l"] Sep 11 00:05:43.704823 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:05:43.704788 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-43f569-openshift-default-5c7f95d694-ccm7g"] Sep 11 00:05:43.705268 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:05:43.704995 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-43f569-openshift-default-5c7f95d694-ccm7g" podUID="d738baa4-79b0-4d1d-80d2-7efbd3d91ade" containerName="istio-proxy" containerID="cri-o://1a686afd4b48cf7efbc6cb2a6fbddf64a81b24ed4b05c347b7f967cc71dbcc49" gracePeriod=30 Sep 11 00:05:48.926925 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:05:48.926892 2574 generic.go:356] "Generic (PLEG): container finished" podID="d738baa4-79b0-4d1d-80d2-7efbd3d91ade" containerID="1a686afd4b48cf7efbc6cb2a6fbddf64a81b24ed4b05c347b7f967cc71dbcc49" exitCode=0 Sep 11 00:05:48.927305 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:05:48.926965 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-43f569-openshift-default-5c7f95d694-ccm7g" event={"ID":"d738baa4-79b0-4d1d-80d2-7efbd3d91ade","Type":"ContainerDied","Data":"1a686afd4b48cf7efbc6cb2a6fbddf64a81b24ed4b05c347b7f967cc71dbcc49"} Sep 11 00:05:48.927305 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:05:48.927030 2574 scope.go:117] "RemoveContainer" containerID="43f25a2779835058242892c1424ca23b40403b94eeb8063ffc32e32b7d258f7a" Sep 11 00:05:48.962429 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:05:48.962406 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-43f569-openshift-default-5c7f95d694-ccm7g" Sep 11 00:05:49.107344 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:05:49.107269 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d738baa4-79b0-4d1d-80d2-7efbd3d91ade-istio-podinfo\") pod \"d738baa4-79b0-4d1d-80d2-7efbd3d91ade\" (UID: \"d738baa4-79b0-4d1d-80d2-7efbd3d91ade\") " Sep 11 00:05:49.107344 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:05:49.107304 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d738baa4-79b0-4d1d-80d2-7efbd3d91ade-istio-data\") pod \"d738baa4-79b0-4d1d-80d2-7efbd3d91ade\" (UID: \"d738baa4-79b0-4d1d-80d2-7efbd3d91ade\") " Sep 11 00:05:49.107344 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:05:49.107338 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d738baa4-79b0-4d1d-80d2-7efbd3d91ade-istiod-ca-cert\") pod \"d738baa4-79b0-4d1d-80d2-7efbd3d91ade\" (UID: \"d738baa4-79b0-4d1d-80d2-7efbd3d91ade\") " Sep 11 00:05:49.107608 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:05:49.107355 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d738baa4-79b0-4d1d-80d2-7efbd3d91ade-credential-socket\") pod \"d738baa4-79b0-4d1d-80d2-7efbd3d91ade\" (UID: \"d738baa4-79b0-4d1d-80d2-7efbd3d91ade\") " Sep 11 00:05:49.107608 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:05:49.107385 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d738baa4-79b0-4d1d-80d2-7efbd3d91ade-workload-certs\") pod \"d738baa4-79b0-4d1d-80d2-7efbd3d91ade\" (UID: \"d738baa4-79b0-4d1d-80d2-7efbd3d91ade\") " Sep 11 00:05:49.107608 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:05:49.107428 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d738baa4-79b0-4d1d-80d2-7efbd3d91ade-istio-envoy\") pod \"d738baa4-79b0-4d1d-80d2-7efbd3d91ade\" (UID: \"d738baa4-79b0-4d1d-80d2-7efbd3d91ade\") " Sep 11 00:05:49.107608 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:05:49.107477 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d738baa4-79b0-4d1d-80d2-7efbd3d91ade-istio-token\") pod \"d738baa4-79b0-4d1d-80d2-7efbd3d91ade\" (UID: \"d738baa4-79b0-4d1d-80d2-7efbd3d91ade\") " Sep 11 00:05:49.107608 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:05:49.107510 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5gsct\" (UniqueName: \"kubernetes.io/projected/d738baa4-79b0-4d1d-80d2-7efbd3d91ade-kube-api-access-5gsct\") pod \"d738baa4-79b0-4d1d-80d2-7efbd3d91ade\" (UID: \"d738baa4-79b0-4d1d-80d2-7efbd3d91ade\") " Sep 11 00:05:49.107608 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:05:49.107545 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d738baa4-79b0-4d1d-80d2-7efbd3d91ade-workload-socket\") pod \"d738baa4-79b0-4d1d-80d2-7efbd3d91ade\" (UID: \"d738baa4-79b0-4d1d-80d2-7efbd3d91ade\") " Sep 11 00:05:49.107946 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:05:49.107736 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d738baa4-79b0-4d1d-80d2-7efbd3d91ade-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d738baa4-79b0-4d1d-80d2-7efbd3d91ade" (UID: "d738baa4-79b0-4d1d-80d2-7efbd3d91ade"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 00:05:49.107946 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:05:49.107753 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d738baa4-79b0-4d1d-80d2-7efbd3d91ade-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d738baa4-79b0-4d1d-80d2-7efbd3d91ade" (UID: "d738baa4-79b0-4d1d-80d2-7efbd3d91ade"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 00:05:49.107946 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:05:49.107804 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d738baa4-79b0-4d1d-80d2-7efbd3d91ade-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d738baa4-79b0-4d1d-80d2-7efbd3d91ade" (UID: "d738baa4-79b0-4d1d-80d2-7efbd3d91ade"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 00:05:49.107946 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:05:49.107911 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d738baa4-79b0-4d1d-80d2-7efbd3d91ade-istio-data" (OuterVolumeSpecName: "istio-data") pod "d738baa4-79b0-4d1d-80d2-7efbd3d91ade" (UID: "d738baa4-79b0-4d1d-80d2-7efbd3d91ade"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 00:05:49.107946 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:05:49.107943 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d738baa4-79b0-4d1d-80d2-7efbd3d91ade-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d738baa4-79b0-4d1d-80d2-7efbd3d91ade" (UID: "d738baa4-79b0-4d1d-80d2-7efbd3d91ade"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 00:05:49.108211 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:05:49.108144 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d738baa4-79b0-4d1d-80d2-7efbd3d91ade-istiod-ca-cert\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 00:05:49.108211 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:05:49.108163 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d738baa4-79b0-4d1d-80d2-7efbd3d91ade-credential-socket\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 00:05:49.108211 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:05:49.108176 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d738baa4-79b0-4d1d-80d2-7efbd3d91ade-workload-certs\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 00:05:49.108211 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:05:49.108190 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d738baa4-79b0-4d1d-80d2-7efbd3d91ade-workload-socket\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 00:05:49.108211 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:05:49.108203 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d738baa4-79b0-4d1d-80d2-7efbd3d91ade-istio-data\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 00:05:49.109408 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:05:49.109385 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d738baa4-79b0-4d1d-80d2-7efbd3d91ade-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d738baa4-79b0-4d1d-80d2-7efbd3d91ade" (UID: "d738baa4-79b0-4d1d-80d2-7efbd3d91ade"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 00:05:49.109692 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:05:49.109670 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d738baa4-79b0-4d1d-80d2-7efbd3d91ade-istio-token" (OuterVolumeSpecName: "istio-token") pod "d738baa4-79b0-4d1d-80d2-7efbd3d91ade" (UID: "d738baa4-79b0-4d1d-80d2-7efbd3d91ade"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 00:05:49.109928 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:05:49.109909 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d738baa4-79b0-4d1d-80d2-7efbd3d91ade-kube-api-access-5gsct" (OuterVolumeSpecName: "kube-api-access-5gsct") pod "d738baa4-79b0-4d1d-80d2-7efbd3d91ade" (UID: "d738baa4-79b0-4d1d-80d2-7efbd3d91ade"). InnerVolumeSpecName "kube-api-access-5gsct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 00:05:49.109981 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:05:49.109912 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d738baa4-79b0-4d1d-80d2-7efbd3d91ade-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d738baa4-79b0-4d1d-80d2-7efbd3d91ade" (UID: "d738baa4-79b0-4d1d-80d2-7efbd3d91ade"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 00:05:49.209689 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:05:49.209653 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d738baa4-79b0-4d1d-80d2-7efbd3d91ade-istio-podinfo\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 00:05:49.209689 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:05:49.209685 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d738baa4-79b0-4d1d-80d2-7efbd3d91ade-istio-envoy\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 00:05:49.209876 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:05:49.209699 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d738baa4-79b0-4d1d-80d2-7efbd3d91ade-istio-token\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 00:05:49.209876 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:05:49.209713 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5gsct\" (UniqueName: \"kubernetes.io/projected/d738baa4-79b0-4d1d-80d2-7efbd3d91ade-kube-api-access-5gsct\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 00:05:49.934842 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:05:49.934811 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-43f569-openshift-default-5c7f95d694-ccm7g" Sep 11 00:05:49.934842 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:05:49.934819 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-43f569-openshift-default-5c7f95d694-ccm7g" event={"ID":"d738baa4-79b0-4d1d-80d2-7efbd3d91ade","Type":"ContainerDied","Data":"76995e235ba3c8a284ff1cbcefa4565da096734c62a383d310e431572f092789"} Sep 11 00:05:49.935352 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:05:49.934903 2574 scope.go:117] "RemoveContainer" containerID="1a686afd4b48cf7efbc6cb2a6fbddf64a81b24ed4b05c347b7f967cc71dbcc49" Sep 11 00:05:49.956982 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:05:49.956954 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-43f569-openshift-default-5c7f95d694-ccm7g"] Sep 11 00:05:49.960278 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:05:49.960256 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-43f569-openshift-default-5c7f95d694-ccm7g"] Sep 11 00:05:51.094731 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:05:51.094694 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d738baa4-79b0-4d1d-80d2-7efbd3d91ade" path="/var/lib/kubelet/pods/d738baa4-79b0-4d1d-80d2-7efbd3d91ade/volumes" Sep 11 00:05:54.511019 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:05:54.510987 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d7a539-fb5d4d5c6-nn6gl"] Sep 11 00:05:54.511520 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:05:54.511211 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d7a539-fb5d4d5c6-nn6gl" podUID="9d8b9058-927e-4b3c-aec9-c94e233d7253" containerName="maas-api" containerID="cri-o://06217fcc15fbe098f591cce5ecbf875de15263f89866a709a444b6059996918b" gracePeriod=30 Sep 11 00:05:54.542016 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:05:54.541985 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d7a539-29818080-kjzfg"] Sep 11 00:05:54.544752 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:05:54.544724 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-d7a539-29818080-kjzfg"] Sep 11 00:05:54.762524 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:05:54.762469 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d7a539-fb5d4d5c6-nn6gl" Sep 11 00:05:54.871633 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:05:54.871597 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d8b9058-927e-4b3c-aec9-c94e233d7253-maas-api-tls\") pod \"9d8b9058-927e-4b3c-aec9-c94e233d7253\" (UID: \"9d8b9058-927e-4b3c-aec9-c94e233d7253\") " Sep 11 00:05:54.871834 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:05:54.871715 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d8b9058-927e-4b3c-aec9-c94e233d7253-openshift-service-ca\") pod \"9d8b9058-927e-4b3c-aec9-c94e233d7253\" (UID: \"9d8b9058-927e-4b3c-aec9-c94e233d7253\") " Sep 11 00:05:54.871834 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:05:54.871794 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d8b9058-927e-4b3c-aec9-c94e233d7253-metrics-certs\") pod \"9d8b9058-927e-4b3c-aec9-c94e233d7253\" (UID: \"9d8b9058-927e-4b3c-aec9-c94e233d7253\") " Sep 11 00:05:54.871958 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:05:54.871842 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-whvbg\" (UniqueName: \"kubernetes.io/projected/9d8b9058-927e-4b3c-aec9-c94e233d7253-kube-api-access-whvbg\") pod \"9d8b9058-927e-4b3c-aec9-c94e233d7253\" (UID: \"9d8b9058-927e-4b3c-aec9-c94e233d7253\") " Sep 11 00:05:54.872082 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:05:54.872052 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d8b9058-927e-4b3c-aec9-c94e233d7253-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9d8b9058-927e-4b3c-aec9-c94e233d7253" (UID: "9d8b9058-927e-4b3c-aec9-c94e233d7253"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 00:05:54.872229 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:05:54.872211 2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d8b9058-927e-4b3c-aec9-c94e233d7253-openshift-service-ca\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 00:05:54.873862 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:05:54.873833 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d8b9058-927e-4b3c-aec9-c94e233d7253-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9d8b9058-927e-4b3c-aec9-c94e233d7253" (UID: "9d8b9058-927e-4b3c-aec9-c94e233d7253"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 00:05:54.874011 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:05:54.873989 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d8b9058-927e-4b3c-aec9-c94e233d7253-kube-api-access-whvbg" (OuterVolumeSpecName: "kube-api-access-whvbg") pod "9d8b9058-927e-4b3c-aec9-c94e233d7253" (UID: "9d8b9058-927e-4b3c-aec9-c94e233d7253"). InnerVolumeSpecName "kube-api-access-whvbg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 00:05:54.874093 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:05:54.874076 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d8b9058-927e-4b3c-aec9-c94e233d7253-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "9d8b9058-927e-4b3c-aec9-c94e233d7253" (UID: "9d8b9058-927e-4b3c-aec9-c94e233d7253"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 00:05:54.969289 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:05:54.969252 2574 generic.go:356] "Generic (PLEG): container finished" podID="9d8b9058-927e-4b3c-aec9-c94e233d7253" containerID="06217fcc15fbe098f591cce5ecbf875de15263f89866a709a444b6059996918b" exitCode=0 Sep 11 00:05:54.969465 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:05:54.969302 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d7a539-fb5d4d5c6-nn6gl" Sep 11 00:05:54.969465 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:05:54.969331 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d7a539-fb5d4d5c6-nn6gl" event={"ID":"9d8b9058-927e-4b3c-aec9-c94e233d7253","Type":"ContainerDied","Data":"06217fcc15fbe098f591cce5ecbf875de15263f89866a709a444b6059996918b"} Sep 11 00:05:54.969465 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:05:54.969388 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d7a539-fb5d4d5c6-nn6gl" event={"ID":"9d8b9058-927e-4b3c-aec9-c94e233d7253","Type":"ContainerDied","Data":"3a8298ebb9e73abfd5d0cdcfaa45753e1f2298cfca07a43d301f8a9cab8335d4"} Sep 11 00:05:54.969465 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:05:54.969411 2574 scope.go:117] "RemoveContainer" containerID="06217fcc15fbe098f591cce5ecbf875de15263f89866a709a444b6059996918b" Sep 11 00:05:54.972894 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:05:54.972873 2574 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d8b9058-927e-4b3c-aec9-c94e233d7253-metrics-certs\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 00:05:54.973011 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:05:54.972896 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whvbg\" (UniqueName: \"kubernetes.io/projected/9d8b9058-927e-4b3c-aec9-c94e233d7253-kube-api-access-whvbg\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 00:05:54.973011 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:05:54.972909 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d8b9058-927e-4b3c-aec9-c94e233d7253-maas-api-tls\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 00:05:54.979479 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:05:54.979462 2574 scope.go:117] "RemoveContainer" containerID="06217fcc15fbe098f591cce5ecbf875de15263f89866a709a444b6059996918b" Sep 11 00:05:54.979836 ip-10-0-138-128 kubenswrapper[2574]: E0911 00:05:54.979816 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06217fcc15fbe098f591cce5ecbf875de15263f89866a709a444b6059996918b\": container with ID starting with 06217fcc15fbe098f591cce5ecbf875de15263f89866a709a444b6059996918b not found: ID does not exist" containerID="06217fcc15fbe098f591cce5ecbf875de15263f89866a709a444b6059996918b" Sep 11 00:05:54.979907 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:05:54.979844 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06217fcc15fbe098f591cce5ecbf875de15263f89866a709a444b6059996918b"} err="failed to get container status \"06217fcc15fbe098f591cce5ecbf875de15263f89866a709a444b6059996918b\": rpc error: code = NotFound desc = could not find container \"06217fcc15fbe098f591cce5ecbf875de15263f89866a709a444b6059996918b\": container with ID starting with 06217fcc15fbe098f591cce5ecbf875de15263f89866a709a444b6059996918b not found: ID does not exist" Sep 11 00:05:54.991060 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:05:54.991032 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d7a539-fb5d4d5c6-nn6gl"] Sep 11 00:05:54.998612 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:05:54.998588 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d7a539-fb5d4d5c6-nn6gl"] Sep 11 00:05:55.094436 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:05:55.094356 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3394f36c-47c5-4ea7-a9a8-264bf006cb45" path="/var/lib/kubelet/pods/3394f36c-47c5-4ea7-a9a8-264bf006cb45/volumes" Sep 11 00:05:55.094753 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:05:55.094739 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d8b9058-927e-4b3c-aec9-c94e233d7253" path="/var/lib/kubelet/pods/9d8b9058-927e-4b3c-aec9-c94e233d7253/volumes" Sep 11 00:05:56.472967 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:05:56.472933 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-56r7l"] Sep 11 00:06:01.396432 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:06:01.396395 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-d7a539-openshift-default-7c74c59b78-q7s7n"] Sep 11 00:06:01.397049 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:06:01.396673 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-d7a539-openshift-default-7c74c59b78-q7s7n" podUID="1f58d38e-bf62-44f6-9522-2ada85451d15" containerName="istio-proxy" containerID="cri-o://393310600ba877303bbbb18665b2078d73065ba1658618e1611fe8db21354889" gracePeriod=30 Sep 11 00:06:01.759592 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:06:01.759551 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4094ce-pxzbw"] Sep 11 00:06:01.760084 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:06:01.760063 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d738baa4-79b0-4d1d-80d2-7efbd3d91ade" containerName="istio-proxy" Sep 11 00:06:01.760189 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:06:01.760086 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="d738baa4-79b0-4d1d-80d2-7efbd3d91ade" containerName="istio-proxy" Sep 11 00:06:01.760189 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:06:01.760109 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7c5b79b-f87f-4fc7-9734-e52a01fb7023" containerName="istio-proxy" Sep 11 00:06:01.760189 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:06:01.760117 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7c5b79b-f87f-4fc7-9734-e52a01fb7023" containerName="istio-proxy" Sep 11 00:06:01.760189 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:06:01.760147 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75191a66-c747-4414-8840-c3c7748fe55a" containerName="maas-api" Sep 11 00:06:01.760189 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:06:01.760157 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="75191a66-c747-4414-8840-c3c7748fe55a" containerName="maas-api" Sep 11 00:06:01.760189 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:06:01.760167 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5311cf89-76b5-4321-a46c-39fafaf1b802" containerName="maas-api" Sep 11 00:06:01.760189 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:06:01.760175 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="5311cf89-76b5-4321-a46c-39fafaf1b802" containerName="maas-api" Sep 11 00:06:01.760537 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:06:01.760193 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d8b9058-927e-4b3c-aec9-c94e233d7253" containerName="maas-api" Sep 11 00:06:01.760537 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:06:01.760201 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d8b9058-927e-4b3c-aec9-c94e233d7253" containerName="maas-api" Sep 11 00:06:01.760537 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:06:01.760300 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7c5b79b-f87f-4fc7-9734-e52a01fb7023" containerName="istio-proxy" Sep 11 00:06:01.760537 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:06:01.760314 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d8b9058-927e-4b3c-aec9-c94e233d7253" containerName="maas-api" Sep 11 00:06:01.760537 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:06:01.760324 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="d738baa4-79b0-4d1d-80d2-7efbd3d91ade" containerName="istio-proxy" Sep 11 00:06:01.760537 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:06:01.760337 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="75191a66-c747-4414-8840-c3c7748fe55a" containerName="maas-api" Sep 11 00:06:01.760537 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:06:01.760346 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="d738baa4-79b0-4d1d-80d2-7efbd3d91ade" containerName="istio-proxy" Sep 11 00:06:01.760537 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:06:01.760356 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="5311cf89-76b5-4321-a46c-39fafaf1b802" containerName="maas-api" Sep 11 00:06:01.762965 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:06:01.762943 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4094ce-pxzbw" Sep 11 00:06:01.765125 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:06:01.765106 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zrcdn\"" Sep 11 00:06:01.765218 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:06:01.765108 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 00:06:01.770003 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:06:01.769980 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4094ce-pxzbw"] Sep 11 00:06:01.840984 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:06:01.840944 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e509de4d-ec35-4c96-88a5-4c4d1126a123-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-4094ce-pxzbw\" (UID: \"e509de4d-ec35-4c96-88a5-4c4d1126a123\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4094ce-pxzbw" Sep 11 00:06:01.941737 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:06:01.941696 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e509de4d-ec35-4c96-88a5-4c4d1126a123-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-4094ce-pxzbw\" (UID: \"e509de4d-ec35-4c96-88a5-4c4d1126a123\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4094ce-pxzbw" Sep 11 00:06:01.942361 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:06:01.942339 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e509de4d-ec35-4c96-88a5-4c4d1126a123-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-4094ce-pxzbw\" (UID: \"e509de4d-ec35-4c96-88a5-4c4d1126a123\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4094ce-pxzbw" Sep 11 00:06:02.077616 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:06:02.077528 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4094ce-pxzbw" Sep 11 00:06:02.349009 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:06:02.348982 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4094ce-pxzbw"] Sep 11 00:06:02.350975 ip-10-0-138-128 kubenswrapper[2574]: W0911 00:06:02.350945 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode509de4d_ec35_4c96_88a5_4c4d1126a123.slice/crio-a3325719f4cce4e3ba8bb75d5f804378a45f67c88cc408de6e4a346123ef069d WatchSource:0}: Error finding container a3325719f4cce4e3ba8bb75d5f804378a45f67c88cc408de6e4a346123ef069d: Status 404 returned error can't find the container with id a3325719f4cce4e3ba8bb75d5f804378a45f67c88cc408de6e4a346123ef069d Sep 11 00:06:03.027513 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:06:03.027466 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4094ce-pxzbw" event={"ID":"e509de4d-ec35-4c96-88a5-4c4d1126a123","Type":"ContainerStarted","Data":"942121b1c98e5cc757998eccbd3b1ee762a2718d3c08de80fa187bd90683ca1a"} Sep 11 00:06:03.027940 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:06:03.027573 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4094ce-pxzbw" event={"ID":"e509de4d-ec35-4c96-88a5-4c4d1126a123","Type":"ContainerStarted","Data":"a3325719f4cce4e3ba8bb75d5f804378a45f67c88cc408de6e4a346123ef069d"} Sep 11 00:06:03.043911 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:06:03.043861 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4094ce-pxzbw" podStartSLOduration=2.043846059 podStartE2EDuration="2.043846059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 00:06:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:06:03.041991168 +0000 UTC m=+1754.429593928" watchObservedRunningTime="2026-09-11 00:06:03.043846059 +0000 UTC m=+1754.431448818" Sep 11 00:06:04.034836 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:06:04.034804 2574 generic.go:356] "Generic (PLEG): container finished" podID="e509de4d-ec35-4c96-88a5-4c4d1126a123" containerID="942121b1c98e5cc757998eccbd3b1ee762a2718d3c08de80fa187bd90683ca1a" exitCode=0 Sep 11 00:06:04.035236 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:06:04.034897 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4094ce-pxzbw" event={"ID":"e509de4d-ec35-4c96-88a5-4c4d1126a123","Type":"ContainerDied","Data":"942121b1c98e5cc757998eccbd3b1ee762a2718d3c08de80fa187bd90683ca1a"} Sep 11 00:06:05.176048 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:06:05.176023 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4094ce-pxzbw" Sep 11 00:06:05.274280 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:06:05.274246 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e509de4d-ec35-4c96-88a5-4c4d1126a123-maas-api-service-ca\") pod \"e509de4d-ec35-4c96-88a5-4c4d1126a123\" (UID: \"e509de4d-ec35-4c96-88a5-4c4d1126a123\") " Sep 11 00:06:05.274673 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:06:05.274631 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e509de4d-ec35-4c96-88a5-4c4d1126a123-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e509de4d-ec35-4c96-88a5-4c4d1126a123" (UID: "e509de4d-ec35-4c96-88a5-4c4d1126a123"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 00:06:05.375368 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:06:05.375290 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e509de4d-ec35-4c96-88a5-4c4d1126a123-maas-api-service-ca\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 00:06:06.049657 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:06:06.049608 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4094ce-pxzbw" event={"ID":"e509de4d-ec35-4c96-88a5-4c4d1126a123","Type":"ContainerDied","Data":"a3325719f4cce4e3ba8bb75d5f804378a45f67c88cc408de6e4a346123ef069d"} Sep 11 00:06:06.049657 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:06:06.049657 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4094ce-pxzbw" Sep 11 00:06:06.049874 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:06:06.049687 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a3325719f4cce4e3ba8bb75d5f804378a45f67c88cc408de6e4a346123ef069d" Sep 11 00:06:06.652337 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:06:06.652309 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-d7a539-openshift-default-7c74c59b78-q7s7n" Sep 11 00:06:06.687164 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:06:06.687101 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f58d38e-bf62-44f6-9522-2ada85451d15-workload-socket\") pod \"1f58d38e-bf62-44f6-9522-2ada85451d15\" (UID: \"1f58d38e-bf62-44f6-9522-2ada85451d15\") " Sep 11 00:06:06.687164 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:06:06.687130 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f58d38e-bf62-44f6-9522-2ada85451d15-istio-data\") pod \"1f58d38e-bf62-44f6-9522-2ada85451d15\" (UID: \"1f58d38e-bf62-44f6-9522-2ada85451d15\") " Sep 11 00:06:06.687373 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:06:06.687181 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f58d38e-bf62-44f6-9522-2ada85451d15-istio-token\") pod \"1f58d38e-bf62-44f6-9522-2ada85451d15\" (UID: \"1f58d38e-bf62-44f6-9522-2ada85451d15\") " Sep 11 00:06:06.687373 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:06:06.687213 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f58d38e-bf62-44f6-9522-2ada85451d15-istiod-ca-cert\") pod \"1f58d38e-bf62-44f6-9522-2ada85451d15\" (UID: \"1f58d38e-bf62-44f6-9522-2ada85451d15\") " Sep 11 00:06:06.687373 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:06:06.687233 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bkpvw\" (UniqueName: \"kubernetes.io/projected/1f58d38e-bf62-44f6-9522-2ada85451d15-kube-api-access-bkpvw\") pod \"1f58d38e-bf62-44f6-9522-2ada85451d15\" (UID: \"1f58d38e-bf62-44f6-9522-2ada85451d15\") " Sep 11 00:06:06.687373 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:06:06.687277 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f58d38e-bf62-44f6-9522-2ada85451d15-istio-envoy\") pod \"1f58d38e-bf62-44f6-9522-2ada85451d15\" (UID: \"1f58d38e-bf62-44f6-9522-2ada85451d15\") " Sep 11 00:06:06.687373 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:06:06.687338 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f58d38e-bf62-44f6-9522-2ada85451d15-istio-podinfo\") pod \"1f58d38e-bf62-44f6-9522-2ada85451d15\" (UID: \"1f58d38e-bf62-44f6-9522-2ada85451d15\") " Sep 11 00:06:06.687373 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:06:06.687368 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f58d38e-bf62-44f6-9522-2ada85451d15-credential-socket\") pod \"1f58d38e-bf62-44f6-9522-2ada85451d15\" (UID: \"1f58d38e-bf62-44f6-9522-2ada85451d15\") " Sep 11 00:06:06.687714 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:06:06.687395 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f58d38e-bf62-44f6-9522-2ada85451d15-workload-certs\") pod \"1f58d38e-bf62-44f6-9522-2ada85451d15\" (UID: \"1f58d38e-bf62-44f6-9522-2ada85451d15\") " Sep 11 00:06:06.687714 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:06:06.687416 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f58d38e-bf62-44f6-9522-2ada85451d15-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1f58d38e-bf62-44f6-9522-2ada85451d15" (UID: "1f58d38e-bf62-44f6-9522-2ada85451d15"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 00:06:06.687714 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:06:06.687429 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f58d38e-bf62-44f6-9522-2ada85451d15-istio-data" (OuterVolumeSpecName: "istio-data") pod "1f58d38e-bf62-44f6-9522-2ada85451d15" (UID: "1f58d38e-bf62-44f6-9522-2ada85451d15"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 00:06:06.687714 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:06:06.687629 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f58d38e-bf62-44f6-9522-2ada85451d15-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1f58d38e-bf62-44f6-9522-2ada85451d15" (UID: "1f58d38e-bf62-44f6-9522-2ada85451d15"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 00:06:06.687714 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:06:06.687684 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f58d38e-bf62-44f6-9522-2ada85451d15-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1f58d38e-bf62-44f6-9522-2ada85451d15" (UID: "1f58d38e-bf62-44f6-9522-2ada85451d15"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 00:06:06.687989 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:06:06.687822 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f58d38e-bf62-44f6-9522-2ada85451d15-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1f58d38e-bf62-44f6-9522-2ada85451d15" (UID: "1f58d38e-bf62-44f6-9522-2ada85451d15"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 00:06:06.688082 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:06:06.688053 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f58d38e-bf62-44f6-9522-2ada85451d15-workload-certs\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 00:06:06.688209 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:06:06.688085 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f58d38e-bf62-44f6-9522-2ada85451d15-workload-socket\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 00:06:06.688209 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:06:06.688099 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f58d38e-bf62-44f6-9522-2ada85451d15-istio-data\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 00:06:06.688209 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:06:06.688112 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f58d38e-bf62-44f6-9522-2ada85451d15-istiod-ca-cert\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 00:06:06.688209 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:06:06.688125 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f58d38e-bf62-44f6-9522-2ada85451d15-credential-socket\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 00:06:06.689607 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:06:06.689574 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f58d38e-bf62-44f6-9522-2ada85451d15-istio-token" (OuterVolumeSpecName: "istio-token") pod "1f58d38e-bf62-44f6-9522-2ada85451d15" (UID: "1f58d38e-bf62-44f6-9522-2ada85451d15"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 00:06:06.689607 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:06:06.689579 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f58d38e-bf62-44f6-9522-2ada85451d15-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1f58d38e-bf62-44f6-9522-2ada85451d15" (UID: "1f58d38e-bf62-44f6-9522-2ada85451d15"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 00:06:06.689903 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:06:06.689883 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f58d38e-bf62-44f6-9522-2ada85451d15-kube-api-access-bkpvw" (OuterVolumeSpecName: "kube-api-access-bkpvw") pod "1f58d38e-bf62-44f6-9522-2ada85451d15" (UID: "1f58d38e-bf62-44f6-9522-2ada85451d15"). InnerVolumeSpecName "kube-api-access-bkpvw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 00:06:06.689969 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:06:06.689886 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1f58d38e-bf62-44f6-9522-2ada85451d15-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1f58d38e-bf62-44f6-9522-2ada85451d15" (UID: "1f58d38e-bf62-44f6-9522-2ada85451d15"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 00:06:06.789228 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:06:06.789198 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f58d38e-bf62-44f6-9522-2ada85451d15-istio-podinfo\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 00:06:06.789228 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:06:06.789226 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f58d38e-bf62-44f6-9522-2ada85451d15-istio-token\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 00:06:06.789228 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:06:06.789236 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkpvw\" (UniqueName: \"kubernetes.io/projected/1f58d38e-bf62-44f6-9522-2ada85451d15-kube-api-access-bkpvw\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 00:06:06.789451 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:06:06.789246 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f58d38e-bf62-44f6-9522-2ada85451d15-istio-envoy\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 00:06:07.057814 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:06:07.057780 2574 generic.go:356] "Generic (PLEG): container finished" podID="1f58d38e-bf62-44f6-9522-2ada85451d15" containerID="393310600ba877303bbbb18665b2078d73065ba1658618e1611fe8db21354889" exitCode=0 Sep 11 00:06:07.058007 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:06:07.057833 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-d7a539-openshift-default-7c74c59b78-q7s7n" Sep 11 00:06:07.058007 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:06:07.057864 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-d7a539-openshift-default-7c74c59b78-q7s7n" event={"ID":"1f58d38e-bf62-44f6-9522-2ada85451d15","Type":"ContainerDied","Data":"393310600ba877303bbbb18665b2078d73065ba1658618e1611fe8db21354889"} Sep 11 00:06:07.058007 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:06:07.057915 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-d7a539-openshift-default-7c74c59b78-q7s7n" event={"ID":"1f58d38e-bf62-44f6-9522-2ada85451d15","Type":"ContainerDied","Data":"006ecb114db21d593d18ff9b9c59b123c339caba50229a9a13667374617c6a3b"} Sep 11 00:06:07.058007 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:06:07.057939 2574 scope.go:117] "RemoveContainer" containerID="393310600ba877303bbbb18665b2078d73065ba1658618e1611fe8db21354889" Sep 11 00:06:07.068426 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:06:07.068407 2574 scope.go:117] "RemoveContainer" containerID="880166dc726b9c67540e69d95a3af0699bcd62e1d21aaf0f71a68df5d9404d7a" Sep 11 00:06:07.076993 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:06:07.076973 2574 scope.go:117] "RemoveContainer" containerID="393310600ba877303bbbb18665b2078d73065ba1658618e1611fe8db21354889" Sep 11 00:06:07.077304 ip-10-0-138-128 kubenswrapper[2574]: E0911 00:06:07.077278 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"393310600ba877303bbbb18665b2078d73065ba1658618e1611fe8db21354889\": container with ID starting with 393310600ba877303bbbb18665b2078d73065ba1658618e1611fe8db21354889 not found: ID does not exist" containerID="393310600ba877303bbbb18665b2078d73065ba1658618e1611fe8db21354889" Sep 11 00:06:07.077416 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:06:07.077318 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"393310600ba877303bbbb18665b2078d73065ba1658618e1611fe8db21354889"} err="failed to get container status \"393310600ba877303bbbb18665b2078d73065ba1658618e1611fe8db21354889\": rpc error: code = NotFound desc = could not find container \"393310600ba877303bbbb18665b2078d73065ba1658618e1611fe8db21354889\": container with ID starting with 393310600ba877303bbbb18665b2078d73065ba1658618e1611fe8db21354889 not found: ID does not exist" Sep 11 00:06:07.077416 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:06:07.077337 2574 scope.go:117] "RemoveContainer" containerID="880166dc726b9c67540e69d95a3af0699bcd62e1d21aaf0f71a68df5d9404d7a" Sep 11 00:06:07.077837 ip-10-0-138-128 kubenswrapper[2574]: E0911 00:06:07.077796 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"880166dc726b9c67540e69d95a3af0699bcd62e1d21aaf0f71a68df5d9404d7a\": container with ID starting with 880166dc726b9c67540e69d95a3af0699bcd62e1d21aaf0f71a68df5d9404d7a not found: ID does not exist" containerID="880166dc726b9c67540e69d95a3af0699bcd62e1d21aaf0f71a68df5d9404d7a" Sep 11 00:06:07.077950 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:06:07.077830 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"880166dc726b9c67540e69d95a3af0699bcd62e1d21aaf0f71a68df5d9404d7a"} err="failed to get container status \"880166dc726b9c67540e69d95a3af0699bcd62e1d21aaf0f71a68df5d9404d7a\": rpc error: code = NotFound desc = could not find container \"880166dc726b9c67540e69d95a3af0699bcd62e1d21aaf0f71a68df5d9404d7a\": container with ID starting with 880166dc726b9c67540e69d95a3af0699bcd62e1d21aaf0f71a68df5d9404d7a not found: ID does not exist" Sep 11 00:06:07.088612 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:06:07.088573 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-d7a539-openshift-default-7c74c59b78-q7s7n"] Sep 11 00:06:07.095929 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:06:07.095904 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-d7a539-openshift-default-7c74c59b78-q7s7n"] Sep 11 00:06:09.094622 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:06:09.094591 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1f58d38e-bf62-44f6-9522-2ada85451d15" path="/var/lib/kubelet/pods/1f58d38e-bf62-44f6-9522-2ada85451d15/volumes" Sep 11 00:06:12.455623 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:06:12.455581 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4094ce-pxzbw"] Sep 11 00:06:12.458148 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:06:12.458124 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-4094ce-pxzbw"] Sep 11 00:06:13.095741 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:06:13.095709 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e509de4d-ec35-4c96-88a5-4c4d1126a123" path="/var/lib/kubelet/pods/e509de4d-ec35-4c96-88a5-4c4d1126a123/volumes" Sep 11 00:06:14.886529 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:06:14.886495 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-56r7l"] Sep 11 00:06:19.449587 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:06:19.449550 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-4094ce-openshift-default-7558d5c9fb-vp9l5"] Sep 11 00:06:19.450026 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:06:19.449935 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-4094ce-openshift-default-7558d5c9fb-vp9l5" podUID="e85f9c50-8a5c-4191-9441-9b30ca611ffb" containerName="istio-proxy" containerID="cri-o://221fd91fca6256fbabd3db5d34ad3c7b8bcb0b9dd81ae92a05d5145c578f58af" gracePeriod=30 Sep 11 00:06:23.776351 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:06:23.776313 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-56r7l"] Sep 11 00:06:24.710544 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:06:24.710515 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-4094ce-openshift-default-7558d5c9fb-vp9l5" Sep 11 00:06:24.856961 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:06:24.856867 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2dcvn\" (UniqueName: \"kubernetes.io/projected/e85f9c50-8a5c-4191-9441-9b30ca611ffb-kube-api-access-2dcvn\") pod \"e85f9c50-8a5c-4191-9441-9b30ca611ffb\" (UID: \"e85f9c50-8a5c-4191-9441-9b30ca611ffb\") " Sep 11 00:06:24.856961 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:06:24.856935 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e85f9c50-8a5c-4191-9441-9b30ca611ffb-credential-socket\") pod \"e85f9c50-8a5c-4191-9441-9b30ca611ffb\" (UID: \"e85f9c50-8a5c-4191-9441-9b30ca611ffb\") " Sep 11 00:06:24.857510 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:06:24.856970 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e85f9c50-8a5c-4191-9441-9b30ca611ffb-workload-socket\") pod \"e85f9c50-8a5c-4191-9441-9b30ca611ffb\" (UID: \"e85f9c50-8a5c-4191-9441-9b30ca611ffb\") " Sep 11 00:06:24.857510 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:06:24.857050 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e85f9c50-8a5c-4191-9441-9b30ca611ffb-istio-token\") pod \"e85f9c50-8a5c-4191-9441-9b30ca611ffb\" (UID: \"e85f9c50-8a5c-4191-9441-9b30ca611ffb\") " Sep 11 00:06:24.857510 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:06:24.857134 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e85f9c50-8a5c-4191-9441-9b30ca611ffb-istio-data\") pod \"e85f9c50-8a5c-4191-9441-9b30ca611ffb\" (UID: \"e85f9c50-8a5c-4191-9441-9b30ca611ffb\") " Sep 11 00:06:24.857510 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:06:24.857196 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e85f9c50-8a5c-4191-9441-9b30ca611ffb-istiod-ca-cert\") pod \"e85f9c50-8a5c-4191-9441-9b30ca611ffb\" (UID: \"e85f9c50-8a5c-4191-9441-9b30ca611ffb\") " Sep 11 00:06:24.857510 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:06:24.857224 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e85f9c50-8a5c-4191-9441-9b30ca611ffb-workload-certs\") pod \"e85f9c50-8a5c-4191-9441-9b30ca611ffb\" (UID: \"e85f9c50-8a5c-4191-9441-9b30ca611ffb\") " Sep 11 00:06:24.857510 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:06:24.857253 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e85f9c50-8a5c-4191-9441-9b30ca611ffb-istio-podinfo\") pod \"e85f9c50-8a5c-4191-9441-9b30ca611ffb\" (UID: \"e85f9c50-8a5c-4191-9441-9b30ca611ffb\") " Sep 11 00:06:24.857510 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:06:24.857294 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e85f9c50-8a5c-4191-9441-9b30ca611ffb-istio-envoy\") pod \"e85f9c50-8a5c-4191-9441-9b30ca611ffb\" (UID: \"e85f9c50-8a5c-4191-9441-9b30ca611ffb\") " Sep 11 00:06:24.857510 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:06:24.857216 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e85f9c50-8a5c-4191-9441-9b30ca611ffb-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e85f9c50-8a5c-4191-9441-9b30ca611ffb" (UID: "e85f9c50-8a5c-4191-9441-9b30ca611ffb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 00:06:24.857510 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:06:24.857316 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e85f9c50-8a5c-4191-9441-9b30ca611ffb-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e85f9c50-8a5c-4191-9441-9b30ca611ffb" (UID: "e85f9c50-8a5c-4191-9441-9b30ca611ffb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 00:06:24.857510 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:06:24.857370 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e85f9c50-8a5c-4191-9441-9b30ca611ffb-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e85f9c50-8a5c-4191-9441-9b30ca611ffb" (UID: "e85f9c50-8a5c-4191-9441-9b30ca611ffb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 00:06:24.857510 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:06:24.857397 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e85f9c50-8a5c-4191-9441-9b30ca611ffb-istio-data" (OuterVolumeSpecName: "istio-data") pod "e85f9c50-8a5c-4191-9441-9b30ca611ffb" (UID: "e85f9c50-8a5c-4191-9441-9b30ca611ffb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 00:06:24.858050 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:06:24.857553 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e85f9c50-8a5c-4191-9441-9b30ca611ffb-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e85f9c50-8a5c-4191-9441-9b30ca611ffb" (UID: "e85f9c50-8a5c-4191-9441-9b30ca611ffb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 00:06:24.858050 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:06:24.857724 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e85f9c50-8a5c-4191-9441-9b30ca611ffb-istio-data\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 00:06:24.858050 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:06:24.857744 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e85f9c50-8a5c-4191-9441-9b30ca611ffb-istiod-ca-cert\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 00:06:24.858050 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:06:24.857757 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e85f9c50-8a5c-4191-9441-9b30ca611ffb-workload-certs\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 00:06:24.858050 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:06:24.857770 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e85f9c50-8a5c-4191-9441-9b30ca611ffb-credential-socket\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 00:06:24.858050 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:06:24.857783 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e85f9c50-8a5c-4191-9441-9b30ca611ffb-workload-socket\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 00:06:24.859249 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:06:24.859226 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e85f9c50-8a5c-4191-9441-9b30ca611ffb-istio-token" (OuterVolumeSpecName: "istio-token") pod "e85f9c50-8a5c-4191-9441-9b30ca611ffb" (UID: "e85f9c50-8a5c-4191-9441-9b30ca611ffb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 00:06:24.859428 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:06:24.859413 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e85f9c50-8a5c-4191-9441-9b30ca611ffb-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e85f9c50-8a5c-4191-9441-9b30ca611ffb" (UID: "e85f9c50-8a5c-4191-9441-9b30ca611ffb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 00:06:24.859609 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:06:24.859586 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e85f9c50-8a5c-4191-9441-9b30ca611ffb-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e85f9c50-8a5c-4191-9441-9b30ca611ffb" (UID: "e85f9c50-8a5c-4191-9441-9b30ca611ffb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 00:06:24.859682 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:06:24.859595 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e85f9c50-8a5c-4191-9441-9b30ca611ffb-kube-api-access-2dcvn" (OuterVolumeSpecName: "kube-api-access-2dcvn") pod "e85f9c50-8a5c-4191-9441-9b30ca611ffb" (UID: "e85f9c50-8a5c-4191-9441-9b30ca611ffb"). InnerVolumeSpecName "kube-api-access-2dcvn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 00:06:24.958399 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:06:24.958360 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e85f9c50-8a5c-4191-9441-9b30ca611ffb-istio-podinfo\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 00:06:24.958399 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:06:24.958395 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e85f9c50-8a5c-4191-9441-9b30ca611ffb-istio-envoy\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 00:06:24.958605 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:06:24.958408 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dcvn\" (UniqueName: \"kubernetes.io/projected/e85f9c50-8a5c-4191-9441-9b30ca611ffb-kube-api-access-2dcvn\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 00:06:24.958605 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:06:24.958422 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e85f9c50-8a5c-4191-9441-9b30ca611ffb-istio-token\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 00:06:25.175004 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:06:25.174914 2574 generic.go:356] "Generic (PLEG): container finished" podID="e85f9c50-8a5c-4191-9441-9b30ca611ffb" containerID="221fd91fca6256fbabd3db5d34ad3c7b8bcb0b9dd81ae92a05d5145c578f58af" exitCode=0 Sep 11 00:06:25.175004 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:06:25.174956 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-4094ce-openshift-default-7558d5c9fb-vp9l5" Sep 11 00:06:25.175192 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:06:25.175003 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-4094ce-openshift-default-7558d5c9fb-vp9l5" event={"ID":"e85f9c50-8a5c-4191-9441-9b30ca611ffb","Type":"ContainerDied","Data":"221fd91fca6256fbabd3db5d34ad3c7b8bcb0b9dd81ae92a05d5145c578f58af"} Sep 11 00:06:25.175192 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:06:25.175055 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-4094ce-openshift-default-7558d5c9fb-vp9l5" event={"ID":"e85f9c50-8a5c-4191-9441-9b30ca611ffb","Type":"ContainerDied","Data":"0958aeef5a3070629862d7c6bb3c008591c4433aeec837a2543d92c1d9c246ca"} Sep 11 00:06:25.175192 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:06:25.175078 2574 scope.go:117] "RemoveContainer" containerID="221fd91fca6256fbabd3db5d34ad3c7b8bcb0b9dd81ae92a05d5145c578f58af" Sep 11 00:06:25.184846 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:06:25.184829 2574 scope.go:117] "RemoveContainer" containerID="86d66a2854991aac386fda6ac27492cd24fbbe529fba777c342907572eee3fd6" Sep 11 00:06:25.193019 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:06:25.192985 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-4094ce-openshift-default-7558d5c9fb-vp9l5"] Sep 11 00:06:25.194111 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:06:25.194089 2574 scope.go:117] "RemoveContainer" containerID="221fd91fca6256fbabd3db5d34ad3c7b8bcb0b9dd81ae92a05d5145c578f58af" Sep 11 00:06:25.194412 ip-10-0-138-128 kubenswrapper[2574]: E0911 00:06:25.194391 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"221fd91fca6256fbabd3db5d34ad3c7b8bcb0b9dd81ae92a05d5145c578f58af\": container with ID starting with 221fd91fca6256fbabd3db5d34ad3c7b8bcb0b9dd81ae92a05d5145c578f58af not found: ID does not exist" containerID="221fd91fca6256fbabd3db5d34ad3c7b8bcb0b9dd81ae92a05d5145c578f58af" Sep 11 00:06:25.194471 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:06:25.194423 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"221fd91fca6256fbabd3db5d34ad3c7b8bcb0b9dd81ae92a05d5145c578f58af"} err="failed to get container status \"221fd91fca6256fbabd3db5d34ad3c7b8bcb0b9dd81ae92a05d5145c578f58af\": rpc error: code = NotFound desc = could not find container \"221fd91fca6256fbabd3db5d34ad3c7b8bcb0b9dd81ae92a05d5145c578f58af\": container with ID starting with 221fd91fca6256fbabd3db5d34ad3c7b8bcb0b9dd81ae92a05d5145c578f58af not found: ID does not exist" Sep 11 00:06:25.194471 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:06:25.194448 2574 scope.go:117] "RemoveContainer" containerID="86d66a2854991aac386fda6ac27492cd24fbbe529fba777c342907572eee3fd6" Sep 11 00:06:25.194776 ip-10-0-138-128 kubenswrapper[2574]: E0911 00:06:25.194756 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86d66a2854991aac386fda6ac27492cd24fbbe529fba777c342907572eee3fd6\": container with ID starting with 86d66a2854991aac386fda6ac27492cd24fbbe529fba777c342907572eee3fd6 not found: ID does not exist" containerID="86d66a2854991aac386fda6ac27492cd24fbbe529fba777c342907572eee3fd6" Sep 11 00:06:25.194829 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:06:25.194783 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86d66a2854991aac386fda6ac27492cd24fbbe529fba777c342907572eee3fd6"} err="failed to get container status \"86d66a2854991aac386fda6ac27492cd24fbbe529fba777c342907572eee3fd6\": rpc error: code = NotFound desc = could not find container \"86d66a2854991aac386fda6ac27492cd24fbbe529fba777c342907572eee3fd6\": container with ID starting with 86d66a2854991aac386fda6ac27492cd24fbbe529fba777c342907572eee3fd6 not found: ID does not exist" Sep 11 00:06:25.196422 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:06:25.196404 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-4094ce-openshift-default-7558d5c9fb-vp9l5"] Sep 11 00:06:27.094182 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:06:27.094149 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e85f9c50-8a5c-4191-9441-9b30ca611ffb" path="/var/lib/kubelet/pods/e85f9c50-8a5c-4191-9441-9b30ca611ffb/volumes" Sep 11 00:06:49.365611 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:06:49.365578 2574 scope.go:117] "RemoveContainer" containerID="abefd27950f53a51cebaa336b23b234e74d81d79bc426d30870f6b1e148995b5" Sep 11 00:07:44.992503 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:07:44.992465 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-56r7l"] Sep 11 00:08:10.575004 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:08:10.574961 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-56r7l"] Sep 11 00:08:17.479193 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:08:17.479111 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-56r7l"] Sep 11 00:08:23.532317 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:08:23.532283 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-56r7l"] Sep 11 00:08:31.669731 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:08:31.669695 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-56r7l"] Sep 11 00:08:45.580624 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:08:45.580584 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-56r7l"] Sep 11 00:08:53.278610 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:08:53.278568 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-56r7l"] Sep 11 00:09:59.207441 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:09:59.207361 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2tjm4"] Sep 11 00:09:59.207906 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:09:59.207787 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e85f9c50-8a5c-4191-9441-9b30ca611ffb" containerName="istio-proxy" Sep 11 00:09:59.207906 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:09:59.207799 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="e85f9c50-8a5c-4191-9441-9b30ca611ffb" containerName="istio-proxy" Sep 11 00:09:59.207906 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:09:59.207809 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f58d38e-bf62-44f6-9522-2ada85451d15" containerName="istio-proxy" Sep 11 00:09:59.207906 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:09:59.207815 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f58d38e-bf62-44f6-9522-2ada85451d15" containerName="istio-proxy" Sep 11 00:09:59.207906 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:09:59.207822 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7c5b79b-f87f-4fc7-9734-e52a01fb7023" containerName="istio-proxy" Sep 11 00:09:59.207906 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:09:59.207829 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7c5b79b-f87f-4fc7-9734-e52a01fb7023" containerName="istio-proxy" Sep 11 00:09:59.207906 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:09:59.207845 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e509de4d-ec35-4c96-88a5-4c4d1126a123" containerName="revoke-keys" Sep 11 00:09:59.207906 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:09:59.207850 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="e509de4d-ec35-4c96-88a5-4c4d1126a123" containerName="revoke-keys" Sep 11 00:09:59.207906 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:09:59.207863 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f58d38e-bf62-44f6-9522-2ada85451d15" containerName="istio-proxy" Sep 11 00:09:59.207906 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:09:59.207868 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f58d38e-bf62-44f6-9522-2ada85451d15" containerName="istio-proxy" Sep 11 00:09:59.207906 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:09:59.207873 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e85f9c50-8a5c-4191-9441-9b30ca611ffb" containerName="istio-proxy" Sep 11 00:09:59.207906 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:09:59.207878 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="e85f9c50-8a5c-4191-9441-9b30ca611ffb" containerName="istio-proxy" Sep 11 00:09:59.207906 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:09:59.207884 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d738baa4-79b0-4d1d-80d2-7efbd3d91ade" containerName="istio-proxy" Sep 11 00:09:59.207906 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:09:59.207889 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="d738baa4-79b0-4d1d-80d2-7efbd3d91ade" containerName="istio-proxy" Sep 11 00:09:59.208393 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:09:59.207943 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="e509de4d-ec35-4c96-88a5-4c4d1126a123" containerName="revoke-keys" Sep 11 00:09:59.208393 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:09:59.207949 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f58d38e-bf62-44f6-9522-2ada85451d15" containerName="istio-proxy" Sep 11 00:09:59.208393 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:09:59.207954 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="e85f9c50-8a5c-4191-9441-9b30ca611ffb" containerName="istio-proxy" Sep 11 00:09:59.208393 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:09:59.207960 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="e85f9c50-8a5c-4191-9441-9b30ca611ffb" containerName="istio-proxy" Sep 11 00:09:59.208393 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:09:59.207969 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7c5b79b-f87f-4fc7-9734-e52a01fb7023" containerName="istio-proxy" Sep 11 00:09:59.210872 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:09:59.210856 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2tjm4" Sep 11 00:09:59.212866 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:09:59.212838 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-5wwxc\"" Sep 11 00:09:59.222285 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:09:59.222253 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2tjm4"] Sep 11 00:09:59.331924 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:09:59.331873 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66q2b\" (UniqueName: \"kubernetes.io/projected/377cc334-c6f9-4cdc-8dbe-d0fc466ce9f0-kube-api-access-66q2b\") pod \"kuadrant-operator-controller-manager-5769f99d9f-2tjm4\" (UID: \"377cc334-c6f9-4cdc-8dbe-d0fc466ce9f0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2tjm4" Sep 11 00:09:59.332119 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:09:59.332026 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/377cc334-c6f9-4cdc-8dbe-d0fc466ce9f0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-2tjm4\" (UID: \"377cc334-c6f9-4cdc-8dbe-d0fc466ce9f0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2tjm4" Sep 11 00:09:59.432678 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:09:59.432597 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/377cc334-c6f9-4cdc-8dbe-d0fc466ce9f0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-2tjm4\" (UID: \"377cc334-c6f9-4cdc-8dbe-d0fc466ce9f0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2tjm4" Sep 11 00:09:59.432874 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:09:59.432697 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66q2b\" (UniqueName: \"kubernetes.io/projected/377cc334-c6f9-4cdc-8dbe-d0fc466ce9f0-kube-api-access-66q2b\") pod \"kuadrant-operator-controller-manager-5769f99d9f-2tjm4\" (UID: \"377cc334-c6f9-4cdc-8dbe-d0fc466ce9f0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2tjm4" Sep 11 00:09:59.433056 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:09:59.433033 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/377cc334-c6f9-4cdc-8dbe-d0fc466ce9f0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-2tjm4\" (UID: \"377cc334-c6f9-4cdc-8dbe-d0fc466ce9f0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2tjm4" Sep 11 00:09:59.441699 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:09:59.441671 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-66q2b\" (UniqueName: \"kubernetes.io/projected/377cc334-c6f9-4cdc-8dbe-d0fc466ce9f0-kube-api-access-66q2b\") pod \"kuadrant-operator-controller-manager-5769f99d9f-2tjm4\" (UID: \"377cc334-c6f9-4cdc-8dbe-d0fc466ce9f0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2tjm4" Sep 11 00:09:59.524051 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:09:59.523947 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2tjm4" Sep 11 00:09:59.810784 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:09:59.810754 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2tjm4"] Sep 11 00:10:00.462024 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:10:00.461988 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2tjm4" event={"ID":"377cc334-c6f9-4cdc-8dbe-d0fc466ce9f0","Type":"ContainerStarted","Data":"a2a2de6dcad9e37304dd2ad1d904ed13fc55c4c0c948cf200f74bcf404c8e524"} Sep 11 00:10:00.462442 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:10:00.462044 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2tjm4" event={"ID":"377cc334-c6f9-4cdc-8dbe-d0fc466ce9f0","Type":"ContainerStarted","Data":"d94d4f2288f77ca197c830ccc077032e69e807727263e649e06610924e1b6545"} Sep 11 00:10:00.462442 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:10:00.462068 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2tjm4" Sep 11 00:10:00.486133 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:10:00.486078 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2tjm4" podStartSLOduration=1.486052352 podStartE2EDuration="1.486052352s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 00:09:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:10:00.482783899 +0000 UTC m=+1991.870386658" watchObservedRunningTime="2026-09-11 00:10:00.486052352 +0000 UTC m=+1991.873655184" Sep 11 00:10:11.470246 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:10:11.470213 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2tjm4" Sep 11 00:10:18.821662 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:10:18.821609 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-56r7l"] Sep 11 00:10:27.027156 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:10:27.027118 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-56r7l"] Sep 11 00:10:33.027181 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:10:33.027131 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-56r7l"] Sep 11 00:10:43.752355 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:10:43.752321 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-56r7l"] Sep 11 00:11:01.823825 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:11:01.823783 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-56r7l"] Sep 11 00:11:09.727377 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:11:09.727344 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-56r7l"] Sep 11 00:11:17.725062 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:11:17.724978 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-56r7l"] Sep 11 00:11:17.820908 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:11:17.820866 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-8f6fd9d5d-x87hn"] Sep 11 00:11:17.821375 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:11:17.821362 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f58d38e-bf62-44f6-9522-2ada85451d15" containerName="istio-proxy" Sep 11 00:11:17.824698 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:11:17.824674 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-8f6fd9d5d-x87hn" Sep 11 00:11:17.827509 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:11:17.827483 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 11 00:11:17.827675 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:11:17.827527 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 11 00:11:17.827675 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:11:17.827485 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 00:11:17.827675 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:11:17.827487 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-9tsnw\"" Sep 11 00:11:17.827675 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:11:17.827489 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Sep 11 00:11:17.835749 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:11:17.835715 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-8f6fd9d5d-x87hn"] Sep 11 00:11:17.872955 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:11:17.872916 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/81db168e-3b9c-4a73-8104-9449d49918f7-metrics-certs\") pod \"maas-api-8f6fd9d5d-x87hn\" (UID: \"81db168e-3b9c-4a73-8104-9449d49918f7\") " pod="odh-ai-gateway-infra/maas-api-8f6fd9d5d-x87hn" Sep 11 00:11:17.873159 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:11:17.872984 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/81db168e-3b9c-4a73-8104-9449d49918f7-maas-api-tls\") pod \"maas-api-8f6fd9d5d-x87hn\" (UID: \"81db168e-3b9c-4a73-8104-9449d49918f7\") " pod="odh-ai-gateway-infra/maas-api-8f6fd9d5d-x87hn" Sep 11 00:11:17.873159 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:11:17.873018 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/81db168e-3b9c-4a73-8104-9449d49918f7-openshift-service-ca\") pod \"maas-api-8f6fd9d5d-x87hn\" (UID: \"81db168e-3b9c-4a73-8104-9449d49918f7\") " pod="odh-ai-gateway-infra/maas-api-8f6fd9d5d-x87hn" Sep 11 00:11:17.873268 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:11:17.873150 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cjc5\" (UniqueName: \"kubernetes.io/projected/81db168e-3b9c-4a73-8104-9449d49918f7-kube-api-access-6cjc5\") pod \"maas-api-8f6fd9d5d-x87hn\" (UID: \"81db168e-3b9c-4a73-8104-9449d49918f7\") " pod="odh-ai-gateway-infra/maas-api-8f6fd9d5d-x87hn" Sep 11 00:11:17.974417 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:11:17.974377 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/81db168e-3b9c-4a73-8104-9449d49918f7-metrics-certs\") pod \"maas-api-8f6fd9d5d-x87hn\" (UID: \"81db168e-3b9c-4a73-8104-9449d49918f7\") " pod="odh-ai-gateway-infra/maas-api-8f6fd9d5d-x87hn" Sep 11 00:11:17.974628 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:11:17.974430 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/81db168e-3b9c-4a73-8104-9449d49918f7-maas-api-tls\") pod \"maas-api-8f6fd9d5d-x87hn\" (UID: \"81db168e-3b9c-4a73-8104-9449d49918f7\") " pod="odh-ai-gateway-infra/maas-api-8f6fd9d5d-x87hn" Sep 11 00:11:17.974628 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:11:17.974450 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/81db168e-3b9c-4a73-8104-9449d49918f7-openshift-service-ca\") pod \"maas-api-8f6fd9d5d-x87hn\" (UID: \"81db168e-3b9c-4a73-8104-9449d49918f7\") " pod="odh-ai-gateway-infra/maas-api-8f6fd9d5d-x87hn" Sep 11 00:11:17.974628 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:11:17.974498 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cjc5\" (UniqueName: \"kubernetes.io/projected/81db168e-3b9c-4a73-8104-9449d49918f7-kube-api-access-6cjc5\") pod \"maas-api-8f6fd9d5d-x87hn\" (UID: \"81db168e-3b9c-4a73-8104-9449d49918f7\") " pod="odh-ai-gateway-infra/maas-api-8f6fd9d5d-x87hn" Sep 11 00:11:17.975305 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:11:17.975244 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/81db168e-3b9c-4a73-8104-9449d49918f7-openshift-service-ca\") pod \"maas-api-8f6fd9d5d-x87hn\" (UID: \"81db168e-3b9c-4a73-8104-9449d49918f7\") " pod="odh-ai-gateway-infra/maas-api-8f6fd9d5d-x87hn" Sep 11 00:11:17.977001 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:11:17.976979 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/81db168e-3b9c-4a73-8104-9449d49918f7-maas-api-tls\") pod \"maas-api-8f6fd9d5d-x87hn\" (UID: \"81db168e-3b9c-4a73-8104-9449d49918f7\") " pod="odh-ai-gateway-infra/maas-api-8f6fd9d5d-x87hn" Sep 11 00:11:17.977095 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:11:17.977016 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/81db168e-3b9c-4a73-8104-9449d49918f7-metrics-certs\") pod \"maas-api-8f6fd9d5d-x87hn\" (UID: \"81db168e-3b9c-4a73-8104-9449d49918f7\") " pod="odh-ai-gateway-infra/maas-api-8f6fd9d5d-x87hn" Sep 11 00:11:17.981880 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:11:17.981860 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cjc5\" (UniqueName: \"kubernetes.io/projected/81db168e-3b9c-4a73-8104-9449d49918f7-kube-api-access-6cjc5\") pod \"maas-api-8f6fd9d5d-x87hn\" (UID: \"81db168e-3b9c-4a73-8104-9449d49918f7\") " pod="odh-ai-gateway-infra/maas-api-8f6fd9d5d-x87hn" Sep 11 00:11:18.137112 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:11:18.137067 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-8f6fd9d5d-x87hn" Sep 11 00:11:18.412038 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:11:18.412010 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-8f6fd9d5d-x87hn"] Sep 11 00:11:18.414439 ip-10-0-138-128 kubenswrapper[2574]: W0911 00:11:18.414401 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81db168e_3b9c_4a73_8104_9449d49918f7.slice/crio-844ecd76b4dad9b58174e06a8e77030825ed57faae37b7d7fc31e223bf0b1493 WatchSource:0}: Error finding container 844ecd76b4dad9b58174e06a8e77030825ed57faae37b7d7fc31e223bf0b1493: Status 404 returned error can't find the container with id 844ecd76b4dad9b58174e06a8e77030825ed57faae37b7d7fc31e223bf0b1493 Sep 11 00:11:18.415861 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:11:18.415842 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 00:11:18.949102 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:11:18.949068 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-8f6fd9d5d-x87hn" event={"ID":"81db168e-3b9c-4a73-8104-9449d49918f7","Type":"ContainerStarted","Data":"844ecd76b4dad9b58174e06a8e77030825ed57faae37b7d7fc31e223bf0b1493"} Sep 11 00:11:19.956857 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:11:19.956804 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-8f6fd9d5d-x87hn" event={"ID":"81db168e-3b9c-4a73-8104-9449d49918f7","Type":"ContainerStarted","Data":"69e053bf785a9a680bcf152f998987604201142e0033ba99982c0eb3bf462eda"} Sep 11 00:11:19.957311 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:11:19.956900 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-8f6fd9d5d-x87hn" Sep 11 00:11:19.974879 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:11:19.974822 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-8f6fd9d5d-x87hn" podStartSLOduration=2.404846858 podStartE2EDuration="2.974805505s" totalImagesPullingTime="569.958647ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 00:11:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:11:19.971894656 +0000 UTC m=+2071.359497417" watchObservedRunningTime="2026-09-11 00:11:19.974805505 +0000 UTC m=+2071.362408318" Sep 11 00:11:25.427342 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:11:25.427305 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-56r7l"] Sep 11 00:11:25.796820 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:11:25.796768 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-8f6fd9d5d-x87hn"] Sep 11 00:11:25.797155 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:11:25.797122 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-8f6fd9d5d-x87hn" podUID="81db168e-3b9c-4a73-8104-9449d49918f7" containerName="maas-api" containerID="cri-o://69e053bf785a9a680bcf152f998987604201142e0033ba99982c0eb3bf462eda" gracePeriod=30 Sep 11 00:11:25.803054 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:11:25.803032 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-8f6fd9d5d-x87hn" Sep 11 00:11:25.832338 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:11:25.832306 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-796ffbb698-krczc"] Sep 11 00:11:25.835903 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:11:25.835886 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-796ffbb698-krczc" Sep 11 00:11:25.847561 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:11:25.847461 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-796ffbb698-krczc"] Sep 11 00:11:25.952889 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:11:25.952843 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/267e9a25-0155-4ec0-9691-4b584544a2c9-metrics-certs\") pod \"maas-api-796ffbb698-krczc\" (UID: \"267e9a25-0155-4ec0-9691-4b584544a2c9\") " pod="odh-ai-gateway-infra/maas-api-796ffbb698-krczc" Sep 11 00:11:25.953056 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:11:25.952908 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98b6c\" (UniqueName: \"kubernetes.io/projected/267e9a25-0155-4ec0-9691-4b584544a2c9-kube-api-access-98b6c\") pod \"maas-api-796ffbb698-krczc\" (UID: \"267e9a25-0155-4ec0-9691-4b584544a2c9\") " pod="odh-ai-gateway-infra/maas-api-796ffbb698-krczc" Sep 11 00:11:25.953056 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:11:25.952972 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/267e9a25-0155-4ec0-9691-4b584544a2c9-maas-api-tls\") pod \"maas-api-796ffbb698-krczc\" (UID: \"267e9a25-0155-4ec0-9691-4b584544a2c9\") " pod="odh-ai-gateway-infra/maas-api-796ffbb698-krczc" Sep 11 00:11:25.953056 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:11:25.953033 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/267e9a25-0155-4ec0-9691-4b584544a2c9-openshift-service-ca\") pod \"maas-api-796ffbb698-krczc\" (UID: \"267e9a25-0155-4ec0-9691-4b584544a2c9\") " pod="odh-ai-gateway-infra/maas-api-796ffbb698-krczc" Sep 11 00:11:25.996594 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:11:25.996551 2574 generic.go:356] "Generic (PLEG): container finished" podID="81db168e-3b9c-4a73-8104-9449d49918f7" containerID="69e053bf785a9a680bcf152f998987604201142e0033ba99982c0eb3bf462eda" exitCode=0 Sep 11 00:11:25.996809 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:11:25.996632 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-8f6fd9d5d-x87hn" event={"ID":"81db168e-3b9c-4a73-8104-9449d49918f7","Type":"ContainerDied","Data":"69e053bf785a9a680bcf152f998987604201142e0033ba99982c0eb3bf462eda"} Sep 11 00:11:26.050411 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:11:26.050348 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-8f6fd9d5d-x87hn" Sep 11 00:11:26.054653 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:11:26.054608 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/267e9a25-0155-4ec0-9691-4b584544a2c9-maas-api-tls\") pod \"maas-api-796ffbb698-krczc\" (UID: \"267e9a25-0155-4ec0-9691-4b584544a2c9\") " pod="odh-ai-gateway-infra/maas-api-796ffbb698-krczc" Sep 11 00:11:26.054771 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:11:26.054690 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/267e9a25-0155-4ec0-9691-4b584544a2c9-openshift-service-ca\") pod \"maas-api-796ffbb698-krczc\" (UID: \"267e9a25-0155-4ec0-9691-4b584544a2c9\") " pod="odh-ai-gateway-infra/maas-api-796ffbb698-krczc" Sep 11 00:11:26.054822 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:11:26.054806 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/267e9a25-0155-4ec0-9691-4b584544a2c9-metrics-certs\") pod \"maas-api-796ffbb698-krczc\" (UID: \"267e9a25-0155-4ec0-9691-4b584544a2c9\") " pod="odh-ai-gateway-infra/maas-api-796ffbb698-krczc" Sep 11 00:11:26.054877 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:11:26.054851 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98b6c\" (UniqueName: \"kubernetes.io/projected/267e9a25-0155-4ec0-9691-4b584544a2c9-kube-api-access-98b6c\") pod \"maas-api-796ffbb698-krczc\" (UID: \"267e9a25-0155-4ec0-9691-4b584544a2c9\") " pod="odh-ai-gateway-infra/maas-api-796ffbb698-krczc" Sep 11 00:11:26.055473 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:11:26.055442 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/267e9a25-0155-4ec0-9691-4b584544a2c9-openshift-service-ca\") pod \"maas-api-796ffbb698-krczc\" (UID: \"267e9a25-0155-4ec0-9691-4b584544a2c9\") " pod="odh-ai-gateway-infra/maas-api-796ffbb698-krczc" Sep 11 00:11:26.057134 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:11:26.057111 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/267e9a25-0155-4ec0-9691-4b584544a2c9-maas-api-tls\") pod \"maas-api-796ffbb698-krczc\" (UID: \"267e9a25-0155-4ec0-9691-4b584544a2c9\") " pod="odh-ai-gateway-infra/maas-api-796ffbb698-krczc" Sep 11 00:11:26.057265 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:11:26.057241 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/267e9a25-0155-4ec0-9691-4b584544a2c9-metrics-certs\") pod \"maas-api-796ffbb698-krczc\" (UID: \"267e9a25-0155-4ec0-9691-4b584544a2c9\") " pod="odh-ai-gateway-infra/maas-api-796ffbb698-krczc" Sep 11 00:11:26.071814 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:11:26.071781 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-98b6c\" (UniqueName: \"kubernetes.io/projected/267e9a25-0155-4ec0-9691-4b584544a2c9-kube-api-access-98b6c\") pod \"maas-api-796ffbb698-krczc\" (UID: \"267e9a25-0155-4ec0-9691-4b584544a2c9\") " pod="odh-ai-gateway-infra/maas-api-796ffbb698-krczc" Sep 11 00:11:26.156140 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:11:26.156104 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/81db168e-3b9c-4a73-8104-9449d49918f7-maas-api-tls\") pod \"81db168e-3b9c-4a73-8104-9449d49918f7\" (UID: \"81db168e-3b9c-4a73-8104-9449d49918f7\") " Sep 11 00:11:26.156343 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:11:26.156162 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/81db168e-3b9c-4a73-8104-9449d49918f7-metrics-certs\") pod \"81db168e-3b9c-4a73-8104-9449d49918f7\" (UID: \"81db168e-3b9c-4a73-8104-9449d49918f7\") " Sep 11 00:11:26.156343 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:11:26.156267 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6cjc5\" (UniqueName: \"kubernetes.io/projected/81db168e-3b9c-4a73-8104-9449d49918f7-kube-api-access-6cjc5\") pod \"81db168e-3b9c-4a73-8104-9449d49918f7\" (UID: \"81db168e-3b9c-4a73-8104-9449d49918f7\") " Sep 11 00:11:26.156471 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:11:26.156377 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/81db168e-3b9c-4a73-8104-9449d49918f7-openshift-service-ca\") pod \"81db168e-3b9c-4a73-8104-9449d49918f7\" (UID: \"81db168e-3b9c-4a73-8104-9449d49918f7\") " Sep 11 00:11:26.156807 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:11:26.156774 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81db168e-3b9c-4a73-8104-9449d49918f7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "81db168e-3b9c-4a73-8104-9449d49918f7" (UID: "81db168e-3b9c-4a73-8104-9449d49918f7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 00:11:26.158272 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:11:26.158246 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81db168e-3b9c-4a73-8104-9449d49918f7-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "81db168e-3b9c-4a73-8104-9449d49918f7" (UID: "81db168e-3b9c-4a73-8104-9449d49918f7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 00:11:26.158652 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:11:26.158623 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81db168e-3b9c-4a73-8104-9449d49918f7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "81db168e-3b9c-4a73-8104-9449d49918f7" (UID: "81db168e-3b9c-4a73-8104-9449d49918f7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 00:11:26.158725 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:11:26.158655 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81db168e-3b9c-4a73-8104-9449d49918f7-kube-api-access-6cjc5" (OuterVolumeSpecName: "kube-api-access-6cjc5") pod "81db168e-3b9c-4a73-8104-9449d49918f7" (UID: "81db168e-3b9c-4a73-8104-9449d49918f7"). InnerVolumeSpecName "kube-api-access-6cjc5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 00:11:26.174904 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:11:26.174869 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-796ffbb698-krczc" Sep 11 00:11:26.257905 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:11:26.257869 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cjc5\" (UniqueName: \"kubernetes.io/projected/81db168e-3b9c-4a73-8104-9449d49918f7-kube-api-access-6cjc5\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 00:11:26.257905 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:11:26.257903 2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/81db168e-3b9c-4a73-8104-9449d49918f7-openshift-service-ca\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 00:11:26.258163 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:11:26.257914 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/81db168e-3b9c-4a73-8104-9449d49918f7-maas-api-tls\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 00:11:26.258163 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:11:26.257925 2574 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/81db168e-3b9c-4a73-8104-9449d49918f7-metrics-certs\") on node \"ip-10-0-138-128.ec2.internal\" DevicePath \"\"" Sep 11 00:11:26.462121 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:11:26.462086 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-796ffbb698-krczc"] Sep 11 00:11:26.463102 ip-10-0-138-128 kubenswrapper[2574]: W0911 00:11:26.463073 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod267e9a25_0155_4ec0_9691_4b584544a2c9.slice/crio-848e7c7a07019bb489c7962a17d25fc7fc668f1d192db7ffd3e751061b4a8dcf WatchSource:0}: Error finding container 848e7c7a07019bb489c7962a17d25fc7fc668f1d192db7ffd3e751061b4a8dcf: Status 404 returned error can't find the container with id 848e7c7a07019bb489c7962a17d25fc7fc668f1d192db7ffd3e751061b4a8dcf Sep 11 00:11:27.005591 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:11:27.005549 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-796ffbb698-krczc" event={"ID":"267e9a25-0155-4ec0-9691-4b584544a2c9","Type":"ContainerStarted","Data":"76cd31917b0d683ca0799725c5ac2d0436ee53c196635417a246b24a3ec4761e"} Sep 11 00:11:27.005782 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:11:27.005624 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-796ffbb698-krczc" Sep 11 00:11:27.005782 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:11:27.005678 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-796ffbb698-krczc" event={"ID":"267e9a25-0155-4ec0-9691-4b584544a2c9","Type":"ContainerStarted","Data":"848e7c7a07019bb489c7962a17d25fc7fc668f1d192db7ffd3e751061b4a8dcf"} Sep 11 00:11:27.006981 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:11:27.006958 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-8f6fd9d5d-x87hn" Sep 11 00:11:27.007115 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:11:27.006976 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-8f6fd9d5d-x87hn" event={"ID":"81db168e-3b9c-4a73-8104-9449d49918f7","Type":"ContainerDied","Data":"844ecd76b4dad9b58174e06a8e77030825ed57faae37b7d7fc31e223bf0b1493"} Sep 11 00:11:27.007115 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:11:27.007025 2574 scope.go:117] "RemoveContainer" containerID="69e053bf785a9a680bcf152f998987604201142e0033ba99982c0eb3bf462eda" Sep 11 00:11:27.028563 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:11:27.028519 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-796ffbb698-krczc" podStartSLOduration=1.584789191 podStartE2EDuration="2.028503432s" totalImagesPullingTime="443.714241ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 00:11:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:11:27.027093718 +0000 UTC m=+2078.414696477" watchObservedRunningTime="2026-09-11 00:11:27.028503432 +0000 UTC m=+2078.416106191" Sep 11 00:11:27.042621 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:11:27.042550 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-8f6fd9d5d-x87hn"] Sep 11 00:11:27.045153 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:11:27.045131 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-8f6fd9d5d-x87hn"] Sep 11 00:11:27.094153 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:11:27.094074 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81db168e-3b9c-4a73-8104-9449d49918f7" path="/var/lib/kubelet/pods/81db168e-3b9c-4a73-8104-9449d49918f7/volumes" Sep 11 00:11:32.627392 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:11:32.627352 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-56r7l"] Sep 11 00:11:33.018989 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:11:33.018961 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-796ffbb698-krczc" Sep 11 00:11:41.217689 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:11:41.217656 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-56r7l"] Sep 11 00:11:48.524610 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:11:48.524572 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-56r7l"] Sep 11 00:11:57.520607 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:11:57.520572 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-7ql2b_16430c51-3465-4ed3-a41c-c759af1e0eb1/manager/0.log" Sep 11 00:11:57.649982 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:11:57.649953 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-9xxkk_9a68eaf6-bccb-4fd2-808d-c7546b395981/manager/0.log" Sep 11 00:11:58.259716 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:11:58.259686 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-55d55f8464-vjvhz_0ae1eec4-b5f9-457c-854c-316bda56d48a/copy-manifests/0.log" Sep 11 00:11:58.267472 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:11:58.267445 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-55d55f8464-vjvhz_0ae1eec4-b5f9-457c-854c-316bda56d48a/manager/0.log" Sep 11 00:11:58.756057 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:11:58.756017 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-796ffbb698-krczc_267e9a25-0155-4ec0-9691-4b584544a2c9/maas-api/0.log" Sep 11 00:11:58.992934 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:11:58.992898 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29818080-6xrvk_198384a6-ba0b-4d39-956d-f33800f5b531/cleanup/0.log" Sep 11 00:12:00.017876 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:12:00.017842 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-dml9m_264aca46-d6b4-4981-9392-7092015b0d7e/manager/0.log" Sep 11 00:12:00.140021 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:12:00.139989 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-sv4gw_c7ab7425-af9d-4839-aba0-e4ca2d298860/manager/0.log" Sep 11 00:12:00.392060 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:12:00.391982 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-2tjm4_377cc334-c6f9-4cdc-8dbe-d0fc466ce9f0/manager/0.log" Sep 11 00:12:00.525919 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:12:00.525886 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-56r7l_3c63a0c1-85ba-47d5-9450-91f8809eb2f0/limitador/0.log" Sep 11 00:12:01.165313 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:12:01.165277 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-cdc5j_d88224c2-9ecd-42a5-ac77-820af96f724a/discovery/0.log" Sep 11 00:12:01.278480 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:12:01.278447 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-55f89b848f-6pl58_c9537e38-fd66-4e10-8f78-81987455a894/kube-auth-proxy/0.log" Sep 11 00:12:01.514152 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:12:01.514120 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-2jhsv_e1034bfd-cf35-4a42-90ac-6518eed5ff4f/istio-proxy/1.log" Sep 11 00:12:01.634302 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:12:01.634263 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-wbdst_645e6258-646a-4a26-83e8-4cb7af5347d7/payload-pre-processing/0.log" Sep 11 00:12:02.226684 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:12:02.226654 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-nm2sz_df5991e1-0d2b-4b74-864f-d263b942fbea/main/0.log" Sep 11 00:12:11.534748 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:12:11.534710 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-75vlp_3a0ec520-b33d-49cf-9714-1662f5910e5b/global-pull-secret-syncer/0.log" Sep 11 00:12:11.681799 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:12:11.681760 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-dl8bc_7ad8f728-76e5-47a2-8ab2-681ac07591f8/konnectivity-agent/0.log" Sep 11 00:12:11.780981 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:12:11.780951 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-128.ec2.internal_577bf0a7d48f3dd63acd7590896a721f/haproxy/0.log" Sep 11 00:12:16.382492 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:12:16.382462 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-dml9m_264aca46-d6b4-4981-9392-7092015b0d7e/manager/0.log" Sep 11 00:12:16.405518 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:12:16.405484 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-sv4gw_c7ab7425-af9d-4839-aba0-e4ca2d298860/manager/0.log" Sep 11 00:12:16.475162 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:12:16.475132 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-2tjm4_377cc334-c6f9-4cdc-8dbe-d0fc466ce9f0/manager/0.log" Sep 11 00:12:16.501133 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:12:16.501103 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-56r7l_3c63a0c1-85ba-47d5-9450-91f8809eb2f0/limitador/0.log" Sep 11 00:12:17.996565 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:12:17.996532 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b5e6b6fa-bb37-4770-93b3-26b897d01b78/alertmanager/0.log" Sep 11 00:12:18.028526 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:12:18.028485 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b5e6b6fa-bb37-4770-93b3-26b897d01b78/config-reloader/0.log" Sep 11 00:12:18.048064 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:12:18.048042 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b5e6b6fa-bb37-4770-93b3-26b897d01b78/kube-rbac-proxy-web/0.log" Sep 11 00:12:18.067034 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:12:18.067008 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b5e6b6fa-bb37-4770-93b3-26b897d01b78/kube-rbac-proxy/0.log" Sep 11 00:12:18.096296 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:12:18.096272 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b5e6b6fa-bb37-4770-93b3-26b897d01b78/kube-rbac-proxy-metric/0.log" Sep 11 00:12:18.116335 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:12:18.116303 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b5e6b6fa-bb37-4770-93b3-26b897d01b78/prom-label-proxy/0.log" Sep 11 00:12:18.142927 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:12:18.142897 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b5e6b6fa-bb37-4770-93b3-26b897d01b78/init-config-reloader/0.log" Sep 11 00:12:18.222248 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:12:18.222222 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-qbts4_a4d6f4b4-86db-445b-8ed8-20e0c4d622c2/kube-state-metrics/0.log" Sep 11 00:12:18.237523 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:12:18.237501 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-qbts4_a4d6f4b4-86db-445b-8ed8-20e0c4d622c2/kube-rbac-proxy-main/0.log" Sep 11 00:12:18.256820 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:12:18.256754 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-qbts4_a4d6f4b4-86db-445b-8ed8-20e0c4d622c2/kube-rbac-proxy-self/0.log" Sep 11 00:12:18.280547 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:12:18.280523 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-6fb78998c8-bh5np_cf3a41cb-434f-4206-8bfa-be659892e8d6/metrics-server/0.log" Sep 11 00:12:18.309116 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:12:18.309077 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5d85778b9d-8kc4t_82b63a87-37d1-40bc-973d-478d4a807db3/monitoring-plugin/0.log" Sep 11 00:12:18.430809 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:12:18.430772 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-m2v2w_869ffc54-b1e8-4f0b-bd8e-2c571d1c4584/node-exporter/0.log" Sep 11 00:12:18.448945 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:12:18.448920 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-m2v2w_869ffc54-b1e8-4f0b-bd8e-2c571d1c4584/kube-rbac-proxy/0.log" Sep 11 00:12:18.468317 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:12:18.468295 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-m2v2w_869ffc54-b1e8-4f0b-bd8e-2c571d1c4584/init-textfile/0.log" Sep 11 00:12:18.583527 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:12:18.583445 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-p7gpg_af3034eb-2870-46ab-add1-005b59c04308/kube-rbac-proxy-main/0.log" Sep 11 00:12:18.604528 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:12:18.604495 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-p7gpg_af3034eb-2870-46ab-add1-005b59c04308/kube-rbac-proxy-self/0.log" Sep 11 00:12:18.623686 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:12:18.623631 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-p7gpg_af3034eb-2870-46ab-add1-005b59c04308/openshift-state-metrics/0.log" Sep 11 00:12:18.837241 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:12:18.837162 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-kckq4_061f75eb-fab8-4777-a086-0d6b2c9abd7c/prometheus-operator/0.log" Sep 11 00:12:18.855778 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:12:18.855748 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-kckq4_061f75eb-fab8-4777-a086-0d6b2c9abd7c/kube-rbac-proxy/0.log" Sep 11 00:12:18.889151 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:12:18.889098 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5864cd99d7-dkphk_f8b518fd-885c-42e8-b59f-a1d271e750dc/prometheus-operator-admission-webhook/0.log" Sep 11 00:12:18.917844 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:12:18.917817 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-54dcfc4455-v9d4r_6db03de9-be49-4832-9ce9-5b131047b19b/telemeter-client/0.log" Sep 11 00:12:18.939781 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:12:18.939760 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-54dcfc4455-v9d4r_6db03de9-be49-4832-9ce9-5b131047b19b/reload/0.log" Sep 11 00:12:18.960532 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:12:18.960510 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-54dcfc4455-v9d4r_6db03de9-be49-4832-9ce9-5b131047b19b/kube-rbac-proxy/0.log" Sep 11 00:12:19.009476 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:12:19.009448 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7f945b84b4-9zvwn_008bdffc-746e-448b-9bd0-18155f16320e/thanos-query/0.log" Sep 11 00:12:19.066083 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:12:19.066049 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7f945b84b4-9zvwn_008bdffc-746e-448b-9bd0-18155f16320e/kube-rbac-proxy-web/0.log" Sep 11 00:12:19.136727 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:12:19.136593 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7f945b84b4-9zvwn_008bdffc-746e-448b-9bd0-18155f16320e/kube-rbac-proxy/0.log" Sep 11 00:12:19.156720 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:12:19.156682 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7f945b84b4-9zvwn_008bdffc-746e-448b-9bd0-18155f16320e/prom-label-proxy/0.log" Sep 11 00:12:19.180270 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:12:19.180242 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7f945b84b4-9zvwn_008bdffc-746e-448b-9bd0-18155f16320e/kube-rbac-proxy-rules/0.log" Sep 11 00:12:19.198903 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:12:19.198872 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7f945b84b4-9zvwn_008bdffc-746e-448b-9bd0-18155f16320e/kube-rbac-proxy-metrics/0.log" Sep 11 00:12:21.381339 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:12:21.381304 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-rq2f8/perf-node-gather-daemonset-ldslv"] Sep 11 00:12:21.381962 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:12:21.381936 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81db168e-3b9c-4a73-8104-9449d49918f7" containerName="maas-api" Sep 11 00:12:21.381962 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:12:21.381961 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="81db168e-3b9c-4a73-8104-9449d49918f7" containerName="maas-api" Sep 11 00:12:21.382198 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:12:21.382074 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="81db168e-3b9c-4a73-8104-9449d49918f7" containerName="maas-api" Sep 11 00:12:21.386291 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:12:21.386272 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rq2f8/perf-node-gather-daemonset-ldslv" Sep 11 00:12:21.388757 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:12:21.388583 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rq2f8\"/\"kube-root-ca.crt\"" Sep 11 00:12:21.389276 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:12:21.389255 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-rq2f8\"/\"default-dockercfg-7cn6g\"" Sep 11 00:12:21.389381 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:12:21.389258 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rq2f8\"/\"openshift-service-ca.crt\"" Sep 11 00:12:21.396733 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:12:21.396709 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rq2f8/perf-node-gather-daemonset-ldslv"] Sep 11 00:12:21.573898 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:12:21.573866 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/14182725-5ed5-4ed1-9d8f-f6f63533cddf-podres\") pod \"perf-node-gather-daemonset-ldslv\" (UID: \"14182725-5ed5-4ed1-9d8f-f6f63533cddf\") " pod="openshift-must-gather-rq2f8/perf-node-gather-daemonset-ldslv" Sep 11 00:12:21.574068 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:12:21.573921 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/14182725-5ed5-4ed1-9d8f-f6f63533cddf-lib-modules\") pod \"perf-node-gather-daemonset-ldslv\" (UID: \"14182725-5ed5-4ed1-9d8f-f6f63533cddf\") " pod="openshift-must-gather-rq2f8/perf-node-gather-daemonset-ldslv" Sep 11 00:12:21.574068 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:12:21.573985 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/14182725-5ed5-4ed1-9d8f-f6f63533cddf-proc\") pod \"perf-node-gather-daemonset-ldslv\" (UID: \"14182725-5ed5-4ed1-9d8f-f6f63533cddf\") " pod="openshift-must-gather-rq2f8/perf-node-gather-daemonset-ldslv" Sep 11 00:12:21.574148 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:12:21.574064 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6r2kl\" (UniqueName: \"kubernetes.io/projected/14182725-5ed5-4ed1-9d8f-f6f63533cddf-kube-api-access-6r2kl\") pod \"perf-node-gather-daemonset-ldslv\" (UID: \"14182725-5ed5-4ed1-9d8f-f6f63533cddf\") " pod="openshift-must-gather-rq2f8/perf-node-gather-daemonset-ldslv" Sep 11 00:12:21.574148 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:12:21.574094 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/14182725-5ed5-4ed1-9d8f-f6f63533cddf-sys\") pod \"perf-node-gather-daemonset-ldslv\" (UID: \"14182725-5ed5-4ed1-9d8f-f6f63533cddf\") " pod="openshift-must-gather-rq2f8/perf-node-gather-daemonset-ldslv" Sep 11 00:12:21.594793 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:12:21.594766 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-76f4fbdcf5-vgxpp_a6155b8d-dffb-4ffa-b03b-97754b95bce4/console/0.log" Sep 11 00:12:21.674479 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:12:21.674414 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/14182725-5ed5-4ed1-9d8f-f6f63533cddf-podres\") pod \"perf-node-gather-daemonset-ldslv\" (UID: \"14182725-5ed5-4ed1-9d8f-f6f63533cddf\") " pod="openshift-must-gather-rq2f8/perf-node-gather-daemonset-ldslv" Sep 11 00:12:21.674479 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:12:21.674451 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/14182725-5ed5-4ed1-9d8f-f6f63533cddf-lib-modules\") pod \"perf-node-gather-daemonset-ldslv\" (UID: \"14182725-5ed5-4ed1-9d8f-f6f63533cddf\") " pod="openshift-must-gather-rq2f8/perf-node-gather-daemonset-ldslv" Sep 11 00:12:21.674616 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:12:21.674505 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/14182725-5ed5-4ed1-9d8f-f6f63533cddf-proc\") pod \"perf-node-gather-daemonset-ldslv\" (UID: \"14182725-5ed5-4ed1-9d8f-f6f63533cddf\") " pod="openshift-must-gather-rq2f8/perf-node-gather-daemonset-ldslv" Sep 11 00:12:21.674616 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:12:21.674536 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6r2kl\" (UniqueName: \"kubernetes.io/projected/14182725-5ed5-4ed1-9d8f-f6f63533cddf-kube-api-access-6r2kl\") pod \"perf-node-gather-daemonset-ldslv\" (UID: \"14182725-5ed5-4ed1-9d8f-f6f63533cddf\") " pod="openshift-must-gather-rq2f8/perf-node-gather-daemonset-ldslv" Sep 11 00:12:21.674616 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:12:21.674552 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/14182725-5ed5-4ed1-9d8f-f6f63533cddf-sys\") pod \"perf-node-gather-daemonset-ldslv\" (UID: \"14182725-5ed5-4ed1-9d8f-f6f63533cddf\") " pod="openshift-must-gather-rq2f8/perf-node-gather-daemonset-ldslv" Sep 11 00:12:21.674616 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:12:21.674582 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/14182725-5ed5-4ed1-9d8f-f6f63533cddf-podres\") pod \"perf-node-gather-daemonset-ldslv\" (UID: \"14182725-5ed5-4ed1-9d8f-f6f63533cddf\") " pod="openshift-must-gather-rq2f8/perf-node-gather-daemonset-ldslv" Sep 11 00:12:21.674781 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:12:21.674610 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/14182725-5ed5-4ed1-9d8f-f6f63533cddf-proc\") pod \"perf-node-gather-daemonset-ldslv\" (UID: \"14182725-5ed5-4ed1-9d8f-f6f63533cddf\") " pod="openshift-must-gather-rq2f8/perf-node-gather-daemonset-ldslv" Sep 11 00:12:21.674781 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:12:21.674655 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/14182725-5ed5-4ed1-9d8f-f6f63533cddf-sys\") pod \"perf-node-gather-daemonset-ldslv\" (UID: \"14182725-5ed5-4ed1-9d8f-f6f63533cddf\") " pod="openshift-must-gather-rq2f8/perf-node-gather-daemonset-ldslv" Sep 11 00:12:21.674781 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:12:21.674709 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/14182725-5ed5-4ed1-9d8f-f6f63533cddf-lib-modules\") pod \"perf-node-gather-daemonset-ldslv\" (UID: \"14182725-5ed5-4ed1-9d8f-f6f63533cddf\") " pod="openshift-must-gather-rq2f8/perf-node-gather-daemonset-ldslv" Sep 11 00:12:21.685487 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:12:21.685461 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6r2kl\" (UniqueName: \"kubernetes.io/projected/14182725-5ed5-4ed1-9d8f-f6f63533cddf-kube-api-access-6r2kl\") pod \"perf-node-gather-daemonset-ldslv\" (UID: \"14182725-5ed5-4ed1-9d8f-f6f63533cddf\") " pod="openshift-must-gather-rq2f8/perf-node-gather-daemonset-ldslv" Sep 11 00:12:21.697378 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:12:21.697361 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rq2f8/perf-node-gather-daemonset-ldslv" Sep 11 00:12:21.969923 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:12:21.969890 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rq2f8/perf-node-gather-daemonset-ldslv"] Sep 11 00:12:21.972663 ip-10-0-138-128 kubenswrapper[2574]: W0911 00:12:21.972622 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod14182725_5ed5_4ed1_9d8f_f6f63533cddf.slice/crio-a275bf281f1e65d7e12b21144e429156db53f38506f0eb2c27fd9b61d0ccefcd WatchSource:0}: Error finding container a275bf281f1e65d7e12b21144e429156db53f38506f0eb2c27fd9b61d0ccefcd: Status 404 returned error can't find the container with id a275bf281f1e65d7e12b21144e429156db53f38506f0eb2c27fd9b61d0ccefcd Sep 11 00:12:22.331250 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:12:22.331154 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rq2f8/perf-node-gather-daemonset-ldslv" event={"ID":"14182725-5ed5-4ed1-9d8f-f6f63533cddf","Type":"ContainerStarted","Data":"7c9df6293f2c4aa8d8d65f040ead8aa9b8264af219b215378ab2c2afa2843cf7"} Sep 11 00:12:22.331250 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:12:22.331209 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rq2f8/perf-node-gather-daemonset-ldslv" event={"ID":"14182725-5ed5-4ed1-9d8f-f6f63533cddf","Type":"ContainerStarted","Data":"a275bf281f1e65d7e12b21144e429156db53f38506f0eb2c27fd9b61d0ccefcd"} Sep 11 00:12:22.331250 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:12:22.331250 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-rq2f8/perf-node-gather-daemonset-ldslv" Sep 11 00:12:22.348416 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:12:22.348366 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-rq2f8/perf-node-gather-daemonset-ldslv" podStartSLOduration=1.348349349 podStartE2EDuration="1.348349349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 00:12:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 00:12:22.345473552 +0000 UTC m=+2133.733076322" watchObservedRunningTime="2026-09-11 00:12:22.348349349 +0000 UTC m=+2133.735952108" Sep 11 00:12:23.451053 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:12:23.451020 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-6bbbz_6baf5c56-fb47-42e5-bc9e-d41ff149c33d/dns/0.log" Sep 11 00:12:23.468118 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:12:23.468094 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-6bbbz_6baf5c56-fb47-42e5-bc9e-d41ff149c33d/kube-rbac-proxy/0.log" Sep 11 00:12:23.532703 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:12:23.532676 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6wz5g_5d6d9866-cf11-4f49-877a-ecc951c01e42/dns-node-resolver/0.log" Sep 11 00:12:24.144769 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:12:24.144735 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-fckpk_cbaa689c-2a82-4b1a-b61f-4e8f43efa7d5/node-ca/0.log" Sep 11 00:12:25.436320 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:12:25.436291 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-cdc5j_d88224c2-9ecd-42a5-ac77-820af96f724a/discovery/0.log" Sep 11 00:12:25.458274 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:12:25.458247 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-55f89b848f-6pl58_c9537e38-fd66-4e10-8f78-81987455a894/kube-auth-proxy/0.log" Sep 11 00:12:25.510379 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:12:25.510353 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-2jhsv_e1034bfd-cf35-4a42-90ac-6518eed5ff4f/istio-proxy/0.log" Sep 11 00:12:25.526292 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:12:25.526267 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-2jhsv_e1034bfd-cf35-4a42-90ac-6518eed5ff4f/istio-proxy/1.log" Sep 11 00:12:25.567908 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:12:25.567871 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-wbdst_645e6258-646a-4a26-83e8-4cb7af5347d7/payload-pre-processing/0.log" Sep 11 00:12:26.477472 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:12:26.477442 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-48dgb_47d30ad8-0b9d-4c37-942d-b31a444e8420/serve-healthcheck-canary/0.log" Sep 11 00:12:26.979402 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:12:26.979373 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-l25dt_3ebfe403-2315-4056-8daa-58d73e056140/kube-rbac-proxy/0.log" Sep 11 00:12:27.001406 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:12:27.001370 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-l25dt_3ebfe403-2315-4056-8daa-58d73e056140/exporter/0.log" Sep 11 00:12:27.017027 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:12:27.017003 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-l25dt_3ebfe403-2315-4056-8daa-58d73e056140/extractor/0.log" Sep 11 00:12:28.347262 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:12:28.347235 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-rq2f8/perf-node-gather-daemonset-ldslv" Sep 11 00:12:29.181862 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:12:29.181831 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-7ql2b_16430c51-3465-4ed3-a41c-c759af1e0eb1/manager/0.log" Sep 11 00:12:29.500338 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:12:29.500307 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-9xxkk_9a68eaf6-bccb-4fd2-808d-c7546b395981/manager/0.log" Sep 11 00:12:29.802498 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:12:29.802407 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-55d55f8464-vjvhz_0ae1eec4-b5f9-457c-854c-316bda56d48a/manager/0.log" Sep 11 00:12:29.820980 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:12:29.820954 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-55d55f8464-vjvhz_0ae1eec4-b5f9-457c-854c-316bda56d48a/copy-manifests/0.log" Sep 11 00:12:31.957089 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:12:31.957047 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-cdc5j_d88224c2-9ecd-42a5-ac77-820af96f724a/discovery/0.log" Sep 11 00:12:31.968259 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:12:31.968214 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-55f89b848f-6pl58_c9537e38-fd66-4e10-8f78-81987455a894/kube-auth-proxy/0.log" Sep 11 00:12:32.000603 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:12:32.000564 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-2jhsv_e1034bfd-cf35-4a42-90ac-6518eed5ff4f/istio-proxy/0.log" Sep 11 00:12:32.016588 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:12:32.016533 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-2jhsv_e1034bfd-cf35-4a42-90ac-6518eed5ff4f/istio-proxy/1.log" Sep 11 00:12:32.050952 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:12:32.050918 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-wbdst_645e6258-646a-4a26-83e8-4cb7af5347d7/payload-pre-processing/0.log" Sep 11 00:12:34.030205 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:12:34.030168 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-cdc5j_d88224c2-9ecd-42a5-ac77-820af96f724a/discovery/0.log" Sep 11 00:12:34.047279 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:12:34.047240 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-55f89b848f-6pl58_c9537e38-fd66-4e10-8f78-81987455a894/kube-auth-proxy/0.log" Sep 11 00:12:34.086403 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:12:34.086364 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-2jhsv_e1034bfd-cf35-4a42-90ac-6518eed5ff4f/istio-proxy/0.log" Sep 11 00:12:34.101969 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:12:34.101946 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-2jhsv_e1034bfd-cf35-4a42-90ac-6518eed5ff4f/istio-proxy/1.log" Sep 11 00:12:34.139184 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:12:34.139155 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-wbdst_645e6258-646a-4a26-83e8-4cb7af5347d7/payload-pre-processing/0.log" Sep 11 00:12:37.736730 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:12:37.736700 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-f47cx_43e0f2f1-cd70-4612-828f-6cd98e1b77c2/kube-multus-additional-cni-plugins/0.log" Sep 11 00:12:37.757454 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:12:37.757431 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-f47cx_43e0f2f1-cd70-4612-828f-6cd98e1b77c2/egress-router-binary-copy/0.log" Sep 11 00:12:37.777461 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:12:37.777433 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-f47cx_43e0f2f1-cd70-4612-828f-6cd98e1b77c2/cni-plugins/0.log" Sep 11 00:12:37.800049 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:12:37.800027 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-f47cx_43e0f2f1-cd70-4612-828f-6cd98e1b77c2/bond-cni-plugin/0.log" Sep 11 00:12:37.821381 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:12:37.821357 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-f47cx_43e0f2f1-cd70-4612-828f-6cd98e1b77c2/routeoverride-cni/0.log" Sep 11 00:12:37.840556 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:12:37.840536 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-f47cx_43e0f2f1-cd70-4612-828f-6cd98e1b77c2/whereabouts-cni-bincopy/0.log" Sep 11 00:12:37.859001 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:12:37.858977 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-f47cx_43e0f2f1-cd70-4612-828f-6cd98e1b77c2/whereabouts-cni/0.log" Sep 11 00:12:38.306255 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:12:38.306226 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-hnjsh_ed113529-7712-4e4f-838a-835f3d6cf1b0/kube-multus/0.log" Sep 11 00:12:38.414487 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:12:38.414456 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-v87ts_6df7e606-319b-4b9a-b6da-a850ecc9d267/network-metrics-daemon/0.log" Sep 11 00:12:38.427808 ip-10-0-138-128 kubenswrapper[2574]: I0911 00:12:38.427779 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-v87ts_6df7e606-319b-4b9a-b6da-a850ecc9d267/kube-rbac-proxy/0.log"